Project – VR Sword Defense Video Game

Introduction

In this project, we constructed a virtual reality (VR) video game. The video game uses several peripherals to give the user an immersive gaming experience. This includes providing visual, auditory, and tactile feedback from the game. The game features a headpiece with a TFT display. The display shows incoming projectiles that the player must deflect/eliminate with a sword. Upon impact, the player receives an unpleasant sound through the earpiece and tactile feedback via the vibration motors in the headpiece. A magnetometer at the top of the headpiece causes the screen to change based on the direction the player is facing. The objective of the game is to survive as long as possible. The goal of the project is to provide an enjoyable yet safe immersive gaming experience.

High Level Design

In the game, the user wears a headpiece with a TFT display on it. This TFT displays projectiles (balls) that are moving towards the player. There can be a projectile incoming on the left or the right of the screen. Additionally, the TFT displays the player�s score and health. The score is how long in seconds that the user has survived the game. The health begins at 5 and decrements every time the user gets �hit� by one of the balls. When the health reaches 0, the game terminates. There is a game reset button which allows the player to continue with another round of the game.

The headpiece features a magnetometer, which determines which way the headpiece is pointing. Based on this, the screen that the TFT is displaying will change. This gives the impression to the player that the game�s screen changes based on the direction that the player is facing. This is a pivotal part of the immersive experience. In total, there are 4 possible screens. The figure below illustrates what the virtual environment looks like for one ball. The �stages� are meant to illustrate the ball increasing in size as it gets closer to the user. Screens 1-3 illustrate different screens the user is not currently looking at.

To combat the incoming projectiles, the player is armed with a sword. The sword contains accelerometers at its tip to detect when the sword has been swung and when the player is swinging the sword left/right. When the ball is close enough, the player must swing the sword in the correct direction to deflect/eliminate the incoming projectile.

If the player gets hit by a ball, there is auditory and tactile feedback. The auditory feedback consists of a sharp, slightly distasteful, sound to indicate to the user that he or she has been hit. To allow for tactile feedback, the headpiece has vibration motors placed on the light, right and back side. When the user gets hit from the left, right, or back, then the appropriate vibration motor will vibrate. If the user gets hit from the front, all three vibration motors will vibrate.

There were hardware software tradeoffs in the form of adding hardware complexity tended to add software complexity. In general, changing hardware created more software work, whereas software changes didn�t change much of the hardware.

This project was the result of brainstorming between the group members with the goal of creating a game that involved swinging a physical sword to control a video game.

Program/Hardware Design

Hardware Description:

A full hardware schematic is available in Appendix B. For the following, refer to the schematic to find the specific �port pins� being mentioned.

The first piece of hardware to be constructed was the sword. Two 1-dimensional accelerometers were soldered together at a 90 degree angle to essentially create a 2-dimensional accelerometer, capable of noting the general direction of a sword swing. These accelerometers were placed in the tip of a recycled, light up, plastic sword with wires from the accelerometers coming out of the handle of the sword. The accelerometers provided values between 0 and 3.3V, corresponding to the force being applied to the accelerometer. As such they were connected to the ADC inputs of the PIC32 for interpretation via 330Ω resistors.

The PIC32 port pins could not supply the required 70mA of current to power the three vibration motors. A simple current driver was constructed using three 2N3904 transistors with the collector connected to power via a vibration motor. The base was connected to one of the port pins of the PIC32 via a 1kΩ resistor. This pin was internally pulled down, and pulled high when the program told the motor to vibrate.

The vibration motors were placed in the helmet to provide the user tactile feedback when being hit by a ball. There were three motors in total: left, back, and right. Being hit from the front triggered a special condition. The screen and circuitry for the rest of the device were attached to the visor of the helmet. The screen was quite close to the users face when wearing the helmet, so a Fresnel lens was attached to the front of a pair of safety glasses to help the user focus on the screen.

The simplest part of the schematic was the restart button. This was a push button connected to ground on one side and a PIC32 port pin via a 330Ω resistor. This pin was internally pulled high. The restart button is meant to signal the software to restart the game when pushed after the game has ended.

The audio output circuit takes the audio signal from the DAC, passes it through a low-pass filter, puts it into an audio jack, and outputs the sound to an external speaker. The audio jack�s left and right audio pins are tied together since there was no differentiation between left and right audio. Since the design only required a short sound upon being hit, an audio filter to increase the quality of sound was not constructed.

Software Description:

The software design consists of 5 protothreads, 1 interrupt service routine, 2 helper functions for I2C, and 1 main function. First, we will describe the main function and all the peripherals it sets up along with the variables it initializes. Then we will discuss the purpose of the interrupt service routine and the need for it. Lastly, we will go through the 5 protothreads, describe each one�s functions, and how they intertwine. When describing the communication over I2C with the magnetometer in one of the protothreads, we will describe the read and write I2C functions in detail.

The main function first sets up timers 2 and 3. Timer 2 is set with a period of 400000 so that it completes 100 cycles every second. Timer 3 is set with a period of 909 so that it runs at a rate of 44 KHz. Timer 2 is then set up to be used to trigger an interrupt, while Timer 3 is used to initiate DMA transfer for audio. Then, the main function sets up the SPI channel so that the PIC32 can send data to the DAC in order to produce sound. Next, the I2C channel is set up and I2C1 is enabled. Along with this, we write certain control bits in the magnetometer to bring the magnetometer to active mode from standby mode so that its registers can be read. Next, the ADC is initialized with auto alternate mode, which allows the two specified analog input channels to be read in an alternate fashion via subsequent calls of the ReadADC10 function. Then, certain pins are designated as digital outputs for controlling the vibration motor current drivers. The DMA channel is initialized with Timer 3. The protothreads and TFT are then initialized and system wide interrupts are enabled. The variables for the game (health, score, restart) and the valid bits for the all the screen are initialized to appropriate values before the game commences. Then the 5 protothreads are scheduled round robin.

The interrupt service routine, which executes at 100 Hz, is used to read and process analogy inputs from the accelerometers on the sword. First the ISR checks if the previous sword swing has been processed. If not, then the ISR just ends. Otherwise, we are accepting new sword input. In this case, we read from the ADC to obtain the value for the vertical and horizontal accelerometers. If the vertical accelerometer�s value indicates that the sword is swinging downward, then the horizontal accelerometer value is checked and we determine if the sword is swinging left or right. Then, the accept_sword_input variable is set to false so that the next time the ISR occurs we will not accept further sword input until the current swing is processed. We need to use an ISR to read sword input rather than simply reading it during a thread because the time scale of the accelerometer is too fast to be read by a msec timed thread.

Next we will describe the five protothreads: change screen, display screen, update screen, sword, and sec. The first protothread, change screen, reads from the magnetometer using i2c read. The i2c read function is modeled off of the i2c read function from the Blimp-F-O project (http://people.ece.cornell.edu/land/courses/ece4760/FinalProjects/f2015/bjr96_jl2628/bjr96_jl2628/bjr96_jl2628/index.html). The i2c read function starts an I2C transaction by transmitting the start bit. Then the i2c read function write the slave address of the magnetometer for writing followed by the register to read from. Then the i2c read function restarts the i2c transaction and then it writes the slave address of the magnetometer for reading. Then the i2c read function reads the register and does not acknowledge the result to end the transaction. Based on the values read from the magnetometer, the change screen thread determines if the current screen should change. If it should, then the current thread is cleared. Lastly, the change screen thread displays the player�s stats, such as health and score.

The next protothread is the display screen thread. The display screen thread simply checks if the left and right sides of the current screen are valid (i.e. if they have a projectile that should be displayed). If is is valid, then this thread draws the projectile onto the screen.

The next protothread is the update screen thread. The update screen thread iterates through all of the screens and checks if any of the projectiles have hit the player. If one of the projectiles has hit the player, then that projectile is eliminated off of its respective screen. The health counter is then decremented. Furthermore, the DMA transfer for the hit sound is initiated. Lastly, the appropriate vibration motor is turned on. For example, if the ball that hit the player is on the screen that is behind where the player is currently looking, then the vibration motor on the rear of the helmet will vibrate. If the user was hit from the left, the left vibration motor will vibrate. If the user is hit from the front, all three motors vibrate. After a quarter of a second, all vibration motors are turned off.

The next protothread is the sword thread. The sword thread uses the variables set by the ISR to handle the sword event. Specifically, if there was a left sword swing detected and there is a valid projectile on the left of the current screen, then that projectile is marked as invalid and disappears off the screen. The same is true for the right sword swing and a projectile on the right side of the screen. Once the sword input is processed, the accept_sword_input variable is marked as true so that the ISR can process further accelerometer input.

The last protothread is the sec thread. This thread keeps a running count of the number of seconds that the game has been active. These number of seconds also correspond to the score of the player. If the player�s health hits 0, then this thread clears the screen, displays �GAME OVER� with the score, and then waits in a while loop for the restart button to be pressed. If the restart button is pressed, then the game variables are reinitialized. This means that all screens are marked as invalid (i.e. there are no valid projectiles on any screen), the health is reinstated at 5, and the score begins again at 0. The screen is then cleared for a new game.

Results

Because this was a video game, a lot of the testing dealt with user experience. This meant that if the game looked like it was functioning correctly, then it was functioning correctly. The major aspects of this testing dealt with making sure we were reading from the accelerometers fast enough to capture the sword motions and to make sure we were reading from the magnetometer fast enough to enable smooth transitions of the screen. Lastly, we needed to ensure the important protothreads were getting enough CPU time to render the game and make the visuals look believable. We describe these three aspects of testing further.

First, we worked with the accelerometers. Because of the small time scale of the sensor output when the accelerometer was perturbed about an inertial state, we needed to embed the ADC reading in an interrupt service routine. After slight experimenting with the frequency of the ISR, we determined that 100 Hz was sufficient to capture the behavior of the accelerometer. Having the ISR frequency be lower than this would have resulted in missing information, but having it much higher would cause unnecessary CPU time wasted when the ISR is triggered unnecessarily often. For the accelerometer, when the sensor accelerates in one direction, we see a spike in the output voltage, but when the sensor slows down (decelerates), there is an equivalent negative spike in voltage. It was important to sample fast enough to ensure we were able to capture the first spike so that we could accurately determine the direction of the sword motion.

Next, we worked with the magnetometer. The magnetometer didn�t need to be read as frequently as the accelerometer. This is because we only needed to read the magnetometer as often as we wanted to allow the screen to change. This meant that we could put the code that read from the magnetometer via I2C in the protothread that was in charge of changing which screen the user was viewing. This allowed us to couple the magnetometer reading and the rate of screen change, since these two values should match up. Specifically, it does not do us any good if we read from the magnetometer more often than we are updating which screen the user is looking at. Additionally, we seemed to be getting a significant amount of magnetic interference from the room as the magnetometer was somewhat unreliable at higher degrees of resolution.

Lastly, we needed to ensure the protothreads weren�t yielding for too long and that the game rendering had minimal flicker and glitches. The first place to check was the protothread that actually printed the projectiles on the screen. This thread had to execute often enough that the motion didn�t appear extremely rigid. Next came the protothread that handled the screen updates. This protothread only had to occur as frequently as the protothread that read magnetometer data/changed screens and the protothread that handled sword swings (since this protothread marked certain screens as valid/invalid). The final product was animated smoothly and was responsive in both that the screens changed properly when rotating the helmet and the sword destroyed the balls when the proper swing was performed. There was a noticeable amount of flicker whenever the balls progressed, but this did not impede the gameplay (the most important aspect). The screen was somewhat hard to focus on without the use of the focusing glasses. This would be solved in future iterations.

The board was powered by a relatively short 5V power chord. As such the range of motion was limited for the user. Additionally, the game involves swinging a sword at virtual objects, therefore some caution should be taken to ensure the sword won�t connect with anything in the physical realm. The game is designed to allow for rotational movement, but since the screen impedes the users vision, lateral movement should not be made while the helmet is worn. Excessive rotational movement in a single direction will likely cause the user to become tangled in the power cord, so it is advised to try to make balanced rotational movements. These safety considerations were taken into account when designing, constructing, and testing the device.

To use the system, plug the power cord into the board and place the helmet on the user�s head. Place the sword in the users hands with the yellow arrow facing the user and pointed at the ceiling. Wear the focusing lenses for better visual interaction with the game. Insert the earbuds and flip the power switch. The game will start automatically. The screen displays the user�s score (game time), health, and current screen. The user should rotate their head to look for balls on the screen. Note: try to keep the helmet level for best results. Upon finding a ball, swing downwards and left/right (corresponding to which side of the screen the ball is on). If the swing was successful, the ball will be destroyed. When a ball hits, a sound will be played and the user�s health will be decremented. Upon the health reaching zero, the game will end and the user�s score will be displayed. Press the restart button located above the visor to start a new game. Flip the power switch again to turn the system off.

Conclusions

General Reflections:

The project was a great opportunity to not only gain experience with implementing the hardware and software of an embedded system, but also to design the whole project. It is this design portion that is crucial takeaway from this project. While the previous lab projects had most of the design complete, this term project was completely ours. Going through the design gave us an appreciation for the parts of the previous lab projects that were already provided. For example, in the lab projects, many hardware details had already been thought out and plenty of suggestions were provided along with all the components we would need. In this project, we had to design the hardware and then select hardware components that matched our needs. On the software side, the previous lab projects often had a suggested software implementation, right down to what threads should do what. In this project, we had the opportunity to design how the software would function and interact with the hardware from scratch. It was a very valuable experience.

In addition to the benefits gained from the design aspect, another less tangible but still important skill gained was time management. Because the scale of the project was 5 weeks, it was important to set goal and make sure we were achieving those goals at the specified times. Without this time management, it would have been impossible to complete the project while attempting to shove everything in the last two weeks.

The design lived up to most of our expectations. We had initially planned for six possible screens and a corresponding six vibration motors, but due to the unreliability of the magnetometer at that degree of resolution in addition to a lack of viable port pins for the motors, we dropped the number of possible screens to four. This provided a better user experience than if we had used six possible directions. We also had originally planned for there to only be one ball with a sword that would simply have to detect motion. During the project, we increased the complexity of this section by adding an additional accelerometer and corresponding ball. With this, we were able to make the user�s experience more interactive.

Safety Concerns:

As noted previously, there is the potential to accidentally hit real world objects with the sword. It�s recommended to clear a play area first and only make rotational movements while playing. Additionally, there is a chance the power cord may tangle the user if they make multiple rotations in the same direction. This can be mitigated by making balance rotational movements.

Issues Faced:

There were 4 slight hiccups faced throughout the project.

The first dealt with the sword and accelerometer. The original plan for the sword consisted of a single vertical swing, for which the single accelerometer would have sufficed. After implementing this, we decided to augment the design and allow for multiple balls on the screen. We wanted to add a left, center, and right ball on each screen. The accelerometer we had would only give us information in one dimension, so our solution was two pair two accelerometers, offset by 90 degrees, such that we could get information over a 2D plane. Even with this plan, it was difficult to get the resolution to support a left, center, and right ball, since the boundary for the center ball was difficult to calibrate. In the end, we decided to only include the left and right balls, which was already an improvement over the original design. If we had planned to incorporate multiple balls per screen from the beginning, then we would have most likely purchased a gyroscopic sensor in place of the accelerometer.

The second issue we faced was in finding the current to drive the vibration motors. The current they drew was too much to be sourced from the pins of the PIC32. The solution was to use simple current drivers. The current drivers consisted of transistors, whose gate was driven by the output of the pin. The source and drain of the transistor were connected to an external power supply in series with the vibration motors. This setup worked very well.

The third was that we ran out of viable port pins in the final stages of construction. We were faced with a decision to either include sound in the project or have four motors instead of three. We decided to take out a motor in order to include sound.

The last issue we faced was in interfacing with the magnetometer via I2C. There was a delivery delay causing the magnetometer to be delivered very late in the term. The ultimate solution to setting up the magnetometer was to write the appropriate control bits during the setup phase. To accomplish there, there was some simple trial and error and re-reading of the data sheet. After adding this setup, the magnetometer worked very well.

Further Improvements:

There are several potential improvements to the system. The sword could be made wireless if some from of wireless protocol were used to transmit data between the accelerometer sensor on the sword and the PIC32. The sword could also be programmed to have different movements do different things. For example if an �X� was coming at the user instead of a ball, an �X� motion could be made with the sword to destroy it. Additionally, if a more sophisticated magnetometer were used then more distinct screens would be possible and this would allow for smoother screen changes as the direction the player is facing changes. With a very sophisticated magnetometer, it may be possible to perform gradual, smooth screen updates more similar to the real world than the choppy method of updating the players entire field of view immediately upon reaching a threshold. The display and graphics of the game could be improved to make the projectiles look more realistic. Finally, the helmet interface could be improved so the user did not have to strain as much to focus on the screen. These would be the first improvements to the system given more time.

Schematic

Source: Project – VR Sword Defense Video Game