Sign Language Learner

These inputs then feed into our box which includes all hardware wiring inside as well as our development board. The development board houses the pic32 processor and the tft display. On top of the box we have mounted a button and a switch. The button starts the callibration sequence and the switch changes between the learn and play modes. Both the button and the switch are attached to pins that have internal pull-down resistors enabled via the pic32. A full diagram of the schematic can be found here.

Software Design

i2c_helper.h

This file was originally created by Desmond Caulley (dc686 @cornell.edu), Nadav Nehoran ([email protected]), and Sherry Zhao ([email protected]) for use with their self-balancing robot project. You can read more about their project via the link in the references. This file provides several functions for communicating with the IMU, including a function that returns all the accelerometer and gyroscope readings in an array. For our project, we didn’t need nearly as much sensitivity as this function provided. Therefore, our version of the function ignores the lower bits in the accelerometer readings. Other than this, nothing major has been changed.

sign_language_learner.c

On startup, the main function sets up the five ADC inputs, TFT display, IMU, input pins, and threads. It also calls the init_signs() function to add the data on all 26 signs to the sign array. Then, it initializes and schedules the four threads.

Sensors Thread

The Sensors Thread, protothread_sensors, is responsible for polling the sensor values and saving them. This thread runs every 120ms to ensure that the user receives prompt feedback and that quick motions are detected. The thread polls the contact points, IMU data, and all five ADC inputs. In addition to saving the raw ADC values, the thread also handles debouncing of each finger’s bend level. By requiring several consistent readings before altering the recorded bend, the program lowers the effect of noise.

Calibrate Thread

The Calibration Thread, protothread_calibrate, is used to calibrate the accelerometer and bend levels for the specific user. It is run automatically at startup, and can be run again at any time by pressing the “calibrate” button. It has three stages; each describes a hand position for the user to make and prompts them to press “calibrate” when ready. Then 20 samples of the current sensor data are taken.

Each position corresponds to a level of bend in the user’s fingers. Every finger is calibrated independently of the others to account for differences in how easy or far they can be bent. The ADC values from each sensor are recorded and averaged over all the readings to define the boundary points between the different levels of bend. These boundaries are used to determine the positions of the fingers during the learn and game modes.

In addition, the first calibration position requires the user to place their hand flat on the table in front of them, with their fingers pointed straight ahead. This is used to calibrate the accelerometer readings from the IMU. Since the user could be facing any direction, it’s necessary to calibrate each time to make sure their hand is facing the correct direction relative to their body. Once calibration is complete, the thread yields until “calibrate” is pressed again.

Learn Thread

The Learn Thread, protothread_learn, is used to run the logic for “learn” mode. This thread only runs when the switch is in the lower position and calibration is complete. When run, the thread displays a letter, an image of the correct hand position, and a pair of indicators. Below is an example of what the tft displays during the learn mode.

The first indicator is shaped like a hand; individual fingers turn red if they are incorrectly positioned, and white if they are correct. We chose to use the colors red and white because colorblind users would still be able distinguish them easily. Red dots on the fingers appear when a contact point at the indicated location is connected incorrectly. The second indicator, a circle with two lines through it, represents the hand’s orientation. If the x, y, or z orientation is incorrect, the corresponding line or circle turns red.

While a sign is being displayed, the thread checks the user’s hand position every 120ms. If the user is doing the sign correctly, a green circle will appear in the middle of the screen. An additional circle appears for every consecutive tick that the user is correct for; a mistake resets this. Once they reach five consistent readings, the sign is considered complete and the next letter in the alphabet is displayed.

To check if a sign is correct, the sign’s required bend levels are compared to the last recorded bend levels for each finger. The bend levels are defined as 0 (unbent), 1, and 2 (completely bent). Additionally, a finger position can be designated as “either 1 or 2”, which is useful for signs where the level of bend can vary while still appearing correct. Similarly, the contact point states and accelerometer values are compared to their ideal values. The accelerometer values have large deadbands (20 for x and y, 30 for z) to decrease the required level of accuracy while still rejecting signs that are blatantly wrong.

For signs that require a motion (j and z), more checks are hardcoded in. A single green circle will be displayed when the starting position is correct. After this, the gyroscope is used to detect a hand movement or rotation in the specified direction. For multi-stage motions, a timer gives the user 10 cycles to finish all movements, and an additional circle is added for each correct motion. Once the movement stops, the final orientation is checked; if it is correct, the sign is complete.

Game Thread

The Game Thread, protothread_game, is used to run the logic for “play” mode. This thread only runs when the switch is in the upper position and calibration is complete. The game thread works similarly to the learn thread, and uses the same functions to check if a sign is being formed correctly. However, there are a few differences.

First, the letters are displayed in a random order instead of alphabetically, and the pictures are not shown. This increases the difficulty for the player. However, the error indicators are still visible to prevent the game from becoming too frustrating due to the level of precision required for some of the signs. Secondly, there is a time limit for each sign. If the timer bar (displayed at the bottom of the screen) runs out before the sign is completed, the words “GAME OVER” are displayed. Three seconds after a game over, a new game starts. The timer starts at a little less than 10 seconds, and gets steadily faster as the game continues. Finally, the number of signs the player has correctly completed is displayed in the upper left hand corner of the screen as their score. An example of the tft during the game mode is shown below.

Results

Our final product was able to detect the hand gesture with fairly close accuracy to the standard ASL letters.

In learning mode, our device was able to give accurate feedback in regards to corrections that needed to be made to signal the letter correctly. The images displayed for each letter were very clear, and allowed the user to get a good understanding of what his/her hand should look like. All 26 letters of the alphabet were tested and worked properly, and the display showed any changes or adjustments made by the user as soon as they were made. The game mode successfully displayed a green bar shrinking to signal time left, as well keeping score of the letters correctly signaled and slowly decreasing the time given to signal correctly.

The only safety concern we had was a potential fire hazard by having the wiring inside a cardboard box. To deal with this, we made sure we covered all the bare wire connections with electrical tape to avoid heat conduction, particularly the push button and switch as they come in direct contact with the cardboard. No interference with other designs needed to be accounted for.

All members of the group were able to use this device. This device is particularly designed for people with hearing impairment, so all instructions are either displayed on the screen or written in the box completely avoiding the use of sounds as intended. However, the glove is only one size so it’s not user friendly to people whose hand size isn’t compatible with the glove. Conveniently, all members of our group have similar sized hands and were all able to use it.