ASL Detection for Real World Implementation
School: Academy For Tech & Classics
Area of Science: Computer Science & Linguistics
As initially stated in our proposal, Computer Science is a form of advocacy, and that’s exactly what we are accomplishing for this year’s project for the 2013-2014 Supercomputing Challenge. American Sign Language is one of our Nation’s most common languages, not just exclusive for the disabled, yet the technology for its detection in computers has become dwindling, especially when focused on real world implementation. This past year, Google designed the “Sign Language Interpreter App” for their video calling systems which allows users with special needs to appoint another physical user to speak on their behalf during the video call. This program is merely a tool to have another human assist in communication, rather than having the device take the user input and give the appropriate translated output, much like Voice commands within user accessibility today. Although a distinct step forward, we as team #1 ask the question “Is American Sign Language detection currently possible for real world consumer level implementation?”
The solution is rather simple in theory. Collect documentation on different methods of visual user input techniques, whether already implemented in programs or merely hypothetical, and compare them until the strongest techniques are found that correlate with widely distributed hardware. Things such as gestural tracking gloves are a rarity in most Households and can easily cost thousands making them rather pointless to use when currently concerning current real world situations - which is a main element we consider. With the collected and seemingly successful methods, we then need to hopefully implement them into a single project, (If one method is significantly better) to demonstrate if our question is possible to answer. Simplicity is key.
Progress to Date:
Multiple methods have been collected, and out of a possible eleven methods, two have shown to have true promise. The first method, which we have already started to design in Python 2.7, is a “Contour Comparison System” which takes real-time user input via webcam and compares the given gesture to a library of preselected images in contour. The library being used currently is compiled from independent sources, but we plan on pairing with New Mexico School of the Deaf to take our own images and convert them. An advantage to doing this is that there are regional differences is the styles of fingerspelling within ASL, and we must incorporate our immediate audience to the product. The comparison is currently being done with the OpenCV library, but we are exploring possibly using a form of neural networking, an idea originally from Drew Einhorn who reviewed our proposal, to optimize the image selection process. Also we are in the process of trying to eliminate visual background noise from within the contour gestural space, which is lowering the program’s comparison rate.
The second method is using a trace and search algorithm, and is utilizing the depth sensory feature of the Xbox 360’s Kinect sensor when connected to a computer. The main technique being implemented is recognition of the palm of the hand which is then isolated due to horizontal and vertical projections. The main library originally being used for our code written in Processing 1.5.1, was CLNUI, due to it overall having the best distinguishable features and support, since most of its updates came from PrimeSense, the original creator of the Kinect. Yet, due to being unofficially supported and many apparently non-debuggable errors, we are now looking at switching to the OpenKinect library, and restarting the process of development.
We expect to have multiple independent models recognizing user input and displaying the correct output with a slim margin of error. This would prove that ASL detection and real world implementation is possible even at a consumer level. We hope that this program will serve as a catalyst for further research and programs to be made to create true user accessibility, disabled or not.
 "Sign Language Interpreter App." - Google Help. 2013 Google Inc., n.d. Web. 05 Dec. 2013.
 Hill, David J. "Smart Gloves Turn Sign Language Gestures Into Vocalized Speech." Forbes. Forbes Magazine, 20 Sept. 2012. Web. 05 Dec. 2013.
 Liu, Jiayang, Zhen Wang, Jehan Wickramasuriya, and Venu Vasudevan. "UWave: Accelerometer-based Personalized Gesture Recognition and Its Applications."Rice.edu. Pervasive Platforms & Architectures Lab Applications & Software Research Center & Motorola Labs, n.d. Web. 03 Dec. 2013.
 OpenCV | OpenCV. N.p., n.d. Web. 05 Dec. 2013. .
 Szeliski, Richard. Computer Vision: Algorithms and Applications. London: Springer, 2011. Print.
 "You Are the Control â Microsoft Kinect." PrimeSense. N.p., n.d. Web. 05 Dec. 2013.
 OpenKinect. N.p., n.d. Web. 05 Dec. 2013. .
 Foley, James D., and Dam Andries Van. Fundamentals of Interactive Computer Graphics. Reading, MA: Addison-Wesley Pub., 1982. Print.
Sponsoring Teacher: NA
Mail the entire Team