New Mexico Supercomputing Challenge

ASL Detection for Real World Implementation

Team: 1

School: Academy For Tech & Classics

Area of Science: Computer Science & Linguistics

Problem Definition:
     As initially stated in our proposal, Computer Science is a form of advocacy, and that’s exactly what we are accomplishing for this year’s project for the 2013-2014 Supercomputing Challenge. American Sign Language is one of our Nation’s most common languages, not just exclusive for the disabled, yet the technology for its detection in computers has become dwindling, especially when focused on real world implementation. This past year, Google designed the “Sign Language Interpreter App” for their video calling systems which allows users with special needs to appoint another physical user to speak on their behalf during the video call[1]. This program is merely a tool to have another human assist in communication, rather than having the device take the user input and give the appropriate translated output, much like Voice commands within user accessibility today. Although a distinct step forward, we as team #1 ask the question “Is American Sign Language detection currently possible for real world consumer level implementation?”

Problem Solution:
     The solution is rather simple in theory. Collect documentation on different methods of visual user input techniques, whether already implemented in programs or merely hypothetical, and compare them until the strongest techniques are found that correlate with widely distributed hardware. Things such as gestural tracking gloves[2] are a rarity in most Households and can easily cost thousands making them rather pointless to use when currently concerning current real world situations[3] - which is a main element we consider. With the collected and seemingly successful methods, we then need to hopefully implement them into a single project, (If one method is significantly better) to demonstrate if our question is possible to answer. Simplicity is key.

Progress to Date:
     Multiple methods have been collected, and out of a possible eleven methods, two have shown to have true promise. The first method, which we have already started to design in Python 2.7, is a “Contour Comparison System” which takes real-time user input via webcam and compares the given gesture to a library of preselected images in contour. The library being used currently is compiled from independent sources, but we plan on pairing with New Mexico School of the Deaf to take our own images and convert them. An advantage to doing this is that there are regional differences is the styles of fingerspelling within ASL, and we must incorporate our immediate audience to the product. The comparison is currently being done with the OpenCV library[4], but we are exploring possibly using a form of neural networking, an idea originally from Drew Einhorn who reviewed our proposal, to optimize the image selection process. Also we are in the process of trying to eliminate visual background noise from within the contour gestural space, which is lowering the program’s comparison rate.
     The second method is using a trace and search algorithm,[5] and is utilizing the depth sensory feature of the Xbox 360’s Kinect sensor when connected to a computer. The main technique being implemented is recognition of the palm of the hand which is then isolated due to horizontal and vertical projections. The main library originally being used for our code written in Processing 1.5.1, was CLNUI, due to it overall having the best distinguishable features and support, since most of its updates came from PrimeSense[6], the original creator of the Kinect. Yet, due to being unofficially supported and many apparently non-debuggable errors, we are now looking at switching to the OpenKinect[7] library, and restarting the process of development.

Expected Results:
     We expect to have multiple independent models recognizing user input and displaying the correct output with a slim margin of error. This would prove that ASL detection and real world implementation is possible even at a consumer level[8]. We hope that this program will serve as a catalyst for further research and programs to be made to create true user accessibility, disabled or not.

Works Cited:
[1] "Sign Language Interpreter App." - Google Help. 2013 Google Inc., n.d. Web. 05 Dec. 2013.
[2] Hill, David J. "Smart Gloves Turn Sign Language Gestures Into Vocalized Speech." Forbes. Forbes Magazine, 20 Sept. 2012. Web. 05 Dec. 2013.
[3] Liu, Jiayang, Zhen Wang, Jehan Wickramasuriya, and Venu Vasudevan. "UWave: Accelerometer-based Personalized Gesture Recognition and Its Applications." Pervasive Platforms & Architectures Lab Applications & Software Research Center & Motorola Labs, n.d. Web. 03 Dec. 2013.
[4] OpenCV | OpenCV. N.p., n.d. Web. 05 Dec. 2013. .
[5] Szeliski, Richard. Computer Vision: Algorithms and Applications. London: Springer, 2011. Print.
[6] "You Are the Control â Microsoft Kinect." PrimeSense. N.p., n.d. Web. 05 Dec. 2013.
[7] OpenKinect. N.p., n.d. Web. 05 Dec. 2013. .
[8] Foley, James D., and Dam Andries Van. Fundamentals of Interactive Computer Graphics. Reading, MA: Addison-Wesley Pub., 1982. Print.

Team Members:

  Noah Caulfield
  Dmitri Malygin-Voorhees

Sponsoring Teacher: NA

Mail the entire Team

For questions about the Supercomputing Challenge, a 501(c)3 organization, contact us at: consult1314 @

New Mexico Supercomputing Challenge, Inc.
Post Office Box 30102
Albuquerque, New Mexico 87190
(505) 667-2864

Supercomputing Challenge Board of Directors
Board page listing meetings and agendas
If you have volunteered for the Challenge, please fill out our In Kind form.