Michael Darmstadt and Marius Martin, graduates of the dual degree program in Information Systems, have been awarded the Prize of the University Council of the Mainz University of Applied Sciences for their outstanding academic achievements. Michael Darmstadt and Marius Martin received the award, which is endowed with €2,000, for their bachelor's thesis Anwendung neuronaler Netze zur Echtzeiterkennung von Gebärden der deutschen Gebärdensprache (Application of neural networks for real-time recognition of German sign language signs). The thesis, supervised by Prof. Dr. Anett Mehler-Bicher, was awarded the best possible grade of 1.0.
Focus on suitability for everyday use
AI-based real-time translation systems now exist for numerous languages. Sign language translation, on the other hand, is still in its early stages. For the approximately 140,000 people who rely on sign language interpreters in Germany, computer-assisted translation can significantly ease the communicative barrier between them and hearing people. Due to the general complexity of sign languages, machine interpretation is challenging and different approaches need to be tested first. It should be noted that there is not only one sign language, but sign languages differ like spoken languages. For example, there is a specific German sign language.
The goal of the bachelor's thesis by Michael Darmstadt and Marius Martin is the analysis of the question of whether and how well neural networks are suitable for the real-time recognition of signs. This is to be evaluated by means of a prototype yet to be implemented. The work can be classified in the research field of computational linguistics, i.e. the machine processing of natural languages, since it deals in particular with the aspect of sign language, which represents a natural language. The focus is on the recognition of individual signs and must be distinguished from the translation into a spoken language.
Machine learning methods based on neural networks are used in most research approaches that address the recognition and translation of signs with the help of computers. In these, the software does not have to be explicitly programmed; instead the computer learns from examples. Since a human cannot describe the signs that precisely and program a computer to recognize them, a self-learning machine based on neural networks is the logical approach. The thesis also puts a special emphasis on the suitability of a potential system for everyday use.
3,000 sample videos
To determine the extent to which a neural network is suitable for recognizing individual signs, a software prototype was created using the multimedia machine learning framework MediaPipe. For the prototype, eight different variants of neural networks were trained on over 3,000 sample videos for 30 different signs. The best models were implemented in the prototype and experiments were conducted regarding sign recognition. The best model achieved a recognition rate of 96.7% for the known signs in a controlled environment.
Basis for further research
"The thesis is directed at highly contemporary research and implementation approaches and is outstanding," said Prof. Dr. Anett Mehler-Bicher. "With 30 signs, however, it is still a very limited vocabulary. In addition, the prototype is only capable of recognizing individual signs. Contextual sentences cannot be recognized at the moment. Nevertheless, the results of this thesis can serve as a basis for further research. For the approximately 140,000 people who depend on sign language interpreters in Germany, a computer-assisted translation could significantly ease the communicative barrier between them and hearing people. This bachelor's thesis by Michael Darmstadt and Marius Martin is an important first step in this direction and consequently of great interest to society."
The award was endowed by the University Council in 2009. Prof. Dr. Susanne Weissman, President of Mainz University of Applied Sciences, emphasized at the award ceremony, "This award is a special distinction for students and graduates, with which our university honors outstanding academic achievements. With this year's award, we are pleased to recognize a thesis that has great potential to be pursued as a possible research project."
To watch a video demonstration of the results of the thesis, visit
https://github.com/Tachionstrahl/SignLanguageRecognition/raw/master/docs/demo.mp4
Other theses nominated for the 2021 University Council Award:
Sophia Lassimo: Limits to Growth: A Systematic Investigation of the Constraints at Frankfurt Airport
(Master's thesis, Advisor: Prof. Dr. Anna Rosinus, School of Business)
Ina-Maria Bernsdorf: "Moment" - Lässt sich mit einer architektonischen Form in Gestalt eines Raums eine Legende erzählen, die erst in 200 Jahren ihre ganze Kraft entwickelt? (Can an architectural form in the shape of a room be used to tell a legend that will only develop its full potential in 200 years?)
(Bachelor’s thesis, Advisor: Prof. Andreas Kaiser, School of Design)
Lilly Gothe: "transform – Potenziale für Nachhaltigkeit an der Hochschule Mainz” (transform - Potentials for Sustainability at Mainz University of Applied Sciences)
(Bachelor’s thesis, Advisor: Prof. Charlotte Schröner, School of Design)
Lukas Haas: "Bestimmung von Trassierungselementen aus in QGIS digitalisiertem grafischen Entwurf” (Determination of routing elements from graphic design digitized in QGIS).
(Bachelor’s thesis, Advisor: Prof. Dr. Renate Czommer, School of Engineering)
Viviane Urmann: "Experimentelle Untersuchungen zu den Frischbetoneigenschaften und der Druckfestigkeit von Polyurethan-Beton” (Experimental investigations of fresh concrete properties and pressure resistance of polyurethane concrete).
(Bachelor’s thesis, Advisor: Prof. Dr.-Ing. Heiko Merle, School of Engineering)
