A Speech and Lip Reading System with Generating Text to Assist Hearing-Impaired People

Author : Tsung-Han Tsai, Yu Syuan Wu, Pin Kuan Li

Abstract :Hearing impairment is a common physical disability that affects daily life and social interactions. Current assistive tools, such as hearing aids and sign language interpretation, are more familiar to individuals with congenital hearing loss but may not be as convenient or effective for those with acquired hearing loss. Additionally, some patients find hearing aids uncomfortable to wear. Therefore, there is a need to develop more effective assistive technology. This research aims to develop a system that automatically converts speech to text, utilizing speech recognition and natural language processing techniques. The system will be tested for accuracy and practicality. Additionally, we will explore lip-reading technology, which helps individuals with hearing loss interpret mouth movements. Using video analysis and deep learning algorithms, the system will detect lip movements and convert them into text. The entire system is implemented on the NVIDIA Jetson AGX Xavier, ensuring efficient real-time processing for practical use

Keywords :Lip Reading, Audio Visual Speech Recognition, Deep Learning.

Conference Name :International Conference on Pattern Recognition and Image Processing (ICPRIP-25)

Conference Place Seoul, South Korea

Conference Date 23rd Aug 2025