Next-gen hearing aids use AI to ‘read’ lips — even if people wear face masks!

GLASGOW, Scotland — The next generation of hearing aids will be able to read lips even if people are wearing face masks, reducing ambient noise, a new study explains.

An international team of engineers and computing scientists developed the technology, which pairs radio-frequency sensing with artificial intelligence for the first time, along with traditional hearing aid technology, to identify lip movements. It could help tackle the “cocktail party effect,” a common shortcoming of traditional hearing aids as the broad spectrum of amplification can make it difficult for users to focus on specific sounds, like conversation with one specific person in a crowd.

New “smart” hearing aids would combine conventional audio amplification with a second device to collect additional data for improved performance, unlike previous attempts using cameras to lip read which raised privacy issues and did not work with masks.

In a new paper, published in the journal Nature Communications, the University of Glasgow-led team outline how they set out to harness cutting-edge sensing technology to read lips.

‘Teaching’ smart hearing aids to understand speech

It preserves privacy by collecting only radio-frequency data, with no accompanying video footage. To develop the system, the researchers asked male and female volunteers to repeat the five vowel sounds (A, E, I, O, and U) first while unmasked and then while wearing a surgical mask. As the volunteers repeated the vowel sounds, a dedicated radar sensor and a Wi-Fi transmitter scanned their faces using radio-frequency signals. Researchers also scanned their faces while their lips remained still.

Then, the team used the 3,600 samples of data collected during the scans to “teach” machine learning and deep learning algorithms how to recognize the characteristic lip and mouth movements associated with each vowel sound. Since radio-frequency signals can easily pass through the volunteers’ masks, the algorithms could also learn to read masked users’ vowel formation.

The system proved to be capable of correctly reading the volunteers’ lips most of the time. The learning algorithms also correctly interpreted Wi-Fi data up to 95 percent of the time for unmasked lips, and 80 percent for masked. Meanwhile, the system correctly interpreted radar data up to 91 percent of the time without a mask, and 83 percent of the time with a mask.

hearing aid technology
(Credit: University of Glasgow)

Providing ‘transformative benefits’ for the hearing-impaired

“Around five percent of the world’s population – about 430 million people – have some kind of hearing impairment, says Dr. Qammer Abbasi of the University of Glasgow’s James Watt School of Engineering in a university release.

“Hearing aids have provided transformative benefits for many hearing-impaired people. A new generation of technology which collects a wide spectrum of data to augment and enhance the amplification of sound could be another major step in improving hearing-impaired people’s quality of life.”

“With this research, we have shown that radio-frequency signals can be used to accurately read vowel sounds on people’s lips, even when their mouths are covered. While the results of lip-reading with radar signals are slightly more accurate, the Wi-Fi signals also demonstrated impressive accuracy,” Dr. Abbasi continues.

“Given the ubiquity and affordability of Wi-Fi technologies, the results are highly encouraging which suggests that this technique has value both as a standalone technology and as a component in future multimodal hearing aids.”

“This technology is an outcome from two research projects funded by the Engineering and Physical Sciences Research Council (EPSRC), called COG-MHEAR and QUEST,” adds co-author Professor Muhammad Imran, head of the University of Glasgow’s Communications, Sensing and Imaging research group.

“Both aim to find new methods of creating the next generation of healthcare devices, and this development will play a major role in supporting that goal.”

The research team received funding from the Engineering and Physical Sciences Research Council (EPSRC) for this project.

South West News Service writer Sarah Ward contributed to this report.

YouTube video