Murmurations
A low power wearable Silent Speech Interface that uses piezoelectric sensors on the face to turn jaw and lip micromovements into text in real time for people who can no longer rely on their voice.
Project Overview
Murmurations is a low power wearable Silent Speech Interface that uses piezoelectric force sensors on the face to capture jaw and lip micromovements and map them to text using adapted automatic speech recognition models. It is aimed at people who can no longer rely on their voice after a stroke, surgery or neurodegenerative disease.
Instead of recording audible speech, the device senses how the skin and muscles around the mouth and jaw move when a person mouths words. Piezoelectric sensors, embedded in a conformal mask that can be tailored to each user via 3D scanning, capture these deformations, as shown in Figure 1. The project builds on Automatic Speech Recognition (ASR) models that were originally designed for audio to map the sensor data to text, enabling truly silent communication.
Figure 1a: Closeup view of the Murmurations device.
Figure 1b: Side view of the Murmurations gasket fit.
Motivation and Context
Silent Speech Interfaces sit at the intersection of assistive technology and human computer interaction. They are particularly relevant for people who cannot rely on their voice after stroke, surgery or neurodegenerative disease, but also for situations where speaking out loud is not possible or desirable, such as noisy environments, shared spaces or privacy sensitive contexts.
Most existing systems rely on electrically-based sensing such as surface EMG or EEG. While powerful in the lab, these approaches often suffer from low signal to noise ratio, sensitivity to both external noise and changes in ambient conditions. Mechanical sensing methods, such as strain and force sensors, have recently been shown to capture articulatory motion more robustly with simpler hardware. Murmurations builds on this line of work by exploring commercially available piezoelectric sensors as the primary sensing modality.
From a modeling perspective, the project is also a testbed for bringing modern audio ASR architectures to a new sensing modality. In particular, it aims to explore the Zipformer model architecture [1], which has proved to be effective for audio based speech recognition on constrained platforms such as mobile phones and wearables, and apply it to the task of speech recognition from facial micromovements. Zipformer transcribes audio to text by predicting phoneme sequences, which allows the system to handle virtually any word or sentence and naturally extend to multiple languages.
System Architecture
The device currently uses two piezoelectric sensors, one positioned under the lip on the chin and the other near the corner of the mouth on the cheek, a configuration that captures complementary movements from speech articulation. The sensors connect to a high resolution 24-bit ADC, which samples the signals at 8 kHz before sending them to an ESP32-S3 microcontroller. The ESP32 performs basic digital filtering and streams the data to a host computer or phone over WiFi. For preliminary experiments, an I2S microphone is used to record audio in parallel. The audio is not used during inference, but it provides a reference for aligning sensor data with the spoken transcription in non-self-aligned machine learning models. The complete system architecture is shown in Figure 2.
Figure 2: Block diagram of the Murmurations system, from facial micromovements to sensor front end, embedded platform and ASR based decoder.
Wearable Design
To mount the piezoelectric sensors on the face and host the electronics, the device is implemented as a hollow wearable mask. The mask is 3D printed in hard plastic and houses both the sensors and the electronics, as shown in Figure 3.
Figure 3: 3D printed hollow mask.
The mask is designed to be comfortable to wear for extended periods of time, thanks to a soft silicone gasket that fits snugly around the face and can be tailored to each user via 3D scanning and custom molding. A 3D printed mold is used to cast the silicone gasket directly into the mask, as shown in Figure 4. The rigid shell contains several cavities that are filled with silicone to create the gasket and keep it in place.
Figure 4a: 3D printed mold for the silicone gasket.
Figure 4b: Cast mask with silicone gasket and embedded piezos.
By casting the silicone gasket into the mask, the piezoelectric sensors are embedded into the gasket, as depicted in Figure 4. This creates a better mechanical impedance match between the face and the device, which minimizes signal loss due to piezo saturation and improves the overall performance of the system.
Machine Learning Pipeline
The machine learning pipeline for Murmurations is based on the Zipformer model architecture [1], which has shown strong performance for audio based speech recognition. The model takes a sequence of MEL spectrograms as input, together with the audio transcription of the corresponding recording, and outputs a string of text. By leveraging this architecture, the recorded data does not need to be manually aligned; only the transcription of each recording is required.
However, training a Zipformer based model typically requires several hundred hours of sensor data. Since the project is still in its early stages, we first validated the idea using a simplified model consisting of a 1D-ResNet that operates on fixed length segments. Because this model is not self-aligned, we use the Short-Time-Energy of the audio recorded in parallel to locate speech segments and align the sensor data with the corresponding transcription, as shown in Figure 5.
Figure 5: Short-Time-Energy of the audio recorded in parallel to the sensor data.
On a biphonemic vocabulary consisting of 42 homophonic words, this setup reaches around 97.1% classification accuracy when processing both ADC channels jointly, and recording around 50 instances of each word with a 70%-15%-15% split into training, validation and test sets, as shown in Figure 6.
Figure 6: Confusion matrix of the 1D ResNet classifier over 42 homophonic words.
References
[1] Z. Yao, L. Guo, X. Yang, W. Kang, F. Kuang, Y. Yang, Z. Jin, L. Lin, and D. Povey, "Zipformer: A faster and better encoder for automatic speech recognition," arXiv preprint arXiv:2310.11230, 2024. [Online]. Available: https://arxiv.org/abs/2310.11230.