Facial Gesture Recognition System

A 3D wireframe model of a human face, with triangular facets forming its structure. The face is depicted in a neutral expression and is slightly turned to one side. A blue circle highlights one of the eyes. The background is solid black.

Detects eye movements and facial gestures to aid communication for individuals with Severe Speech and Motor Impairment (SSMI) throughout the day and night.

Team Size:
5
Location:
Australia
Founders:
Azmi CİBİ, Ahmet Gökçe BOZAN, Şafak Akinci, Ali Goktogan
Program(s):

The Problem

Severe Speech and Motor Impairments (SSMI) is a condition characterised by profound limitations in speech and motor functions. Individuals with SSMI often experience significant challenges in articulating and expressing themselves verbally, coupled with limitations in motor control that may affect various aspects of movement and coordination. Hence, their interaction and communication with the surrounding environment and community can be minimal. At different stages of Motor Neuron Disorders (MND), patients encounter challenges in verbal communication and limb mobility. While some patients may retain full or partial control over their facial muscles, allowing for the generation of Facial Gestures (FG), or nonverbal communication cues, the specific facial muscles under control can vary among patients. Although some expensive eye-gaze tracking solutions help those patients, there is no ready-to-use solution to utilise FGs as a communication modality for patients with SSMI.

The Solution

Our solution aimed to help people with SMMI by introducing a new communication modality via a novel, affordable, stand-alone sensor with embedded electronics and software that monitors and detects the patient's eye movements and FGs in the daytime and complete darkness without being attached to a dedicated computer. Using our previous experience detecting eye motions of people with SMMI to enable them to interact with the environment, such as calling for assistance, was a success. However, there was a subset of patients whose eyes were difficult to track due to either their eye shape or the overall morphology of their face. To address this rare but existing issue, in addition to eye tracking, we also add the FG recognition function to our system. Furthermore, we made our new system a stand-alone smart sensor with its own internal, embedded processing system to function with and without being connected to a user's computer.