Facial expression recognition through muscle synergies and estimation of facial keypoint displacements through a skin-musculoskeletal model using facial sEMG signals
The development of facial expression recognition (FER) and facial expression generation (FEG) systems is essential to enhance human-robot interactions (HRI). The facial action coding system is widely used in FER and FEG tasks, as it offers a framework to relate the action of facial muscles and the r...
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2025-02-01
|
| Series: | Frontiers in Bioengineering and Biotechnology |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/fbioe.2025.1490919/full |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1823856558219460608 |
|---|---|
| author | Lun Shu Victor R. Barradas Zixuan Qin Yasuharu Koike |
| author_facet | Lun Shu Victor R. Barradas Zixuan Qin Yasuharu Koike |
| author_sort | Lun Shu |
| collection | DOAJ |
| description | The development of facial expression recognition (FER) and facial expression generation (FEG) systems is essential to enhance human-robot interactions (HRI). The facial action coding system is widely used in FER and FEG tasks, as it offers a framework to relate the action of facial muscles and the resulting facial motions to the execution of facial expressions. However, most FER and FEG studies are based on measuring and analyzing facial motions, leaving the facial muscle component relatively unexplored. This study introduces a novel framework using surface electromyography (sEMG) signals from facial muscles to recognize facial expressions and estimate the displacement of facial keypoints during the execution of the expressions. For the facial expression recognition task, we studied the coordination patterns of seven muscles, expressed as three muscle synergies extracted through non-negative matrix factorization, during the execution of six basic facial expressions. Muscle synergies are groups of muscles that show coordinated patterns of activity, as measured by their sEMG signals, and are hypothesized to form the building blocks of human motor control. We then trained two classifiers for the facial expressions based on extracted features from the sEMG signals and the synergy activation coefficients of the extracted muscle synergies, respectively. The accuracy of both classifiers outperformed other systems that use sEMG to classify facial expressions, although the synergy-based classifier performed marginally worse than the sEMG-based one (classification accuracy: synergy-based 97.4%, sEMG-based 99.2%). However, the extracted muscle synergies revealed common coordination patterns between different facial expressions, allowing a low-dimensional quantitative visualization of the muscle control strategies involved in human facial expression generation. We also developed a skin-musculoskeletal model enhanced by linear regression (SMSM-LRM) to estimate the displacement of facial keypoints during the execution of a facial expression based on sEMG signals. Our proposed approach achieved a relatively high fidelity in estimating these displacements (NRMSE 0.067). We propose that the identified muscle synergies could be used in combination with the SMSM-LRM model to generate motor commands and trajectories for desired facial displacements, potentially enabling the generation of more natural facial expressions in social robotics and virtual reality. |
| format | Article |
| id | doaj-art-11b4b508a72f4da5a69669b0e0f3d339 |
| institution | Kabale University |
| issn | 2296-4185 |
| language | English |
| publishDate | 2025-02-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Bioengineering and Biotechnology |
| spelling | doaj-art-11b4b508a72f4da5a69669b0e0f3d3392025-02-12T07:26:28ZengFrontiers Media S.A.Frontiers in Bioengineering and Biotechnology2296-41852025-02-011310.3389/fbioe.2025.14909191490919Facial expression recognition through muscle synergies and estimation of facial keypoint displacements through a skin-musculoskeletal model using facial sEMG signalsLun Shu0Victor R. Barradas1Zixuan Qin2Yasuharu Koike3Department of Information and Communications Engineering, Institute of Science Tokyo, Yokohama, JapanInstitute of Integrated Research, Institute of Science Tokyo, Yokohama, JapanDepartment of Information and Communications Engineering, Institute of Science Tokyo, Yokohama, JapanInstitute of Integrated Research, Institute of Science Tokyo, Yokohama, JapanThe development of facial expression recognition (FER) and facial expression generation (FEG) systems is essential to enhance human-robot interactions (HRI). The facial action coding system is widely used in FER and FEG tasks, as it offers a framework to relate the action of facial muscles and the resulting facial motions to the execution of facial expressions. However, most FER and FEG studies are based on measuring and analyzing facial motions, leaving the facial muscle component relatively unexplored. This study introduces a novel framework using surface electromyography (sEMG) signals from facial muscles to recognize facial expressions and estimate the displacement of facial keypoints during the execution of the expressions. For the facial expression recognition task, we studied the coordination patterns of seven muscles, expressed as three muscle synergies extracted through non-negative matrix factorization, during the execution of six basic facial expressions. Muscle synergies are groups of muscles that show coordinated patterns of activity, as measured by their sEMG signals, and are hypothesized to form the building blocks of human motor control. We then trained two classifiers for the facial expressions based on extracted features from the sEMG signals and the synergy activation coefficients of the extracted muscle synergies, respectively. The accuracy of both classifiers outperformed other systems that use sEMG to classify facial expressions, although the synergy-based classifier performed marginally worse than the sEMG-based one (classification accuracy: synergy-based 97.4%, sEMG-based 99.2%). However, the extracted muscle synergies revealed common coordination patterns between different facial expressions, allowing a low-dimensional quantitative visualization of the muscle control strategies involved in human facial expression generation. We also developed a skin-musculoskeletal model enhanced by linear regression (SMSM-LRM) to estimate the displacement of facial keypoints during the execution of a facial expression based on sEMG signals. Our proposed approach achieved a relatively high fidelity in estimating these displacements (NRMSE 0.067). We propose that the identified muscle synergies could be used in combination with the SMSM-LRM model to generate motor commands and trajectories for desired facial displacements, potentially enabling the generation of more natural facial expressions in social robotics and virtual reality.https://www.frontiersin.org/articles/10.3389/fbioe.2025.1490919/fullfacial expression recognitionsEMGmuscle synergymusculoskeletal modelfacial keypoints estimation |
| spellingShingle | Lun Shu Victor R. Barradas Zixuan Qin Yasuharu Koike Facial expression recognition through muscle synergies and estimation of facial keypoint displacements through a skin-musculoskeletal model using facial sEMG signals Frontiers in Bioengineering and Biotechnology facial expression recognition sEMG muscle synergy musculoskeletal model facial keypoints estimation |
| title | Facial expression recognition through muscle synergies and estimation of facial keypoint displacements through a skin-musculoskeletal model using facial sEMG signals |
| title_full | Facial expression recognition through muscle synergies and estimation of facial keypoint displacements through a skin-musculoskeletal model using facial sEMG signals |
| title_fullStr | Facial expression recognition through muscle synergies and estimation of facial keypoint displacements through a skin-musculoskeletal model using facial sEMG signals |
| title_full_unstemmed | Facial expression recognition through muscle synergies and estimation of facial keypoint displacements through a skin-musculoskeletal model using facial sEMG signals |
| title_short | Facial expression recognition through muscle synergies and estimation of facial keypoint displacements through a skin-musculoskeletal model using facial sEMG signals |
| title_sort | facial expression recognition through muscle synergies and estimation of facial keypoint displacements through a skin musculoskeletal model using facial semg signals |
| topic | facial expression recognition sEMG muscle synergy musculoskeletal model facial keypoints estimation |
| url | https://www.frontiersin.org/articles/10.3389/fbioe.2025.1490919/full |
| work_keys_str_mv | AT lunshu facialexpressionrecognitionthroughmusclesynergiesandestimationoffacialkeypointdisplacementsthroughaskinmusculoskeletalmodelusingfacialsemgsignals AT victorrbarradas facialexpressionrecognitionthroughmusclesynergiesandestimationoffacialkeypointdisplacementsthroughaskinmusculoskeletalmodelusingfacialsemgsignals AT zixuanqin facialexpressionrecognitionthroughmusclesynergiesandestimationoffacialkeypointdisplacementsthroughaskinmusculoskeletalmodelusingfacialsemgsignals AT yasuharukoike facialexpressionrecognitionthroughmusclesynergiesandestimationoffacialkeypointdisplacementsthroughaskinmusculoskeletalmodelusingfacialsemgsignals |