Enhancing Facial Expression Research – Fujitsu Laboratories uses Online User Group Studies

With the expectation that the rise of AI will increase human-machine collaboration, Fujitsu Laboratories has a long history of facial expression recognition technology research to detect various mental states. The goal is to enable machines to interpret human reaction and behavior more effectively and with greater accuracy. Based on Action units (AUs) – the fundamental actions of individual muscles or groups of muscles, Fujitsu Laboratories’ recognition technology can detect subtle changes in facial expression with a high degree of accuracy. For the past few years, Fujitsu Laboratories has published research papers and conducted pilot projects with this Action Units-based recognition technology and associated AI models to detect human emotion via facial expression under different scenarios, including interaction between people in group settings, user engagement and response during remote meetings, etc.

Demo of Fujitsu Laboratories’ Facial Expression Recognition Technology

Applying the facial expression recognition technology to a group meeting

In order to extend this research and develop more advanced and accurate models for facial expression recognition technology, Fujitsu Laboratories designed and conducted several online experiments. These included cognitive games, which require different levels of user interaction and mental engagement to complete the tasks, as well as self-input surveys. The idea is to map the user’s task performance with their facial expression and self-evaluation results. The cognitive games and tasks were designed based on neuroscience, psychology, and behavior science. Fujitsu Laboratories conducted the experiment with Japanese participants and resulting data helped build these models.

The cognitive task used for the online user group study

The participant’s recorded facial data while performing the cognitive task

As a next step in the research, Fujitsu Laboratories decided to recruit users across different demographic categories to fine tune the AI models to be more inclusive and trustworthy. This approach posed several challenges. The first challenge was the limited time available to complete the experiments and meet the research project milestone – as little as 3 weeks.

The second challenge was how to recruit users across demographics, who would be willing to record their facial video data while performing the tasks. An additional constraint was that participants would need a certain level of computer operation skills. To accelerate the recruiting process, Fujitsu Laboratories identified and recruited LiveMinds – an online participant recruiting platform – as the partner for this project. LiveMinds’ platform is based on behavioral recruitment, which can identify candidates based on their online activities to meet the requirements on a large scale. LiveMinds also had experience recruiting participants for online video recording that the experiment needed.

Given the geographic distribution of the participants and to ensure a smooth user experience and timely communication, Fujitsu Laboratories built a dedicated web site to digitalize and automate the process fully, allowing the users to do the experiments remotely on their own, including downloading the applications and uploading their data and surveys.

Eventually, over 120 heterogeneous users were recruited by LiveMinds to conduct the online experiment, and 90 of them had successfully completed all 6 tasks and uploaded their video data, task scores and survey results. The whole project was initiated and finished within 3 weeks – thanks to LiveMinds’ efficient online recruiting platform and Fujitsu Laboratories’ automated and digitalized process. With the diversified data collected, Fujitsu Laboratories was able to build novel facial expression recognition models with higher performance and better accuracy targeted at different mental states.

Towards the end of the project, the world was impacted by the Covid-19 pandemic, which curtailed all in-person activities. It would therefore be a major challenge for brands that rely on field-based surveys or focus groups to understand consumer feedback. Fujitsu Laboratories believes using facial expression recognition technology coupled with online experiments will provide an innovative solution for enterprises to conduct future online market research, including focus groups, and help brands get insights from users without requiring physical presence and irrespective of geographic boundaries. In general, Fujitsu Laboratories believes the pandemic has accelerated the adoption of digital transformation across many domains, including ResearchTech and innovative solutions like the Fujitsu facial expression recognition technology can be used to quantify the participant engagement for online surveys, user study, or focus groups. Fujitsu Laboratories will soon announce the research results from the study and make the technology available for pilot projects.

For more information, please contact our research team at contact-ferecog@cs.jp.fujitsu.com

Related Articles:

• Press Release
• ZDNet Article