The ability to estimate a person’s emotions precisely from their facial expressions
Facial recognition technology itself is not a new concept, with images captured by cameras being used to identify people in various fields in recent years. However, this technology is now undergoing a real transformation with the advent of facial expression recognition technology: estimating human emotions by recognizing various facial expressions from facial images. It is a major research theme for Fujitsu Laboratories, as Junya Saito, a researcher of the Trusted AI Project, explains,
“Mainstream research and practical applications for facial expression recognition technology were initially based on relatively straightforward facial expression changes such as smiling and showing anger. This was too basic for our requirement and we wanted to take the concept much further. Our ambition was to apply the technology much more widely, for example analyzing customer responses in the marketing field and the observation of patients in the medical field. This meant that we needed to develop a technology that could capture much more subtle facial expression changes, including consent or confusion. One of the challenges was how to define these subtle facial expressions objectively, as how we look and interpret facial expressions can vary significantly from person to person.”
The Fujitsu Laboratories’ team came up with a novel approach, using "Action Units" that are based on anatomical knowledge and describe facial activities in relation to facial muscles.
Flow of facial expression recognition technology under development at Fujitsu Laboratories (Detection of Action Units and estimation of emotional and psychological state based on the detection)" Action Units (AUs) are the fundamental actions of facial muscles, consisting of about 40 types of different actions. They are used in the field of facial expression research in psychology. Although the emotions conveyed by facial expressions vary from person to person, Action Units make it possible to quantify each expression objectively, because AUs are based on human muscle movement. We started our research and development on facial expression recognition technology with this as the springboard,” continues Junya.
In 2018, Fujitsu Laboratories launched a joint research project with Carnegie Mellon University (CMU) in the United States to accelerate its research activities. The collaboration majored on how to detect subtle facial expression changes based on Action Units. Koichiro Niinuma, a researcher now working at Fujitsu Laboratories of America (FLA), played an important part in co-ordinating the research with CMU, as he elaborates.
"At Fujitsu Laboratories, we have been researching technologies to sense, quantify and analyze human emotion, expression, behavior and motivation for a long time. However, for this application, we knew that we needed to broaden the scope considerably to encompass not just the technology base but also the psychology aspects, before we could succeed in connecting facial expressions with emotions. This was the driver behind our collaboration with CMU, which is a leading research institute in this field, working on a joint research and development basis.”
International Recognition: Achieved No. 1 recognition accuracy in an academic competition
A major milestone for Fujitsu Laboratories came in October 2020, when the technology achieved the top spot for recognition accuracy at a competition staged by the IEEE (Institute of Electrical and Electronics Engineers) during an international conference. This reflected our long-standing work behind improving the accuracy in facial expression recognition, its applications, and practical use, which has successfully contributed to the highly accurate recognition of Action Units.
"In the competition at the international academic conference held by IEEE in October 2020, Fujitsu Laboratories' technology achieved the No. 1 recognition accuracy. For Junya and the team, this was a very important moment, “We are proud to have achieved the world’s highest level of accuracy, and believe that this is our strong point.”
The technology is now being applied as part of a practical Proof of Concept (PoC) with Best Life Promotion Ltd., which provides health services to Fujitsu Group employees and their families, as well as other medical insurers, companies, and individuals. In this demonstration experiment, Fujitsu Laboratories’ facial expression recognition technology is used as a means of providing insured persons’ information to public health nurses, who provide specific health guidance remotely.
“As working from home becomes ever more widespread due to the COVID-19 pandemic, so too does the likelihood of our being confronted with trying to guess the feelings and physical conditions of others during online communication. Facial expression recognition technology can help us to solve such problems by showing speakers’ subtle facial expressions, even though we cannot meet them in person. With this technology, online meetings become much more like actual face-to-face meetings. We are currently putting this to the test, carrying out PoCs in a variety of business areas including workplace re-evaluation and healthcare,” Junya continues.
R & D teamwork in Japan and the United States
Our facial expression recognition technology project is currently being driven by two groups; one that develops basic technologies for detecting action units from facial images, and the other involved in conducting applied research to progress these basic technologies towards practical applications. From the US-based team, Koichiro Niinuma is one of the researchers from the first group. With his extensive experience in image recognition projects, he is focused on the research and development aspects, as well as how to incorporate advanced technologies. He explains,
“Just after joining Fujitsu Laboratories, I was mainly engaged in R & D on computer vision systems, such as biometric authentication algorithm development and AR (augmented reality)/VR (virtual reality). When I heard about this new research and development project, I immediately volunteered and moved to the United States in 2018. My current mission is to provide feedback to Japan on the technologies resulting from our academic research with CMU.”
Junya Saito is Koichiro’s counterpart in Japan in the basic technology research and development group, expanding,
“Since I joined Fujitsu Laboratories, I have consistently been involved in R & D related to human beings. At first, I was engaged in speech synthesis technology, and two years ago, I joined the research and development team involved with facial expression recognition technology based on Action Units. Now, I am in charge of basic research to refine the technologies that have been fed back from the United States.”
Wide-ranging Applications from education to manufacturing
The technologies developed mainly by these two researchers are handed over to Ryosuke Kawamura, a member of the Trusted AI Project. He is in charge of both basic and applied technologies, acting as the intermediary between the basic technologies and the applied technologies. Ryosuke’s research background includes estimating psychological states from facial expressions while at university.
Ryosuke explains, “When we tried to put facial expression recognition technology into practical use, we discovered that the main challenges involved any features associated with the face, such as glasses, hairstyles, and the presence or absence of a beard. As a result, we are working to improve the robustness of our recognition models to counter the effects of these features. For practical applications, we are conducting research in collaboration with universities and educational institutions. The main focus is how to enhance teaching effectiveness by using our facial expression recognition technology, using it to estimate a student’s psychological state and the degree of their concentration during e-learning.”
The application of facial expression recognition is not limited just to education, but also has potential for a wide variety of business fields. Several demonstration experiments are already underway at Fujitsu Laboratories. Sachihiro Youoku from the Trusted AI Project is at the center of our activities looking into how to apply this technology in the field.
“As for my own research background, I started in mechanical engineering and was involved with tactile touch panels development. After that, I shifted my subject focus to 'human-based research and development', for example, measuring stress levels from a person’s pulse. My current work directly reflects this experience, which now involves conducting joint research and PoC activity to estimate the concentration levels of employees engaged in their work at a production site, as well as drivers at the wheel, all based on their facial expressions. This technology is intended to encourage employees to improve safety management,” explains Sachihiro.
Working towards enhancing our technology and better understanding human psychology
Although we have already achieved some tremendous results at international academic competitions and with real-world PoCs, there is still a considerable way to go before this technology can become commercially available.
“Achieving the No. 1 level of accuracy in the competition was an important milestone, but we now need to make further improvements for practical applications. For example, the accuracy of some Action Units is still low. In the future, we want to enhance this technology to include greater Action Units’ recognition accuracy,” continues Junya.
"The challenge with developing application technologies is to create models that inspire confidence in the general public. For example, in online lessons at school, students’ gestures let us know their state of mind, whether they are 'seriously concentrating' or 'spacing out'. We have to match these behaviors to our psychological estimation correctly, otherwise we create problems and undermine confidence in the technology. By collecting extensive video data in Japan, the United States, and China, and conducting in-depth evaluation by many experts, we are able to create diverse and highly objective training data and build a very convincing model. In the future, we will collect even more data and realize consistently better models,” Sachihiro explains.
Our facial expression recognition project has the full strength of Fujitsu Laboratories behind it. It is backed by an advanced, comprehensive and unified R & D organization that unifies inputs from academic activities such as joint research with overseas educational and research institutions that possess leading-edge knowledge, as well as the essential business activities that commercialize the results of R & D.
The Covid-19 pandemic has triggered unprecedented changes and challenges in both society and our daily lives. At Fujitsu Laboratories, we continue to focus on finding solutions to the challenges facing humanity, with our facial expression recognition technology representing an important and successful example.
