Working towards a "Connected World" from "Palm Vein Authentication" to "Multimodal Biometric Authentication"

Biometrics is widely used in everyday life, for example to unlock smartphones and laptops. Among the many different types of authentication, such as fingerprint authentication, facial authentication, and palm vein authentication, Fujitsu Laboratories has been focusing on one in particular - developing palm vein authentication technology.

This technology was first commercialized in 2003 as a technology to read and identify each person's palm veins pattern by using near infrared light. Since then, its contactless and stable accuracy in particular have been widely recognized, and many banks in Japan and overseas now use this technology to authenticate ATM users. Today, it is the most widely used authentication method in the world, with 94 million users in 60 countries.

It is possible to identify a person just from their biometric information, such as fingerprints and veins, because biometric information is unique to an individual. However, biometrics are generally used in conjunction with identities, so there are restrictions on their use.

"The reason why we do not adopt authentication with just a single type of biometric information is down to its limited accuracy, despite the advantage of not having to enter private ID information, " explains Narishige Abe, the Project Manager in the Digital Innovation Core Unit. He has been working on neural networks since before Deep Learning made its appearance. After working on UI, UX, and image processing techniques such as the tangible user interface, he is now playing an important role in project managing the Authentication and Settlement Project.

"At present, the view is that the maximum number of people who can be identified by palm vein authentication is around 10,000, taking the errors in authentication into account. If we want to identify a single person using biometric authentication alone and incorporate this into a wider range of applications in real life, we need to increase this number to between 1 million and 10 million units.” Narishige continues.

Fujitsu Laboratories has taken a Multimodal Biometric Authentication approach, which combines multiple biometrics to identify individuals. We elected to use facial authentication as an additional biometric authentication to complement Fujitsu's specialty, palm vein authentication technology. Narishige explains the reason behind this decision,

“This single most important element for us is the user perspective. It is anticipated that biometrics will be used for payments as well as for entry and exit applications. So we considered carefully what type of information can be most easily and naturally obtained without calling users attention to it; for example, when people hold their palms over the sensor, when shopping or passing through a gate. We concluded that face authentication was the best option. By integrating different types of authentication, we have successfully extended the accuracy to the 1 million users scale. "

At Fujitsu Laboratories, research into facial recognition has been conducted for many years, but the challenge has been the lack of high accuracy compared to other biometrics. However, with recent rapid progress in research on neural networks and AI/machine learning, the accuracy of authentication has improved dramatically.

"A key feature of our technology is not only identifying one person out of 1 million in real time, but also realizing the same convenience as using the conventional palm vein authentication. Our technology allows users to make hands free payments (ID-less authentication) via the Internet when they shop. It’s no longer necessary to use a smartphone or ID cards to identify ourselves. So, I think this new method will be very convenient and greatly improve the user experience (UX)." Narishige elaborates.

Narishige took advantage of the Fujitsu Laboratories “study abroad “program to conduct research on the combination of biological technologies and cryptography at Stanford University. He had long wanted to make the most of this experience and knowledge, including research on UI and UX. These were the drivers that encouraged him to put Multimodal Biometrics Technology into practical use.

In March 2020, we started live verification tests of this technology at a Lawson store in the Fujitsu Shin-Kawasaki Technology Square. The plan involved implementing a brand new approach, using Multimodal biometrics at the entry and exit gates as well at payment points.

The Authentication and Settlement Project at Fujitsu Laboratories, which is responsible for the development of Multimodal Biometric Authentication technologies, consists of three groups: the Core Technology Group, which develops the main technologies; the Suite Development Group, which assembles technologies developed by the Core Technologies Group into easy-to-use products; and the "Practice" Group, which verifies with the business units whether systems and services created by the Suite Development Group fulfil the user-friendly brief. The aim is to accelerate the development cycle through the cooperation of these groups, achieving the practical application in the shortest possible period of time.

However, integrating several authentication technologies is never easy. In order to make a solution practical for everyday use, it is necessary to improve the accuracy and collation speed for each piece of biological information. Tomoaki Matsunami, an expert in such fields as image processing and pattern recognition, and a specialist in face recognition in the Core Technology Group, explains.

“The biggest benefit of biometrics is convenience, but as the number of people we need to identify increases, processing time becomes a challenge. We use machine learning AI models for authentication, but the dilemma is that the higher the accuracy, the longer the processing time.”

Lina Septiana from Indonesia, who joined Fujitsu Laboratories after studying image processing on a doctoral course, became a member of the Core Technology Group in May 2020. She adds, “The most challenging task is how to improve accuracy while reducing the processing time. While academic research at graduate level is focused on the work of individuals, organizational research requires teams to work together to solve problems. This difference is refreshing and something that I very much enjoy. “

The first prototype was built in 2017. In the beginning, however, it was sometimes difficult to identify a user’s face because the faces of other people in the vicinity were unintentionally captured as well.

"To enable users to input facial information naturally, we placed a camera that captures facial information close to the palm-vein sensor. In this way, users’ information, including their face, could be identified without them being aware simply via a series of natural movements while they held up their palm," Tomoaki says.

Fujitsu Laboratories has also developed a technology that enables some images to be selected that are suitable for narrowing down the number of the acquired images. This works even when the user approaches the sensor while turning their face in different directions.

In terms of optimizing the processing speed, Tomoaki explains the process of developing *‘distillation’ technology in deep learning, which narrows down the candidates.

“Based on complex and highly accurate learning models, we generate and extract lightweight models without losing accuracy. By doing this, we have achieved a model that can run at high speed even on a general GPU (Graphics Processing Unit). The processing speed has been increased to the level where the face can be identified within approximately 1 second, which is the same amount of time for putting the palm on the sensor. "

This distillation technology is the result of Fujitsu’s and Fujitsu Laboratories’ continuing research and development on a global scale, working with their overseas organizations. “In this day and age, many of the issues in business and daily life that we now see in Japan are no longer limited to this country. Therefore, it is important to share research and development to solve problems not only in Japan, but also overseas, refining the different AI technologies and utilizing them globally,” adds Narishige.

*Distillation... one of the methods used to compress and lighten deep learning models that often requires huge computational resources.

While researchers focus on refining the core technology, the Suite Development group plays an important role in practical applications. Yūichi Hanada, a member of the group working with Tomoaki Matsunami on the issues of convenience, recalls,

“In order actually to launch a product, it is important to avoid putting stress on the user during face recognition. Not only did we pay attention to the camera installation, but also worked to solve the problems both on the server and client sides. "

In the trial with LAWSON, Multimodal Biometric Authentication technology is used when users pass through the gate. Yūichi and other members of the Suite Development Group concentrated on how to improve the device’s design to give a better user experience by tilting the device’s surface by 10 degrees. As a result, users can hold their hands up more easily to be recognized. "How can we go through the gate and authenticate with it in one go? We kept on repeating it by trial and error from the user's point of view,” says Yūichi.

Yūichi says that the members of Suite Development Group, which has the mission of turning core technology into an easy-to-use system and putting it into practice, must have a wide range of knowledge, including programming, databases, and networks, as well as the ability to communicate well.

“The Suite Development Group plays a key role in bridging the work of the Core Technology Group with the Implementation Group, clarifying anything that we don't understand in the core technology. Researchers produce output from various perspectives. Therefore, in order to understand and combine their thoughts, it is necessary to summarize the development based on their different viewpoints and requirements. I think my experience has allowed me to develop the skill to help different groups to work together. Like our project manager, Mr. Abe, I worked at Stanford University for one year under the study abroad program of Fujitsu Laboratories. My experience in countries where languages and cultures are different to ours has helped me realize the importance of communication in research activities. "

The immediate goal is to commercialize Fujitsu’s Multimodal Biometric Authentication technology in 2021, and apply it practically. The teams plan to use the feedback from the demonstration experiment in future research and development.

Lina comments, “What I am learning about this technology is useful for society, which in turn makes my current research very rewarding. Our most recent challenge is to keep improving accuracy and to develop technologies that can process data at high speed.”

The field of image processing , on the other hand, is a research theme that Tomoaki has worked on since his university days, using this knowledge in his daily research activities. “Under the Covid-19 pandemic, a lot of people are forced to wear masks, and so there is an increasing requirement to identify a face with a mask. Social problems emerge one after another, and we have to solve them. That is the difficulty of research, but also the most interesting part.”

From the perspective of the Suite Development Group, Yūichi explains,

“After we accomplish this authentication method for a single person, the next step will be to apply it for multiple people. We want to make Multimodal Biometric Authentication even easier to use and introduce it to the world, while solving the unique problems that arise as the usage scene expands. "

There are an increasing number of opportunities to apply Multimodal Biometric Authentication in various fields, although the current research and development activities are in the field of payments. Narishige Abe, who leads the project, is looking to the future,

“Multimodal Biometric Authentication can be used wherever there is a need for identification, such as showing a driver's license. Currently, each user is required to register their biometrics for use in each situation. If we realize a world where different authentication infrastructures are combined with each other in the future, we can create a win-win relationship where users, service providers, and technology providers can use biometric information in all situations at the same time. "

Of course, it is important to take measures against information leakage and to consider privacy, because of the unchanging nature of biological information. In response, Fujitsu Laboratories developed a technology in 2019 for verifying encrypted biological information without decryption, and is continuing research on this from a security standpoint as well.

"Even if it were leaked, the biometric information in the database would not be easily deciphered, because it is encrypted. Through such efforts to ensure safety, we are working to promote biometric authentication technologies.” comments Narishige.

Fujitsu Laboratories will continue to accelerate its efforts, focusing on commercializing the technology in 2021 to realize a world that is ever more connected through biometrics.

Titles, numerical values, and proper nouns in this document were accurate at the time of publication.