Exploring the potential of continuous authentication: our research team’s input on personalized services, suspect detection, and emerging applications

Hidetsugu:Continuous authentication technology is able to maintain the authentication state at all times in order to track an individual accurately. It does this by linking the results of biometric authentication to the person captured by the camera. Even if the person is temporarily hidden by an obstacle or moves across multiple cameras with non-overlapping fields of view, this tracking is not interrupted. As the individual is always in an authenticated state, it is called 'continuous authentication.'

Junji:Up until now, we have been researching and developing multi-biometric authentication. This combines multiple factors such as facial recognition and fingerprints, as a seamless means of authentication that does not require the hassle involved in presenting or inputting something. To further enhance the convenience aspects, we aimed to realize a new authentication technology that users do not need to be conscious of, maintaining an authenticated state anytime and anywhere. This would allow us to create a world where personalized services and unprecedented security can be provided in accordance with people's actions. This aspiration was the inspiration for the development.

Yuichi:From March 2023 to June 2024, we conducted a trial experiment at Misawa Homes housing exhibition (*1)(*2). We explored services that could be provided through continuous authentication in shared office spaces and had various customers experience them first-hand. For example, if a user entered without performing authentication at the reception, continuous authentication detected this and issued an alert. After authentication, we could trigger services such as automatically unlocking a PC and playing the user's favorite music while they were sitting on the sofa, all by combining tracking and authentication. Additionally, we demonstrated various applications, such as integration with IoT devices and detecting human conditions like 'falling down' using the behavioral analysis technology Actlyzer (*3), which then issues alerts.

Yuichi:Currently, the primary focus is on single spaces, such as individual rooms within a house. Although the area is limited, we have managed to continue tracking individuals even in complex environments like Misawa Homes housing exhibition, where there are many objects placed around the area involved, and people can temporarily become hidden behind things like potted plants. Moving forward, we are working on solving potential challenges to expand the target area. As part of this effort, we conducted a trial experiment at an airport last year.

Hidetsugu:Yes, we conducted a trial experiment at Chubu Centrair International Airport. The airport operator currently tracks suspicious individuals by having personnel visually monitor footage from security cameras. However, they were keen to try out an automated tracking system. Unlike the limited space at Misawa Homes exhibition, where everyone in the area is continuously authenticated, the airport scenario involves tracking only specific suspicious individuals due to the large number of people moving through the airport. If a suspicious item is left unattended, the system can automatically identify the person who placed the item and track their current location from the video footage.

Yuichi:For example, in the context of airports, implementing a continuous authentication system at each airport could allow for continuous tracking of an individual moving from say Haneda Airport to Naha Airport. Additionally, we could consider a system that enables tracking at international airports when someone travels abroad. This is something we hope to explore in the future.

Mingxie:We achieved the highest accuracy in the world in the MOT Challenge, a global competition for person tracking technology in video footage (*4). During tracking, individuals may temporarily become obscured by other people or obstacles, causing interruptions or errors in tracking. However, we developed a technology that can continue tracking by focusing on changes in characteristics such as head movements, walking patterns, clothing, and belongings. Our ability to implement continuous authentication in real-time using multiple camera feeds is also a strength of our technology.

Hidetsugu:Person tracking is performed using a combination of different features relating specifically to an individual known as feature quantities that quantify the characteristics of individuals captured by cameras. However, when the camera angle changes, these feature quantities also change, leading to tracking errors. Additionally, feature quantities are easily influenced by clothing, which can result in mistaking individuals with similar clothing or losing track of a person if they change clothes.

Hidetsugu:We developed technologies to extract feature quantities that remain consistent even when the camera angle changes and are not influenced by clothing. In terms of technology validation, we received cooperation from Misawa Homes and Chubu Centrair International Airport. In the trial experiment with Misawa Homes, we conducted a validation experiment in a space simulating a shared office, where tracking individuals with similar clothing was a challenge. By incorporating new information such as location data in addition to visual appearance, we improved the tracking accuracy of individuals with similar clothing. In the trial experiment with Chubu Centrair International Airport, we tracked passengers within the airport. Here, too, there were many individuals with similar clothing, so we introduced the location information technology developed in the Misawa Homes experiment and a newly developed model that is not dependent on clothing, thereby improving tracking accuracy.

Mingxie:It is a technology that utilizes movement characteristics in addition to visual feature quantities. For example, if a person enters and exits, and then re-enters with different clothing, it can be difficult to identify them based solely on appearance. However, by combining behavioral feature quantities, we can capture unique movement characteristics of the individual and recognize them. We are conducting research and development accurately to identify the same person even if their appearance changes, such as their clothing. This technology is expected to be applicable for re-identifying individuals, who appear on different days with different clothing, in locations such as airports and train stations. The technology to identify the same person using feature quantities that are not dependent on appearance is being advanced through joint research with Tokyo University of Technology.

Yuichi:In continuous authentication, I am involved in the development of the entire system, from transmitting camera footage to person detection, matching algorithms, and displaying results, as well as applications that use ReID (*5) authentication results, such as displaying user locations on a map. To ensure widespread use of this system, we considered it essential to make it easy to build and deploy, maintain real-time performance, and be unaffected by external environmental changes. For ease of deployment and construction, we prepared a container-based distributed execution environment to facilitate the creation of identical environments. Additionally, we developed an integrated monitor that anyone can use easily, which is employed in various demonstrations and trial experiments. Regarding real-time performance and adaptation to external environmental changes, we distributed the processing load using the distributed execution environment. Even when the number of tracked individuals increases and the processing load becomes heavy, we prioritize tracking targets instantly, keeping the delay within 10fps and 100 milliseconds.

Junji:To obtain information for identifying individuals from cameras, the consent of the person is required. When we exhibited continuous authentication at MWC2024 Barcelona, we considered having attendees experience the demonstration at the exhibition venue. However, under General Data Protection Regulation, clear consent from the individual and evidence of this consent are necessary to obtain personal information. Therefore, we decided on a number of methods to obtain the necessary consent and retain evidence of it through our legal department and local office before attending the exhibition. We took measures to notify attendees via posters about the different methods of obtaining consent for those who wished to register their information and experience the person tracking demonstration with continuous authentication, those who wished to experience the demonstration without registering, and those who wished to observe from outside the demonstration area.

Junji:The person tracking technology is available on the Fujitsu Research Portal as 'Multi-Camera Tracking.' For a demonstration or to test our Multi-Camera Tracking, please contact us here. Additionally, by logging into the portal, you can use others advanced Fujitsu technologies for free through web applications and web APIs. We encourage those who are interested to try it out.

Mingxie:At the Technology Hall in Fujitsu Technology Park, we have set up interactive exhibits. Those who experienced it have given feedback such as, 'It's amazing that the system can continuously track a person even when they are hidden behind a pillar, and to understand their actions,' and 'It seems useful for monitoring services in places like theme parks.' There have also been inquiries about applying the technology in real-world scenarios and connecting it to trial experiments.

Hidetsugu:Currently, continuous authentication is targeted at limited spaces, such as specific enclosed areas in commercial facilities or airports. In the future, we aim to expand its application to different facilities and outdoor areas, eventually covering entire cities. To achieve this, it will be important to address not only technical challenges but also privacy and legal considerations, and we plan to take the necessary measures.

Yuichi:To target a wider range of areas in the future, further technological advancements are necessary. The challenges will include system aspects such as processing capacity and network bandwidth. Even if we process the footage from numerous cameras in a distributed manner across multiple continuous authentication systems, there are many issues to address. These range from ID linkage between continuous authentication systems, determining which system will handle the processing, and predicting processing loads to prevent network congestion, to redistributing processing tasks. Additionally, it is crucial to consider how to elevate this technology into a service, not just the technology itself.

Mingxie:Continuous authentication boasts world-class accuracy in person tracking technology and is easy to implement with existing cameras. However, to apply it to more facilities, it needs to evolve into a more user-friendly technology that can flexibly adapt to various environments. For example, by building a system that can handle changes in location, time, and appearance, we aim to further develop it as a technology that provides safe and secure environments in various facilities.

Junji:There are an increasing number of trial experiments related to this technology overseas. Moving forward, we plan to investigate and consider measures for privacy-related laws and regulations in different countries, such as Europe and the United States, when obtaining information to identify individuals from camera images in public spaces.