A research career inspired by drumming and a desire to make real changes to benefit society!

Development of a drum performance CG character application

During my student days, I formed a band with friends and played the drums. Drumming is an instrument where it’s really important to have an image of the movements involved. However, when I first started playing the drums, simply looking at the sheet music often left me confused about which hand to use to hit which drum part. So I thought it would be great to have a model that shows you how to play the drums from 360 degrees when practicing, and I created a CG character for drum performance support on a computer as part of my university research. By inputting sheet music information such as notes into the computer, the CG character performs the piece. Since it is CG, you can view it from any angle, and you can freely change the playing speed. This made it easier to visualize which hand to use to hit which drum part, and it was particularly useful for beginners in their practice.

In my graduate school research, I developed this idea further and created an app that allows users to have a session with the CG drum character. When a human plays an instrument, the CG character also plays the drums in sync with that tempo. To have a session, it is necessary not only to estimate the tempo in real-time from the sound of the instrument being played but also to predict the tempo a little ahead. Therefore, by learning the actual performer’s movements of keeping the rhythm in addition to the sound of the instrument, the app can predict whether the tempo will slow down or speed up, allowing the CG character to match the human performance tempo more accurately in real-time. This research also motivated me to start learning the bass guitar, and gave me the motivation to practice a new instrument, making it a very interesting experience.

Detecting suspicious behavior using AI technology

One of the most rewarding aspects of research in a corporate setting is the ability to enhance technology until it is ready for societal implementation. After joining Fujitsu, I was involved in the R&D of video analysis (*1). We received a request from a client, who wanted to analyze footage from security cameras to identify suspicious individuals, as well as to track and locate persons potentially involved in incidents or accidents. Traditionally, this was done entirely through visual inspection. However, a system approach that could analyze video using AI and filter results based on conditions such as "people wearing red clothes" would clearly be extremely helpful in identifying suspicious individuals. My responsibilities included not only developing the core technology for video analysis but also designing and developing a comprehensive demonstration system. This can import real data for analysis and display the results in an easily understandable manner for the client. There were instances where we had to receive video data on-site from the client and demonstrate the accuracy of our analysis on the spot.

The system's effectiveness depended not only on the accuracy of the video analysis but also on the speed of the analysis, the speed of searching for specific individuals in the footage, and the display functionality. To improve analysis speed, we implemented parallel processing using multiple machines and optimized the processing efficiency. For the search functionality, we investigated, selected, and fine-tuned various search methods to locate individuals quickly among thousands. Additionally, we continuously communicated with the client to refine the display features. By enhancing these aspects beyond just the accuracy of the video analysis, we were able to build a system that was practical for real-world use. During demonstrations, we never knew in advance what kind of footage the client would provide. Therefore, our development team prepared by shooting test videos under various anticipated conditions, both indoors and outdoors. We honed our search techniques, determining which individual characteristics to start with and how to combine them effectively. It was incredibly gratifying when we successfully met the client's requirements for video analysis during the demonstration. This success also led to securing the contract, reinforcing my belief that technology holds value only when it can be practically utilized.

Traveling by car

On holidays, I enjoy participating in festivals and events. This June, I participated in an event celebrating the 100th anniversary of Kawasaki City. It was a mystery-solving event around Kawasaki, and I had a great time taking part with my child. Additionally, since I worked hard to get my driver's license last year, I am keen to travel by car and conquer all the prefectures of Japan with my family. I am particularly interested in lesser-known B-grade spots. Regarding music, when I was previously with the Fujitsu Social Science Laboratory, I joined the company's light music club and often had jam sessions with other members in the studio. Recently, I've been busy with work and home matters, so I have had to put my electronic drums away, but I hope to start playing again someday.

Accelerating innovation through the societal implementation of technology

At the Converging Technologies Laboratory, where I currently work, we aim to solve complex social issues by integrating digital technology with the humanities and social sciences (*2). I am conducting research on behavior change technologies, focusing on crime prevention and countermeasures against phone fraud. Specifically, I am working on prevention and damage mitigation strategies to protect the elderly from falling victim to fraud. Our approach is divided into two parts. The first involves the development of an AI system that allows users to simulate conversations involved in special fraud. By using this system, individuals can experience scenarios such as refund scams over the phone, and we analyze their responses along with vital information obtained through millimeter-wave sensors. This helps us estimate whether the person is likely to be deceived or not. The second element involves providing feedback after the experience. An AI avatar of a criminal psychology expert appears and gives direct feedback to the user. We are also conducting a pilot experiment in collaboration with Amagasaki City, where elderly individuals participate in these simulations (*3).

In tackling this research theme, we not only build the core technology but also develop user-friendly systems and improve them based on feedback, following a comprehensive process. Being involved from the research phase to the stage where people can actually use the technology provides a great sense of fulfillment. I have always challenged myself to pursue what I am passionate about. By building on this experience, I aim to continue contributing to the development of future technologies.

Messages from colleagues

The technology created by Shuji is entirely 'original.' By not being confined to the boundaries of academic knowledge and common sense, and by continuously challenging and realizing unknown territories with free thinking and an insatiable curiosity, he reminds us of the excitement of research. I hope that you will continue to advance your research with that same curiosity and creativity, paving the way for the future. (Takeshi Konno, Converging Technologies Laboratory)

• (*1)
  Multi AI video engine
• (*2)
  Fujitsu Converging Technologies
• (*3)
  Fujitsu, Toyo University, and Amagasaki City leverage AI technology and psychological research in trial to protect senior citizens from phone fraud

• Notes
  (*1) only available in Japanese