Skip to main content

Fujitsu

Global

Archived content

NOTE: this is an archived page and the content is likely to be out of date.

Fujitsu Develops Next-Generation User Interface for Intuitive Touch-Based Operations

Enables innovative handling of data by directly touching actual objects

Fujitsu Laboratories Ltd.

Kawasaki, Japan, April 03, 2013

Fujitsu Laboratories Limited today announced the development of next-generation user interface technology that enables intuitive, fingertip operations. This new technology can accurately detect where the user's finger is and what it is touching, and uses off-the-shelf cameras and projectors. This eliminates the need for expensive equipment and overcomes imprecise tracking—issues posed by conventional technologies.

The new technology uses the cameras together with a projector to enable a user to trace a finger across a document on a table, copy it as digital data, and display it. This simple interaction with objects makes the technology an interface between people and ICT services, thereby expanding the ways in which ICT is employed.

Background

In recent years, considerable work has been completed in developing intuitive user interfaces with the aim of improving the usability of PCs and smartphones. Commercial applications have been developed for hand gesture-based screen control technologies and speech interaction technologies, and new user interfaces have begun to be employed as replacements for the mouse, keyboard and touchscreen. At the same time, to test the viability of enhanced user interfaces, actually touching and moving objects has been suggested as a potential area of development. Until now, however, this approach has required special sensors to be embedded into objects, which has served as a major roadblock in making the technology practical.

Technological Issues

If it were possible to perform contactless detection of touch operations on real world objects, this would obviate the need to embed objects with special sensors. Current commercial gesture technologies function predominantly on operations performed in the air without touching anything. When a user's hands draw near to background objects, these technologies have difficulty differentiating between the hands and background. As a result, this approach has been unsuitable for detecting touch operations. In addition, technologies that use infrared and other special devices to measure distance have also begun to be employed in user interfaces. However, these technologies lack the resolution to detect subtle changes in finger movements, in addition to being large and costly.

Newly Developed Technology

Fujitsu Laboratories has developed technology that can accurately and rapidly detect finger operations on actual objects using an ordinary camera. The features of the new technology are as follows.

1) Recognizing and converting coordinates of physical objects and ICT equipment

Fujitsu has developed a technology that uses a camera to automatically measure irregularly shaped objects on a table and then automatically adjust the coordinate systems for the camera, projector and the actual objects. This makes it possible for the software to take the finger movements and touching of objects made by the user, and then match them with the digital display projected onto physical objects.

Figure 1: Recognizing and transforming coordinates of physical objects with ICT equipmentFigure 1: Recognizing and transforming coordinates of physical objects with ICT equipment

2) Stabilization of finger recognition technology

This Fujitsu technology recognizes the shape of the user's fingers by extracting the fingers' colors and contour features. Other technology controls the color and lighting of the camera image, depending on the surrounding ambient light, along with technology that corrects for differences among individual fingers. A stable extraction of the form of the user's fingers, one that is minimally impacted by the environment and individual variations, can be obtained.

Figure 2: Finger recognition stabilization technologyFigure 2: Finger recognition stabilization technology

3) Accurate and rapid fingertip recognition technology

By enhancing images of the user's fingertips, the technology is able to obtain a level of precision that is sufficient for touch detection, even using low-resolution images that can be captured on a regular webcam. Moreover, the technology is capable of fingertip tracking speeds of 300 mm per second, thereby enabling it to follow natural finger movements.

Figure 3: Accurate and rapid fingertip recognition technologyFigure 3: Accurate and rapid fingertip recognition technology

Results

A variety of operations are possible using the newly developed technology, one being the ability to copy important parts of a document in digital format by simply tracing one's finger across the physical document while it is placed on a table. Another is the ability to project copied data onto a tabletop and then expand and shrink the projected image by tracing with a finger. In addition, the technology can capture graphic data from handwritten sticky-notes that have been attached to a table, and enable the user to move around these newly digitalized sticky-notes with a finger over a physical surface. It would then be possible to group and rearrange these digitalized notes or perform other useful operations with them. This new technology makes it possible to easily have interactions with actual objects and serve as an interface between people and ICT services, thereby helping to expand the ways in which ICT is employed.

Future Plans

Fujitsu Laboratories plans to evaluate the new system and its software applications in real usage environments with the aim of commercializing in fiscal 2014.

About Fujitsu

Fujitsu is the leading Japanese information and communication technology (ICT) company offering a full range of technology products, solutions and services. Over 170,000 Fujitsu people support customers in more than 100 countries. We use our experience and the power of ICT to shape the future of society with our customers. Fujitsu Limited (TSE:6702) reported consolidated revenues of 4.5 trillion yen (US$54 billion) for the fiscal year ended March 31, 2012. For more information, please see http://www.fujitsu.com.

About Fujitsu Laboratories

Founded in 1968 as a wholly owned subsidiary of Fujitsu Limited, Fujitsu Laboratories Limited is one of the premier research centers in the world. With a global network of laboratories in Japan, China, the United States and Europe, the organization conducts a wide range of basic and applied research in the areas of Next-generation Services, Computer Servers, Networks, Electronic Devices and Advanced Materials. For more information, please see: http://jp.fujitsu.com/labs/en.

Press Contacts

Public and Investor Relations Division
Inquiries

Company:Fujitsu Limited

Technical Contacts

Media Processing Systems Laboratories
Media Service System Lab.

E-mail: E-mail: ftui-2013@ml.labs.fujitsu.com
Company:Fujitsu Laboratories Ltd.


All company or product names mentioned herein are trademarks or registered trademarks of their respective owners. Information provided in this press release is accurate at time of publication and is subject to change without advance notice.

This press release has been revised as of December 17, 2018.

Date: 03 April, 2013
City: Kawasaki, Japan
Company: Fujitsu Laboratories Ltd., , , , , , , , , ,