Fujitsu Develops Eye Tracking Technology

Kawasaki, Japan, October 02, 2012

Fujitsu Laboratories Ltd. today announced development of eye tracking technology that takes advantage of compact, inexpensive cameras and light-emitting diodes (LED) embedded in PCs. Up until now, tracking eye movements required a specialized camera and LED, the prohibitive cost and size of which made it difficult to use in PCs and other general-purpose products. Moreover, less expensive equipment was of insufficient quality, and blurred images meant poor detection accuracy. Now, however, using a near-infrared LED and the type of compact, inexpensive camera that comes as a standard feature in many PCs, Fujitsu Laboratories has developed image processing technology that can overcome blurred images to accurately detect the pupil of the eye. As a result, an eye tracking device can be embedded into the frame of a PC screen without detracting from the PC's design, enabling a new and natural non-contact PC interface in which, for example, screen zooming and scrolling based on where one is looking are possible.

Details on this technology will be exhibited at CEATEC Japan 2012, opening October 2 (Tuesday) at Makuhari Messe in Chiba, Japan.

Background

Eye tracking technology that can grasp where a user is looking and manipulate a PC screen—such as automatic zooming or scrolling—has attracted high expectations as a new interface that requires little effort on the part of the user. This kind of interface, however, had required a large and expensive specialized camera as well as an LED, limiting actual usage applications and preventing its widespread adoption to ordinary PCs.

Technological Issues

The corneal reflection method, which uses an LED to emit near-infrared light and a camera to track the reflection on the user's eye, presents a non-invasive, highly accurate way to calculate the direction of an individual's line of sight. In this method, which negates the need to wear goggles, the image of the eye captured by the camera is analyzed to calculate where a person is looking. It tracks the pupil's movements that change depending on the direction of the user's gaze, and also detects the invisible near-infrared light from the LED that is reflected off the cornea. The position of the reflection is not affected by the line of sight, so the direction of the gaze can be calculated from the positional relationship between the pupil and the corneal reflection. This method requires accurate detection of the corneal reflection and the middle of the pupil from the camera image, which is why up until now large, expensive, high-performance cameras and LEDs were needed.

Another issue is that cameras and LEDs need to be embedded into a PC or other consumer product in a way that is both inexpensive and that does not detract from the PC's design. When using a low-cost, compact camera and LED, however, the sensitivity of the camera and the strength of the LED are low, resulting in a blurry image of the pupil and a low degree of corneal reflection. As a result, it is difficult to accurately detect the pupil and the corneal reflection and calculate their position from the image taken.

The Newly Developed Technology

Fujitsu Laboratories has developed eye tracking technology that is small enough to be embedded in a PC using a compact, inexpensive camera and an LED. Using a near-infrared LED and the type of general-purpose, compact camera that is already widely embedded in PCs, an eye tracking module with an overall width of 7mm was created (see figure 3). In addition, the processing required to detect an individual's line of sight from the image captured by the camera is accomplished using software, minimizing hardware costs. The features of this technology are described below.

1. Extracting pupil and corneal reflection candidates

The image captured with the compact camera is run through image processing software. As the pupil is darker than the rest of the eye and, conversely, the corneal reflection is lighter, the software uses this characteristic to extract candidates having these qualities from the eye area. The size of the pupil varies in accordance with ambient light, and that is taken into account to extract candidates of varying sizes in order to prevent viable candidates from mistakenly going undetected.

2. Narrowing down to the correct candidate

From the candidates extracted, rules for consistency are applied regarding the relationship between the camera, LED, and the position of the face, as well as with such factors as the size of the pupil just before the image capture, in order to narrow down to the candidate with the highest degree of consistency. For example, one rule is that the pupil size does not ordinarily change significantly from one image to the next during image processing. This ensures the image of the pupil size will not differ greatly from right before the image was taken to right after, so the difference will also not be very significant. As a result of this type of image processing, the pupil and corneal reflection are accurately detected, and from their positional relationship, the direction of sight is calculated. This process enables accurate detection of the pupil and corneal reflection even when the condition of the image is poor, thereby enabling the line of sight to be calculated.

Results

Using this technology, a non-contact sensor that detects a person's eye movements can be embedded in a PC without detracting from the PC's design. The result is a new interface that can automatically scroll or zoom the screen based on where a user is looking. Moreover, because it can detect where on a screen a user is looking, it opens up the potential for applications based on an understanding of what catches a user's attention and interest.

Future Plans

Fujitsu Laboratories is proceeding with the verification of this technology with the aim of commercializing it within fiscal 2012. It will also conduct further R&D on new interfaces and other application technologies that take advantage of eye tracking capabilities.