Fujitsu Technology Uses a Single Camera to Enable a Panoramic Understanding of Remote Field Sites

Kawasaki, Japan, March 17, 2016

Fujitsu Laboratories Ltd. today announced that it has developed a technology that generates synthesized 3D panoramic screens to fully understand conditions throughout a worksite for on-site maintenance, inspection and other work. Using time series images taken from a worker-side camera, when combined with augmented reality (AR) technology, remotely supporting operators can use that image to provide precise, omnidirectional instructions to field workers.

In recent years, a shortage of skilled workers in Japan has put a spotlight on technology that enables skilled workers to provide remote work support to less-experienced field workers. Remote support technology using camera images from the worker, however, typically have a narrow scope and are often blurry, making it difficult for a remote support operator to grasp conditions at the worksite.

This remote work support technology developed by Fujitsu Laboratories uses worker-side camera images to infer a worker's position and orientation, and generates a three-dimensional image arranged from multiple images. In this way, remote operators can understand worksite conditions from any perspective, and, as AR information on a worker's position and orientation, it is possible to offer precise guidance to on-site workers.

With this technology, skilled worker's instructions can be efficiently conveyed in accordance with actual onsite conditions in the field in an easy-to-understand way, so even less-experienced workers can expand their operational scope. In addition, by using this technology skilled workers for whom field work would be difficult because of age or other reasons can effectively utilize their skills, enabling improved flexibility in leveraging the talents of employees.

Background

With Japan facing a labor shortage brought on by low birthrates and an aging society, the areas of construction, maintenance, and inspection are facing a lack of skilled workers, a serious and ever-growing problem. Accordingly, remote work support technology that enables skilled workers in an operations center to provide remote work support to less-experienced field workers is attracting attention. In particular, there are high expectations for the use of smart devices, such as tablets and head-mounted displays, to enable the provision of real-time support while grasping conditions in the field (Figure 1).

Issues

While there are technologies that capture still images of the field site with a smart device's built-in camera, transmit them to the remote support operator at a remote location, and enable the field worker to then receive instructions, the problem is that the narrow field of view of most cameras makes it difficult for the remote support operator to get a good grasp of overall conditions in the field. On the other hand, when transmitting live video, blurred images caused by the worker's irregular movements can leave remote support operators feeling dizzy.

Also, as these technologies cannot directly convey the worker's position and orientation within the work environment to the remote support operator, the two of them need to interrupt the task at hand to discuss such issues as the place being addressed in the instructions and the direction in which the worker should be looking. This then requires a discussion which takes time and reduces efficiency.

About the Technology

Fujitsu Laboratories has developed 3D image synthesis technology that uses images captured by a worker-side camera to generate a 3D panoramic image to convey the entire scene of a field site in a way that is easy to grasp. It has also developed user interface and work instruction AR presentation technology that, by using camera images and sensors to estimate a worker's position in real time, enables precise instructions to be conveyed directly to the field site from a remote location (Figure 2).

Key features of the technology are as follows.

1. Online 3D image synthesis technology for field site images

This technology extracts a number of characteristic points in a scene from multiple images shot at the worksite to tracks the camera's position and orientation in real time, and continuously estimates the worker's position and orientation. Using the estimate results, clean images that are not blurred are selected and placed at the appropriate spot and orientation in the 3D scene, and then transmitted to the remote support operator. On the remote support operator side, camera parameters such as focal length and image center are used to lay out the images in three-dimensional space in real time, with image sizes adjusted to take into account the distance between the camera and the characteristic points. By sequentially placing each selected image in three-dimensional space, it immediately and accurately displays a 3D image with a strong sense of presence that captures the entire scene of a work site over a wide scope (Figure 3).

2. Worker activity monitoring and AR presentation technology for work guidance

In order to faithfully reproduce the three-dimensional spatial structure of the actual worksite using a generated 3D image, this technology incorporates information from multiple sensors to make a highly precise estimate of the worker's position and orientation, which is used to depict a virtual scene that places the worker as a 3D model in a synthesized panoramic image. This allows the remote support operator to view the scene from any desired perspective and to monitor the worker's actions. In addition, the remote support operator can give guidance in the form of pointers or notes placed in the 3D image. This guidance is immediately conveyed to the worker's smart device and is presented as AR information in real time, corrected for the worker's position and orientation. Furthermore, if that AR information is outside of the worker's visual scope, the worker can be directed to that area, which improves the effectiveness of remote support.

Results

This technology makes it possible for a remote support operator to view a 3D bird's-eye image of a worksite from any perspective and, while understanding in real time where a worker is standing and facing, to provide instructions. Because this allows the worker to receive precise guidance from the remote support operator, less-experienced workers can expand their scope of operations. In addition skilled workers who may find it difficult to work in the field can effectively utilize their skills, enabling improved flexibility in leveraging the talents of employees.

Future Plans

Fujitsu Laboratories plans to begin testing this technology in real-world facilities maintenance and inspections, with the goal of a practical implementation during fiscal 2016. In addition to field work, Fujitsu Laboratories plans to apply the technology to a variety of areas, including distribution, construction, and retailing.