Skip to main content

Fujitsu

Global

Archived content

NOTE: this is an archived page and the content is likely to be out of date.

Fujitsu Technology Puts Big Data to Use in Minutes

Develops distributed data processing technology that dramatically reduces disk accesses

Fujitsu Laboratories Ltd.

Kawasaki, Japan, April 05, 2012

Fujitsu Laboratories today announced that it has developed new parallel distributed data processing technology that enables pools of big data as well as continuous inflows of new data to be efficiently processed and put to use within minutes.

The amount of large-volume, diverse data, such as sensor data and human location data, continues to grow, and various data processing technologies are being developed to enable these pools and streams of big data to be quickly analyzed and put to use. When the priority is on high-speed performance, methods that process the data in memory are used, but when dealing with very large volumes of data, disk-based methodologies are typically used as volumes are too large to process in memory. When using disk-based techniques, however, if the objective is to immediately reflect the newly received data in the analytical results, many disk accesses are necessary. This results in the problem that analytical processing cannot keep pace with the volume of data flowing in.

To address this problem, Fujitsu Laboratories has developed technology that slashes the number of disk accesses by approximately 90% compared to previous levels(1) by dynamically reallocating data on disks to match trends in data accesses. Whereas producing analytic results of new data could take several hours in the past, with this new technique results are available in minutes. This development excels at both volume and velocity when processing big data, an objective that has been difficult to achieve until now.

This technology will be one of the technologies underpinning human-centric computing, which will provide relevant services for every location.

Background

In recent years, the amount of large-volume, diverse data, particularly chronological data such as sensor data and human location data, continues to grow at an explosive pace. There is a strong demand to take this type of "big data" and efficiently extract valuable information that can be put to immediate use in delivering services, such as various navigation services.

A number of data-processing techniques have emerged for handling big data (Figure 1). One of these, parallel batch processing(2), as in Hadoop(3), has become a focus of attention. In parallel batch processing, the dataset is divided and quickly processed by multiple servers.

Another technology that has also received interest is complex event processing (CEP)(4), which handles a stream of incoming data in real time. This has the benefit of being extremely fast because it processes data in memory.

Figure 1: How Fujitsu's new technique compares with existing techniques.

Technological Issues

The goal of extracting valuable information more quickly, from larger datasets, requires a data-processing technology that is disk-based and can quickly produce analytic results. While there are both batch and incremental disk-based processing techniques, obtaining analytic results from either one quickly (responsiveness) remains a problem.

Because batch techniques perform a batch process on a snapshot of the data, there will always be a fixed lag-time before new information can be reflected in the analytic results.

Conversely, with incremental processing, new data is processed consecutively as it arrives, but updating the analytic results directly requires the disk to be accessed numerous times. This creates a bottleneck for analytic processing overall, which ultimately cannot keep up with the pace of incoming data (Figure 2). Quickly reflecting new data in analytic results, therefore, required addressing the problem of reducing the number of disk accesses.

Figure 2: Problems with batch and incremental processing

Fujitsu's Newly Developed Technology

Fujitsu has developed a technology it calls "adaptive locality-aware data reallocation," which dramatically reduces the number of accesses, along with distributed parallel middleware for incremental processing.

With adaptive data localization, data is optimally allocated by the following three steps (Figure 3):

  1. Record data-access history: Records sets of continuously accessed data.
  2. Calculate optimal allocation: Based on step 1, group sets of data that tend to be accessed continuously.
  3. Reallocate data dynamically: Based on step 2, specify a location on disk for data belonging to a group and allocate it there.

This makes it possible to acquire desired data through a fewer number of continuous accesses, not numerous random accesses, which vastly increases overall throughput in a distributed-processing system. Also, by monitoring and automatically recognizing patterns of data access, this technology can gradually accommodate the hard-to-anticipate data characteristics of social-infrastructure systems.

Figure 3: Adaptive locality-aware data reallocation

Results

This technology can perform analytic processing on big data using incremental processing while accepting data as quickly as it arrives, allowing for rapid analytic processing of current data.

This technology was used in the analytical processing portion of an electronic commerce recommendation system, where it was shown to operate with about one-tenth the number of disk accesses of previous technologies. Consequently, whereas batch processing had conventionally been used for analytical processing of large data volumes, incremental processing is now suitable. This greatly reduces the time required for new data to be reflected in analytical results. When applied to analytic processes that had been run as overnight batches because of the hours-long processing time required with batch processing, this technology can be used to utilize analytical results in a matter of minutes.

Figure 4: Results using this technology

Future Plans

Fujitsu Laboratories will move forward to make further performance enhancements to the technology and conduct verification testing with the aim of applying it to commercial products and services in fiscal 2013.


  • [1] Enables disk accesses to be reduced to approximately 90% from previous levels

    Rate of disk I/O operations compared to previous techniques when used for analytic processing in a recommendation system.

  • [2] Parallel batch processing

    A technique in which massive data sets are converted to batches, which are processed in parallel.

  • [3] Hadoop

    Apache Hadoop. Developed and released by the Apache Software Foundation (ASF), Apache Hadoop is an open-source framework for efficiently performing distributed parallel processing of massive volumes of data.

  • [4] Complex event processing (CEP)

    A method of extracting valuable information from a stream of big data in real time. By processing data in memory in accordance with pre-defined rules (queries), the data can be processed in real time.

About Fujitsu Laboratories

Founded in 1968 as a wholly owned subsidiary of Fujitsu Limited, Fujitsu Laboratories Limited is one of the premier research centers in the world. With a global network of laboratories in Japan, China, the United States and Europe, the organization conducts a wide range of basic and applied research in the areas of Next-generation Services, Computer Servers, Networks, Electronic Devices and Advanced Materials. For more information, please see: http://jp.fujitsu.com/labs/en.

Press Contacts

Public and Investor Relations Division
Inquiries

Company:Fujitsu Limited

Technical Contacts

Cloud Computing Research Center

E-mail: E-mail: aidp@ml.labs.fujitsu.com
Company:Fujitsu Laboratories Ltd.


All other company or product names mentioned herein are trademarks or registered trademarks of their respective owners. Information provided in this press release is accurate at time of publication and is subject to change without advance notice.

Date: 05 April, 2012
City: Kawasaki, Japan
Company: Fujitsu Laboratories Ltd., , , , , , , , , ,