File System and Power Management Enhanced for Supercomputer Fugaku

3. Power management functions

Fugaku was ranked No. 1 in the TOP500 list of the world’s supercomputers announced at International Supercomputing Conference (ISC) High Performance 2020 Digital held in June 2020 [8]. This achievement verified the high program execution performance of Fugaku [8]. The power consumption on executing this benchmark test was 28,335 kW [9], which corresponds to the power consumed by about 70,000 households given a power consumption of 400 W for a typical household. Operating the Fugaku requires a reduction in unnecessary power consumption (enhanced power savings) as well as the provision of maximum computing power given a limited amount of power that can be consumed. A key element in meeting these requirements is operations management software.

In this section, we first present an overall picture of power management functions that we have newly designed and developed as part of the functional enhancements being made to Fugaku operations management software. We then describe power-consumption measurement functions that lead to power-control and optimization policies in Fugaku together with some measurement results.

3.1 Power management functions in Fugaku

The design and development of the Fugaku aimed to integrate hardware and software and enable effective use of power in the execution of applications.

On the hardware side, we designed and implemented mechanisms for dynamic power control and power measurement for CPUs, memory devices, etc. within compute nodes. Here, power control means dynamically changing the state of a device from software within a compute node according to the characteristics of the application (job) being executed to reduce power consumption and optimize computing performance per unit amount of power. Power measurement, on the other hand, aims to confirm and evaluate how power consumption in a compute node or device actually changed as a result of those power control measures.

Next, on the software side, we adopted Power API [10] as a power-control API for using the above mechanisms from within a compute node and designed and implemented associated software.

As part of Fugaku’s operations management software, we designed and implemented power management functions that use Power API to link with job operation and control and perform power measurements. Figure 4 shows the operation nodes of power management functions and their configuration.

Figure 4 Operation nodes of power management functions and their configuration.

In Figure 4, function (1) is “job power consumption statistical information” that measures and consolidates the power consumed by a job. The power consumption of a job is recorded as job-related statistical information together with the number of compute nodes used, total execution time, etc. The charge levied on each user can be calculated on the basis of such information. This function is described in more detail in the following subsections. Next, function (2) is the “power capping job scheduler” [11] that controls job execution so that system power consumption does not exceed the target value. Finally, function (3) is the “power-saving mode switching function” that performs device control according to whether or not the compute node is executing a job as a means of power control that does not impact job execution performance. These power management functions are described in detail in a Fujitsu white paper [12].

3.2 Issues in measuring and evaluating power consumption

In system design, if the processing for executing jobs is uniform throughout, it is desirable that job execution exhibits the same performance and power consumption. In Fugaku, however, individual differences between compute nodes and different types of compute nodes will invariably lead to variation in power consumption.

In a supercomputer made up of multiple compute nodes, power consumption differs from one compute node to another due to variations in transistor characteristics originating in the semiconductor manufacturing process of CPUs and memory devices, optimization of operating voltage, etc. In addition, the typical method of using supercomputers is to run multiple jobs simultaneously on the system—it is rare to execute a job that uses all of the compute nodes in the system at one time. As a consequence, the compute nodes to be used whenever executing a certain job will differ depending on what compute nodes are already being used by other jobs, which means variation in power consumption each time that job executes.

In addition, some compute nodes in Fugaku serve a dual role as a compute node and I/O node—such a node will perform file I/O processing for storage or other purposes in addition to computational processing. This variability originates in the unique system design of Fugaku in which compute & I/O nodes and ordinary compute nodes have the following differences.

• Number of assistant cores
  An ordinary compute node mounts two assistant cores. On the other hand, a compute & I/O node mounts two more to handle I/O processing for a total of four assistant cores.
• Mounted devices
  A compute & I/O node mounts a storage device itself as well as a PCI Express (PCIe) device to connect to external storage.

3.3 Introduction of estimated power

As described above, a job uses different compute nodes every time it executes, so the power consumption of a job cannot be uniquely specified due to variation in the power consumption of compute nodes. To resolve this issue, we are designing and introducing “estimated power” in Fugaku as a power indicator unaffected by variation in power consumption in addition to measured power consumption (measured power) and promoting its use in evaluating job power consumption by the user.

1) Requirements

Our aim in introducing estimated power is to use it in collecting job statistical information in relation to fair power consumption and in optimizing job power consumption by the user based on power consumption trends. With this in mind, the following requirements must be met in the design of estimated power.

• （a）The value of estimated power must be determined solely on the content of processing performed by the application program
• （a-1）It must not be affected by variation in power consumption due to individual differences between compute nodes
• （a-2）Variation in power consumption due to different types of compute nodes must be eliminated
• （b）Estimated power must increase/decrease with increase/decrease in measured power

2) Design

To satisfy requirements (a-1) and (b), we check the activity of CPU and memory circuits and calculate estimated power based on those utilization rates. Next, to satisfy requirement (a-2) when calculating estimated power, we eliminate from the total value of the target compute nodes the power consumed by any assistant cores used in operations other than those of that job and by PCIe devices that may or may not be mounted depending on the type of compute node. Furthermore, given that estimated power is calculated from the activity of various types of circuits, it is also possible to calculate power consumption in units of circuit blocks within a CPU and thereby provide power information of even higher granularity as an additional feature.

3.4 Evaluation of estimated power

We evaluated the validity of estimated power in terms of the following two items:

• Variation in estimated power
• Relationship between estimated power and measured power

In this evaluation, we used benchmarks included in the HPC Challenge Benchmark that measures a more realistic level of performance in a high performance computing (HPC) system. Since the power consumption of a compute node is heavily affected by the utilization rates of CPUs, memory, etc., we used the DGEMM benchmark that performs matrix-matrix multiplication as a routine generating high CPU load. We also used the STREAM benchmark that measures sustainable memory bandwidth by performing array vector operations as a routine generating high memory load. Moreover, to generate an even higher memory load in this benchmark, we changed the MULTIPLY instructions to XOR instructions in the SCALA operation, a vector operation that multiplies each element of a vector by a constant.

To evaluate change in estimated power due to differences in CPU and memory load factors, we used a total of 48 computing cores and varied the number of cores for executing the DGEMM and STREAM benchmarks. In this way, we executed programs with different load patterns each on 192 compute nodes in parallel and measured estimated power and measured power per compute node. We used Power API [10] installed in Fugaku operations management software to measure estimated power and measured power for a CPU operation frequency of 2.0 GHz.

1) Variation in estimated power

Table 2 lists the average value and standard deviation of estimated power and measured power per compute node for each load pattern. In comparison with measured power, these results show that variation in estimated power was kept to about 1/10 for all load patterns. In other words, estimated power can eliminate variation in compute nodes and thereby satisfy requirement (a).

2) Relationship between estimated power and measured power

Figure 5 shows the relationship between average estimated power and measured power across all load patterns. Here, the correlation coefficient R is nearly 1 and estimated power and measured power have a nearly proportional relationship. These results show that estimated power increases/decreases with increase/decrease in measured power thereby satisfying requirement (b).

Figure 5 Relationship between estimated power and measured power.