OS Enhancement in Supercomputer Fugaku

3. Initiatives to improve performance at the OS level: Minimization of OS noise

Fugaku improves application performance during large-scale parallel execution by reducing OS noise. This section describes the mechanism of performance degradation due to OS noise and the noise reduction method utilizing assistant cores. Furthermore, it introduces McKernel, a lightweight kernel (LWK) for HPC, and describes the noise reduction effect.

3.1 What is OS noise?

The value of supercomputers lies in their ability to run their users’ computing applications at great speed. On the other hand, to operate properly as a computer system, a lot of management processing is also required, and the basic software that executes it is the OS.

When running computing applications on large-scale supercomputers, this OS management process may delay the execution of the applications, and this may hinder performance improvement. Such application execution delays due to non-application processes, such as OS daemons, kernel daemons, and interrupt processing, are collectively called OS noise.

Figure 1 shows how OS noise delays application execution. Multiple node execution forms the basis of modern HPC applications, but when multiple nodes work together to perform calculations, there is often a process of coordination before the next process is begun, to allow the exchange of data between the nodes. This process of coordination is called synchronization processing. When OS noise occurs at a given node, the execution of the application at that node is delayed, and this delay is propagated to other nodes by way of synchronization processing. For this reason, the larger the number of nodes and the larger the system, the greater the delay even even when OS noise is low.

Figure 1 Delay due to OS noise.

Figure 2 shows the relationship between node scale and delay due to noise for parameters assuming an environment without OS noise countermeasures. The slowdown (execution delay) model equation was derived from reference [1]. This is the result of calculating the degree of performance degradation due to OS noise using model equations for two benchmarks (HPCG and HPL) that are often used in supercomputers, and LQCD, which is a Fugaku target application. Factors related to application delay are maximum noise length (D), noise generation interval (T_n), and application synchronization interval (T_a). Taking D, T_n, and T_a as constants, the delay increases as the number of nodes increases. There are applications whose performance degrades by more than 60% (D = 1 ms, T_n = 600 s, T_a = 250 μs) when the number of nodes exceeds 100,000. Therefore, OS noise control is essential for Fugaku, which has more than 150,000 nodes.

Figure 2 Relationship between node scale and delay due to noise.

Fugaku requires a clear grasp of the effectiveness of tuning for applications that require fine-grained inter-node synchronization in order to achieve the ultimate in performance. To this end, it is necessary to minimize local performance fluctuations. Development was carried out with the challenging target value for maximum noise length of 10 μs in order to reduce performance fluctuations to less than 1% for the application synchronization interval T_a of 1 ms at all nodes.

3.2 OS noise countermeasure 1: Utilization and effect of assistant cores

In Fugaku, the impact of OS noise is reduced by separating the CPU resources used to execute OS tasks, such as system daemons, kernel daemons and interrupt processing, which are sources of OS noise, from the CPU resources used to run applications.

Fugaku deploys 50 cores per node for compute nodes and 52 cores per node for I/O & compute nodes. Of these, 48 cores are allocated for application computation and are called compute cores. The remaining 2 or 4 cores are assigned to OS tasks for the system and are called assistant cores. The allocation (binding) of OS tasks to the cores is implemented using the Linux kernel’s cgroup function.

Figure 3 shows the results of OS noise measurements when the binding of the OS tasks to the assistant cores is enabled and disabled. The application uses Fixed Work Quanta (FWQ) [2]. FWQ is a benchmark that measures the magnitude of OS noise by repeatedly executing given processing. The magnitude of OS noise is called the noise length. The noise length is the difference between the execution time of each run and the shortest execution time. The horizontal axis of the graph shows the number of times the measurement is performed. The vertical axis shows the execution time of the process and, on rare occasions, this is increased by delays caused by OS noise. When binding was disabled, maximum execution time was 35.504 ms and maximum noise length D was 29.37 ms, indicating large variation in execution time. On the other hand, when binding was enabled, maximum execution time was 6.129 ms and maximum noise length D was 1.13 μs even when measurement was performed 60,000 times. These results confirm that the effect of OS noise was reduced by separately binding the application to the compute cores and the OS tasks, which are the source of OS noise, to the assistant cores. The OS used for measurement was RHEL8.2GA (RHSA-2020:2427).

Figure 3 Comparison of OS noise with and without binding to assistant cores.

As mentioned above, although the aim is to reduce OS noise while using a general-purpose server OS such as RHEL, OS noise still cannot be completely eliminated. This is because there are kernel threads (e.g. kswapd, kworker) that cannot be bound to assistant cores: a restriction for the sake of support of various use cases in Linux.

Such unavoidable OS noise that is inherent to Linux implementations can be reduced by adopting a lightweight OS (LWK) other than Linux. This is suitable for HPC applications that emphasize performance in large-scale environments. Typical LWKs include Argo [3] from the Argonne National Laboratory, Hobbes [4] from Sandia National Laboratories, mOS [5] from Intel, and McKernel [6] from RIKEN. McKernel is an LWK whose development, from scratch, was initially taken on by the University of Tokyo and later taken over by RIKEN. Fugaku supports job execution by McKernel in order to reduce OS noise to the greatest extent possible.

A conceptual diagram of McKernel is shown in Figure 4.McKernel runs on a CPU and memory that are independent of Linux, and independently of Linux, it implements OS functions that are performance critical in HPC applications, such as process management, signal processing, memory management, and synchronization processing. This allows for efficient process and memory management. OS functions not provided by McKernel are executed on the Linux side. Daemons and interrupt processing, which are sources of OS noise, are processed on the OS cores (assistant cores) where Linux is running and do not affect the app cores (compute cores) where McKernel is running. If an HPC application uses only OS functions provided by McKernel, it is not affected by OS noise caused by Linux.

Figure 4 Conceptual diagram of McKernel.

3.3 OS noise countermeasure 2: Adoption and effect of McKernel

Figure 5 shows the results of noise measurement using the OS noise measurement program FWQ on Linux and McKernel. Note that in Figure 3, the vertical scale is on the order of milliseconds (ms), whereas in this figure it is on the order of microseconds (μs). Using the same data as in Figure 3, the measurement results on Linux, where the OS tasks were bound to assistant cores, were maximum execution time of 6.129 ms and maximum noise length D of 1.13 μs. OS noise on the Linux OS, which is not visible on the order of milliseconds employed in Figure 3, is clearly visible on the very small order of microseconds used in Figure 5. On the other hand, on McKernel, the maximum execution time is 6.128 ms and the maximum noise length D is 0.10 μs, which is extremely small even when viewed on the order of microseconds. This confirms that the effect of OS noise in McKernel is even smaller than in Linux.

Figure 5 OS noise comparison between Linux and McKernel.

3.4 Whole-system evaluation by performance degradation model equation

Lastly, the effect of these OS noise countermeasures was compared using performance deterioration simulation for each of the benchmarks shown in Figure 2.

Figure 6 and Figure 7 show the calculation results of performance degradation due to OS noise, when assistant core binding is enabled, and when jobs are executed on McKernel, respectively. As shown in Figure 2, the maximum delay without noise countermeasures was 60%, whereas performance degradation could be suppressed to 0.1% when binding to the assistant cores was enabled, and to 0.009% when the job was executed on McKernel.

Figure 6 OS noise reduction effect of assistant core binding.

Figure 7 OS noise reduction effect of McKernel job execution environment.

5. Conclusion

This article described initiatives to improve the performance of the Fugaku OS at the OS level and its support of various job execution environments to meet diverse user needs.

Through these efforts to improve performance at the OS level, we were able to reduce performance degradation to 0.009% or less even at the Fugaku scale of 150,000 nodes or more, thereby establishing the foundation for high-speed execution of applications. Going forward, we would like to further reduce OS noise on Linux.

Further, regarding the job execution environment, we would like to contribute to the improvement of user convenience by having Fugaku support the execution of user-specified McKernels in the same way as the latest containers and KVM do.

All company and product names mentioned herein are trademarks or registered trademarks of their respective owners.