Concurrency and Computation: Practice and Experience Theoretical Computer Science

TofuD Node Mapping Optimization for 2D Pencil Decomposition on a p × q Node Grid With Two All‐to‐All Transpose Steps

Yoshinobu Yamamoto, Tatsunobu Kokubo, Masato Matsui, Takahiro Katagiri, Yoshiyuki Tsuji
Wiley · 2026-11-01 · DOI 10.1002/cpe.71032

Abstract

ABSTRACT This study proposes a node‐mapping strategy on the six‐dimensional (6D) Tofu Interconnect D (TofuD) mesh/torus network to reduce communication overhead in the two‐step global all‐to‐all communication inherent to two‐dimensional (2D) pencil decomposition. The core idea is to embed the 2D domain decomposition within a virtual three‐dimensional (3D) node configuration on TofuD, thereby minimizing inter‐node communication costs. We demonstrate the method's effectiveness on a 3D fast Fourier transform (3D‐FFT) and on a direct numerical simulation (DNS) of turbulent channel flow (TCF). Compared to a conventional virtual 2D node configuration on TofuD, the proposed mapping strategy yields significant performance gains. In particular, the TCF DNS run achieved 91% parallel efficiency with strong scaling from 5760 to 41,472 nodes, sustaining 1148 TFLOPS. This result sets a new record—more than four times the previous DNS benchmark for TCF. The node‐mapping tool developed in this study will be publicly released to facilitate reproducibility and wider adoption.

Read at publisher →