Concurrency and Computation: Practice and Experience Theoretical Computer Science
TofuD Node Mapping Optimization for 2D Pencil Decomposition on a p × q Node Grid With Two All‐to‐All Transpose Steps
Abstract
ABSTRACT This study proposes a node‐mapping strategy on the six‐dimensional (6D) Tofu Interconnect D (TofuD) mesh/torus network to reduce communication overhead in the two‐step global all‐to‐all communication inherent to two‐dimensional (2D) pencil decomposition. The core idea is to embed the 2D domain decomposition within a virtual three‐dimensional (3D) node configuration on TofuD, thereby minimizing inter‐node communication costs. We demonstrate the method's effectiveness on a 3D fast Fourier transform (3D‐FFT) and on a direct numerical simulation (DNS) of turbulent channel flow (TCF). Compared to a conventional virtual 2D node configuration on TofuD, the proposed mapping strategy yields significant performance gains. In particular, the TCF DNS run achieved 91% parallel efficiency with strong scaling from 5760 to 41,472 nodes, sustaining 1148 TFLOPS. This result sets a new record—more than four times the previous DNS benchmark for TCF. The node‐mapping tool developed in this study will be publicly released to facilitate reproducibility and wider adoption.