Communications AI & Computing Компьютерные науки (информатика)
Phase transitions reveal hierarchical structure in deep neural networks
Аннотация
Abstract Training deep neural networks (DNNs) involves navigating a high-dimensional, non-convex loss landscape, yet much of the phenomenology of this process remains poorly understood. Three observations remain puzzling: phase transitions reminiscent of statistical physics, the ubiquity of saddle points, and mode connectivity between solutions. Here we show these three phenomena share a common geometric origin in the loss and error landscapes. Analytically, we demonstrate that phase transitions in DNN learning are governed by saddle points induced by L2 regularisation at concave regions in the error landscape. We introduce the Pathfinder algorithm, which repurposes the L2 regulariser as a systematic probe of landscape geometry. Using MNIST, we establish the saddle-point correspondence, a hierarchy of accuracy basins, and flat connecting paths between minima. We further demonstrate hierarchical phase transitions in a sigmoid network on Fashion-MNIST, a ReLU network on MNIST, and a convolutional network on synthetic data. These results suggest hierarchical basin structure is a robust property of overparameterised DNNs.