Knowledge and Information Systems Human-Computer Interaction

Mean activation curvature for scalable second-order optimization in deep networks

Hyunseok Seung, Jaewoo Lee, Hyunsuk Ko
Springer Nature · 2026-05-31 · DOI 10.1007/s10115-026-02781-7

Abstract

Abstract Second-order methods can accelerate deep neural network training, but their adoption is limited by the cost and instability of estimating and inverting curvature matrices. We revisit Kronecker-factored Fisher approximations via an empirical structural analysis of activation and gradient statistics in modern architectures. Across a range of models, we find that activation statistics capture most of the effective curvature directions, while gradient statistics mainly act as a global or diagonal rescaling. Based on this observation, we propose mean activation curvature , a scalable curvature surrogate that yields two optimizers, MAC and SMAC , offering different trade-offs between expressiveness and efficiency. We further extend the construction to self-attention layers with a structured approximation that retains the role of attention scores, and provide a convergence analysis under standard assumptions. Experiments on vision and language models show that MAC / SMAC matches or improves the accuracy of existing second-order baselines while reducing training time and memory usage. Code: github.com/hseung88/mac .

Read at publisher →