From Geometry to Generalization: Why Row Normalization Can Beat Adam and Muon

Abstract

Different optimizers can fit the same training data while selecting classifiers with substantially different geometries, but whether these differences lead to provably different population performance remains poorly understood. We show that row-wise normalization can achieve strictly higher population accuracy than Adam and Muon in high-dimensional multiclass classification. Under a Gaussian-cloud representation model, row-normalized methods preserve the population class geometry, whereas Adam- and Muon-associated geometries incur a nonvanishing distortion. This yields strict population-accuracy orderings in both deterministic full-batch and random-reshuffling stochastic settings. In particular, row-normalized gradient methods outperform full-batch Adam and exact-SVD Muon, while row-normalized SGD with momentum outperforms stochastic exact-SVD Muon under random reshuffling. For random-reshuffling Adam, we derive an AdamProxy that approximates its one-epoch displacement in the high-memory regime and prove a corresponding advantage for row-normalized SGD with momentum.

Chulhee Yun

Associate Professor

I am an Associate Professor at KAIST AI. I am interested in optimization and machine learning theory.