Abstract: Neural Architecture Search (NAS) automates the design of deep network architectures, but its cost is dominated by the need to train every candidate network. To reduce this cost, we propose a training-free proxy for image classification accuracy that is grounded in statistical estimation theory. A network is scored by the uniformity of the eigenvalue spectrum of its Fisher Information Matrix at initialization, computed from random inputs alone. When combined with other proxies, it estimates a network’s accuracy without any training, which greatly reduces the cost of standard NAS algorithms. Notably, this combination finally outperforms the simple FLOPs baseline, which previous training-free proxies had struggled to beat despite considerable effort from the community. We also discuss broader aspects of NAS and propose an evaluation metric that measures how well a proxy identifies the best networks. We show that this metric is more informative for practical NAS than existing metrics.

Bio: Ondřej Týbl is a Postdoctoral Researcher at the Czech Technical University in Prague, working in the field of deep learning and neural architecture search. He obtained his PhD in Probability Theory from Charles University, Prague, where he studied stochastic equations and filtering methods under the supervision of Prof. Bohdan Maslowski. During his research stays abroad — at King’s College London with Prof. Markus Riedle and at EPFL Lausanne with Prof. Xue-Mei Li — he focused on stochastic modeling and mathematical foundations of learning systems.