Neural Architecture Search (NAS) is an automated machine learning (AutoML) technique that uses algorithms to discover the optimal neural network architecture for a given task, rather than relying on manual design by human researchers. Traditionally, designing neural network architectures has been a time-consuming, expertise-intensive process of trial and error, with researchers experimenting with different layer configurations, activation functions, connection patterns, and hyperparameters. NAS automates this exploration, often finding architectures that outperform hand-designed models.
NAS operates across three dimensions: a search space that defines the set of possible architectures (types of layers, connections, operations), a search strategy that explores this space efficiently (reinforcement learning, evolutionary algorithms, gradient-based methods, or Bayesian optimization), and a performance estimation strategy that evaluates candidate architectures without fully training each one from scratch (using techniques like weight sharing, early stopping, or proxy tasks). Pioneering work by Google Brain in 2017 demonstrated that NAS could discover architectures rivaling or surpassing the best human-designed models for image classification.
Key NAS approaches include:
- NASNet – Used reinforcement learning to search for optimal cell structures, achieving state-of-the-art results on ImageNet.
- DARTS (Differentiable Architecture Search) – Made the search space continuous, enabling gradient-based optimization that is orders of magnitude faster.
- EfficientNet – Combined NAS with compound scaling to create a family of highly efficient image classification models.
- Once-for-All (OFA) – Trains a single super-network that supports many sub-networks, enabling deployment across devices with different resource constraints.
While NAS has traditionally required enormous computational resources (thousands of GPU hours), recent advances in efficient NAS methods have made it more accessible. NAS can yield models that are smaller, faster, and more accurate than off-the-shelf architectures, which is particularly valuable for edge deployment on mobile devices and IoT hardware where resources are constrained.