A hyperparameter is a configuration variable that is set before a machine learning model begins training, as opposed to model parameters (like weights and biases) that are learned automatically during the training process. Hyperparameters control the behavior of the training algorithm and the structure of the model itself, directly influencing how well the model learns, how quickly it converges, and how well it generalizes to new, unseen data.
Common hyperparameters in machine learning include:
- Learning rate: Controls how large the steps are during gradient descent optimization. Too high causes instability, too low causes slow convergence.
- Batch size: The number of training examples processed before updating model weights.
- Number of epochs: How many times the entire training dataset is passed through the model.
- Number of layers and neurons: Defines the architecture depth and width of neural networks.
- Regularization strength: Controls techniques like L1/L2 regularization to prevent overfitting.
- Dropout rate: The fraction of neurons randomly disabled during training to improve generalization.
Finding the optimal combination of hyperparameters, a process called hyperparameter tuning or optimization, is a critical step in developing high-performing machine learning models. Techniques range from simple grid search and random search to more sophisticated approaches like Bayesian optimization, genetic algorithms, and automated machine learning (AutoML) platforms that explore the hyperparameter space efficiently.
For software teams building AI-powered applications, understanding hyperparameters matters for achieving production-quality model performance. Poorly chosen hyperparameters can result in models that underfit (too simple), overfit (too complex), or train prohibitively slowly. Systematic hyperparameter optimization is a standard part of any serious machine learning workflow.