A loss function (also called a cost function or objective function) is a mathematical function that measures how far a machine learning model's predictions are from the actual target values. It quantifies the "error" or "cost" of the model's current state, providing the signal that guides the training process. The goal of training a machine learning model is to find the set of parameters (weights) that minimizes the loss function, resulting in predictions that are as accurate as possible.
Different types of problems require different loss functions. For regression tasks (predicting continuous values), common choices include Mean Squared Error (MSE), which penalizes large errors heavily, and Mean Absolute Error (MAE), which treats all errors equally. For classification tasks (predicting categories), Cross-Entropy Loss (also called Log Loss) is standard: it measures the difference between predicted probability distributions and actual class labels. Binary Cross-Entropy is used for two-class problems, while Categorical Cross-Entropy handles multi-class scenarios.
The loss function plays a central role in the training loop of neural networks and other models:
- Forward pass: The model makes predictions on a batch of training data.
- Loss calculation: The loss function computes how wrong those predictions are compared to the ground truth.
- Backpropagation: Gradients of the loss with respect to each model parameter are calculated.
- Optimization: An optimizer (like SGD, Adam, or RMSprop) uses those gradients to update the model parameters, reducing the loss.
Choosing the right loss function is an important design decision that affects model behavior. Custom loss functions can encode domain-specific priorities, for example penalizing false negatives more than false positives in medical diagnosis systems. The loss function you pick essentially tells the model what "good" looks like, so it's worth thinking carefully about what you're optimizing for.