In artificial intelligence and machine learning, a weight is a numerical parameter within a neural network that determines the strength of the connection between neurons in different layers. Weights are the core learnable parameters that a model adjusts during training to minimize prediction errors. Together with biases, weights define how input data is transformed as it passes through the network, ultimately shaping the model's ability to recognize patterns, make decisions, and generate outputs.
During the training process, weights are initialized (often randomly or using strategies like Xavier or He initialization) and then iteratively updated through a process called backpropagation. The optimization algorithm (such as stochastic gradient descent, Adam, or RMSprop) calculates how much each weight contributed to the overall error and adjusts it accordingly. Over thousands or millions of iterations, the weights converge toward values that allow the model to accurately map inputs to desired outputs.
The concept of weights is fundamental to virtually every type of neural network architecture:
- Feedforward networks use weights to connect input, hidden, and output layers
- Convolutional neural networks (CNNs) apply weight matrices as filters to detect visual features
- Recurrent neural networks (RNNs) share weights across time steps to process sequential data
- Transformer models like GPT and BERT use attention weights to determine how tokens relate to each other
Modern large language models can contain billions or even trillions of weights, which is why training them requires massive computational resources. Understanding how weights function matters for tasks like model fine-tuning, transfer learning, pruning for deployment on edge devices, and diagnosing issues like overfitting or vanishing gradients during the development process.