Quantization is a technique used in machine learning and deep learning to reduce the precision of numerical values in a model, typically converting 32-bit floating-point numbers to lower-precision formats such as 16-bit, 8-bit, or even 4-bit integers. This process shrinks model size and memory requirements while speeding up inference, making it possible to deploy large AI models on resource-constrained devices like smartphones, edge hardware, and embedded systems.
There are several approaches to quantization, including post-training quantization (PTQ) and quantization-aware training (QAT). Post-training quantization applies precision reduction to an already-trained model, making it the simpler and faster approach. Quantization-aware training, on the other hand, simulates lower-precision arithmetic during the training process itself, typically yielding better accuracy at the cost of additional training time and complexity.
Quantization has become especially important with the rise of large language models (LLMs), where model sizes can reach hundreds of billions of parameters. Techniques like GPTQ, GGML, and AWQ have made it practical to run powerful AI models on consumer hardware. While quantization inevitably introduces some loss of precision, modern methods can preserve most of a model's performance while reducing its footprint by 50-75% or more.