What is Quantization (in AI)? — AI Glossary

Quantization in AI reduces the numerical precision of a model’s weights — for example, converting 32-bit floating-point numbers to 4-bit integers — making models dramatically smaller and faster to run while preserving most of their capability. A Llama 3 70B model in full FP32 precision would require ~280GB of VRAM to run. Quantized to 4-bit (Q4), the same model fits in ~35GB — achievable on a single high-end consumer GPU. Quantization is the primary reason powerful open-source AI models are now running on laptops.

How Quantization Works

Neural network weights are typically stored as 32-bit or 16-bit floating-point numbers during training. These provide high precision but consume a lot of memory. Quantization maps these high-precision values to a smaller set of values — integers rather than floats:

• FP32 (full precision): 4 bytes per weight. Most accurate, most memory-hungry.
• FP16 / BF16: 2 bytes per weight. Standard for modern GPU training and inference. Negligible quality loss.
• INT8: 1 byte per weight. ~2x memory reduction from FP16. Small quality loss on most tasks.
• INT4: 0.5 bytes per weight. ~4x reduction from FP16. Noticeable quality loss on very complex tasks, but often acceptable for practical use.
• INT2 / Binary: Extreme compression. Significant quality degradation. Research area.

The key challenge: how do you map thousands of different float values to just 16 (4-bit) or 256 (8-bit) discrete values with minimal quality loss? Advanced techniques like GPTQ, AWQ, and GGUF address this through calibration datasets and clever mapping strategies.

Quantization Formats for LLMs

If you’ve downloaded open-source models, you’ve encountered these formats:

• GGUF: The dominant format for CPU/GPU inference with llama.cpp, Ollama, and LM Studio. Supports mixed quantization (different layers at different precisions).
• GPTQ: GPU-optimized post-training quantization. Used with transformers library and AutoGPTQ.
• AWQ: Activation-aware weight quantization — smarter than GPTQ at preserving important weights. Generally slightly better quality at same bit width.
• EXL2: Used with ExLlamaV2, offers flexible mixed-precision per-layer quantization for maximum quality at a given memory budget.

For most users running models locally via Ollama or LM Studio, Q4_K_M (4-bit, medium variant) is the recommended sweet spot — good balance of size reduction and quality retention for most use cases. Combined with model distillation, quantization makes AI accessible on consumer hardware.

Post-Training vs. Quantization-Aware Training

There are two approaches to quantization:

• Post-Training Quantization (PTQ): Quantize an already-trained model. Fast and flexible, but accuracy loss is higher at extreme quantization levels. GPTQ and AWQ are PTQ methods.
• Quantization-Aware Training (QAT): Train the model with simulated quantization from the start, so it adapts to reduced precision. Better quality, but requires full training access (less feasible for large models). Used by companies like Google for their production edge deployments.

For the AI community using open-source models, PTQ (specifically GGUF Q4 variants) has been transformative — enabling state-of-the-art models to run on the hardware people already own. This democratization of inference is a significant force in the open-source AI ecosystem and shapes broader AI infrastructure choices.

Key Takeaways

• Quantization reduces weight precision (e.g., FP16 → INT4) to shrink model size and speed up inference.
• 4-bit quantization reduces memory by ~4x with modest quality loss for most tasks.
• GGUF is the dominant format for local inference; GPTQ and AWQ for GPU-optimized serving.
• Q4_K_M is the recommended format for most local LLM users balancing quality and memory.
• Quantization enables frontier-class models to run on consumer hardware — a major democratizing force.

Frequently Asked Questions

Does quantization hurt model quality?

For 8-bit: minimal quality loss, usually imperceptible. For 4-bit: small but measurable degradation, especially on complex reasoning tasks. For creative and conversational tasks, Q4 models often perform comparably to the full-precision original in practical use.

What is the difference between quantization and pruning?

Quantization reduces the precision of existing weights. Pruning removes weights entirely (setting them to zero), reducing the number of active parameters. Both are model compression techniques, and they’re often combined — prune unimportant weights, then quantize what remains.

Can I run a quantized model on a Mac?

Yes. Apple Silicon Macs (M1-M4) are excellent for running quantized LLMs via llama.cpp, Ollama, or LM Studio. The unified memory architecture means GPU and CPU share memory, so a Mac with 32-64GB of RAM can run impressively large quantized models.

What does Q4_K_M mean?

In GGUF naming: Q4 = 4-bit quantization, K = uses k-means quantization (a smarter grouping strategy), M = medium quality variant within the K family. Q4_K_M is one of the most popular formats because it offers excellent quality at 4-bit with optimized GGUF implementation.

Is quantization only useful for inference, or also training?

Primarily inference. Training requires higher precision to compute accurate gradients. However, techniques like QLoRA (Quantized Low-Rank Adaptation) enable fine-tuning quantized models by keeping base weights in 4-bit while training small LoRA adapters in higher precision — a popular approach for consumer-GPU fine-tuning.

---

Want to go deeper? Browse more terms in the AI Glossary or subscribe to our newsletter for daily AI concepts explained in plain English.

Free download: Get the Beginners in AI Report — free daily coverage of AI efficiency, local models, and infrastructure developments.

Sources

• Grokipedia
• Stanford HAI AI Index Report

You May Also Like

• What Is Artificial Intelligence
• AI Glossary
• AI for Dummies
• How Does AI Work
• Essential AI Skills

---

Get free AI tips daily → Subscribe to Beginners in AI

Sources

This article draws on official documentation, product pages, and industry reporting. Specific sources are linked inline throughout the text.

Last reviewed: April 2026