Making a Model Smaller: Quantization and Pruning for Beginners

There's a specific moment familiar to anyone who has pulled a model off Hugging Face for the first time. You find the checkpoint, you write ten lines of loading code, and then you watch your machine throw an out-of-memory error before a single token comes out. This is the practical side of modern model deployment. The same model can be highly capable yet too large for the available memory, too slow for real-time use, or simply impractical to run outside a high-end GPU setup. If we want models to move beyond the hardware they were trained on, we need techniques that reduce their resource requirements without throwing away the capabilities that made them useful in the first place.

Quantization and pruning are two of the most important ways to do that. Neither is new, but together they're the reason a model that once needed a huge amount of GPU resources can end up running on a phone. People conflate them constantly, although they solve different problems.

The core idea: models carry more precision than they need

A trained neural network seems very complex. But underneath, it is just a very large pile of numbers. Every one of those numbers has to be stored, loaded into memory, and multiplied against other numbers every time the model produces an output. By default, most of them are stored as 32-bit floating point values.

The problem is that neural networks are tolerant of imprecision. A weight of 0.7341826 and a weight of 0.734 usually push the network toward the same output. This is exactly what both quantization and pruning exploit, just from opposite directions.

• Quantization keeps every weight, but stores each one using fewer bits.
• Pruning keeps full precision, but removes the weights that aren't contributing much.

They are not competing techniques. They are usually stacked.

Quantization: fewer bits per number

We can think of it like re-encoding a photo as a JPEG. You lose some of the fine detail, but the image looks essentially the same to a human, and the file is a fraction of the size. Quantization does the equivalent thing to a model's weights: it maps a range of floating-point values onto a much smaller set of representable values, and stores a small scale factor so the originals can be approximately reconstructed.

The common precision levels are:

• FP16 / BF16. The easy win. Half the memory of FP32, and the accuracy hit is close to unnoticeable. Most training and inference has quietly moved here as a default.
• INT8. Roughly a quarter the size of FP32. This is where you start needing to be a bit more careful.
• INT4. Roughly an eighth the size. This is the interesting territory for large language models specifically, because it's what turns a 70-billion-parameter model from "needs a multi-GPU server" into "runs on a single consumer card."

There are two very different ways to get there:

• Post-Training Quantization (PTQ): take a model that's already trained, and quantize it afterward, without touching the training loop at all. This is fast and requires almost no extra infrastructure, which is why it's the default most people reach for first. The risk is accuracy loss, especially once you push below 8-bit.
• Quantization-Aware Training (QAT): simulate the effect of quantization during training, so the model's weights are nudged toward values that survive the rounding well. More expensive, but it recovers most of the accuracy PTQ gives up, particularly at 4-bit and below.

In practice, for LLMs specifically, these are rarely implemented from scratch. Methods like GPTQ, AWQ, or bitsandbytes, which are a more careful, calibrated version of PTQ, are used commonly. They are specifically built to keep quality intact at low bit-widths rather than naively rounding every weight the same way.

A minimal example of what this looks like in practice with Hugging Face:

from transformers import AutoModelForCausalLM, BitsAndBytesConfig

quant_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype="bfloat16",
    bnb_4bit_quant_type="nf4",
)

model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-3B-Instruct",
    quantization_config=quant_config,
    device_map="auto",
)

That's it. No training loop, no manual calibration step. The model loads directly into 4-bit precision, and the memory footprint drops immediately.

Pruning: remove the numbers you don't need

Quantization makes each weight cheaper. Pruning takes a blunter approach; it deletes weights outright, on the assumption that a trained network has a lot of dead weight in it. So, removing the connections that barely affect the output can save space and computing resources.

Two flavors show up constantly:

• Unstructured pruning removes individual weights, typically the ones closest to zero, since a weight near zero contributes almost nothing regardless of what it's multiplied against. This can hit very high compression ratios, but the result is a sparse matrix full of holes, and standard hardware isn't built to skip over holes efficiently. Without sparsity-aware kernels, you can prune 80% of a model's weights and see almost no speedup, because the hardware is still doing dense math.
• Structured pruning removes entire neurons, channels, or layers instead of scattered individual weights. It's less surgical, and the compression ratio you can hit before accuracy falls apart is usually lower. But the result stays dense and regularly shaped, which means it actually runs faster on ordinary hardware without needing anything special.

Unstructured pruning may look better on paper, but structured pruning is usually what actually makes something faster in deployment.

A workflow that holds up reasonably well in practice:

1. Train (or start from) a full model.
2. Score every weight or structure by some importance measure; raw magnitude is the simplest and a fine starting point.
3. Remove the lowest-scoring fraction, targeting some sparsity level.
4. Fine-tune briefly, so the surviving weights can absorb what was lost.
5. Repeat in small increments rather than pruning everything in one pass.

A word of caution

Pruning too aggressively, or without fine-tuning afterward, can noticeably hurt accuracy. It's worth treating your pruning ratio as a tunable hyperparameter: prune a bit, check accuracy, prune more if you can afford to, and stop when the accuracy drop is no longer acceptable for your use case.

Quantization vs. pruning: how do they compare?

The honest answer to "which one should I use" is usually both, in that order: prune first to remove genuine redundancy, then quantize what's left. They're attacking different axes of the same waste, and the savings from each roughly compound rather than compete.

Conclusion

To wrap it up, let's imagine what would happen if we couldn't shrink these models. An experiment that you could normally run twenty times in an afternoon might become something you can afford to run only once or not at all, simply because each run takes too long or demands more memory than your hardware has available. That's where quantization and pruning matter: they make models smaller and faster, without the usual trade-off where one comes at the cost of the other. Together, they turn model efficiency from a nice-to-have into something that can determine whether a model is practical to experiment with and deploy on the hardware you actually have.

References

1. Hugging Face. Quantization overview. Transformers documentation. huggingface.co/docs/transformers/en/quantization/overview
2. Hugging Face. Making LLMs even more accessible with bitsandbytes, 4-bit quantization and QLoRA. Hugging Face Blog. huggingface.co/blog/4bit-transformers-bitsandbytes
3. Merve. Introduction to Quantization cooked in 🤗. Hugging Face Blog, 2023. huggingface.co/blog/merve/quantization
4. Liang, T., et al. Pruning and Quantization for Deep Neural Network Acceleration: A Survey. arXiv:2101.09671
5. Ultralytics. Pruning and quantization in computer vision: A quick guide. ultralytics.com/blog/pruning-and-quantization-in-computer-vision-a-quick-guide
6. Deepgram. Model Pruning, Distillation, and Quantization, Part 1. deepgram.com/learn/model-pruning-distillation-and-quantization-part-1

---

Want to share your own experience? Every member can write here: reach out and we'll help you publish your first post.