"Exploring LLM Quantization Formats and Methods"

View profile for Huizi Mao

Deep Learning @ NVIDIA | Ex Co-Founder and CTO @ OmniML

More and more LLM models are released in native quantized formats. This blog (https://lnkd.in/gfTcp5yM) provides a brief overview of #LLM #quantization formats and methods, plus insights into the "native quantization" of DeepSeekV3.1 and GPT-OSS.

  • table
Kyle Sayers

Model Optimization @ Red Hat

1mo

Great overview of existing quantized models. I'll also mention that you're not limited to just the quantization formats provided on release day. Other techniques like GPTQ activation ordering, mixed precision quantization, and Hadamard transforms should be used to boost recovery and performance. You can apply these techniques yourself using tools like https://github.com/vllm-project/llm-compressor, which is an open source project which I helped to develop. This was the tool used to compress the Llama4 family of models and can be used to quantize your own model weights/ architectures with different schemes/formats.

To view or add a comment, sign in

Explore content categories