ORIGINAL REDDIT POST
Question about Quant versus Size.
Sorry if this is asked a lot, but I was wondering if there is any clear winner on the Quantization versus Model Size debate? I can run Qwen3.6 27b at Q8, Laguna at Q6, and the new Deepseek Flash at Q3 bit. I am in the process of testing, but is there a clear…
Sorry if this is asked a lot, but I was wondering if there is any clear winner on the Quantization versus Model Size debate? I can run Qwen3.6 27b at Q8, Laguna at Q6, and the new Deepseek Flash at Q3 bit. I am in the process of testing, but is there a clear formula or winner for choosing between higher quant, especially with long tasks? Or is there a place to find quant specific benchmarks? Thanks.
Collected discussion
I’m no expert so take this with a grain of salt but from what I understand, more parameters is better up to a certain point. It depends on the model as to where that exact point is, but from what I’ve heard, it is generally around Q4 or a smart 3 bit quant and anything less and you’re sacrificing too much quality to be worth the extra parameters
Man, I was hoping for a < 128GB MoE that beats Qwen3.6. I guess I just gotta keep waiting. Thanks for the help.
I would be right on the line in that case. I guess the only thing I can do is run a series of tests and see.
Laguna released new NVFP4, INT4, and FP8 checkpoints of S 2.1 yesterday. So far, in my use, excessive thinking and looping is greatly reduced. I would give it another shot if you are looking for a coding model.
Can offloading be applied to any of these models or only MoE models?
This is all correct, and can be coupled with the probabilistic nature of LLMs plus perceived and expected performance makes fair testing on real workflows a challenge. I remember when the Omnicoder 9B model was all the rage, and they dropped Omnicoder 2 one day...a bunch of people tried it, extolled it's virtues of how much better it was, then it turned out it was the original model and was just a labelling mistake by the uploaders.
It's difficult to compare apples to apples across models for a few different reasons: Models vary in their sensitivity to quantization. I'll fully admit to not understanding this entirely and I believe it to be an active field of study (see things like Quantization Aware Training - QAT), but some models handle quantization down to Q1/Q2 far better than others. One person's Q8 isn't the same as another unfortunately. Unsloth publishes iMatrix driven Q8_K_XL quantizations that have higher total bpw than say a standard Q8_0. Add on top of that the ability to call something NVFP4 and really many of the weights are Q8 or even BF16...the waters get very muddy. Some models are natively not BF16. DeepSeek V4 Flash for example is natively 4-bit. So you can't really do an "apples to apples" run off between Q8 Qwen-3.6-27B and Q8 Deepseek V4 Flash because it doesn't really exist for DS4. Models vary in size of parameters and generally we end up in a VRAM limited state - not a model limited state. So given an RTX Pro 6000 you can run Q8 Qwen-3.6-27B, but you can only run Q2-ish Deepseek v4 Flash, and some-form of 4-bit-ish Laguna-S-2.1...so naturally, that's what you would be comparing instead of Q4 across the board. Some models can make up for lower quantizations with higher reasoning effort / max_tokens settings. If you give AngelSlim-HY3-IQ1 a metric ton of reasoning tokens to work with, it can still be competent, even though it's squished down to somewhere in the 1-2bit range (their IQ1 is a weird beast). But if you turn thinking off, it falls apart rather hilariously (like it's really, really bad)...and unfortunately, this is different model to model (just like #1). With that said, I think the answer to your question is "no there's no clear formula" because of those reasons. One minor thought though - you can compare within the same model. So you have enough VRAM for Q8 27B, but you could also test it at Q6 / Q4 / Q2 / Q1 and see it degrade for yourself. Don't do just one run of whatever task though - run 5 or 10 runs of each one and then the quantization artifacts start to stick out more (but this is very time consuming).
CPU offloading. Won't be as fast as 100% VRAM, but higher quality outputs far outweigh a model that can't finish the task regardless of time taken. Deepseek v4 flash has had no trouble with any task I've given it while every other local model that's usable for me fails to varying degrees.
Qwen3.8 27B is announced to be released in about a week. I'm assuming it will be very hard to beat for coding by anything under 128GB, unless qwen smiles on us and realeases their 122B a10B model.
yea laguna was a dissapointment for me You can try Qwen 3.5 122B tho, its still pretty solid - and compare its output with 27B dense.
It is very hard to generalize the answer to this, but I frequently find the cut off to be at 4-5 bit. If you have less than 256G VRAM that means you are going to be running Qwen 27B-122B or Gemma 31B
DeepSeek and Qwen 3.6 27B should be somewhat similar - but Qwen is gonna be A LOT more consistent with Q3 vs Q8, laguna is not even worth bringing to the table as it would struggle even agaisnt Qwen 35b.
Models from ds and qwen are 4bit built now so it’s sorta not a thing in the same way fron 3 more baths ago