Deepseek V4 Flash 0731 KV Cache precision

I believe the repo warns to only try f8 if necessary for memory constraints. I’m already seeing quite a bit of degradation and I’m running IQ2_M as well, with 262K unquantised context. I expect f8 K/V would make things worse.

48Gb VRAM and 64Gb DDR5.

My friend has B300 system from the company that runs deepseekv4 , K3 ,glm52. DS4pro use like 3G vram per 1M session with FP8 kvquant. it's very cheap to run.

How

llama-perplexity. Take the model you want at native kv cache and then generate logits (big file). Then have llama-perplexity compare that to what a quantized model or a kv cache quantization generates. Here's an example that I did from my notes: GENERATE THE LOGITS (THIS IS DIFFERENT FROM THE NEXT COMMAND) /home/user/llm/q8_llama/llama.cpp/build/bin/llama-perplexity \ -m /home/user/llm/models/Qwen3.5-2B/Qwen3.5-2B-BF16.gguf \ -f /home/user/llm/wikitext-2-raw/wiki.test.raw -t 8 -c 512 --chunks 200 \ -fa on --cache-type-k bf16 --cache-type-v bf16 --no-mmap -ngl 999 -np 1 \ --kl-divergence-base /home/user/llm/models/Qwen3.5-2B/Qwen3.5-2B-BF16.logits Now run a comparison with KV cache quant /home/user/llm/q8_llama/llama.cpp/build/bin/llama-perplexity \ -m /home/user/llm/models/Qwen3.5-2B/Qwen3.5-2B-BF16.gguf \ -f /home/user/llm/wikitext-2-raw/wiki.test.raw -t 8 -c 512 --chunks 200 \ -fa on --cache-type-k q8_0 --cache-type-v q8_0 --no-mmap -ngl 999 -np 1 \ --kl-divergence --kl-divergence-base /home/user/llm/models/Qwen3.5-2B/Qwen3.5-2B-BF16.logits Pay very close attention because the first command used --kl-divergence-base whereas the second command uses --kl-divergence --kl-divergence-base That will give you this kind of output (this is just an example not that exact result) ====== Perplexity statistics ====== Mean PPL(Q) : 12.572844 ± 0.157398 Mean PPL(base) : 12.554549 ± 0.156918 Cor(ln(PPL(Q)), ln(PPL(base))): 99.97% Mean ln(PPL(Q)/PPL(base)) : 0.001456 ± 0.000291 Mean PPL(Q)/PPL(base) : 1.001457 ± 0.000292 Mean PPL(Q)-PPL(base) : 0.018294 ± 0.003683 ====== KL divergence statistics ====== Mean KLD: 0.001067 ± 0.000005 Maximum KLD: 0.037869 99.9% KLD: 0.009374 99.0% KLD: 0.004592 95.0% KLD: 0.002812 90.0% KLD: 0.002172 Median KLD: 0.000887 10.0% KLD: 0.000053 5.0% KLD: 0.000010 1.0% KLD: -0.000000 0.1% KLD: -0.000014 Minimum KLD: -0.000059 ====== Token probability statistics ====== Mean Δp: -0.001 ± 0.004 % Maximum Δp: 10.814% 99.9% Δp: 4.677% 99.0% Δp: 2.905% 95.0% Δp: 1.588% 90.0% Δp: 0.951% 75.0% Δp: 0.192% Median Δp: -0.000% 25.0% Δp: -0.190% 10.0% Δp: -0.953% 5.0% Δp: -1.559% 1.0% Δp: -2.898% 0.1% Δp: -4.798% Minimum Δp: -7.554% RMS Δp : 0.944 ± 0.006 % Same top p: 97.545 ± 0.069 %

DS4F should only use 6gb for 1m context. if that is not the case, there is room for improvement in implementation.

They told us not to fuck around with kv quants. You did the confirmation so all my points to you.

I don't think anyone should quantize the KV with DS4F. I checked the the quality impact (PPL, KLD, Same TopP) for swhitching from BF16 KV to Q8 KV, and it appears significant. Very much in contrast to Qwen 397B. Here are the results for DS4F: ====== Perplexity statistics ====== Mean PPL(Q) : 5.877076 ± 0.042497 Mean PPL(base) : 5.839660 ± 0.041730 Cor(ln(PPL(Q)), ln(PPL(base))): 95.74% Mean ln(PPL(Q)/PPL(base)) : 0.006387 ± 0.002100 Mean PPL(Q)/PPL(base) : 1.006407 ± 0.002114 Mean PPL(Q)-PPL(base) : 0.037416 ± 0.012318 ====== KL divergence statistics ====== Mean KLD: 0.145884 ± 0.001043 Maximum KLD: 12.467786 99.9% KLD: 4.535020 99.0% KLD: 1.857870 95.0% KLD: 0.652148 90.0% KLD: 0.349220 Median KLD: 0.032079 10.0% KLD: 0.000093 5.0% KLD: 0.000012 1.0% KLD: 0.000000 0.1% KLD: -0.000002 Minimum KLD: -0.000025 ====== Token probability statistics ====== Mean Δp: -0.007 ± 0.031 % Maximum Δp: 99.525% 99.9% Δp: 81.503% 99.0% Δp: 42.054% 95.0% Δp: 14.588% 90.0% Δp: 7.220% 75.0% Δp: 1.066% Median Δp: 0.000% 25.0% Δp: -1.061% 10.0% Δp: -7.112% 5.0% Δp: -14.515% 1.0% Δp: -42.297% 0.1% Δp: -84.157% Minimum Δp: -99.994% RMS Δp : 11.884 ± 0.069 % Same top p: 87.189 ± 0.088 % Here are the results for Qwen 397B: ====== Perplexity statistics ====== Mean PPL(Q) : 3.747980 ± 0.020507 Mean PPL(base) : 3.746773 ± 0.020461 Cor(ln(PPL(Q)), ln(PPL(base))): 99.89% Mean ln(PPL(Q)/PPL(base)) : 0.000322 ± 0.000260 Mean PPL(Q)/PPL(base) : 1.000322 ± 0.000260 Mean PPL(Q)-PPL(base) : 0.001207 ± 0.000975 ====== KL divergence statistics ====== Mean KLD: 0.003552 ± 0.000034 Maximum KLD: 2.220941 99.9% KLD: 0.131591 99.0% KLD: 0.043847 95.0% KLD: 0.014439 90.0% KLD: 0.007836 Median KLD: 0.000866 10.0% KLD: 0.000013 5.0% KLD: 0.000004 1.0% KLD: -0.000000 0.1% KLD: -0.000006 Minimum KLD: -0.000176 ====== Token probability statistics ====== Mean Δp: 0.019 ± 0.005 % Maximum Δp: 39.939% 99.9% Δp: 15.971% 99.0% Δp: 6.618% 95.0% Δp: 2.334% 90.0% Δp: 1.222% 75.0% Δp: 0.233% Median Δp: 0.000% 25.0% Δp: -0.219% 10.0% Δp: -1.183% 5.0% Δp: -2.258% 1.0% Δp: -6.245% 0.1% Δp: -14.757% Minimum Δp: -88.445% RMS Δp : 2.024 ± 0.022 % Same top p: 97.929 ± 0.037 %

If you are happy going down to q2 with your weights, you are happy to go down to at least q8 for kv.

What hardware? f16 with SYCL on Intel B70 seems OK out to 256k on Q3_XXS. Linux, 128GB DDR5.

You can test this so easily

The designers tell you to not fuck around with kv quants. Their shit dont work with it.

Q8 doesn't degrade performance more than 0.1%, if that helps :)

Hell no. DS4 uses tiny KV cache already. Compressing it more wrecks it.