How do you test your setup?

I have a test harness that allows me to provide a source file. I ask the robot for a code review of the work I've been working on recently. I get a pretty clear idea what I think of its capabilities by seeing what it finds and how it suggests the changes.

Ha, it seems to be a good test of how far ahead the model can reason and follow through to the end with accuracy. I used the models that passed the test in practice with coding agents to see if they are smart or just good at chess. It involves a few minor tool calls to feel out issues.

I would keep the test setup boring and repeatable before adding bigger benchmarks. My baseline would be: One short prompt, one medium prompt, one long-context prompt, and one ugly edge case you actually care about. Same model, same quant, same sampler settings, same context size, same batch/concurrency settings. Save tokens/sec, prompt processing speed, TTFT, peak VRAM, power draw if you can, and whether the answer is actually acceptable. Run each test cold once, then warm twice. The cold result catches loading and cache weirdness. The warm results tell you steady state. Keep a tiny markdown log with backend version, driver/CUDA or ROCm version, command line, and hardware notes. The important thing is to test the workload you really run. A synthetic benchmark can say your setup is faster while your actual 16k context coding prompt feels worse. For local changes, I would only change one variable at a time: quant, backend, context size, flash attention, offload, GPU split, CPU threads. If two things change together, the benchmark turns into folklore. A small set of repeatable prompts plus a boring spreadsheet beats vibes. It also makes upgrades less emotional, because you can see whether the new driver or second card helped the thing you actually do.

It’s not a hard number comparison but I’ve been happy with models I choose based on this chess prediction test. Many models fail in odd ways, a few get very close with comfortable consistency, and occasionally the better models just get the board cells reversed and the play correct. I also like those that successfully complete the test in around 5 minutes or less. https://www.reddit.com/r/LocalLLaMA/s/QTh1Qswz7n

Implement a b+tree with insert, update and delete functions, then feed it 100k context of unrelated mumbojumbo, then ask it to add unit tests for the code at beginning. But tbh i just use things and don't waste my time on endlessly benchmarking things. Qwen 3.6 27b at q8 quant with full kv cache works well enough for a big set of tasks.

Compare numbers for what? You didn’t specify. Quality? Time to completion per task? Prefill? Decode? False positives? False negatives? Concurrency? The answers will be different depending on what’s actually useful for you to measure. If you don’t clearly define your objectives you will fail.

I have my own script specifically to test KV cache resiliency. Large context needle-in-haystack is only one of several tests it does. It doesn't test the typical stuff like PPL and KLD, because I rely on the usual suspects (Unsloth, mradermacher, bartowski) to supply high quality quants. I do use KV quantization to allow a bigger context window so I use this script to verify accuracy. I won't accept anything but a perfect score. The tests that really separate the wheat from the chaff are the determinism tests. They catch even the tiniest of inconsistencies (things that PPL and KLD and "benchmarks" do not show). I have learned that there is interplay between model quant, MTP, and KV cache quants - therefore I never just assume that any given recipe is "good enough". Once a specific setup passes this test. I put it to use in real-world (coding, research, chat, translation, etc. etc.) and I am now seldom disappointed. I am currently refining the script so it is more suitable for public consumption and might stick it on Github. On my rig (twin 3060 12G) it takes about 30 minutes, give or take, depending on the model and context size being tested.