can someone create a website where people share specific hardware specs with specific llama cpp flags so we see what works?

I usually Ctrl+F this one a lot. Tweaking the vars is part of the fun for me. You can also find a bunch of examples if you search this sub. https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md

So a 3dmark but for ai tools ;)

Yeah, LLM intake questions would help a lot, especially if the first form is permissive and then the model tightens it into a reproducible checklist. I would make the trust layer about runs more than users at first. Generic vouching can turn into popularity score pretty fast. Better signals would be: exact command and config hash present logs attached or pasted prompt processing and generation speeds split out model file hash included driver and runtime versions included another user reproduced the same run on similar hardware known bad fields flagged, like missing context length or unknown quant Then each entry could have states like incomplete, needs reproduction, reproduced, and stale after dependency changes. If CUDA, ROCm, llama.cpp, or the model file changed, the old result should age out instead of living forever as gospel. The web-of-trust idea is still useful, but I would aim it at "this person reproduced this exact recipe" rather than "this person is trusted in general". More cookbook with receipts, less benchmark leaderboard drama.

Yep, that is exactly the kind of field I would want logged next to any benchmark result. Power mode, power limit, and thermals can move numbers enough that two people with the same GPU are not really testing the same setup. Laptop GPUs make it even messier because the model name hides the wattage range. For a useful comparison, I would rather see: normal vs max performance mode actual board or package power during decode sustained clocks after a few minutes temperature and whether it throttled plugged in vs battery, if it is a laptop Your 60 to 90 tok/s example is a perfect reason to make "power profile" a required field instead of a footnote.

Same. Claude has been setting up all my models lately. Updates llama.cpp/vllm/whatever -> downloads the model -> settings tuning and benchmark -> verifies model works and is coherent, needle in the haystack test, doesn't loop or do anything else funny -> done

Unsloth discloses them on their website for each of the models.

A useful version would need to capture more than the final command, otherwise the data will turn into folklore fast. I would want each entry to require: GPU model, VRAM, RAM, CPU, OS driver, CUDA or ROCm version, llama.cpp commit hash model name, quant, file size, context length exact launch command and any env vars prompt processing tok/s and generation tok/s separately batch size, parallel requests, cache type, and offload settings power limit and rough wall power if measured whether the run was cold start or warm cache The big thing is separating "it runs" from "it runs well for this workload". A 4090 config for short chat prompts, a Mac Studio config for long context, and a dual 3090 config for throughput can all be correct while looking contradictory. If someone built it, I would make the comparison page workload-first: chat, long-context summarization, coding, embeddings, batch inference. Hardware-first tables are useful, but they hide the reason the flags were chosen.

A useful version would need to capture more than the final command, otherwise the data will turn into folklore fast. Two suggestions here: a) Maybe use an LLM to ask (lots of?) questions after a user entered their setup to make sure descriptions are as complete as possible, b) use a web-of-trust or similar or at least some kind of 'users vouching for other users' mechanism on the TBD site so that spam has less of a chance to confuse people

What I’ve started doing is using my pi harness (with access to my llama.cpp config, model folder, etc) with a smarter model from openrouter, typically GLM 5.2 lately. Then point it to the llama.cpp tools README and tell it to “get creative with settings, run a few agentic tests, and some long form content creation, run multiple iterations and settle on what seems like the best balance”. Then just walk away and let it chew on the problem for a bit. There’s probably better ways, but llama.cpp moves so fast that I don’t have time to do much more than that.

Changing the power limit from normal to max on my 5080 laptop pushed decode from 60 to 90 t/s on Gemma4 12B 😅

What i do is just post my environment to gpt 5.6 high or max and tell it to look at my setup and the latest llama.cpp movements and find out the best combo, it does great job.

Yeah

that field list above is the right shape, but the thing that kills these isn't missing fields, it's that a complete row can't tell you when it stopped being true. commit hash, driver, quant all move, and a row from six weeks ago reads exactly like one from tuesday. that's how you get the folklore even with full provenance. not a local guy, mine is hosted stuff on ads data, but every settings wiki i've watched go bad went the same way. nobody deletes anything so the rows that still work get buried under rows that worked once. the fix is making the entry a rerun rather than a report. one small fixed prompt set and seed, pp and tg reported separately, cheap enough that anyone with that hardware redoes it in two minutes. then you can sort by last confirmed instead of by upvotes.

I have been working on something very close to this called LLM Hardware Sift. It uses a standardized llama.cpp benchmark ladder so results from different CPUs, GPUs, APUs, and Apple Silicon machines can be compared more meaningfully. Results are saved locally for review before sharing, and the longer-term goal is a searchable collection of community hardware results rather than isolated benchmark screenshots. It is still an early alpha, but I would genuinely appreciate feedback on what hardware details and comparisons would be most useful. https://github.com/nozzlenaut/llm_hardware_sift

+1 seems obvious that after a few weeks an optimal command exists for specific hardware and need.

https://old.reddit.com/r/LocalLLaMA/comments/1t1m5mn/what_about_a_website_to_share_our_model_settings/ojhdsss/

I've always wanted to see KL divergence for different models across different quantizations. It'd reduced so much of the guesswork needed to download 3 different quants and test them and discard them after a couple of days. If I knew the KL divergence for half the models, I wouldn't bother downloading any of them, and I could spend my time tweaking other flags.

https://github.com/noonghunna/club-3090 it's a bit messy but it helps

You would see a multitude of different combinations, all claiming to be "the best"

Yes, it also could be you!