native 64k+ Gguf model for llama.cpp
I'm doing something a bit silly and trying to get hermes agent fully local on minimal resources. I've got a lenovop520 64gb quad channel ram and an rtx 3060. I'm using qwen3.5 35b with cpu mode so it only takes about 4g of my vram and still gets 30 t/s. I'm…
I'm doing something a bit silly and trying to get hermes agent fully local on minimal resources. I've got a lenovop520 64gb quad channel ram and an rtx 3060. I'm using qwen3.5 35b with cpu mode so it only takes about 4g of my vram and still gets 30 t/s. I'm *trying* to configure a secondary model for LCM context compaction and auxiliary hermes agent tasks. the Problem: it needs to be able to do 64+k natively, otherwise when hermes sends the request for compaction, llama.cpp sees that the request is for a larger context than the native training CTX. using rope flags doesn't change anything, setting 64k context in the flags isn't enough, llama .cpp looks at n_ctx_train and if it's smaller than hermes hard coded request it rejects the task, even models that in native quants can handle 64k sometimes the GGUF is different. I'm looking for a dense, non reasoning model that is quaanized in GGUF that is quantized to handle 64k.
已收录讨论
You can try granite 4.1 3b or phi 4 mini 4b, or use something like gemma e2b/qwen3.5 4b but turn the reasoning off. edit: also, you've probably been looking at weird quants, length should be the same.
I've tried q4_k_M from unsloth mostly, one UD q4 K Xl reasoning model. and I've tried with a reasoning model with reasoning off flag. I can't see exactly what it's putting out but it goes on for so long that I assume it's still doing CoT 'cause it's so baked in. is there a better q4 quant I should look for than q4km?
I just experimented with qwen 3.5 0.8b q4_k_m, unsloth, seems to be working fine. (llama_server location) -m (gguf location) --reasoning off it might be that hermes forces reasoning somehow, but even with reasoning on, it should be near instant. Have you checked your gpu vram usage, to see if it's actually loaded on GPU? Also, you can test it with your 35b to see if it works, if it doesn't work then it must be a setup problem instead.