LLM speed no longer the issue.
At this point LLM speed is not the issue at all. 135 tok/ sec. With Qwen 27b and it's great. The issue is that I have to tell it what I want, when there are already docs for this. Like UI docs. Qwen 27b has vision already but it's vision isn't precise enough…
At this point LLM speed is not the issue at all. 135 tok/ sec. With Qwen 27b and it's great. The issue is that I have to tell it what I want, when there are already docs for this. Like UI docs. Qwen 27b has vision already but it's vision isn't precise enough to break a UI design down into parts or assign particular widths and heights to the components for implementing. Is there a good open source vision model I can pair up with Qwen 27b for user interface design tasks?
Collected discussion
135 tok/ sec. With Qwen 27b and it's great 5090?
The bottle neck is it's vision. The code quality is good, the speed is good. Looking for a vision model
Dual 3090s, a 3rd 3090 now for image generation and vision.
Some low quant? Because that does not sound like q8 with full 16bit kv cache.
Your harness sounds fantastic, would you be willing to share it with the world?
Gemma 12B produces better constructive criticism of screenshots than other alternatives.
Gemma 12B QAT? Thats a vision model?
That's f'n cool.
I will but it will probably be at the end of the month.
It's a multimodal model, I built a tool into my harness that my main model can use to query a multimodal or VL model with image files.
What's the max context you are able to reach? And can you please share your config of llama.cpp? I have similar rig as well! Have you tried nvfp4? It's supposed to be much faster.
I use Gemma 12B QAT on a second machine with the high res settings as described in the recent updates as my Qwen 3.6 27B's eyes, disabling the MMPROJ also gives me back some more space for KV cache. Has been doing a good job on refining the design on my vibecoding sideproject.
2x 5060 ti total 32gb - 60 t/s qwen 3.6 27b q6_k - sharing so it can be reference for someone
You can offload mmproj to the CPU, so while slow, it’s still there on the rare occasion you need it without it eating into your vram use.
Are you using containers or namespaces for the fs?
60T/s and 1000T/s prefill for UD_Q5 on a 3090+3060Ti, bit still wished it was much, much faster
https://huggingface.co/microsoft/Fara1.5-27B Havent tried this yet, but seems interesting
And Q0.1_XSSSSS
i was running 4 full glm models and bunch of small ones on xeons, each cpu gave me 3tks. If you pipe it right and setup a proxy you can design an app over night, especially if you have 1.5tb ram, clear instructions, router with deterministic logic being the brain, model to refactor prompts. So each cpu was responsible for a component in a designated design pattern, each class required a unit test, profiler snapshot, in mem db as global context. So it was slow but highly accurate, it would run a debugger as well. Nothing would overlap, clear separation of concerns, rag db with code for trivial components so llm was no involved. No GPU, mediocre Xeons is all i need. I switched to using small models and making deterministic components the main source of truth. the first question is do i need llm for this, most code is already written, some need refactoring to fit into patterns. I designed a file system specifically for llms, keep it from drifting, prompts don't waste time on guessing,no drifting,self analysis , no files or folders, it works of logic trees, tools like rosylin, metadata, asts. FS also serves as context, test bench, analysis , visualization of repo, idea booster. No decisions or code generation by llms unless it's working in a highly constrained, modular environment. Saves me time, money, no dependance on commercial ai, even though i use kimi or other cheap models to create repo/fs visualizations in webgl to see how repo is performing
Well, if the speed is no longer the issue depends on what you want to do with the LLM. For example on conversational AI you want as much speed as possible, especially if you want to use reasoning. Also, the biggest thing that limits our productivity is the AI speed. If you need to wait for AI to finish their things before you can continue, it slows your productivity down and you waste your lifetime.
Give it some function call / script to query CSS / rendered html values?
Is there any tips to speed it up, I tried for the first time last night and it felt slow (7800x3D, 4080 Super, 32Gb ddr5)
Maybe title the post with the question, not the preface
Get better hardware and run a higher quant.