5700, 48GB RAM and a 3090 24Gb. Best OS and framework/model?
Hi everyone, I have a system which I have been using for gaming, R7 5700X, 48GB DDR4, RTX3090 24GB. But I want to use it for Local AI to reduce my reliance on cloud AI providers (mainly usage limits - accepting some quality loss). I have had it setup with…
Hi everyone, I have a system which I have been using for gaming, R7 5700X, 48GB DDR4, RTX3090 24GB. But I want to use it for Local AI to reduce my reliance on cloud AI providers (mainly usage limits - accepting some quality loss). I have had it setup with Ubuntu Server and was using the machine as a server to connect via WebUI, but for some reason an update broke the NVIDIA drivers and then broke my install so I’m starting again. The question is what platform do I run as there are so many platform to choose from and many differing opinions, I tried Ollama+OpenWebUI and Unsloth Studio. Ollama ran very slow with the models (Gemma 4 and Qwen3.6), also OpenWebUI occasionally was slow with web/MCP, but it was stable. Unsloth however, was quick and search/tool calls worked perfectly but unstable and the models crashed a few times. I’d just like to know, what is everyone else using for this kind of setup, I can’t get a solid sense of what is the go-to setup for this kind of system is (some say Unsloth, or Ollama, or llama.cpp etc). Also what models are people running well on 24Gb VRAM + 48GB RAM? Its primary job is coding/finding info from the web & PDF’s/generating config files. Thank you
已收录讨论
CLI or GUI? GUI - bazzite (a flavor of ubuntu) which has absolute perfect nvidia support. CLI - just ubuntu 26.04 LTS. And every model is built differently I get a lot of good results on llama.cpp for coding but better results on swarms on sglang and sometimes even ollama for fiction.
What makes catchy the fastest distro?
Allegedly kernel optimizations for CPU, even if those improvements are negligible or non existent I would still recommend it. Arch Linux can be little rigid but it has been good to me on my inference rigs
Minor correction: bazzite is based on fedora, not Ubuntu.
Ah my bad thanks. You’d think I’d realize that running it on two machines and Ubuntu on my server.
Cachyos for up to date drivers and good gaming support and llama-cli is faster than ollama.
Llama.cpp + a Linux distro of your choice (I'm on Ubuntu 26.04) Models - Qwen 3.6 27B - coding/STEM Gemma4 31B QAT - chat, topic exploration, writing Gemma4 26B A4B QAT - faster at web search, generic chat I know it's frowned upon here but use LM Studio if you're feeling lazy. You can always move to llama.cpp once you're ready.
If you want to go with linux, CachyOS, Its the fastest linux distro. And for your setup I'd say use Llamacpp's base UI (llama-server) with MCP or LM studio with MCP. And for the model, Qwen 3.6 27B fully on GPU though you could also do Gemma 4 31B. iQ4_Nl is a really good quant. Allows you to have a lot of context size. You can run both in fully gpu with a lot of context size which is good for the reason you use LLM's for. I wouldn't say use context quantization. If you want to get to utilizing the System ram too and running a larger model, GPT OSS 120B at its native MXFP4, though you need to split across gpu and the ram properly and will still have some part of it go to swap/disk, and you should let it go. It shouldn't impact much since its a MoE and only a small portion of it but it means you cant do "--no-mmap" if you dont want to see your whole system crash which would require magic sysrq keys to recover from which I advise you to learn if you want to test your hardware's limits. If you want to go lower than GPT OSS, try Solar open 100B.
So I have done tons of testing with a single 3090. I have found the best setup imho. https://huggingface.co/DavidAU/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-NEO-MAX-MTP-GGUF Run this model with the IQ4XS-low quant with MTP. You can fit full 262K context in VRAM with KVcache at Q4. If you want KVcache at Q8, drop the context by half. This setup hits hard. Model at IQ4XS beats stock qwen 3.6 27B. Ive used LMstudio