Running Deepseek 4 Flash 0731

Have you tried looking into buying an AI datacenter ?

How much would that cost?

Like 4.000 usd where I live

Small correction/addition - I have 8x 3090s, it doesn't work at the speeds OP mentions - at least 25-30t/s I have llama-bench-ed it on 8x 3090s (3.0 x16 + 4.0 x16 mixed, not x1 risers) and the max you can get is 220 t/s pp and 18-20 t/s decode. To run deepseekv4 flash at the speeds it was intended you NEED (it's a non negotiable) hopper+. Ampere does not support fp4/fp8 and there is no roadmap of supporting ampere in the future. So 3080s/3090s/CMP 170hxs do not work. As for vLLM - Upstream vLLM does not support FP8 / sparse-MLA / DeepGEMM on SM < 90 and has declined SM80-class patches ("SM80 support better lives in a fork", PR #40906). https://github.com/Lasimeri/vllm-dsv4-ampere consult here for more numbers. 3080s get into single digit decode.

I have 8x 3090 Ti and I didn't run new Flash yet. I did run old flash a bit but I don't remember the speeds I was getting. However, I get 500 t/s PP and 43 t/s with Nex N2 Pro 397B A17B in exllamav3/ik_llama. I'm confident that DS V4 Flash could run faster than 220 t/s PP and 20 t/s TG on 8x 3090 setup, and it's mostly a matter of software getting there. I will probably find time to run it locally this weekend, I'll share the speeds I'll get. Spark is a separe arch sm121 which has its own compatibility issues, Blackwell gpus are much more expensive. Some jankiness can happen, but V4 Flash should ultimately be runnable on sm86. And models will keep on getting better so I wouldn't buy hardware for a specific model, rather I'd buy it for extrapolation of the trends and whatever your wallet can support. Edit: i posted speeds in the past but it was on initial DS V4 Flash support code https://www.reddit.com/r/LocalLLaMA/comments/1ukv66u/deepseek_v4_flash_on_quad_3090_box/ov1i1wz/

I also wanted to recommend 2x Pro6000 until I saw that OP wants 8-16 simultaneous users, and that requires a whole datacenter at home.

which fork are you using? I see a few https://github.com/AppMana/forks-vllm-consumer-nvidia-platforms https://github.com/Lasimeri/vllm-dsv4-ampere and there might be more.

I get more than that with high concurrency lol (EDIT: on the preview v4 flash, on 0731 I get around 200tps)

Can prefill be ass? cheapest and easiest-to-maintain hardware Choose one, or at least your priorities. I have some cheap-ish setups in mind but they are big, janky and hard to maintain. I am not sure if anything will beat 2x GB10-powered mini-PCs ( Asus GX10, DGX Spark , HP ZGX Nano etc) there. And a lot will depend on where you live, if you want to buy used hardware and how your local used market looks like. 6x R9700 AI or 8x 3090 or 6x MI50 32GB or V100 32GB or 4x cmp 170hx or 6x Intel B70 or 10x RTX 3080 20GB. Pick your poison. Most of those setups are in the range of $5k to $10k, with some outliers. And I didn't check if V4 runs on those older cards at all. It's supported by llama.cpp so they probably do work too, I guess. But if you want to use vllm/sglang, I think you need blackwell for it (not sure if consumer blackwell is enough but I will assume it is for now), so 6x 5090 or 2x RTX 6000 Pro.

I tried new DS V4 Flash 0731 briefly in ik_llama.cpp this quant - https://huggingface.co/bullerwins/DeepSeek-V4-Flash-0731-GGUF/blob/main/DeepSeek-V4-Flash-0731-MXFP4_MOE-BF16.gguf I get about 25 t/s TG with few token question. When pasting 4.7k doc I got 437 t/s prefill and 23.3 t/s TG It's not too bad. exact command to launch llama-server used was ./llama-server -m /path/to/model/dsv4flash0731.gguf -ngl 99 --jinja --no-mmap -c 32768 --host 0.0.0.0 -sm graph --max-gpu 8 -cuda fusion=1 -ger I don't know if graph mode works with V4 Flash and I don't remember what ger does, I just copied it from older run command I had noted down. I expect that if this model will get attention, perf will improve. IDK how it looks with mainline llama.cpp yet, I didn't test that. No idea about concurrency too. This is with GPUs power limited to 320W but they were not close to being well utilized.

I have 1x3090 (soon to be 2x3090) + 128gb ddr5, i'd love for someone to try this

3090 - upstream vllm won't work, they declined sm86 PR adding the support (at least for now). There is vllm fork, with 8 cards it fits but tight without cuda graphs. pp=5 tp=2 fits way better - I can enable cuda graphs there and pp tps is really good ~3k tps, tg is ~45tps. (Note one would lose mtp with pp but I don't trust MTP implemetations yet and my workflow is fine with PP) yet to try llama.cpp next week.

I think 3x 5060 ti total 48gb, and 4x 32gb total 128 gb ram would be sufficient and cheapest. I don't have any idea about t/s ratio. I would be glad if someone with this config shared theirs because i am thinking about this build lately.

Two Sparks not just GB10s. The N1X (when it eventually comes out) doesn't have the ConnextX-7 port.

200tps aggregate? Not realistic for 2xGB10.

You can run native to my surprise on dual RTX pro 6000. I've been doing hacky solutions for like a month and I just decided to try it out on native vllm and it just worked. D flash doesn't work but it doesn't matter cause you get 120 tokens per second even without it. .93 is a million context window for memory utilization I got mine capped at 250k though .8855 mem util

I can say dual Sparks run DeepSeek v4 Flash at a very reasonable speed; over 1000tps prefill, 27tps generation (higher with speculative decoding) and with a 384k context window.

Pretty sure you can do this with 2 sparks

Dual rtx pro6000, only way. Or perhaps some kinda tensor parrallel with multiple large ram drr5 machines. Need to be certain of your usecase. Qwen3.6 27b runs well on a single 9700 at q6. I use 27b as my daily, 120b+ are for thinking only.

1 - 2 sparks (depends on quant). if you want speed dual 6000 pro (i prefer maxq's)

I'm running it comfortably for under $5k all-in on 8x V620s. Not the fastest, definitely better than the performance you're looking for. I don't think you can get much cheaper than that, though.