dual rtx pro 6000s and can't get dspark to work with sglang nor vllm. Any tips?

https://github.com/local-inference-lab/blackwell-llm-docker 200 to 300 TGs and 6500 PPs, context is 256k, you may be able to squeeze another 50k or so.

Ask ChatGPT sol to help you. There is also a discord for pro 6000 owners. I have two as well and got it working with vllm at 1 mil context size at original quant, around 200 t/s a second. I recommend power limiting down to 150 watts. BTW, it’s not that great IMO. Also, make sure you have max thinking enabled or it’s just hot garbage Been trying to get GLM 5.2 running at a decent speed…making progress!

I would get one boring baseline working before trying to make dspark part of the recipe. For dual RTX Pro 6000s I would test in this order: Start from a known-good Blackwell container instead of mixing host packages. The local-inference-lab Blackwell LLM Docker repo has been the least painful path people are sharing right now. Prove vLLM or SGLang works on one card first with the exact quant you want. Add tensor parallel across both cards only after single-card inference is stable. Keep dspark off until you have a clean baseline for prompt processing, generation speed, max context, and VRAM use. Avoid random community NVFP4 builds unless someone posts the exact command, CUDA version, driver, model hash, and launch args. The failures can look like a serving bug when it is really a quant/runtime mismatch. If MXFP4 works with dspark off, I would treat that as the baseline and save the full command. Then change only one thing at a time: dspark on, different backend, different quant, longer context. Otherwise it is impossible to know which layer broke. Also check power and clocks. A moderate power limit can make dual-card setups more stable without costing as much throughput as you would expect.