DeepSeek V4 Flash 0731 - Dual Strix Halo Crashes?
Hey all, Not sure whether best to post here or on the llama.cpp board, however...: I have two Strix Halo systems that run over a (cheap) set of mellanox Connect - X 3 cards for RDMA (Parallel, as it uses USB4 instead of the LAN ports that are 5GBe each) All…
Hey all, Not sure whether best to post here or on the llama.cpp board, however...: I have two Strix Halo systems that run over a (cheap) set of mellanox Connect - X 3 cards for RDMA (Parallel, as it uses USB4 instead of the LAN ports that are 5GBe each) All other models from Step 3.7 Flash (my daily driver) to Laguna S2.1 works amazingly with this setup, and has no issues, other than being slower that a model on a single platform. Deepseek V4 Flash is different, however... I can load it using my launcher, with llama.cpp (ROCm) as the driver for this model (updated to today) but when it hits 4096 prompt processing it keeps crashing and dumps from memory. I have tried Claude code but to no avail, and was wondering if others have tried this at all, and had the same issue with running the model, across two strix halo systems? Thanks in advance and happy to post logs if that helps others who are more technical than me :)
Collected discussion
Thanks for the replies! yes ubatch does seem to work... I am guessing it needs a more mature llama.cpp release, somewhere down the road...
How much do you set it?
I have mine at 256 currently
I was getting errors but reducing the ubatch size fixed it.
I'm running the UD-Q4_K_XL on an EPYC server and it's constantly crashing in Hermes-Agent. Likewise, I'm seeing a vastly lower crash rate when reducing batch-size. I'm thinking it probably has something to do with MLA or Deepseek's particular flavour of context.
I haven't been able to RPC properly without crashing as well, surely it will be sorted out at some point. I can run GLM5.2 just fine on the same setup so the issue seems to be on DeepSeek V4 arch somewhere.
The guys are working on proper tensor parallel RPC support, give'em some time.
opencode works much better with ds4 than claude code