Has anyone been working on a solid setup for DSV4F on x2+ R9700s?
I'm hoping that one of you guys has been working on an inference engine or has somehow found improvements to running DSV4F on RDNA4 multi-GPU setups. I am currently building a custom inference engine in Rust using HIP but its still in the early stages. I'm…
I'm hoping that one of you guys has been working on an inference engine or has somehow found improvements to running DSV4F on RDNA4 multi-GPU setups. I am currently building a custom inference engine in Rust using HIP but its still in the early stages. I'm using vulkanforge and antirez' work on ds4 as inspiration, and likely will be adopting a custom quant like what antirez did. The only issue with it is that it's entirely built out for my setup and has things placed on my system in certain areas to work fast. Currently I have 2x R9700s, Ryzen 5 9600x, 128GB DDR5. My second card is still on PCIE 4 x4 so its majorly bottlenecked. Planning to only put the hot experts on that card since bandwidth between would be minimal, then use the RAM for the cold/missed experts with a design to xfer cold experts to the GPU and swap out the least used ones after multiple misses during cooldown periods between processing. From testing my best case scenario is around 80 tok/s on tg with DFlash but my hope is at least 60 tg.
已收录讨论
Deadcode has been working on a custom hybrid inference with dynamic expert placement, maybe y’all can compare notes. I have dual R9700s with 128GB ddr4 and a 5900x, wanting to try running it soon but will be a little slower. https://discord.gg/qcqRVT6Wz
Yeah honestly this project is yet another one that makes me want to upgrade my machine. I was originally planning to go to a higher end AM5 board with 2 PCIE 5 x8 lanes, but at this point I may just get a threadripper or an EPYC for the RAM. Being limited to 85ish GB/s sucks.
Yep, you are spot on, i confabulated that bit
sweet, ill hop in and reach out. I'm expecting to be done today with the first untested revision of the engine w/o the quant performed yet so that would be an ideal point to compare and see what we have came up with in isolation. My work thus far has basically been all about ruthlessly extracting whatever hardware advantages there are in the R9700s and with Ryzen chips, basically focusing on things like getting P2P inference to work, utilizing resizable BAR, trying to figure out how to utilize Wave32 & native FP8 without balooning the model to a bigger size, and determining which placement is the most efficient. Optimistically, I am expecting 80 tok/s on tg as long as DFlash provides a ~1.5x speedup.
It's a bit of a tossup. Prefill hits nearly every expert, so it's bottlenecked by PCIe. Increasing batch size can help, but you lose experts which hurts prefill and decode. Faster CPU & RAM bandwidth allows better CPU inference for cold experts. If lossy is fine, there's also dropping cold experts that have a low confidence or when hot experts exceed a certain amount. I never tested actual quality on that other than "it didn't produce garbage."
I did something similar with an RTX Pro 6000. My goal was to keep the original quality, so no quantization or dropping experts. One of the biggest improvements was running cold experts on the CPU and transferring the top k hot experts every so often, but that is with an EPYC 9654 and 12-channel memory. May not be useful on a 9600x, but it's an idea. Gets about 100tps single request with dspark and 160tps with 2.
your hot expert placement is backwards. put hot experts on the x16 card. the x4 bottleneck hits every token because every forward pass calls the hot experts, cold experts are only touched on cache misses. also on rdna4 you want persistent kernels for moe. hip launch overhead on rdna is way worse than cdna, so without persistent kernels you're paying a launch tax on every expert invocation. use hipDeviceEnablePeerAccess for direct gpu-to-gpu transfers instead of bouncing through cpu.
https://huggingface.co/salen-00/DeepSeek-V4-Flash-0731-GGUF I based this dynamic quant on the antirez fork back in may, might be of interest to you. cheers
It’s a shame amd don’t give a shit about their products.