Looking for inference compute integration ideas - standard consumer 5090 PC, TB4/5 5090 eGPU, M3U 256gb Studio, & 14th Gen Dell Server
Before you roast me too hard, this is a hobby and all of this is just for fun. Would my stack be much more efficient and efficacious if I sold everything and built a dual Pro 6000 system on a threadripper mobo and threw in a large JBOD? Without a doubt in my…
Before you roast me too hard, this is a hobby and all of this is just for fun. Would my stack be much more efficient and efficacious if I sold everything and built a dual Pro 6000 system on a threadripper mobo and threw in a large JBOD? Without a doubt in my mind. But that's a lot of work so I'm making this post in cope hope of finding some ideas to integrate, or at the very least, just make use of my current hardware. I currently use my 5090 PC + my 14th gen Dell T640 server for all my local AI work but recently picked up a TB4/5 5090 eGPU and a M3 Ultra Mac Studio with 256gb unified mem and am trying to figure out how to integrate them or create a new workflow. My primary use case is agentic coding, lots of workflow automation, and peripheral utilities (TTS, embedding, compression, etc). I use cloud subscriptions for orchestration/spec building and then push that to Qwen3.6 2.7B on the 5090 PC to execute while the Dell server hosts dev envs, local TTS, embedding, compression, and other lightweight/MOE models to support the agentic workflows & persistent memory. The server also hosts 20 or so services and a ~300TB Raidz2 array mostly unrelated to AI. I picked up the Mac Studio 256gb because Qwen3.6 2.7B at NVFP4 (~180k context) on the 5090 PC was still kind of dumb. I wanted to use larger model weights to relieve my cloud subs from spending so much usage on orchestration/validation rather than building. My initial idea was to shift from: Cloud orchestration/spec build —> 5090 PC execution to, Cloud orchestration/spec build —> M3U execution + 5090 PC load balancing slightly dumber parallel inference tasks while the slower M3U is busy. Then I picked up this Aorus RTX 5090 eGPU that can't be fully utilized by my 5090 PC, Dell Server, or Mac Studio. The PC and server don't have the TB4/5 connection required and the Mac Studio doesn't have effective inference engine drivers / kernel optimization available for Nvidia. I do, however, have an older RTX 3080 Razer laptop that can enumerate the 5090 eGPU through its TB3 port but I am not sure what I would use this "node" for besides more parallel/concurrent inferencing. I considered it for multi-step image/video diffusion work or as a training node but neither of those are things I do often or am deeply involved in. So, what would you do in this situation? You have an 8yr old Dell sever (PCIe 3.0), a 5090 consumer PC, a 5090 tb4/5 eGPU connected to a 3080 laptop, and a M3 Mac Studio with 256gb memory. Everything is connected on a 10GBE network but inferencing power is all isolated and independent from each other. I could be wrong, but AFAIK, there are no effective ways to execute tensor parallelism, splitting layers, etc. over network.
已收录讨论
obviously add 5090 to your mac studio to increasse prefill speeds... the other 5090 is useful for comfyui image/video gen setups and nvfp4 qwen 3.6 27b... other than that a single 5090 is not that useful... maybe sell that node altogether and get something else. maybe sell everything and get 2 sparks... they are doing well with the 0731 deepseek v4 flash I heard.
obviously add 5090 to your mac studio to increasse prefill speeds wait, this is possible?! how would that work if the 5090 can't fit the model loaded on the studio? there is software support for this? this would be perfect for me if it were possible
I don't know the specifics but I am pretty sure I have seen somebody doing this on their mac studio around in this sub. maybe ask chatgpt to do a websearch and find the relevant threads about it
uhh, yeah tell the guys in localllama just use the cloud /s jk all good -- ya gotta remember for a lot of people on this sub the reason they're here (like me) is because we value privacy more than the average bear
A project I'm working on has to processes copyright protected books that cloud models refuse to process so you are right that I have reasons beyond cost for needing local models.
Every option they say in their post involves the cloud already.
My workflow is conceptually the same as yours. Cloud for all the thinking, establishing as many deterministic processes as possible, and then providing the scaffolding and validation for the dumb local model's execution. That said, I only have 1 Claude Pro sub and 2 ChatGPT Plus subs, so cutting out local models for the execution really isn't an option for me unless I want to start spending a lot more on subs. Even if I cut my local model token consumption by 75% to account for frontier efficiency, the bill would be thousands per month in API costs or multiple Max 20x subs. With the way I have my feedback loop set up for cloud orchestration/validation combined with very structured execution docs / implementation plans with gates unlocked only by frontier models, local models that exist work for me. Even quantized qwen3.6 27b is fine -- not great or perfect -- but usable.
there is a startup called wici.ai, their product might be what you are looking for.
2x rtx pro 6000 and 1pb of local storage. I still basically only use cloud models for everything. I don't use local models for generating anything, local models only for processing (given x get structured answer y) & retrieval. Anything that shouldn't be uploaded that should be private doesn't get put into local because tbh local models are still not up to par, not worth the effort or maintenance. Local is mostly for training some super secret models. tldr, seriously, just use cloud models and save the hassle that will come just for trying to save some tokens.