Can this be done with a single (budget) card?
I want to use a local model to help write some status reports for my business, and this could contain PHI. Obviously this can't leave our infrastructure. The cloud based records system we currently use offers something like this, but it is stupid expensive…
I want to use a local model to help write some status reports for my business, and this could contain PHI. Obviously this can't leave our infrastructure. The cloud based records system we currently use offers something like this, but it is stupid expensive and isn't very good. I have been testing it with hypothetical data now, and with a decent system prompt, it works much better. My question is this: What hardware would be needed to accomplish the following with a local model: 1.) Take 20K of context (Report) and break it down into a structured format, as well as say, Gemini Flash 3. This can be done at night, and doesn't have to be fast. 2.) Retrieve the structured data and add maybe 2K worth of additional context for making an update The current site I built to help with this could potentially start prefilling about 30 seconds before I'd be ready to send the request and additional context. Basically I would select a customer and it would start loading the previous context which hopefully would be much less than 20K, and I would then spend maybe an extra 30 seconds adding my updates. Right now I have been testing this with test data and obviously Flash 3 is returning results almost instantly. The problem is that in a production environment, this really needs to be fairly quick. If an employee has to spend 2 minutes waiting for the report its going to cause issues. Even 30 seconds I think is going to be an issue. Although if it starts streaming the report in 10ish seconds that would be fine. Also, it has to be decent. Flash 3 right now is giving acceptable results. I currently have no hardware other than our Unraid server: MSI PRO Z790-P WIFI (MS-7E06), 64 GB DDR5, Intel i5-12600K I have been tinkering with some local models, but obviously anything sizeable really doesn't work for chat. I am also assuming that if it can meet the report update requirements, using the same hardware/model would be decent at some basic other agentic tasks. I use Codex now for coding, and would likely continue to do so. My apologies if this seems like a really obvious question, I just have noticed that much of the hardware discussed on this sub is VERY prosumer. I am looking to spend < $1500 now on a card, and maybe add in the future. Not sure if that is feasible. Any thoughts would be hugely appreciated.
已收录讨论
Try hosted Gemma 4 12b / qwen 3.5 9b. If it works for you you can buy any 16gb vram card and it will give you 60+ tps If by structured data you mean json then you would like to have some format repair tool because these models can skip a comma and break output
So the reports are fairly structured Into sections. Probably just would store them in postgres as markdown.
Gemini Flash 3
Can't risk PHI leakage. Most of the time just 1 user. Could be 2 but rarely.
Is this a business you own, or is this your employer?
Are you talking about one of the older V100 cards?
My business.
Rtx 5060ti 16gb can be had for $500 after tax on sale. 2-3x of them should fit your budget. Won't be super fast, but enough to run qwen 3.6 27b, or 35b a3b at decent quant
Kernels are better optimized on cuda. NVIDIA also secured high bandwidth memory well ahead of others. AMD and Intel simply don’t have the high bandwidth GDDR RAM. AMD is likely in a better position than Intel. Intel does not have enough resource to support their GPU. Their own API cannot even run qwen3.5 when B70 released. The first impression was just so bad. I had to use Vulkan to run model on B70 on day 1. If you have more than 1 GPUs it is even more complicated.
Currently looking at the arc pro B70, but it seems to have a lot of chatter around support and such.
If by structured data you mean json then you would like to have some format repair tool because these models can skip a comma and break output Or set an output grammar?
Is CUDA really triple as fast or is that an exaggeration? I've found comparable AMD cards running Vulkan to be pretty quick
Gemini flash or gemma?
This would be a great way to do pre-testing before buying hardware. If it was me, I would install Hermes Agent on a local box (doesn't need to be powerful), then connect it to a smaller (but capable) hosted model as suggested, and then simply start a session and explain what you need to achieve. I think you might be surprised what you can accomplish this way. Based on these results, you may need to choose a more capable model (like Qwen3.6-27B or 35B-A3B) but once you find out what works you'll know what hardware to buy. Your budget is plenty, IMO. I know for a fact I could pop open a Hermes Agent session connected to my local Qwen3.6-27B and work with it to build a plan and then implement this type of thing in an afternoon or so.
What kind of load are we looking at, here? ie how many concurrent users? Any reason why you can't do it programatically? feed a giant context into gemini, shoot out a json response and manage the rest in code?
If you need it done overnight you can do it easily with any good SSD running a model way above your vram+ram at a slow ass speed, I’m able to run the latest deepseek at 1TPS on my gaming pc
You might want to try running the Gemma-4-E2B model on CPU and see how it performs for you, or even Qwen3.5-4B or -2B.
get a 32GB GPU, preferably nvidia. run qwen3.6-35b-a3b or qwen3.6b-27b. it should work fine. if budget is hard limit a Intel or AMD 32GB GPU is the only option, but they run 1/3 to 1/2 the speed of nvidia. local running these models will be slower than cloud API call sure, until you can spend >$4000