How would you set up a 96GB M3 Ultra as a small shared local LLM server?
We already have an M3 Ultra with 96GB and want to use it for internal document jobs submitted by a small team. This would not be 5-10 people generating at the same time. Think of a queue where someone submits a document, waits for it to run, and reviews the…
We already have an M3 Ultra with 96GB and want to use it for internal document jobs submitted by a small team. This would not be 5-10 people generating at the same time. Think of a queue where someone submits a document, waits for it to run, and reviews the output. We want to keep confidential document work local while still using cloud models for deep research and harder jobs. Nothing would be sent, approved, or added to our records without a person reviewing it. The first workflow is financial statements. A scanned or native PDF goes in, and a standard Excel workbook comes out with each value tied to a source page. The basic process would be: GLM-OCR or native parsing reads and extracts the full document A local model reviews the extracted statement and returns a standard structure with source-page references Python applies approved account mappings and checks totals, monthly amounts against YTD, and whether the statement balances The model helps review anything that cannot be mapped or interpreted confidently Anything that does not tie goes to a person for review If this works, we could use the same setup for document classification, CRM cleanup, and first drafts of reports or presentations. I have tested LM Studio, Open WebUI, MLX/oMLX, smaller models, and Ternary-Bonsai 27B. They are fine for one person, but I have not built the shared system yet. For anyone running something similar: Which model and Mac serving setup has been reliable for structured JSON and tool calls? If you moved from a personal setup to a small shared service, what did you use for the job queue, user access, logging, and recovery when something failed? Did running locally actually reduce cloud spending, or was the main benefit keeping the data private? I care more about predictable results and easy recovery than benchmark scores. I am also fine hearing that the Mac is useful for testing but not worth turning into a shared service. Edit: To clarify, the model will review the full extracted document. “Unresolved rows” only refers to the later account-mapping step.
Collected discussion
Good catch. I described that part poorly. The full document still needs to be extracted and reviewed. “Unresolved rows” only refers to the account-mapping step after extraction, not the full workflow. Some jobs may also require comparing several monthly statements together, so context size still matters. Your 35B recommendation and the reasoning-off results are very helpful.
For a shared system we are running openwebui. Cool thing is, you can connect to different machines. For example on the Mac in lmstudio you can set up a Webserver a the. Connect that to openwebui and there you can select the server you want.
I cant keep 35b@q6 from making up numbers, using truncated statements, misinterpreting numbers and not verifying. 27b@q6 does a lot better but unbearably slow. Is there that much difference between 35ba3b@q6 vs q8?
Not for that number, no. The 94 to 30 is IFEval, so it's instruction following rather than JSON validity. A grammar would keep the output parseable, it wouldn't make the model follow the instructions inside it. What made me stop trusting the shape on its own showed up in the same runs. GSM8K gets scored two ways and with thinking on the forgiving extractor came in lower than the strict one, 52 vs 78 at one quant level. Only way that happens is it pulled a number out of the reasoning trace instead of the final answer. Valid shape, wrong value in the field. On a financial statement job that's the failure that actually costs you, and a grammar won't catch it.
That makes sense. Would you have the model return source row or cell IDs plus the proposed account mapping, then let Python retrieve the actual amounts from the OCR output? That seems like the cleanest way to prevent invented numbers from entering the workbook.
Are you using Open WebUI only to select between inference servers, or does your setup also handle queued document jobs? If you have batch jobs, what handles the queue and user separation?
Good to hear. Are you using it with an actual team? If so, what Mac, model, and concurrency settings are working, and do you have a separate queue in front of it for retries and failed jobs?
Are you not using grammars?
On financial statements, the "making up numbers" thing BrodyFriend hit won't go away with a bigger model. The fix is to not let the model emit numbers at all. Use it for classification and account mapping only, which line item belongs to which account, and pull the actual dollar figures straight out of the OCR output in Python. The model proposes the mapping, Python owns the numbers, so a misread or invented figure can't make it into the workbook. And since you're queued rather than latency bound, running each job twice and kicking any disagreeing row to a human is cheap insurance.
One thing that will bite you on the structured JSON part: turn reasoning mode off. I ran a 122B on a 256GB M3 Ultra with thinking forced on vs forced off, same session, same 2048 token generation budget, across four quant levels. At the level I have the full table for, instruction following went from 94% to 30% with thinking on, and the 7 item practical suite landed on exactly 4 of 7 at every level with it on. Decode speed was basically identical either way, so it costs accuracy and buys nothing. For 96GB I would try Qwen3.6-35B-A3B at 8bit before reaching for anything bigger. It did 108 tok/s here and leaves real headroom in 96GB, and since your model only sees unresolved rows rather than whole documents you are not paying for a big context anyway.
omlx is pretty good for a team because of batch processing