Diagnosing local AI errors... with cloud AI
I'm using ClaudeCode as my harness with the DeepSeekAPI directly. I use Deepseek-flash set to max and just brute force its stupidity. I now want to work on private data with local AI. I bought a 24gb Macbook Air M4 as a way to test this. I'm a bit shocked how…
I'm using ClaudeCode as my harness with the DeepSeekAPI directly. I use Deepseek-flash set to max and just brute force its stupidity. I now want to work on private data with local AI. I bought a 24gb Macbook Air M4 as a way to test this. I'm a bit shocked how difficult it's been to keep up with developments and also to get things working reliably. I've chosen Qwen3 9B 4bit(MLX)MLX via oMLX to run locally in OpenCode, but I keep running out of Memory (OOM), or the prompt is too big etc, or context gets too large etc. I point Deepseek at the config to try to fix it, but it just can't seem to do it. Then I started doing it myself and I still can't get it to work. It feels a lot like early linux: Should be cool. Can be cool. But often a lot of screwing around. Is that the way it is? I just need to gauge a bit before I invest more time on this. I can try other approaches to working with my data. I quite like the look of CloakPipe for that. But this wouldn't protect against leaking Alpha in a Hedgefund strategy for example. edit to help anyone who stumbled on this: I went with gemma-4-26B-A4B-it-QAT-MLX-4 via unsloth. The MoE caching keeps ram low. Unsloth is more open and handling things well so far. Online, I'm using Kimi K3 for heavy lifting and DS-F for implementing from Kimi. But I suspect these kinds of offline jobs are too heavy for gemma? Things like: go through 100mb of personal markdown notes and tell me something I missed. This isn't a good candidate for PII redaction. Better for local AI. Look at my student and lesson notes in .csv format. Tell me which students are similar to which other students. This might be a good candidate for PII redaction and an cloud APIs. Look at all my WhatsApp messages from customers. Characterise the interactions with each customer. This might be a good candidate for PII redaction.
已收录讨论
With 24GB you can run way better models with a very usable context window. Long story short - YES it is worth the effort and no it isn't that hard. I don't know this oMLX you speak of, but if it isn't working out for you there are other options. The obvious choice is llama.cpp. Stick with it. Learn a bit more. In a week you'll look back at this post and chuckle.
🤷♂️
Thanks. OK! I'll keep at it. Perhaps oMLX is too untested with the models I'm trying. I've switched to Bionic, which is basically Llama.cpp runtime, and I've chosen Gemma 4 E4B to test with, since it's only ~6gb and came up top in the interface under recommended models. Hopefully this will work!
I have no clue what you're running as far as I know there is no Qwen 3 9b. I don't have a Mac so I'm not sure where it's at in the program you're using so you'll have to look for this stuff. I am on windows and download directly from hugging face usually. You probably can do this as well you'd just have to know what folder to drop the files in (I assume anyways) First, what's your context limit set to currently? I assume it's low and running out fast. Second look through where ever you can download models for qwen 3.5 9b and give that a try over what ever you're running now. Alternatively you can probably run gemma 26b A4B its a bigger model but its a MoE so less active parameters run at a time. I'd suggest giving both a try to see how both work on your device.
Forget the tiny models. With 24GB you can easily run Qwen3.6-27B which is the hands-down best model for your hardware and use case. It won't be lightening fast but it will impress you. Grab a Q4_K_M and set up a 128k context window at Q8_0. If you need "way faster but a bit dumber" then Qwen3.6-35B-A3B. I recommend getting to know llama.cpp: https://llama-cpp.com/getting-started/#how-to-install-llama-cpp-on-your-mac
You picked a smaller one now that's essentially developed for mobile type devices its not going to impress you for data work and I'm not sure if it really can do tool calls? You have more VRAM use a smarter one like many if us have suggested already. The ones I mentioned to you when you said qwen3 9b (which isn't a thing) I suggested: Qwen 3.5 9b (3.5 not 3) Gemma 4 26b A4B (26B A4B not E4B) You could probably get away with at Q4 while still having a decent amount of context Qwen 3.6 35b A3B (note this is 3.6 not 3 and not 3.5) There's also Qwen 3.6 27b, you could probably get this running at Q4 too but I assume it will be quite a bit slower on your device than the two above. It will be smarter though, but you'll get much less context than the 35b A3B would give.
You're linking Qwen 3 32B?
Yes. Well when I mean testing, I do just mean testing. I paused for the longest time before going with this compromise. When dividing RAM bandwidth and size for cost Macs come out cheapest. Only stack memory is cheaper. But I didn't want to go all-in with a 128gb Macbook or Studio at the time and decided to go more with the cloud. My idea was to just go small for practicing and see what RAM prices do. Then use API calls for practicing. What I really want to do is just do a small least bit of work stuff at home and offload the stuff that isn't private to the cloud. I don't know how to setup agents for that yet though. I'm only using Claude(or Pi with nono)+Deepseek for coding and just a separate OpenCode for local. The local jobs I'm interested in are: go through 100mb of personal markdown notes and tell me something I missed. This isn't a good candidate for PII redaction. Better for local AI. Look at my student and lesson notes in .csv format. Tell me which students are similar to which other students. This might be a good candidate for PII redaction and an cloud APIs. Look at all my WhatsApp messages from customers. Characterise the interactions with each customer. This might be a good candidate for PII redaction.
Yes. I've been mostly happy with DeepSeek4-Flash. But watch what it does in its thinking sometimes. Notice that it often gets confused with python venv and everything related to that. It fixes itself, yes, but this is just symptomatic and an example that's easy for me to call on. Something like Kimi won't do that and will more likely go straight to one-shot. Of course it's massively more expensive, but it's something to bear in mind. Basically DS4F works great when it can test the results of what it's made. That's what I've seen with coding. I've had masses of fun with that. It's fantastic. I've build lots and lots of stuff with it. That's not going to change. But I have some new workflows I'm developing now to move from making stuff to handling sensitive data. This is where I'm now looking for local AI. Things like: go through 100mb of personal markdown notes and tell me something I missed. This isn't a good candidate for PII redaction. Better for local AI. Look at my student and lesson notes in .csv format. Tell me which students are similar to which other students. This might be a good candidate for PII redaction and an cloud APIs. Look at all my WhatsApp messages from customers. Characterise the interactions with each customer. This might be a good candidate for PII redaction.
> Yes. Well when I mean testing, I do just mean testing. I'll state again: you can't test on that machine, with that amount of system ram. You can't load the model there. You may test on a remote api on openrouter for 5$ a model like https://openrouter.ai/qwen/qwen3.5-27b for coding or https://openrouter.ai/qwen/qwen3.5-35b-a3b for mundane tasks, it has 4x faster prompt process. > go through 100mb of personal markdown notes whooaa man that is a lot of ctx for a local model, actually Estimate: 27.6 Million Tokens I mean you can do that in small pieces yet a SOTA is gonna be 10X faster and has more than 1million ctx while a cheap local may give you "reliably" 132-264"k ctx.
https://huggingface.co/Qwen/Qwen3-32B-MLX-8bit huggingface says this has 9b.
What a crazy statement to say "brute force DS4 Flashes stupidity". Might be opperator error dog. DS4 flash has been our companies daily driver and its absurd how fast and good the model is. Why dont you try opencode and dsv4 flash and litterally tell it what you want accomplished.
I think part of it is poor management of DeepSeek and relying on the brute force with flash and thinking set to high. If it’s checking itself then it may get it running and thing appear to work. However, it won’t test beyond the obvious. As an example, I got DS to write me a market scanner. It all appeared to work. But when I checked the values i noticed what looked like a small error. So then I got my wallet out and pointed Kimi at it. It found 8 bugs just looking at the code without doing any testing, including missing a core goal of the app. Now, I think that core goal was due to context dropping off the back, so in a way that’s my fault. Likewise, not specifying the testing loop is also my fault. But when you compare to Kimi and likewise models that don’t do this, you start to wonder about DeepSeek. Even so, it’s a good lesson to keep context fresh and specify testing loops explicitly.
> I now want to work on private data with local AI. I bought a 24gb Macbook Air M4 as a way to test this. You got the wrong hw bro, no way you can run properly LLM "for work" with just 24GB of total RAM. For the other readers: if you just want to test local models like 27B you can do that with openrouter for 5$. For OP: to run 27B at home "for work" you start with some 32GB of vRAM, as in 2x 16GB gpu, then you have the common system RAM of the PC. Or you can get those expensive macs with 128-256GB and then run some MoE...
edit: Switched to Gemma 4 E4B and had to use Jinja template: {{ bos_token }}{% for message in messages %}{% if message['role'] == 'system' %}<|turn>system\n{{ message['content'] }}<turn|>\n{% elif message['role'] == 'user' %}<|turn>user\n{{ message['content'] }}<turn|>\n{% elif message['role'] == 'assistant' %}<|turn>model\n{{ message['content'] }}<turn|>\n{% endif %}{% endfor %}{% if add_generation_prompt %}<|turn>model\n{% endif %} That said, still interested in comment from people as to how often they're playing with configs?
What do you mean by qwen 3 9b? As far as I see on hugging face there is no qwen 3 9b? There's 8b listed https://huggingface.co/collections/Qwen/qwen3 Secondly if you actually mean qwen 3 that's like a year and a half old there's better options now. Qwen 3.5 has a 9b do you mean that?