If your data can't legally go to ChatGPT, what are you doing instead?
Question for anyone whose work involves information that can't leave your systems. Client files, patient records, case files, contracts, anything under an NDA or a regulator. What do you actually do about AI right now? My guess is one of three things:…
Question for anyone whose work involves information that can't leave your systems. Client files, patient records, case files, contracts, anything under an NDA or a regulator. What do you actually do about AI right now? My guess is one of three things: nothing, or somebody's quietly pasting things into ChatGPT and not mentioning it, or you got quoted an enterprise number that made it not worth it. I'm a developer and I've been building setups where the documents stay on your own hardware and a model only produces the final output, so nothing sensitive leaves. Before I go further I want to know whether that's a real problem for people or a problem I've invented. So: is data location actually why you're not using this stuff, or is that just the reason that's easiest to say out loud? What would you be doing with it if the data question disappeared tomorrow?.
Collected discussion
There are some (few) european AI providers, and there is also local models and self-hosting if you want to be clean. Many companies just use "sovereign" instances by Google / Microsoft for example (MS copilot = ChatGPT underlying) -- they promise your data stays in europe but US Cloud ACT of course says no to that but still it's generally tolerated. Basically your best options for privacy are chinese open-weight models like Kimi, running on servers you control, also newest models from Mistral seem to be catching up if you don't want to go self-host stuff and just be "europe" hosted. Hope this helps!
Yeah, it does. Do you think a promise of “it will not leave your system” is good enough for most companies? Do you know/heard about companies looking for service providers which help integrate those local running LLMs together with the system which help to operate it easily and get consistent results?
Do you know someone who did that? What is their experience moving away from proprietary AI models? I know a couple of dev who do that, but they are not really businesses with data. The most obvious complain is that "AI gets dumber", because what was sitting in system prompt in vendor solution right now is gone.
I haven't worked with any I can recommend, most are very pricey I've seen. A simple work-around if you want fast time to market, is just let them specify their own models. Many companies have settled on their own AI, so just allow them to configure it. Mostly it's just the URL + API keys and very similar structure of sending prompts to any AI so you'll be able to support most common models easily. Pitch it as "any controls you set on your internal AI now instantly apply" or you can also just build an MCP and THEY can make the AI use YOU instead of the other way around, figure out what best fits your product. ETA: also saves you the trouble of billing per token... And probably you'll want to offer multi-model support & model selection to some extent, but this is the way we decided to go in our own built-in AI. We offer some european models + claude + open AI, or bring your own, and I believe market will mostly use "bring your own" after convos we've had.
Yes, I know several businesses who decided that they could not trust AI with their full customer data. and who were using so many tokens (because they had a huge database). They installed Ollama on their servers. And last time I checked with them, it was working very well.
Most are selfhosting a AI in their local servers in that case and using that 🙂
I use local models. They might not be cutting edge enough for very difficult tasks, but they are good enough for many simpler tasks. They are also several order of magnitude cheaper and you can migrate to newer models in your own terms.