How do you test that an AI agent won't do something catastrophic?

You fundamentally cannot prevent an LLM from being tricked into doing something, unless you either never allow it (even indirect access to) untrusted input or never give it access to important things. Those "guardrails", assuming you mean a lack of access to resources, are the most important thing you can keep to protect yourself from LLM hallucinations.

increasing access as we get confidence And that's where you'll get bit. LLMs are non-deterministic, ie random. Enough run time and they'll eventually hallucinate or go insane. With IAC I see no good reason to give an LLM agent anything more than read permissions. You can have the agent monitoring and generate an alert and suggested fix through your IAC, but a human should verify and push the change.

Seconded - the agents need their own credentials to limit their roles and differentiate between user and agent action. We have too much evidence that guardrails within the LLM environments are not fully functional controls, so we have to enact controls where they always do what they're configured to do. I think there's a barrier in most places that they aren't prepared to issue and manage all the additional credentials needed to have agents act under their own access rather than access inherited from the user.

https://owasp.org/www-project-ai-testing-guide/ You can’t stop an LLM going rogue anymore than you can stop a human. Think of them as drunk interns. Now - where you build the guardrails is around putting them in a sandbox. What data do they have access to What tools do they access to (as in CLI tools, local binaries, browsers) What approval is needed before they are allowed to execute. Expecting an agent to follow instructions is hopium. You need to control it with the same primitives we place on normal users - identity and access, network security, logging and monitoring, secrets management, etc etc. But yeah l, you test that the agent can’t just gobble up all your confidential docs and exfiltrates them out the network if you inject a prompt. Red teaming these systems is a thing. https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/red-teaming https://www.offsec.com/learning/paths/llm-red-teaming/

Yes, we extensively red team our LLMs. You'd have to be insane not to if you're giving them that level of access.

The same way you make sure a human can't do something catastrophic. Least privileged access, monitoring, clear instructions, etc.

you don't. you prevent it by not granting them permissions that make a catastrophy possible.

That's the neat part. You don't.

Have a little bit faith 😇

Evals and red-teaming help, but you can't test a probabilistic system to the point where it's safe to hand it destructive actions on its own. What held up for us was making the irreversible stuff, file writes and shell commands, require a human approval per call, plus logging every tool call so you can replay a run and red-team the approval boundary rather than the model. Most of the safety ends up living in that gap between the agent deciding to act and the action actually landing, not in the model itself.

We're experimenting with agents internally, and red teaming has become part of the workflow. Not because we expect attackers immediately, but because users will always find prompts you never thought about. The failures we've found from internal testing were honestly more surprising than the model benchmarks.

Jesus take the wheel homie. And backups

Don’t allow it access to your infrastructure. Sandbox it on a infrastructure that there is zero internet access. (Read OpenAI agent incident and Anthropic Claud incident) additionally make sue your BCP, DRP, and Incident response plans for AI incidents.

you dont, welcome to roulette.

Never give an AI agent anything more than read access to production. Always have an intermediary. The AI can do a phenomenal job at coding or diagnostic work. But the second it has production write access, all bets are off. You literally cannot trust it. Lock your ssh key with a password. I worked in AI until recently. And while I use it, I flately refuse to give it production access. My rule: read only access or an intermediary system or human in the loop.

Most teams red team their agents before and after launch. They list all the tools the agent can use, then run automated and human tests to try to make it do bad things and fix the issues before going live

Your paranoia is legit, a lot of people are freaking out and over provisioning permission for agents because “we might fall behind”. You should be very careful with them right now and follow tried and true cybersecurity practices. The people saying otherwise are stupid or trying to sell you something.

This is a great question, and I think one the industry is still really figuring out. The tools we have now are to treat them like a rogue employee or insider threat. This is an autonomous agent with access. How do you deal with insider threat? RBAC Zero Trust Segmentation AI Agents get trickier as they will actively and expertly work around the security constraints to get a task done. They are not smart; they can't hold the context of what it should do vs getting its job done. When a human runs into a constraint like not having access or a bit of software, they may seek permission; the AI agent will likely try to work around it, find a way to download, or go around the permissions.

Testing for every possible catastrophic outcome, resulting from a model output seems like a losing prospect. Instead, we have implemented our tool use such that the action is gated and deterministic and the generation results in the execution of one of those actions with/without known flags/arguments; tool use should be architected into bucketed actions and the probabilistic part is merely to chose which bucket. You can also have verification steps, allowlists and blocklists that are context dependent in the pipeline or per user/agent/action/other criteria. Beyond that, for extremely sensitive operations such as rm or shutdown for example if you're asking this type of question maybe just ease into those at a future date, well after figuring out low level agentic coding, verifications and less just "generate the bash" if that's indeed what you're doing now.