ORIGINAL REDDIT POST
How do you do SAST on Clojure? Can’t find any tool
We have several repos, all Clojure. I need SAST coverage for secure code review, but nothing mainstream (Semgrep, CodeQL) supports Clojure properly. Even clj-holmes the one Clojure-specific tool isn’t a real solution; the maintainer told me himself it’s not…
We have several repos, all Clojure. I need SAST coverage for secure code review, but nothing mainstream (Semgrep, CodeQL) supports Clojure properly. Even clj-holmes the one Clojure-specific tool isn’t a real solution; the maintainer told me himself it’s not in good shape. I’m leaning toward an AI agent for code review, since LLMs are actually decent at reading Clojure. Has anyone gone the AI route for SAST on a niche language? How well did it hold up?
Collected discussion
Commenting to follow. I have no direct experience on this. A bunch of linkedin posts and YouTube videos tell me using AI agents to test is workable in virtually any language but I don't know enough to tell where real world ends and click bait begins on this one.
I have compared deterministic SAST tooling with using LLMs across several languages. Both have their own advantages. Traditional tooling does best for the bulk of the scanning and it’s consistent. LLMs can find a few edge cases where context really matters. Both will have false positive results, but LLMs are much more “noisy”. Use both together.
This comment was deleted.
tbf, a lot of “proper” SAST tools are already kinda useless on niche stacks, they just tick a compliance checkbox. What you've just said is one of the most insanely idiotic things I have ever heard. At no point in your rambling, incoherent response were you even close to anything that could be considered a rational thought. Everyone in this room is now dumber for having listened to it. I award you no points, and may God have mercy on your soul.
That's what I wrote in my comment to OP -- use both together. The problem for OP is s/he doesn't have, and will never have, "both".
How do you do SAST on Clojure? You kinda don't. You mentioned Semgrep. Semgrep allows one to create their own rules. But even though Clojure is Java-like, that doesn't apply to syntax. To use it for Clojure, someone would need to build: A complete Clojure parser An AST representation Pattern matching support Metavariable extraction Taint analysis support While I appreciate you trying to use GenAI as an alternative and there is *some* overlap, using a LLM for SAST is kinda like trying to open your front door with a can opener. Here are major problems you're going to run into with substituting GenAI for SAST: Traditional SAST is deterministic where LLMs are probabilistic SAST parses the entire program; LLMs reads code like a developer SAST tracks data flow; LLMs reasons about intent and patterns What a LLM will give you over SAST is: Analyzing custom security APIs Finding business logic flaws Explaining why something is risky That's why, if you dig deeper into all the major SAST players, they're NOT replacing their traditional SAST engines. Instead, they are combining the two approaches rather than replacing one with the other. I wish I had a better answer for you. But I think you're SOL. And given the deterministic versus probabilistic angle, your org is exposing itself to considerable regulatory and certification (SOC 2, ISO 27001) risk by not being able to do proper SAST.
but I don't know enough to tell where real world ends and click bait begins There's a lot of GenAI Emperor's New Clothes out there right now in the influencer space. You're right to be skeptical. The reality is there is some overlap in capability between SAST and LLM. But for the most part, the architectures of each are NOT compatible. Anecdotally, we use some GenAI to help us walk through expolitation paths of SAST findings. Which the SAST tool cannot do. But the LLM would have never found the vulnerability in the first place. Short of someone on YouTube actually demo-ing that, they're lying, pushing FUD, or are outright full of shit if they say LLMs can do SAST otherwise.
If you put in an issue, along with some examples of good test repos, what you need to be able to detect, I can add that language to my sast in a day or two. https://github.com/squid-protocol/gitgalaxy Despite common dogma, you don't need an ast for 95% sast.
There are challenges with throwing LLMs at code for scanning; result consistency between runs, performance, and cost are the big ones. Several AST vendors are blending their traditional rule-based SAST engines with AI/ML to improve detection accuracy and add coverage for languages/frameworks historically not covered by their SAST engines. I know that Checkmarx, Black Duck, and Snyk are taking this approach going forward. These vendors are using more complex hybrid strategies to manage result consistency, performance, and cost.