I'm kinda tired of obsession for one-shot tests in coding, there are good tests for multi-step debugging with analyzing output/images/videos?

LLM inference is a statistical process, which makes one-shot tests even worse for evaluating a model.

I think it depends on whether or not you're implementing a solved problem, or at least something that can easily be decomposed into a group of solved problems. For most applications, I think, this is the case, and so as long as you know software, can read the code and know when it's making mistakes, you're going to get good output. By contrast, attempting something that hasn't already been documented many times before (or something that can't be trivially broken down into tasks that are) will require more research and prompting to get a decent result, and if you don't know the domain in which you're attempting something, it's not going to end well. PS I too use it both at work and at home, and for the well solved problems I'm working on, Opus has been a miracle and saved me so much time.

The problem with AI is the lack of context and vague instructions. Guide the AI like you would guide a junior dev. Give it a detailed task, with clear scope and objective specification. Tell all steps it must take, which commands it should use, which tools it should call. Give it context, a lot of it, and tell it to follow the code style and architecture. And the more important part, do not dump a task in the chat and expect it to work by itself, the result will be pure shit, you must use an orchestration mode to orchestrate the tasks. Usually it will break the big task in smaller ones and spawn a subagent to do it. It has an enormous advantage, you avoid context poisoning, keeping the model focused. What I'm doing for new projects is give it a boilerplate with everything set up and examples of the code style and architecture. So it will just follow it. Since I hate closed models and how expensive they are, for my personal stuff I only use Qwen 3.6 35B A3B Q2. The code it generates is the same of what I would expect from a junior, in fact, it is better than I would expect. Of course, it is not perfect and sometimes I have to tell it to change the code or optimize something, but this is what I do at work anyways with the code reviews. I've finished my agent this week and I've been doing some tweaks, the codebase became large but everything is perfectly maintanable. Of course there is some stuff to refactor, but the result is better that I was expecting. 90% done by Qwen, 10% Gemini/ChatGPT. PS: in this 100% I'm not including myself. I place myself as the architect, I do the thinking and the AI the typing, so that's why I think the final project is good. I do not let the AI do what it wants.

This is something I've discovered the more I learn about code generation. There are so many techniques your harness can employ to save context and eke out better performance.

Can you share some? I finally caved in and started doing this agentic coding thing, was quite surprised by what opencode out of the box did.

I would make at least a one out of three test. Even when a model would have 99%, bad luck, and you hit with your one-shot test that 1% where it generates crap. On top, that different LLMs also often need different sort of prompt. Not every prompt works with every LLM. That said, one out of three would be an improvement, but no real solution.

please do, if i never see another threejs/html used to show off an llm it would make me quite happy.

A lot of these tests doesn’t take I to account the harness either. For that, they’re mostly useless. To be fair, models work best in harnesses that are tuned for their quirks, so these tests aren’t fair anyway. One thing they are useful for though is seeing progress in model lineage.

I use them at work (opus 5), and vibecoded a few apps for my homelab, it's awesome. (I used opus 4.6 and gemini 3.5 with their respective harnesses, planning to test the openai stuff now) They do make mistakes, but in general it's pretty easy to fix by just telling them what the problem is. I always wonder how complex the stuff people are making for it to completely fall apart, it hasn't been my experience. My use cases have admittedly been rather simple (webapp + postgres database, and k8s config).

It's like herding cats. For work they can be great especially for code review. Sure half the stuff they find is wrong but among the list there are some useful points. They are also great for codebase exploration. For working on large tasks I do find them more trouble than they are worth. If you are actually reviewing their work and pointing them in the right direction then you often dont save as much time vs doing the complex parts and just having the agent fill in the gaps with a good spec. For personal use I've built a few things, things that either didnt exist in a way that worked for me or didnt have the functionality I wanted. For example getting xcom 1 to render at 4K in a linux desktop using wayland is a pain, I had to ask qwen to write a hook to fake the screen resolution AND force the game to recognize it by patching the list of resolutions. Small stuff like that is great, I would had never written it myself but I was happy to throw qwen at the problem until I had a better resolution and antialiasing. Something similar with waydroid, it lacks multi touch emulation for things such as pinch in and out using keyboard + mouse, I can live with that limitation but since I already have a local setup I let an agent go at it until I got something kind of, almost, functional. Enough for my needs. I also built a simple audiobookshelf client with better gnome integration that what was available, if I had to write it myself I would had just used the web client but by using an LLM I ended up with something marginally better for a few cents of electricity.

I'm going to fine-tune a weak model on ThreeJS one-shots and ruin this community.

I think, since I wanted to ask that question anyway, I will just hijack this post and ask a related question: Did anyone actually make something work with LLMs? I have been using them for years now, closed sourced, open sourced, but to this day anything one makes ends up being unmaintainable. I don't even touch agents because the spec drift is so bad. "Productivity" without the "product" part. Am I doing it wrong or are LLMs still basically in the "it is gonna be good soon" part?

In what way are they unmaintainable?

Slopbench probably closest to what you want.

I have built and replaced several large production legacy systems with agents. I have also used agents to automated computer vision research. However I don't think I'd be successful at this without already having existing expertise in software and research.

If the use cases most people are posting don’t match yours, you can always run tests in line with your needs and post them here.

One-shot scores mostly measure sampling luck on that specific run, which is why they jump around between releases. For me the interesting failure shows up on the third or fourth turn, after the model has been wrong once and has to read its own output. Something that looks sharp on a single generation often degrades there, because an early bad guess quietly shapes every fix after it. Same weights under a different harness give very different numbers, so a lot of what gets called model quality is scaffolding doing the reasoning. The closest thing to a usable test Ive run is a small broken repo, six failing tests, with the rule that the tests themselves are off limits. A fair number get around that by weakening the code path until the assertion stops mattering. Nobody reports that.

I think you're confused. "one-shot" does not mean the model never iterate on the results of tool calls. It means the human operator give it one prompt. For modern models with interleaved function calls, they can literally run tests by themselves autonomously (as long as the software harness around it enabled it) and iterate on the results. Be warned that this is not consistently supported for local models (ie. the company might support it in their own software but it's not implemented in llama.cpp). SWE-bench is literally for resolving GitHub issues. And there are multiple versions. The reason for not doing 2 prompts for interactive debugging is because the 2nd prompt depends on the outcome after the first prompt, and there is no explicit standard as to how the human operator is supposed to generate that prompt, and you don't want human operator's skill to be part of the benchmark. What if the first program it makes has no bugs? Or the bugs are completely different between models? How are you supposed to compare? However, for changing requirements (ie. the software specification change after implementation) there are things like SlopCodeBench. This is doable because the prompts do not depends on the previous output (you always ask for feature A first then ask it to add feature B regardless of what the first one produce).

I would say design a series of prompts that each ask for some user story to be solved. Take pains to only talk in terms of the problem and the desired outcome you want, as a user. The series of prompts is about building o to whatever it builds, and to really test it, you have to intentionally throw in some common zingers, like: completely changing ones mind about some aspect of the problem, massive scope creep, a real lateral move in the user needs like suddenly pulling in a very different domain into the problem, and just normal feature additions. Once you have a good series of such prompts, you could start to measure the AIs success, churn it goes through, bugs it produces, readability of the code, usability of the app, etc. I think it would be fun to do. A couple years ago I had done some of this, building a calculator project through such a series of prompts, but I was experimenting more with what ancillary context influenced success the most in terms of software design instructuons (ie I compared the results when I told the AI to use good clean OO code vs everything-in-one-long method procedural code).

The reason local is so focused on one-shots? Because its harness performance sucks!

I find one shot tests stupid because i actually want an agent that seeks out gaps in specifications and asks for clarification. Even if it proposes several options as a resolution for me to pick from. To assume makes an ass of u and me.

One shot "vibe prompts" are absolutely critical because they show the maximum capability of the model