[OS] Free, Open Source, AI-driven user testing for iOS, macOS, and the web
Hey MacApps! I'm Alan, I'm a developer that has been building Mac, iOS and web applications for over 30 years. Recently, I started a few Open Source projects and ran into the age old problem of user testing.... Problem: Developers have always been plagued by a problem: how do you user test when your application isn't ready for distribution yet? How do you maintain consistent user testing with different user personas after your application is released? This is the same issue regardless of platform, Mac, iOS or Web - so I built Harness. Not a UI test. A user test. Harness is a native macOS dev tool that drives your iOS Simulator, your macOS app, or a web app the way a real user would. You write a goal in plain English and pick a persona; an LLM agent reads each screen, decides what to do, and pursues the goal — tapping, typing, scrolling, navigating. When something is confusing, ambiguous, or a dead-end, the agent flags it. Every run produces a replayable timeline of screens, actions, and friction events you can scrub through later. No accessibility identifiers, no source-code access, no pre-written plan — the agent reasons from what's on screen. Comparison: Aside from UserTesting.com, I am unfamiliar with any others to compare this to, but if you know of any please post them in the comments. Pricing: Free, Open Source, Universal Binaries for each Release I hope you like it. Feel free to review the code, play with the application, and let me know what features or functionality you might want to add to make it better. For more information, please check out: https://awizemann.github.io/harness/ Or grab the latest release, fully signed universal binary: https://github.com/awizemann/harness/releases/tag/v0.2.0
Collected discussion
As a non-dev, the description was a little confusing, but apparently it is a free app that uses AI to pretend to be a real person clicking through an app, trying to complete a task described in plain English. If anything is confusing or broken, it flags it and records the whole session to replay later.
That is correct! I should have user tested the post.
Just a quick update, there was a small issue with a hard-coded path in the 0.2.0 build. 0.2.1 is now out and can be downloaded here: https://github.com/awizemann/harness/releases/tag/v0.2.1
oh nice, asked basically this question on here last month (link) and ended up just writing janky AppleScript smoke tests because I didn’t trust computer-use on native AppKit. how is the agent doing on SwiftUI vs web — does it actually understand native controls or is it mostly pattern-matching from screenshots?
It uses visual models off of personas to behave like real people, leveraging the llm’s that provide vision. The surface doesn’t matter, it navigates what it sees, not through code.
Cool concept
Thank you!
woah!! this is insane. I'm literally close to releasing a beta of my app and documented everything I've done over the last few months in terms of feature implementation, so i can run through it today and write software tests on them. This is absolutely what I needed and amazing that you're open sourcing this. I'll test this out in the next few days once I ran through my manual test cases :) Only thing I'm worried about: My app is a file backup system. Meaning the testing does include copying and moving large files across drives. Would be a bit scared of giving the agent access to select which files etc. and potentially restoring the original state of things, which means deleting files from my drive etc.
Interesting use case! Treat the agent like a person, and the goals to not do anything destructive. Let me know how it goes!
100 Stars and Growing! Thank you everyone for helping reach this milestone (in less than 24 hours to boot)! Once a project hits 100 stars, it falls into a new status for me: idea validated, maintain & improve. I should be done with the next update tomorrow adding authentication, so you can test gated/authentication needed applications (Mac, iOS, and Web). I have lots of ideas for this application, and I have received several great ones already from Reddit and GitHub. Let's go!
Version 0.3.0 is out! This release ads some amazing features! Credentials, Set-of-Mark targeting, and a web mirror that fills the column: 0.2 cracked the model path open across three providers. 0.3 cracks the agent's targeting and authentication open: pre-stage credentials against any Application, click form elements by numbered badge instead of pixel, and watch the live mirror render the full middle pane instead of an iPad-shaped device bezel. Per-Application credential storage Pre-stage one or more (label, username, password) triples against an Application from its detail panel. Pick a credential at run time on Compose Run; the agent gets a new tool — fill_credential(field: "username" | "password") — on iOS, macOS, and web.
Looks interesting. How can I export the testing results? Do it has some third party integrations?
That’s the next step! I’m also going to connect to GitHub for UX issue testing. What services do you use that you’d like it to connect to?
Super interesting! Can you integrate this into a test suite and use it in a CI/CD pipeline?
Right now, no, but I am contemplating having the engine headless that can take queued actions (the application already allows you to queue actions for robust testing)... I need to do more testing first, and add user authentication (that should be done shortly).
this is cool. does screen reading costs a lot tokens?
I’ve done many under $0.15 using lower powered models, but the process uses screenshots and vision models, so the token use is higher. There are token limits set in the application based on the model selected.
I don’t get it, what’s wrong with boring repeatable reliable free UI tests? I really don’t see how a llm could figure out what to test and tell me if the result is what’s expected.
Give it a shot! I didn’t either, but this started as an experiment to see. My first test found an issue on my website that I never found, because I told it to find strange paths to get to content.
does it have user personas built in? Did not fully understand how the user testing would work
Yes, there are pre-built ones and you can create your own.
Kind of like playwright then?
No, this is vision based, playwright is tag and code based. Plus this is a gui that you run yourself, not a ci/cd pipeline product. Not yet anyway. I’d say they do similar things in very different ways, using very different means.
Is it bring-your-own-api-key or work with OpenAI/Claude subscriptions?
Bring your own. The next update will allow your own endpoints (so if you have the ability to run local vision models, you can).