Bubo: AI code-reviewer that learns from review comments
I'm getting tired of AI code reviewers flooding PRs with noise and repeat findings, not to say false positives. And then learning nothing when a developer explains why a finding is wrong! For me there's a lot going on in code beyond the tech stack: tribal and…
I'm getting tired of AI code reviewers flooding PRs with noise and repeat findings, not to say false positives. And then learning nothing when a developer explains why a finding is wrong! For me there's a lot going on in code beyond the tech stack: tribal and institutional knowledge, business logic that evolved over time, ripple effects across upstream and downstream systems etc. I started those requirements and added simple setup, evidence-backed findings/ LGTM, and most importantly learning from human comments on those findings, so it gets better tuned over time to each repo. I ran a smaller benchmark, not definitive by any means, four open source reviewers on 20 pinned PRs/MRs. Bubo 20/20 7/8 27 findings 0% noise ai-codereviewer 19/20 6/8 118 findings 20% noise ChatGPT-CodeReview 20/20 5/8 75 findings 11% noise Qodo/PR-Agent 19/20 2/8 7 findings not scored Alibaba open-code-review partial run, 4/20 All on GPT-5.5 except Qodo was on 40 (didn't support 5.5). Take it as directional results. I'll be doing more comps in coming days and upload benchmark to the repo. I went with polling (zero steup on repo side) - wasn't trying to change the repo CI/CD's rather have a tool seamlessly work. Roadmap item: I want to route changes to pluggable subject-matter specialists (Skills) instead of one general reviewer, i.e. industry SMEs or just an "Expert Python Guy". Bubo is running in production in two places, a large data processing/ETL codebase and a fintech crypto stack. Would appreciate feedback, particularly on whether the learning from comments part actually holds up. https://github.com/mountainowl/bubo
已收录讨论
benchmark numbers look decent but 20 PRs is tiny sample size. the learning from comments thing is clever though most reviewers just ignore why a finding was dismissed how do you handle when different devs give conflicting feedback on same type of issue
In coming weeks I'm planning on a 1k sample benchmark, still in process of sifting thru. On the conflict resolution, that is a good question btw, agreed upon resolution wins rn. But I know that's not always the right way, defferd to later etc. There are a few more such scenarios - a full solve will need integration to ticketing system to build better memory. Open to feedback and suggestions.