Get an AI code review in 10 seconds (opens in new tab)

Here are the commits, the tasks were not trivial

https://github.com/hofstadter-io/hof/commits/_next/

Social posts and pretty pictures as I work on my custom copilot replacement

https://bsky.app/profile/verdverm.com

But they're fantastic at spotting dumb mistakes or low-hanging fruit for improvements!

And having the AI spot those for you first means you don't waste your team's valuable reviewing time on the simple stuff that you could have caught early.

AI code review does not replace human review. But AI reviewers will often notice little things that a human may miss. Sometimes the things they flag are false positives, but it's still worth checking in on them. If even one logical error or edge case gets caught by an AI reviewer that would've otherwise made it to production with just human review, it's a win.

Some AI reviewers will also factor in context of related files not visible in the diff. Humans can do this, but it's time consuming, and many don't.

AI reviews are also a great place to put "lint" like rules that would be complicated to express in standard linting tools like Eslint.

We currently run 3-4 AI reviewers on our PRs. The biggest problem I run into is outdated knowledge. We've had AI reviewers leave comments based on limitations of DynamoDB or whatever that haven't been true for the last year or two. And of course it feels tedious when 3 bots all leave similar comments on the same line, but even that is useful as reinforcement of a signal.

In order to be time efficient, human review should focus on the 'what' rather than the 'how' in most cases.

AI is a mixed bag. I'm the type of person who is compelled to have a deep understanding of the code they write. Writing my own code vs fixing AI-generated code is a wash timewise, and the AI generated code is so limited (assuming you pared down the uselessly elaborate code and fixed all the critical runtime bugs) as to restrict further iterations. And I'm talking about uploading an architectural blueprint with every function a documented but otherwise empty stub.

AI is a great bellwether. I bounce ideas off AI for a consensus. The closest equivalent is reading StackOverflow comments. I once offhandedly complained that python had no equivalent for setattr at class scope (as __class__ is not defined until after __new__), but AI showed me how to provide a closure in __prepare__ over the class namespace, which was introduced in 3.3 (?) and to which I paid little attention. What a gem.

AI is great for learning. If you follow a textbook or blog or paper and don't understand, AI can clarify. But be careful with less structured learning - it is important to build a full mental model accounting for every possible outcome and explanation, otherwise you're susceptible to hallucinations. I remember my first derivative in which the end result could be obtained via two separate proofs, one of which would imply an incorrect calculus. You've got to play with it until you're satisfied your mental model accounts for all the facts.

Because AI facilitates learning so easily I feel the best skills for a future generation are those pertaining to memory and retention. Ya know, assuming we don't develop individualized and personalized AI that can model your next word and act as a personal memex.

AI is great as a search assistant. I have much better recall when rereading content. Thus, I prefer to ask AI to search for links to content I vaguely recall, rather than ask AI for it's own summary or recollection.

Despite being terrible at writing decent code, AI provides fantastic code review. It catches everything from subtle high-level errors - even potential errors that haven't yet occured - to api mismatches to syntax errors. I actually wish I was as fluent at namespaces and cgroups as AI, and I'm well versed.

AI is interesting at comments. It can be hard to provide germane comments while wading through the weeds. I feel the best comments are written a few days after the code is complete. AI provides that fresh perspective instantly. And if AI can't understand your code, you had better improve your comments.

I'm about to try AI for unit tests. I prefer hypothesis tests. I took a quick glance at the generated code, and it seemed overly complicated. So I'm not optimistic.

Outside of software, AI is great for things you don't really care about. Yeah it might hallucinate, but my back-and-forth with AI is more about holding up a mirror to myself, revealing inner biases. Especially useful for interior decoration / remodeling.

Of course, take everything I said with a grain of salt as security at my company actively discourages AI. So, everything I said applies to free/cheap plans only. And I haven't tried skills yet.

My experience is you can trust any code that is well tested, human or AI generated. And you cannot trust any code that is not well tested (what I call "vibe tested"). But some constraints need to be in natural language, and for that you need a LLM to review the PRs. This combination of code tests and LLM review should be able to ensure reliable AI coding. If it does not, iterate on your PR rules and on tests.

> My company uses some automatic AI PR review bots, and they annoy me more than they help. Lots of useless comments

One way to make them more useful is to ask to list the topN problems found in the change set.

You can also append ".patch" and get a more useful output

Now, I use it as well to review, just like you mention it pulls it via gh, has all the source to reference and then tells me what it thinks. But it can't be left alone.

Similarly people have been trying to pass root cause analyses off as true and they sound confident but have holes like a good Swiss cheese.

Colleague's feedback:

Claude> Address comments on PR #123

The objective of this initial review is to catch those low-hanging fruit that your colleagues would waste cycles on.

LLMs can catch syntax and basic semantics. Peers can spend time on more interesting things like design and relevant biz context.