Codex's precision and attention to detail is crazy when set up correctly

4 pointsditchfieldcaleb5d ago12 comments

Lately I've been working on a Tower Defense game with Codex, in part to learn how game development works and in part to see how far I can get using just Codex, no manual coding at all. I've got my AGENTS md & my CODESTYLE md & six other ALLCAPS md files etc, and am working on some refactoring to keep the codebase clean & file sizes low, etc.

And then I see this in the ExecPlan for my latest refactor:

---

# Observations

- Observation: The refactor made the screenshots pixel-identical after the baseline was recaptured correctly.

Evidence: sha256sum screenshots/before-implementation-x.png screenshots/after-implementation-x.png reported matching hashes for before/after pairs 1, 2, and 3.

---

Which is crazy! I've never told Codex to do an sha compare on before/after screenshots of the app, but I do have instructions in my PLANS.md to take before & after screenshots of the webapp for the game to make sure we avoid frontend regressions (it uses GPT-Image-2 for analysis). So for non-frontend impacting changes, of course nothing should be different between screenshots taken at identical timestamps into the game start.

But doing an explicit SHA compare - that's just...not something I would've ever thought of. Wild.

12 comments

daemonologist5d ago

I theorize that this kind of thing - verification approaches that do already exist but which a human would not usually reach for - is a result of RLVR. I've noticed Claude Opus a couple of times trying to use `nm` to check that its changes made it into the binary (and often tying itself into knots in the process), and have stuffed some extra coaxings along the lines of "don't try to be clever" into its prompt.

TacticalCoder5d ago

> But doing an explicit SHA compare - that's just...not something I would've ever thought of. Wild.

If I'm not mistaken SGI (Silicon Graphics, Inc.) was already doing that to prevent regression 40 years ago: maybe not SHA but they were taking "screenshots" of the entire screen at a time t and some kind of checksum to then verify (without having to compare every single pixel in the happy case) that enhancement/optimization to their rendering pipeline not supposed to change the output indeed did indeed generate the exact same image as before.

It's basically a 40 years old technique: not too sure what's that wild about it.

ditchfieldcalebOP5d ago

Sure, it's been done before, and I'm sure not just limited to SGI, but no one does this for regular apps these days - never heard of it before. I just find it neat that Codex came up with this - not something I ever would have.

Edit: I'm not saying no one does checksums to compare files (lol). I'm saying no one takes screenshots at specific timestamps within an app or game's lifecycle and then compares them to ensure they're identical.

Edit 2: Whoops, looks like I'm wrong and this is apparently a pretty common thing (but not at the startups I've worked at, /shrug). I still think it's cool that Codex is doing it without being told to, though.

kay_o5d ago

> but no one does this for regular apps these days - never heard of it before

Everyone does this to match files as identical, be it sha, md5, or something else. I cannot imagine any other method such that it would first come to mind easily you would be doing to check if two files are the same.

I don't mean to offend but I quite literally mean everyone does this. Every software updater, game patcher, checking if two binary files are identical (pixel perfect/lossless in this case: BMP, PNG created by same encoder off same inputs would qualify, JPG would likely not), all of them do exactly this.

GPT-Analysis or a similarity and image chunk hashing would not be the first thing you turn to if what you wanted was exact identical pixel perfect. I am curious what your background is if this is the case.

1 more reply

tomjakubowski5d ago

https://en.wikipedia.org/wiki/Checksum

1 more reply

j / k navigate · click thread line to collapse