There's a famous 2006 Google Research blog post titled "Nearly All Binary Searches (...) are Broken" [1] due to a commonly occurring bug when implementing binary search.

glibc still has that bug [2].

I just asked ChatGPT 4 to write an implementation of binary search in C and it wrote a bug-free version on the first try.

I mean, this is not conclusive evidence, but I find it conceivable that an AI which despite being trained with buggy code, can still incrementally learn what the different coding constructs actually do, would be able to write more bug-free code than what it was trained with...

[1] https://blog.research.google/2006/06/extra-extra-read-all-ab...

[2] https://sourceware.org/bugzilla/show_bug.cgi?id=2753