How the hell does one A/B test a language model that even the designers don’t fully understand?

Of course, I’m sure that once you start plugging engagement metrics into the model and the model itself conducts A/B tests on its output… hoo boy….

I pasted error messages from code it generated. It kept generating the same compiler error eventually. When I applied the "socratic method" and explained to it the answer based on stack overflow answers. It would at first pretend to understand by transforming the relevant documentation I inserted into it, but once I asked it the original question, it basically ignored all the progress and kept creating the same code with the same compiler errors.