> Let's say I'm a bank, how do I know that my APIs don't allow the unintentional creation of money?

How do you know now?

> Presumably because the engineers designed the system to prevent that.

How do you know the engineers understood the invariants? How do you know they didn't make a mistake in coding these invariants? Banks still don't use formal methods to prove these invariants last I checked, so no matter what, you need to write tests to check any invariants, and tests still can't achieve 100% certainty.

> I'm talking about designing for simplicity so that the behavior of the system can be reliably predicted.

From the page, it sounds like the system is fairly predictable, generating a program based on a schema and a descriptive method name. If it's not predictable then the model needs to be tuned to make it more predictable, just like how any other software development advances.

If you can design your schema to ensure any invariants are preserved, even better.

Finally, don't confuse the first preview version of the product with where this is going. The project as it is is fairly simple and predictable, but a bit limited. It does point the direction towards what is possible though.

You could also have separate AI trained to do fuzz testing of an API description and automatically and instantly generate thousands of tests checking all possible corner cases, and in principle, such systems could be even more robust than human written ones simply because of the breadth of testing and the number of iterations you can rapidly go through to converge on a final product.

Great response that you've written.

It makes me think of Ben Graham's apt observation from his book "Intelligent Investor" (which Warren Buffet and Charlie Munger both know and cite religiously):

"You do NOT want your banker to be an "optimist."

If you do not understand what this means, just ask http://perplexity.ai to explain the idium/phrase/limitless concept(s). No login/signup required [this replaced Google Search, IMHO, for all but the most-specific technical inquiries].