That said, I really do appreciate this exchange and it has helped me clarify some ideas, and I appreciate the time it must take you to write this out. And yes, I'll happily put things on my reading list if that's the best way to learn them.

Let me offer another example that I believe captures more clearly the essence of what you're saying: A model that learns addition from everyday examples might come up with an infinite number of models like mod(a+b, N), as long as N is extremely large.

(Another side note, I think it's likely that something like this does in fact happen in currently SOTA AI.)

And, the fact that human physicists will be quick to dismiss such a model is not because it fails on data, but because it fails a heuristic of elegance or maybe naturalness.

But, those heuristics in turn are learnt from data, from the experience of successful and failing experiments aggregated over time in the overall culture of physics.

You make a distinction between experiment and observation - if this was a fundamental distinction, I would need to agree with your point, but I don't see how it's fundamental.

An experiment is part of the activity of a meta-model, a model that is trained to create successful world models, where success is narrowly defined as making accurate physical predictions.

This implies that the meta-model itself is ultimately trained on physical predictions, even if its internal heuristics are not directly physical and do not obviously follow from observational data.

In the Fermi anecdote that you offer, Fermi was talking from that meta-model perspective - what he said has deep roots in the culture of physics, but what it really is is a successful heuristic; experimental data that disagree with an elegant model would still immediately disprove the model.