What exactly is your basis for thinking it’s crazy to ship things from the other side of the planet? What cost benefit analysis have you done to come to this conclusion? Or are you just appealing to your own amateur intuitions?
If there’s an ecological cost that isn’t being accounted for, the solution is to adjust policies so the cost is reflected, e.g. via a carbon tax. And I’m sure if we had a carbon tax, we would eat more locally grown food because it would be more cost effective. On the other hand, different parts of the world vary widely in terms of what can be grown there and how efficiently it. It’s a complicated problem that can’t be solved top-down, and certainly can’t be solved by some armchair hippie going “the whole system is mad, man!”
