It is kind of odd how we ship grains and soybeans farmed in the rain forest to a factory farm in the US and then end up with huge poo lagoons that in theory would have to be shipped back to the farms in the rain forest to act as fertilizer. Instead people just abandon the land and continue with their slash and burn agriculture on a different plot of land.
This may come as a surprise, but there's plenty of wastage in locally shipped produce too. It's not so much the distance as the act of picking things and putting them into boxes, and taking them out of boxes again that does it.
You haven't tried the vast majority of fruits, veggies, meats, fishes out there and I'm sure you don't feel like you're life is miserable because of it