First, it's calories absorbed, not consumed. That is, if I eat calories in a form that's harder to digest, my body may not get all the calories out of it that are in the food. I think that this is one of the deals with processed food - the processing makes it easier for your body to extract the calories from the food. The calories were already there, but they're processed into a form that your body can use more easily.
Second, some foods (at least for some people) have some effect on metabolism, so that calories consumed is not totally independent of calories burned.
Fish is one of the least calorie dense meats. Vegetables and fruits are some of the least calorie dense foods. It's nearly impossible to eat enough lettuce in a day to go over your daily recommended calories. It just won't fit.
So, in a lot of ways, keto, paleo, vegan, Mediterranean diets are all sneaky ways to cut out a major source of calories. You don't have to count them because just the way you're eating is taking care of that aspect.
But if you want to eat Twinkies exclusively, you can do that as long as you eat only so many.
Yes, paleo and keto (I think) don't exclude fat, but they are excluding a major source of calories. Because we do need some fat in our diet or else we go into rabbit starvation.
There's nothing really in bread or refined sugar that we need that we aren't getting from somewhere else.
So, when you point out where I said those diets cut out all major source rather than a major source, feel free to continue.