How much your body absorbs is a huge variable which seems insane to have to say but people just want to say calories all day.

The problem here is people equate calories in/calories out with the equation "calories eaten - (calories burned from activity + basal metabolism) = weight loss". While energy in/out dictating weight loss is basically true on a biochemical level, that equation is false because your biochemistry undergoes massive changes as a result of what you eat, and gut bacterial metabolism also depends on the substrate provided.

The problem with the 'energy in-energy out' model is that energy is rarely what we need more of. Food cravings persist for the nutrient one needs most, not just the amount of energy needed. Eating less of a conventional diet does reduce the amount of energy one takes in, but also reduces intake of whatever nutrient the subject was craving in the first place. That's why compositional diets work better for so many people, if the subject gets all the nutrients they need, they won't feel the desire to eat more than that, and consequently won't take in more energy than they use.

TL;DR: Total energy in matters, but watching energy-to-vital-nutrient-ratios in one's food is a better way of ensuring the proper levels of both.

Edit: wording