Or perhaps you did invite them, they just couldn't make it and you don't want to rub their nose in how good a time you all had when they may already be feeling disappointed about not being able to attend? Especially in the direct aftermath of the event?

It's called social etiquette and consideration. Sometimes it's misplaced; sometimes it's unneeded. But sometimes it is required.

Humans are not machines. Human interactions do not have many hard rules.

> Either you invite them, or you own the decision not to.

Define "own the decision not to." Surely you don't mean that every time you are planning an event you must preemptively inform everyone who is not invited that they are not invited, because that's clearly ludicrous.