Yeah, exactly. That's how optimization works! Recognizing that in many cases, the work is that was being done is not necessary to produce the result you actually need, that it could have been in a different way.

This can sometimes involve adding complexity (by means of a better data structure or algorithm, e.g. a quadtree), and sometimes involve removing it (non-pessimization[0]).

That being said, it's certainly also possible to achieve a speedup by breaking things, which you are suggesting is the case here. I can't comment on that part.

[0] Casey Muratori - Refterm Lecture Part 1 - Philosophies of Optimization https://www.youtube.com/watch?v=pgoetgxecw8