I think that's part of the issue that polarizes people: it's not consistently slow and it's not consistently fast. So you get people who can't understand how it's usable, and others who can't understand how anyone has a problem with it.
Some other editors are consistently fast, for everyone, always (like [n]vim). Not because they're special but editing text just isn't that big a deal and these editors don't try to do too much graphically, so their performance is great.
Another problem is the combination of extensions probably has a significant impact on performance and resource usage. So person A uses it to edit JavaScript and has no problem, but person B uses it to edit Ruby and has one. It might not even be VS Code's fault, maybe an extension, but it is seen as if it's the editor since it's a whole package and the input latency is what's suffering.
