It implies the persons' leverage is limited to only what they personally do. That's obviously false. A non-manager engineer can have broad scope in putting in a proper architecture, in mentoring others, in cross-team communications. I would go as far as saying there's virtually no engineer whose impact is limited to themselves. They have a harder job since they need to affect change without having official authority to affect change.
The term's very existence it puts people in a certain bin. Why is a CEO not also an "individual contributor"? They're an individual. They contribute. It's just newspeak.
I've never thought of management positions in an organization to reflect something derogatory. But maybe to some.