I am privileged in some ways, and not in others. Generally privileges are treated as being different and hard to compare, because "oppression olympics" (arguing some are more harmful than others) never actually helps the discussion, and generally leads to people forgetting to check their privileges - who am I to speak for the experiences of other minorities?
The fact that you act amused at "not-abused" as a privilege is part of the whole problem, especially when you consider how this intersects with with privilege (or lack thereof) issues.
The concept of intersections between privileges (or the lack thereof) is important - that some combinations of the effects of a privileged society are worse than others.