Abstaining from voting means "I am ok with whatever the rest of you decide". While that's not an active vote in favor, it's close enough.
I don't care about non-voters. If they don't care enough to vote, they're not worth considering when we try to assign "blame" for the current situation.
The American public voted for this, full stop.