The crew was legally right only inasmuch so as the captain can order anyone off his/her plane, legally. If you're brown and a passenger doesn't like you, they can report you to the crew with some trumped up reason and the captain will order you off (which has happened).
What's your legal recourse? None.