Academically speaking, that is a matter of some debate. Pragmatically speaking, it's easy-- are they speaking the language they grew up speaking? Then it's correct. What you call "broken English", assuming native speakers, are just poor (i.e. impoverished) dialects.
To be clear on what I'm saying, an AAVE speaker can attempt to speak some other dialect and fail, producing broken constructions similar to non-native speech. But almost none of the speech labeled as such falls into that category.
