What about regional accents? For a non native speaker of English, understanding English as spoken in Scotland is a challenge. And I heard people born in UK complaining they had sometimes difficulties with American English (for example with real life/social things). This without speaking of people from India who speak an English which is slightly different from US' or UK's English (and equally valuable and "native").
So I guess a definition of "native speaker" is a bit difficult.