How much does that affect a user? I mean, if you take the time to learn how much it strays, are you at that point just as accurate as someone with a more accurate weapon?
All rifles do this a bit; snipers/designated marksmen get very accustomed to their personal weapons, and need to take time to re-acclimatize with any other rifle. For a regular line infantryman in a modern army who is mainly using his rifle for suppressive fire it doesn't significantly reduce his effectiveness.
In real close-to-mid-range combat, this matters even less (unless you're a marksman). It's rare that you'd have time to properly line up the sights and take one really good shot. What really matters in that situation is knock-down power, reliability, and the shooter's training.