The GC doesn't just affect high-performance games. The problem is it doesn't just make things broadly slower; it introduces what can easily become a second-long pause a couple times a minute. Even if your game runs buttery smooth the rest of the time, that's pretty much unacceptable. There are measures to minimize it - object pooling, for example - but they remove most of the benefits of having a GC in the first place.
