When I was developing a VR game, my team noticed that some ultra-sensitive people started to get sick when we would drop around five frames (the vive displays 90, so this is only a ~5% drop). There was more to it (dropping a few frames every few minutes was usually fine, but consistently dropping frames every second or two is an issue).
If GC only causes occasional (maybe every 2 minutes) loss of a frame or two it should be no problem. If you watch benchmark FPS traces that count every frame delay you'll see that occasional stutters happen in basically every game.
Loss of a single frame just isn't noticeable
And some games much later didn't. I recall needing to fire off a shot on each new gun I picked up in Far Cry to be sure I wouldn't get a stutter later at a more critical moment.
Streaming/Loading assets in the main render thread is a very 2005 thing to do.