No, you're wrong, an extremely scant number of applications can completely avoid allocating memory in the old generation. Your best bet would probably be to use the Disruptor approach of preallocating everything then avoiding any subsequent allocations if that's really what you want to accomplish. However, if you honestly believe "most real world applications" can avoid allocating memory in the old generation, I'd sure love to see some graphs of some heap profiles of real world usage of these applications!

CMS does a stop the world collection of the old generation, however it will do an incremental collection of the eden generation. If you've ever looked at a graph of the JVM heap observing CMS running under VisualVM or YourKit, you'll notice a distinct sawtooth pattern to its collections. The diagonal parts of the sawtooth represent incremental collections in eden space. The vertical parts of the sawtooth represent stop-the-world collections of the old generation.

The "small regular sweeps" you're talking about affect the eden generation, not the old generation. Collecting the old generation is a stop the world operation in every JVM GC option except Azul's C4 collector. C4 resorts to all sorts of clever tricks to avoid stopping the world, most notably read barriers and eager pointer updates. On the Azul hardware, these tricks involved hardware transactional memory. On x86 they don't have HTM (at least until Transactional Synchronization Extensions ship on Haswell). What they do, however, is quickly set up GC invariants and a trap on a given page. Whatever thread accesses a page that contains an object which needs to be relocated relocates the object on the GC's behalf and gets the new pointer. All that said, C4 is only available on the Azul JVM, which is commercial.

CMS can't do this. It stops the world at a JVM safe point whenever it needs to collect the old generation.