I think that one of the biggest challenge the GC will have to face in the following months/years is the increase in size of the managed heap memory. Patrick Dussud, one the main developer of Microsoft .NET's GC, mentioned in a podcast (http://www.microsoftpdc.com/2009/FT51) that heaps overs 15 GB could, at this time, cause some serious GC latencies.
There already exists commercial Java based solutions, such as Terracotta's big memory solution (http://www.terracotta.org/bigmemory?src=/index.html), that offers up to 600GB of memory off-heap with no pause time caused by the GC.
I'm planning to work on Mono's generational GC. My approach would be to identify in which specific conditions the GC starts presenting latencies and look for solutions that would allow it to manage bigger heaps.
Thanks you very much!