Modern Collectors
Modern garbage collectors use a combination of the techniques that I've talked about here. They typically have several generations and copy objects between them. Often, they have a small map of intergenerational pointers, so a young generation can be scanned from roots in older generations.
Where determinism is important, collectors often use reference counting. IBM, for example, has a real-time JVM that uses reference counting and cycle detection, rather than tracing. Many of the same techniques apply, however. You can use the same generational approach that reduces the working set for tracing collectors to reduce the size of cycles that need scanning.
The most important research area in garbage collection at the moment is scalability. Azul, for example, provides a Java implementation (on custom hardware) that can handle tens of gigabytes of memory churn per second. Most of us rarely allocate or free more than a few tens of megabytes per second, so simpler approaches give more than adequate performance.