Multi-agent refers to behavior of pedestrians/cyclists, and other cars on the road. This is especially tough in "ambigous" junctions such as roundabouts, and unprotected left turns. There the strategy to negotiate the junctor is highly context dependent, and the information needed to find a strategy is not in the current scene. Drivers in these moments draw on "cultural awareness" of what "should" be done. Observing a history of what people do in these situations may not be sufficient because of the long tail of unique events, or at least unique in terms of how the computer will represent the scene. For example, if the scene is represented by the set of trajectories (or really waympoints), then the set of possibiilties is infinite. All of this assume the car "knows" it's entering and exiting a predefined scenario such as roundabout, real life driving is not so discrete.

On top of this, there's a liability and ethics issue. We accept teenagers for getting drunk and killing people, but we cannot accept an autonomous car that cannot navigate a roundabout which would otherwise be easy for a person, sober or otherwise.