My point is that in an overwhelming number of situations, it's not sufficient to merely identify the locations and possible trajectories of all of the moving objects. If you assume everyone will act "rationally", you will end up killing someone every few days. If you assume everyone will do the worst possible thing, you'll never get anywhere.

This situation forces you to predict behavior reasonably accurately. There's a big not-so-long tail of people doing stupid things. If you drive in a bar area late at night, you will fairly often encounter drunk people in the street. You may notice some swerving on the road or walking erratically and infer that they might be distracted or drunk or a small child and give them a lot of extra room.

Human drivers typically develop something of a sixth sense observing people on the road and know who it's safe to drive past at high speed and who needs lower speed and more room. If humans didn't have this, the road would be a lot more dangerous.

AVs can compensate for this by having (potentially) faster reaction times, and sensors that can see longer distances. There are a lot of scenarios though where better prediction is more important than either of these. Behavior prediction is a very active area of research in the AV world.