Your agent _should_ always be where you are (ie not inside a container, a bastion host, or whatever else) because otherwise that means you aren't actually in possession of the key material and there's plenty of opportunity for much _worse_ surprises than with SSH agents if somebody else has the key material.
Because it's where you are, and you're probably not on a 1970s video terminal link but a laptop or something, the agent could just ask you to OK each request out of band, e.g. popping up a "Really log into machine X?" request. Once such a mechanism existed it could be refined (should it let you say "Yes always to requests from machine X" ? How about "Yes always for the next five minutes" ?) and if necessary SSH auth could even be tweaked to better support any real world behaviours that are popular (e.g. I don't recall off the top of my head if the agent can tell from what it's signing either where you're signing in, or where that sign-in was used, but the binding mechanism in SSH auth could certainly enforce either of those if they're determined to be important and don't exist today)
