This is a longstanding critique of GOFAI; see Hubert Dreyfuss and Phil Agre.
https://pages.gseis.ucla.edu/faculty/agre/critical.html
> I have a model in my head mapping out the world around me, so I know where things are etc and what I can do with all those things.
No you don't; a map is not the territory, and is necessarily wrong, which means that if you had such a model and were actually relying on it you wouldn't be able to do things you obviously can do in real life.
You have an inaccurate memory of the world and you update it, only as much as you need to[0], as you go, in order to do a specific task.
[0] probably a little less than you need to, because you want to save thinking energy