It's not a simplification. It is abstraction. Binding doesn't imply a particular implementation.
A variable binding can disappear entirely. If you bind var x = 42 and never use it, the variable need not exist. If it doesn't exist, then there is no pointer.
If you do use it, constant propagation/folding can again make the variable disappear. If the variable is never assigned, all uses of it can be replaced with 2.
Variables that care captured by lexical closures can be treated differently from ones that are not. Variables captured by closures but not shared among different closures, or not mutated, can be treated differently from mutated variables shared among closures.
The abstraction of binding is more complicated than "every variable is a memory pointer" because it allows more possibilities, which can coexist.