This seems to work through the numbers and equations, albeit for answering a slightly different question:
https://space.stackexchange.com/questions/20054/fuel-needed-...In summary, as you said, the altitude is less important than the base velocity increase and atmospheric density reduction.
The former, because you're pushing maximum mass at t=0 (i.e. all the future fuel you need to burn), so any added velocity at rocket ignition time would compound throughout the rest of the burn cycle (or, to think of it another way, you've already overcome fully-fueled vehicle inertia with the benefit of atmospheric oxygen combustion).
Similar to how a multistage vehicle operates more efficiently, albeit without the benefit of atmospheric oxygen.
The latter, because you're essentially getting atmospheric density reduction for "free" (in terms of saving your on-vehicle propellant), and your propellant efficiency (in terms of propellant:velocity increase) scales better.