I agree with this although I think it can sometimes be a red herring for those new to optimization. It's possible to spend a lot of time digging deeper when that's not really what's needed. For example, you might find a particular database query that's really slow and start looking at what configuration tweaks you can make to your database server. But maybe if you structure the application a bit differently, that query isn't necessary at all.
It's a great skill to be able to peel back layers of abstraction and continuing to optimize all the way down. But it's an equally important skill to be able to know what later really needs optimizing.