Yeah, for me at least, personally believe inversion of control should be used more surgically instead of blanketing the system with it. On the one hand, freeing your application layer from direct dependencies with the lower-level objects conceptually seems like a good idea, but think in practice, this is hardly ever helpful especially when used for every dependency.
At least from my experience, seems like we don't change the objects we use that often, that once a object is set on a reference var, a very, very large majority of them won't change.
And because of this, seems like that for most objects dependencies, we should just new them directly, and if later on we do need to change them, then at that time, we can refactor the code to use more abstraction (like inversion of control) to break the direct dependency, but only for the code that needs it (or if there is a important situation where having a direct dependency could be highly problematic in the future, like to a DB provider).
It's like the performance optimization problem. One guideline that is often quoted is that it's best not to over optimize the performance of your code, until you actually can test it in real-world test cases, because you very often optimize things that aren't the bottleneck. Same with the over usage of inversion of control. Spring makes it so we're using IOC everywhere, but it's just adding unnecessary complexity.
Think that if inversion of control is used, should be used mainly at a higher level, for components instead of on every class like often happens. But even for components, think you should be careful when deciding to do so.
... and agreed, you could just use the factory pattern instead of Spring.