From my POV and experience, the middle ground is often what people refer to as 'integration' tests. Testing (without a browser), hitting endpoints/urls with known payloads and getting expected results catches errors with assumptions made about the interaction between various individual libraries.
At least in the web app world, my views are:
1. Testing the individual libraries gets you one layer of confidence. 2. Testing the interaction of those, usually via URL endpoints as various identities, gets you another layer of confidence. 3. Testing with E2E exposes primarily UI/JS problems.
When the first 2 are strong/solid, you can focus troubleshooting problems in #3 at the client/JS level first. It's not always the case, but it can help reduce concerns about "is this a back-end issue?".
I've been (slowly) trying to write more js component tests (in one case, with jest and vue), as it makes it easier/faster to test many permutations of input/validation/etc all at once. It's yet another 'confidence' area such that, when there are E2E tests, I can narrow down focus even more.
On a couple projects I've been on the past few years, we've found very few problems via E2E tests alone, mostly because there are so many back-end unit and integration tests. The E2E issues that are found are often UI-only (error state changes not rendering, sometimes perf issues, etc).
Sure if you write your own implementation of "string" or "list" you will probably get the API right the first time - those are commonly used and time tested so everyone knows about what the API should be. However almost nobody is writing them, they come with your language for everyone but a few language implementers, or once in a while the company library implementers.
For everyone else we are writing to a business requirement that isn't well understood and may change. However the purpose of a unit test is to assert that no matter what this won't change. So every time you want to make a change all those tests are in the way of the change and need to be fixed.
Everyone writing tests needs to figure out their own middle ground. Because end to end tests have their own problems.
As soon as you start changing the behavior, you have to change the unit tests. If you're adding behavior, you have to add tests. If you're removing behavior, you remove tests. If you're changing the way a procedure works, you change the related unit tests.
Really, any behavior change requires changes to the tests (whatever level they may be, if you want a high degree of test coverage).
