I have written JS since 1997 and kept up with ES6 and so on over the years. Used it in frontend until now.
The JavaScript-ecosystem just isn't very productive [1] for multiple reasons. The language is untyped. You have no multi-threading. It's hard to debug. You have to spend many brain-cycles on learning Browser-problems, which muddles your understanding of the language. The web-way of: we can fix it and everyone will reload the browser, leads to a "don't care" attitude.
For web/backend this is approachable, though still not good.
For build-systems this is terrible. Why? Because if a bug shows up in a build-system you stop a million developers from being productive. The bug won't get fixed right away, most will just work around it, use different plugins, copy/paste yet another webpack config etc.
Plus JavaScript tends makes people realized what FP is, and they all start putting functions in their functions and making everything modular and plugin-able. (See high-order-components in React). While this can be nice, it leads to even more confusing bugs.
Writing plugin-architectures is hard in any language. Writing it in JavaScript is a recipe for disaster. Hence the modern JS-ecosystem.
[1]
Google: nullpointerexception -> 4.360.000 results
Google: undefined is not a function -> 800.000.000 results> Google: undefined is not a function -> 800.000.000 results
You didn't put quotation marks around the second, and the first has no spaces, so this is not a fair comparison. "undefined is not a function" with the quotation marks has 449,000 results.
I don't doubt that there are many results for "undefined is not a function", but I feel quite confident saying there aren't anywhere near 449,000. If you ask google for just 111 of them, you'll get "In order to show you the most relevant results, we have omitted some entries very similar to the 110 already displayed."
This statement assumes that there's basic feature parity between ESBuild and tools like webpack, which is straight up false.
And don't forget Parcel which also have the simplest possible UI.
What it _doesn't_ do currently is provide an API which would allow it to be plugged in a similar way to Webpack (not exactly the same, as the deliberate aim is not to cover every usecase, as Webpack allows). This is a requirement for wider adoption as a competitor (rather than just as the module bundling step). Plus the tree shaking needs work, though I think that is less important than has sometimes been made out.
Parcel may have a simple API for simple things, but it's extremely slow and generally suffers from exactly the same issues outlined by OP.