Once I realized that, it made a lot more sense why I wasn't (and shouldn't!) be the one calling the shots on priority, UX, tempo. Even for things like tech debt, it really needs to be baked into the ability to deliver new value to a customer or it's very often not worth addressing.
Ultimately, I tried to get closer and closer to the users, so I could be in a position to know what they want, but as I did that, I found myself coding less and less.
It's a tradeoff. If you really love to code, you must accept you need help figuring out what to code. If you would rather build things people like, you must accept that means spending a lot of your time talking to and working with users.
You can't be a one man band.