On a basic level, Story Points are there because if you can't tell me how long something will take, I don't want to give you money to go do it.
Everyone has to deal with this, at every level. CEOs think about this, janitors think about this, literally everyone is constantly calculating estimated time to complete a task as a key metric about whether or not they should do the task.
All a Story Point should do is make that a conversation, not an isolated person's random guess. That's it. I don't see what's harmful about that.
I think you are looking at it from an r&d (still evangelistic) prospective, but now picture rewriting a truly ancient system still operating on mainframes(and the original developers and maintainers are no longer among the living) in a modern language where it's not a question if something is worth doing or who will do it-it has to be done. Figuring out how the old system works is a combination of reverse engineering and brute forcing. I've been there and trust me, in those circumstances story point is a synonym for a random number. This is a real life example and in those conditions agile doesn't only crack, it collapses and it mentally crushes on top of you cause you are still forced to stick to everything agile.
What I'm saying is that agile and by implications sorry points work in some extremely specific circumstances but in the vast majority of cases it only adds multiple layers of complexity which does nothing but waste valuable time.
I've seen your argument many, many times before, and it comes from people who have divorced their understanding of value from their understanding of work.
You don't get paid to work, you get paid to provide value, and I'm not saying that's right or how it should be, but that's how it currently is. If, through your labor, you can't provide value above your cost to a company, the work you're doing is not worth the cost. One way to get an idea of how much what you're doing will cost is to ask you and your peers what you think it'll cost, and one way of measuring that cost is a Story Point.