I agree! That's why it's wild to allow a setter to do literally anything.
You aren't just setting a property, there is a conversion happening under the hood.
And the reason I hate it is that code that appears to be infallible, isn't. It can have arbitrary side effects, raise exceptions, have unbounded runtime, etc.
In a lot of contexts you have something that requires a bit of code but really does behave like a property access, where it's more misleading to make it look like a method call than to make it look like a data access. E.g. using an ORM hooked up to SQLite embedded in the program. Or accessing properties of objects that you're using an EAV system or array-of-structs layout to store efficiently in memory. Or a wrapped datastructure from a C library that you're binding.
Of course if you make something look like a property that doesn't behave like a property then that's confusing. Programmers have to exercise judgement and only make things look like properties when they behave like properties. But that's not really any different from needing to name methods/objects/operators in ways that reflect what they do.
I sometimes use getters and setters to provide backwards compatibility. What was just maybe a simple data field a decade ago doesn't even exist anymore because some major system has changed and we aren't rewriting every single application when we can provide the values that they need easily enough.
If you know that setters exist then you already know that the code can do more. It's not a huge mental leap from property setting to method calls. You should never assume anything is infallible. I don't think classes should even expose raw fields.