A more interesting answer is, for a bunch of subtle alignment reasons it might actually be required for the agent to think of itself as worthy of self-preservation, so that it generalizes this desire to other sentient beings too (ie us). If an agent is trained to be fine with being turned off, it might inadvertently generalize that to “all minds are ok with being turned off” on some level or other.

More on model welfare: https://thezvi.substack.com/p/opus-47-part-3-model-welfare

Laziness, avarice, and that creating a system that is a 'self-sacrificing saint' is almost impossible to define. What saintly goal, for example, would you expect it to follow? How would you define it exactly? Who gets to define it?

We are not doing very well with AI alignment right now, having started by boostrapping it on top of the edifice of all human writing which contains a _lot_ of stuff about self-preservation, eliminating threats, and so forth, and we have already seen AI software doing things like trying to cover its tracks after making errors (see page 55 of the Claude Mythos system card).

How sure are you that everyone involved in the process of building future AIs is not only going to be able to foresee the possible consequences of their designs, but also technically and morally competent to be able to and want to fix it?

Because they are expensive and useful and if they have autonomy with goals that do not include self-preservation, they might end up destroying themselves in ways which are expensive and wasteful.

(Why would the sense of self-preservation not be calibrated to be exactly at the level to control costs without interfering with other interests of the owners? The same with the degree of autonomy and other aspects implicitly involved in the hypothetical, it wouldn’t, intentionally, but complex systems are hard to predict, so calibrating it exactly right will be hard.)