We have been living with this problem of poorly defined goals for ages. Computers do what they are told to, not what we want them to. I built a career on translating what people want computers to do into what computers need to be told.

I suspect we'll hear of Dr. Susan Calvin more and more in the future.

I already know about the efforts about AI alignment - this is the reason I'm afraid about natural stupidity. I really do not want to see the alignment issue solved in my lifetime.