I think this is wrong, because in general, when analogy is good, it is typically good because of the tendency toward allowing for reflex responses. It can't be good and bad for the same reason. It needs to be for a different reason or there isn't logical consistency.

I'll try to explain what I mean by that in an empirical context so you can observe that my model makes general predictions about cognition related to analogical reasoning.

If you have an agent with a lookup table that is the perfect bayesian estimates versus an agent which has to compute the perfect bayesian estimates and there is an aspect of judgement related to time to response - which is a very true aspect of our reality - reflex agents actually out-compete the bayesian agent because they get the same estimate, but minimize response time.

So it can't be the reflex itself which makes an analogical structure bad, since that is also what makes it good. It has to be something else, something which is separate from the reflex itself and tied to the observed utilities as a result of that reflex.

> imagine someone buying a product recommended by ChatGPT because "otherwise Sydney would be sad".

Okay. Lets do that.

If Sydney claims that they would be sad if you don't eat the right amount of vitamin C after you describe symptoms of scurvy, it actually isn't unreasonable to take vitamin C. If you did that, because she said she would be sad, presumably you would be better off. Your expected utilities are better, not worse, by taking vitamin C.

> programmed to maximize the profits of Microsoft

This isn't the objective function of the model. That it might be an objective for people who worked on it does not mean that its responses are congruent with actually doing this.

---

I think to fix your point you would need to change it something like "The way anthropomorphism can be problematic is if it causes a human to react with a reflex consideration for the (simulated) feelings of the machine and this behavior ultimately results in negative utility. Ultimately the behavior of the large language model is learned weights which optimize an objective function that corresponds to seeming like a proper response such that it gets good feedback from humans - so imagine someone getting bad advice that seems reasonable and acting on it, like a code change proposal that on first glance looks good, but in actuality has subtle bugs. Yet, when questioning for the presence of bugs, Sydney implies that not trusting their code to work makes them sad... so the person commits the change without testing it thoroughly. Later, the life support has a race condition as a result of the bug. A hundred people die over ten years before the root cause is determined. No one is sure what other deaths are going to happen, because the type of mistake is one that humans didn't make, but AI do, so people aren't used to seeing it."

I think this is better because it actually ties things to the utilities, rather than the speed of the decision making. You can't generalize speed being bad. It fails in most generalized contexts. You can generalize bad utilities being bad.