I'm sure this would be reasonably effective on the specific goal of occupying these users' time for days/weeks/months/or even years until they realized that they were talking with robots, but it seems like it might have significant negative social consequences for some groups.

For instance, imagine somebody dealing with depression who genuinely needs a human to talk to. Like many in modern society, maybe social media is the only human interaction they have until they get shadowbanned based on tenuous reasons. God, what happens then when depressed and/or unstable people only have robots to talk to for years? What happens when they realize that they've been tricked and feel intense anger/sadness/hopelessness?

This is playing with peoples' lives with fire and shouldn't be an acceptable practice.