Yes, I understand that, and I'm dismayed to learn about this.

But the question I'm asking is _why_ do AI crawlers behave in this different way.