undefined | Better HN

0 pointsSajarin14d ago0 comments

People aren't good at detecting AI generated/edited comments, so unsure how effective this policy will be. Though I guess there are still some obvious signs of AI speak like emdashes and sycophantic (it's not X, it's Y!) speech.

Bit of a shameless plug but I wrote a HN AI comment detector game[0] with AI and most of my friends and fellow HN users who tried it out couldn't detect them.

[0]: https://psychosis.hn/

[1]: https://sajarin.com/blog/psychosis/

0 comments

tomhow14d ago

Something I've noticed through moderation is that people are much more easily duped by generated comments if they like the content and/or agree with the point. We've seen several cases where a bot-generated comment has been heavily upvoted and sits at the top of the thread for hours, and any comments calling it out for being generated languish at the bottom of the subthread below other enthusiastic, heavily upvoted replies. This shouldn't be surprising, given what we've seen of LLM chatbots being tuned to be sycophantic, but it's interesting to see it in effect on HN.

This is another reason why it's good to email us (hn@ycombinator.com) rather than commenting when you see generated comments.

dooglius14d ago

Do you have reason to believe that you have a reliable way in these cases of determining whether the comment is generated?

tomhow14d ago

Having been reading generated comments almost daily for over three years now, I have a pretty good sense of it. There's a bunch of signals: how new the account is; how the comments look visually (the capitalization and layout of the paragraphs, particularly when all of one user's comments are displayed in a list). Em-dashes and short, emphatic sentences, make it more obvious of course.

There are cases that are more borderline; usually when someone has used a translation service or has used an LLM to polish up a comment they wrote themselves. For these ones there's less certainty, and whilst we discourage them, we're not as rigid in our aversion to them or as eager to ban accounts that do it.

But ones that are entirely generated are still pretty easy to spot, even just from visual appearance.

vova_hn214d ago

> HN AI comment detector game

Looks cool, but how exactly do you gather proven-to-be human comments?

I think it would be better if you used pre-ChatGPT (Nov 30 2022, I think?) stories.

zahlman14d ago

I appreciate the restraint in not calling your game "AIdle".

foltik14d ago

It’s certainly hard to detect in isolation, but the thing that gives it away is the comment history.

All the AI acounts I’ve seen repeatedly post the exact same cookie cutter top-level comments over and over again. Typically some vapid observation followed by an obviously forced question serving as engagement bait. The paragraphs and sentence structure even looks visually similar across comments when you scroll down the history page.

Just look at a few of these accounts and you’ll easily be able to recognize AI posts on your own.

happyopossum14d ago

> obvious signs of AI speak like emdashes

Some of us were trained/self taught to write that way. Even "it's not X, it's Y" is a legitimate and subjectively effective communication tool, and there are those of us who either by training modeling have picked it up as a habit. It's not Ai that started this, Ai learned it from us.

Crap - I just did it, didn't I? Awww double crap! Did it again...

salicaster14d ago

Forums and comments are not written as formal novels or text. Corporate-speak is also not typically used in these environments unless you are representing corporate.

So I think it's fine to scrutinize commenters who write that way.

Besides, the biggest offense of AI speak is making everything seem like a grand epiphany and revolutionary discovery. Aka engagement bait.

j / k navigate · click thread line to collapse

0 comments

tomhow14d ago

This is another reason why it's good to email us (hn@ycombinator.com) rather than commenting when you see generated comments.

dooglius14d ago

Do you have reason to believe that you have a reliable way in these cases of determining whether the comment is generated?

tomhow14d ago

But ones that are entirely generated are still pretty easy to spot, even just from visual appearance.

vova_hn214d ago

> HN AI comment detector game

Looks cool, but how exactly do you gather proven-to-be human comments?

I think it would be better if you used pre-ChatGPT (Nov 30 2022, I think?) stories.

zahlman14d ago

I appreciate the restraint in not calling your game "AIdle".

foltik14d ago

It’s certainly hard to detect in isolation, but the thing that gives it away is the comment history.

Just look at a few of these accounts and you’ll easily be able to recognize AI posts on your own.

happyopossum14d ago

> obvious signs of AI speak like emdashes

Crap - I just did it, didn't I? Awww double crap! Did it again...

salicaster14d ago

Forums and comments are not written as formal novels or text. Corporate-speak is also not typically used in these environments unless you are representing corporate.

So I think it's fine to scrutinize commenters who write that way.

Besides, the biggest offense of AI speak is making everything seem like a grand epiphany and revolutionary discovery. Aka engagement bait.

j / k navigate · click thread line to collapse