GPT-4 doesn't pay close attention to detail in some cases (opens in new tab)

(github.com)

3 pointsmcaledonensis3y ago2 comments

2 comments

2 comments · 1 top-level

> - Arthur outputs the sum of last two digits that he had seen.

> Here is the sequence in which the actors act: MAMA.

I didn't even get this on the first reading.

It's like that brain teaser "count the number of Fs in 'Finished files are the result of years of scientific study combined with the experience of years'" (https://www.researchgate.net/figure/How-many-Fs-do-you-see-i...). It shows that GPT4 understands the instructions like a human does, and makes the same mistakes.

mcaledonensisOP3y ago

The trick with the F is a good one. Yes, tokenization is very likely one of the contributing factors.

The point of the eval is an attempt to exercise: attention to detail, tokenization, ability to stop, ability to follow instruction and ignore irrelevant instructions, biases, laziness, computational limits, etc.

j / k navigate · click thread line to collapse