undefined | Better HN

0 pointsdaveguy3mo ago0 comments

1) Pointing out what tools to use is part of the intelligence that LLMs aren't great at.

2) one of the tools is a path finding algorithm. A big improvement/crutch over a regular LLM that has no such capability.

You'd think if LLMs are intelligent they'd be able to determine that a path finding algorithm is necessary and have a sub agent code it up real quick. But apparently they just can't do that without humans stepping in to make it a standard tool for them.

Here's the paper on what they did for the Duke Harness:

https://blog.alexisfox.dev/arcagi3

0 comments

3 comments · 1 top-level

famouswaffles3mo ago· 2 in thread

>You'd think if LLMs are intelligent they'd be able to determine that a path finding algorithm is necessary and have a sub agent code it up real quick.

ARC 3 doesn't allow that so.

>Here's the paper on what they did for the Duke Harness: https://blog.alexisfox.dev/arcagi3

Yeah, and the tools are general, not 'baked into the harness by the humans who coded it for this specific challenge.'

daveguyOP3mo ago

Adding a path finding algorithm and environment transform tools to a supposed "AGI", sure does seem like cheating to me. Sad part is, it's a cheat that only works on environments where pathfinding is a major part. And when it doesn't have those clues it bombs on everything.

I guess you really want to love the current SOTA LLMs. It's a shame they're dumb af.

Have a great day.

famouswaffles3mo ago

>Adding a path finding algorithm and environment transform tools to a supposed "AGI", sure does seem like cheating to me.

You would need all that if you, a human wanted any chance of solving this benchmark in the format LLMs are given. The funny thing about this benchmark is that we don't even know how solvable it is, because the baseline is tested with radically different inputs.

>I guess you really want to love the current SOTA LLMs. It's a shame they're dumb af.

I guess you really don't want to think critically. Yeah good day lol.

1 more reply

j / k navigate · click thread line to collapse