undefined | Better HN

0 points85392_school1y ago0 comments

The funny thing is that if your request only needed the top 100's temperature or the top 33's precipitation, it could just read "List of cities by average temperature" or "List of cities by average precipitation" and that would be it, but the top 250 requires reading 184x more pages.

My perspective on this is that if Deep Research can't do something, you should do it yourself and put the results on the internet. It'll help other humans and AIs trying to do the same task.

0 comments

Balgair1y ago

Yeah, that was intentional, well, somewhat.

The project requires the full list of every known city in the western hemisphere and also Japan, Korea, and Taiwan. But that dataset is just maddeningly large, if it is possible at all. Like, I expect it to take me years, as I have to do a lot of translations. So, I figured that I'd be nice and just as for the top 250 for the various models.

There's a lot more data that we're trying to get too and I'm hoping that I can get approval to post it as its a work thing.

therein1y ago

Sounds like the you're having it conduct research and then solve the Knapsack problem for you on the collected data. We should do the same for the traveling salesman one.

How do you validate its results in that scenario? Just take its word for it?

Balgair1y ago

Ahh, no. We'll be doing more research on the data once we have it. Things like ranking and averages and distributions on the data will come later, but first we just need it to begin with.

wyre1y ago

If you have the data, but need to parse all of it, couldn’t you upload it to your LLM of choice (with a large enough context window) and have it finish your project?

Balgair1y ago

I'm sorry I was unclear. No, I do not have the data yet and I need to get it.

XenophileJKO1y ago

Well remember listing/ranking things are structurally hard for these models because you have to keep track of what it has listed and what it hasn't, etc.

j / k navigate · click thread line to collapse

0 comments

Balgair1y ago

Yeah, that was intentional, well, somewhat.

There's a lot more data that we're trying to get too and I'm hoping that I can get approval to post it as its a work thing.

therein1y ago

Sounds like the you're having it conduct research and then solve the Knapsack problem for you on the collected data. We should do the same for the traveling salesman one.

How do you validate its results in that scenario? Just take its word for it?

Balgair1y ago

Ahh, no. We'll be doing more research on the data once we have it. Things like ranking and averages and distributions on the data will come later, but first we just need it to begin with.

wyre1y ago

If you have the data, but need to parse all of it, couldn’t you upload it to your LLM of choice (with a large enough context window) and have it finish your project?

Balgair1y ago

I'm sorry I was unclear. No, I do not have the data yet and I need to get it.

XenophileJKO1y ago

Well remember listing/ranking things are structurally hard for these models because you have to keep track of what it has listed and what it hasn't, etc.

j / k navigate · click thread line to collapse