undefined | Better HN

0 pointsmake34y ago0 comments

don't you pretrain on very silar tasks explicitely

0 comments

We discuss this a bit in Section D.2 (HOW UNSEEN ARE THE HELD-OUT TASKS?). From our perspective,

a) The tasks we test on are very different, particularly tasks like BIG-Bench that we didn't even have access to until several days ago (and none of us read).

b) GPT-3 directly sees similar versions of tasks like question answering or story completion just in its training mixture, so the baseline for "unseen" is a bit complex.

stellaathena4y ago

Minor correction: I (Stella Biderman) am a contributor to BigBench, have read many of its tasks, and have had access to it for months. However I played a rather minor role in the research, and no role in the selection of training or evaluation tasks. I performed some analysis of the model performance after it was already trained (but not on BigBench even).

j / k navigate · click thread line to collapse

0 comments

srush4y ago

We discuss this a bit in Section D.2 (HOW UNSEEN ARE THE HELD-OUT TASKS?). From our perspective,

a) The tasks we test on are very different, particularly tasks like BIG-Bench that we didn't even have access to until several days ago (and none of us read).

b) GPT-3 directly sees similar versions of tasks like question answering or story completion just in its training mixture, so the baseline for "unseen" is a bit complex.

stellaathena4y ago

j / k navigate · click thread line to collapse