> A minority of the problems in the exams were seen by the model during training
A minority can be 49%. They do mention they tested against newly available practice exams, but those are often based on older real exam questions which may have been discussed extensively in forums that were in the training data. Now that it is for-profit ClosedAI we have to somewhat treat each claim as if it were made adversarially, assuming minority may mean 49% when it would benefit them one way and .1% when it serves their look better for sales pitch to the Microsoft board, etc.
> A minority of the problems in the exams were seen by the model during training; for each exam we run a variant with these questions removed and report the lower score of the two. We believe the results to be representative. For further details on contamination (methodology and per-exam statistics), see Appendix C.
From the results before and after removing some of the duplicates it doesn't seem to have hurt its performance badly though. Sometimes the score increases, so the substring approach may be helping it by excluding question variants with matching substring that it memorized but then the real test varied somewhere outside of the sampled substrings and had a different answer (or it random chance that the extrapolated score increased with some questions removed).
