I think its a commonly accepted opinion that a person's ability to admit when they don't know something is a strength. It would be interesting to a see how much a constraint for known validity impacts the ability of a model to synthesize responses. In other words, where do we draw the line between knowing and "thinking we know" something (i.e., do we draw the line at 95% confidence)? Surely we do not hold each other to a strict requirement of 100% confidence, but there must be some acceptable range.
There's a lot of fan fiction out there in the Star Trek & Wars universes, maybe even some crossover stories. If the GPT training model includes that, then it might be fairly confident about an unfortunate incident on the Millennium Falcon involving Luke Skywalker and tribbles. Or does it understand the idea of "canon"?
