I don't think there's any claim you could make about the language model, factual or otherwise, that would resolve my primary hangup. As a non-native learner of a new language, I do not have a trained bullshit detector for that language. I cannot, by virtue of still being a novice, determine if the sentence structure sounds "weird," and I certainly can't determine if that weirdness is a limitation of the language model, some hallucination, whatever. So, by learning from that model, I would pick up any mistakes it makes and fold those into my own speech patterns.

If I'm going to pick up speech patterns at all, I would really rather pick them up from a native speaker of the language, since at the very least I'll make the sorts of mistakes that a human might make. I want to sound like human, not like a language model. Language models sound like the average of several humans at best, and a strange program trying to imitate human speech at worst.

Once I'm fluent in a language, enough to recognize when the language model itself is probably making a mistake, then I might become comfortable using it. But not as my first introduction to the nuances of the language, when I'm still building my own internal representation up from scratch. After all, my goal is to converse with other human speakers. Shouldn't that be my personal training corpus? I'm a neural network too, and I don't want to feed myself bad data.