That idea has stuck with me ever since and I've tried ways of both disproving and affirming it, now I'm mostly in the "affirming it" phase. As a quick example, one can just think of a sentence like "I love you", which means different things for different people, even if spoken in the same language, and, most important of all, it might mean (and in many cases it does mean) different things to different people who have lived together for decades (as husband and wife, for example, or as parents and children). On the other hand, more "neutral" sentences like "turn off the lights" seem to mean the same thing for a larger proportion of speakers.
All this ends up being a centuries long conversation, and the current reference is merely the latest overlay. For instance, Wittgenstein mentioned that one must "throw away the ladder after he has climbed up it", riffing on Schopenhauer's scorn of those who carried books on their backs instead of leaving them behind as climbed ladder steps, and apparently it was Plato who spread the idea that there was a knowledge ladder to be climbed ...