One interesting tell that I might have heard in Jamie Johnson's documentary
"Born Rich" was talk of holidays.
If you talk about holidaying with one's family in some exotic locale or of having a summer home in the Hamptons, the person you're speaking with can surmise that you're not poor.
The rich kids in that documentary also regularly go out to bars and think nothing of spending hundreds or even thousands of dollars on champagne.
It's definitely not just about how one talks that signals to others that one is part of the upper class.. but once one is established as in the upper class, then things such as one's manner of speech might distinguish you.