Stop Words as Social Signals (a result that goes against IR common knowledge) (opens in new tab)

Amazing that [to, to, on, my, to, on, be, on, to, to ... etc] can reveal so much. One of the linked articles ( http://homepage.psy.utexas.edu/homepage/faculty/pennebaker/r... ) describes how the relative frequency of the use of "I" in communication is a powerful indicator of status (less "I" => higher status).

One of the two linked papers (AnnualReview.pdf) discusses formal settings, including those with disparities in power:

'Brown & Levinson’s (1987) polite-ness theory takes into account an individual’s efforts to preserve the “face(s)” of others with whom one communicates. For example, they propose impersonalizing the speaker and hearer by avoiding the pronouns I and you, using past tense to create distance and time, diminishing the force of speech by using hedge words such as perhaps, using slang to convey ingroup membership, and using inclusive forms (we and let’s) to include speaker and hearer.'

He seems to be claiming that in the phrase I don't feel comfortable signing up for a $7,000 extra flight without talking to you guys , it's the word to that signals the manager/subordinate relationship. It seems to me that ... without talking ... you is the best clue.

In English, subjunctive case usually requires a to, and people asking permission use the subjunctive a lot. Is that the signal he's finding? Is that the best NLP can do?

I was always uncomfortable with stopwords. Ignoring some data smells like overfitting, or perhaps a naïve algorithm (as in, it has no concept of structure – which is often the case, for computational reasons).

That said, the last time I dabbled in IR, my algorithm choked on stopwords. I should revisit it.

Good Lord that is counterintuitive!

See also SpotSigs ("...stopwords may however be very good indicators of the actual interesting parts of a web page...")

http://infoblog.stanford.edu/2008/08/spotsigs-are-stopwords-...

Outside the old post-2001 TIA/NSA efforts, what can this be used for?

Maybe for determining social hierarchies for mind share in product recommendations or advertising?

Yeah, I can't think of one noncreepy way to use this.

I think the following phrase is unnncessary "a result that goes against IR common knowledge"

IR aims to retrieve information and not sentiments from the text. This kind of work is related to sentiment analysis, where use of stops words as signal is common. Thus the whole argument that some holy criteria in IR has been shown to false is clearly incorrect. Even original authors don't make such assertions.