That doesn't factor in line delivery. You can have the words say/mean one thing (e.g. "I'm fine.") and the delivery say/mean another (defensive, distraught, etc.).
It also does not account for where stresses, emphasis, pauses, etc. are placed to enhance the delivery of a given text.
How do you get sentiment analysis to properly annotate an audiobook that has a dramatic reading, or something akin to the narration of the Game of Thrones or Harry Potter books where the narrators switch characters, accents, manarisms to portray the written content?