undefined | Better HN

0 pointsalbert_e1y ago0 comments

There is SSML for speech markup to indicate various characters of speech like whispers, pronunciation, pace, emphasis, etc.

With LLMs proving to be very good at generating code, it may be reasonable to assume they can get good at generating SSML as well.

Not sure if there is a more direct way to channel the interpretation of the tone/context/emotion etc from prose into generated voice qualities.

If we train some models on ebooks along with their professionally produced human-narrated audiobooks, with enough variety and volume of training data, the models might capture the essence of that human-interpretation of written text? Just maybe?

Amazon with its huge collection of Audible + Kindle library -- if it can do this without violating any rights -- has a huge corpus for this. They already have "whispersync" which is a feature that syncs text in a kindle ebook with words in corresponding audible audiobook.

0 comments

1 comments · 1 top-level

micw1y ago

Good points, thank you! I just tested it. While ChatGPT was very good in adding generic (textual) annotations, the result for generating SSML where very poor (lack of voice names, lack of distinction between narrator and character etc).

Probably the results with a model trained for this plus human audit could lead to very good results.

j / k navigate · click thread line to collapse