awesome project though! How did you build the voice encoder for each character?
This stuff has gotten a lot more accessible. You can download Tacotron and give it a try.