Llasa: Llama-Based Speech Synthesis (opens in new tab)

> Interesting that there isn't a mention of Orpheus as prior art either

Llasa-3b (https://huggingface.co/HKUSTAudio/Llasa-3B) came out before Orpheus (https://huggingface.co/canopylabs/orpheus-3b-0.1-ft).

> it's the exact same thing.

They're very similar, but they're not the exact same thing.

Llasa uses xcodec2, a much simpler, lossless 16khz wav codec. This makes it superior for one-shot voice cloning.

Orpheus' 24khz snac codec is lossy which makes it difficult to use for zero-shot cloning as the reference audio gets degraded during tokenization. You can test this here: https://huggingface.co/spaces/Gapeleon/snac_test

But when finetuned on 50+ audio samples, it produces much cleaner 24khz audio than Llasa, and the snac model is much easier to run on consumer hardware than xcodec2 (87t/s for realtime speech, which can be achieved on an RTX3080 for example)

Oh, sure, for the well-known models that are already on there.

I just wish that new research would always spell it out in full instead of these silly block diagrams labelled with just e.g. "Cross Attention" and not the exact parameters, number of heads, layer sizes, etc.

Also some of these diagrams use a + for concatenation and some use it for addition, that's another headache to figure out, having layer sizes would make it clear.

If you're doing a home lab voice assistant 1B is nice, because on a 12gb gpu you can run a moderately competent 7b LLM and two 1b models; 1 for speech to text and also text to speech, plus some for the wake word monitor. Maybe in a couple of years we can combine all this into a single ~8b model that runs efficiently on 12gb gpu. Nvidia doesn't seem very incentivized right now to sell consumer GPUs that can run all this on a single consumer grade chip when they're making so much money selling commercial grade 48gb cards.