Jarvis: A Voice Virtual Assistant in Python (OpenAI, ElevenLabs, Deepgram) (opens in new tab)

I implemented this! All local models. And I packaged it up so people can install it with one click: https://apps.microsoft.com/detail/9NC624PBFGB7

The speech recognition part needs work for sure, but when it works you can see the potential. It's very different from the way it feels to talk to Siri or even ChatGPT's voice mode. It won't be long before we are having real conversations with our computers.

TTS and STT models have decent support for streaming in chunks, but the accuracy drops the smaller the chunk size. Current state of LLMs are pretty limited in their ability to handle streaming inputs due to attention window constraints. There is some emerging research into attention sinks and caching initial tokens that look promising. I don't think we're quite there yet though.

You can do the "almost-realtime" part, all locally. I tinkered with a Python script for a few hours that used Whisper to speech-to-text, fed that into a local Mistral model (don't recall which), and then piped the output into text-to-speech.

It wasn't really streamed, though. Audio input was buffered, fully evaluated to a string, then fed into the LLM and the full text was converted back to audio.

The Whisper speech-to-text was pretty real-time, the LLM was not. I was barely scraping by on hardware specs, though.

Available as a phone line API (https://www.vocode.dev) and OS project (https://github.com/vocodedev/vocode-python)

This has happened already. It was maybe about 7 months ago and I believe it was a twitter link posted here. They took it further and streamed it to twilio to create a live phone call.

Any existing stream api for llm input?

IANAIPL but I find it difficult to believe that trademark is valid, considering it's never been used in trade. There is no Jarvis digital assistant software sold either fictionally or IRL. Even if the trademark were somehow upheld, I don't see how there could be any damages.

Trademarks are industry specific. If you made a fictional AI character named Jarvis and tried selling media based on that character THEN Marvel has a case. Creating a talking AI Assistant named Jarvis would be an expensive court case, which Marvel/Disney has to cash to pursue, but it would be a legal stretch with a lot of moneyed interests willing to back the non-Marvel/Disney side.

And just for those who felt the need to check:

> Computer application software that may be downloaded via global computer networks and electronic communication networks for use in connection with mobile computers, mobile phones, and tablet computers, namely, software for use as a voice controlled personal digital assistant

But there are many apps called Jarvis, so I am not sure how that is supposed to work?

What about Jenkins? oh yeah nope.

Um what about Jeeves? oh yeah nope.

Ok we need more butler names.

What about Smithers? Or Jeffrey?

Uh oh I hope I'm not in trouble

There's three fantastic niche players in the speech-to-text market right now that you should check out: - Deepgram (cheap and dirty, but accuracy quite poor) - Speechmatics (a bit more pricey, but fantastic accuracy) - Assembly AI (just announced Series C funding of $50m)

Exactly, I don't think this project uses VAD for pausing LLM generation or interruptions in general which is key to good assistant interactions

Deepgram advertised itself as being the fastest, and I wanted to focus on limiting response delay so I chose it. I hope I did not get misled.

Nice work! Very impressed.

Do you happen to know anything about any open source voice identification software?

I’ve noticed with ChatGPT voice and any other voice driven assistant that a massive problem is the background voices and noise. One solution could be advanced pre-processing to ID your voice only.

Another idea I’ve had is using something professional with PTT:

https://sheepdogmics.com/products/quick-disconnect-mic-tubel...

Problem with this is Deepgram's accuracy (but agree their speed/latency is excellent). We used to use them too, but eventually we got so frustrated with poor accuracy we switched to Speechmatics - would definitely recommend checking them out.

Great! What do you guys have in mind in terms of products using these tools. Yeah unfortunately it's hard to shave on latency.

How did you find these? I was literally looking for tutorials all day long and could not find something. These projects look insane!

Exactly my thought, I was like "Jarvis has got to be just a 2030 version of an LLM".

Yeah I actually considered making a spotter AI using computer vision in a game like ARMA 3 or Squad but kind of difficult. I made a spotter for ground vehicles on aerial imagery using YOLOv5 here: https://github.com/AlexandreSajus/Military-Vehicles-Image-Re...

There's a French defense company, Preligens, that actually does this currently

That was exactly my thought haha, I want Jarvis at home. You could easily modify my code to run a local LLM instead

Yeah I had the same issue so I used (stole) this answer on StackOverflow: https://stackoverflow.com/questions/46734345/python-record-o... Basically there's a library that records until it detects a silence