Show HN: Open source framework OpenAI uses for Advanced Voice (opens in new tab)

As costs of humanlike communications decrease, so will Sybil attacks and spam.

The IRS is notorious for resistance to tech change, don't be surprised if they unplug the phones and force you to walk in to ask your question.

What is the value add here? Save sometime for technocrats and technoadjacents for a whole of 3 years before victims of spam adapt?

Also this has been solved already just mail your question like the rest of mortals.

They'll just put up "captchas" or whatever.

The point if phone lines is to waste the client's time. Not to have the client waste their time.

We do this exact thing at getvibrato.com. You can schedule calls like these, or even do more advanced automation with Zapier.

> LiveKit is too expensive

Most of it is open source, especially the clients, although they do feel quite ad hoc hacked together (a possible side effect of WebRTC evolution).

Would totally agree on OpenAI moving away. The description of the agent here sounds like a big hack just to get around the fact temporarily the model server expects audio over sockets instead.

Field CTO — hi @Sean-Der :wave:

Fractional CTO sounds like a disaster lol

Currently it does: all audio is sent to the model.

However, we are working on turn detection within the framework, so you won't have to send silence to the model when the user isn't talking. It's a fairly straight forward path to cutting down the cost by ~50%.

You dont get charged per hour with the openai realtime api, only for tokens from detected speech and response

There’s a ton of complexity under the “relatively simple use case” when you get to a global, 200M+ user scale.

I had no idea! <3 Thank you for sharing this, made my weekend.

You guys are honestly the best

Yes. Kyutai released an opened model called moshi : https://github.com/kyutai-labs/moshi

There's also llama-omni and a few others. None of them are even close to 4o from an LLM standpoint. But moshi is called a "foundational" model and U'm hopeful it will be enhanced. Also there's not yet support for those on most backends like llamacpp / ollama etc. So I'd say we're in a trough but we'll get there.

There’s Ultravox as well (from one of the creators of WebRTC): https://github.com/fixie-ai/ultravox

Their model builds a speech-to-speech layer into Llama. Last I checked they have the audio-in part working and they’re working on the audio-out piece.

When I asked advanced voice mode it said that it receives input as audio and generates text as output.

I'm working on a PR now :)

Both input and output are audio. This post is about bridging WebRTC audio I/O with an API that itself operates on simple TCP socket streams of raw PCM. For reliability and efficiency you want end users to connect with compressed loss-tolerant Zoom-style streams, and that goes through a middleman which relays to the model API.

Who did you ask? ChatGPT? Not sure if you understand LLMs but its knowledge is based on the training data, it can't reason about itself, it can only hallucinate in this case, sometimes correctly, most times incorrectly.

It's using the same model/engine. I don't have knowledge of the internals, but a different subsystem/set of dedicated resources though for API traffic versus first-party apps.

One thing to note is there is no separate TTS-phase here, it's happening internally within GPT-4o, in the Realtime API and Advanced Voice.

We had our playground (https://playground.livekit.io) up for a few days using our key. Def racked up a $$$$ bill!