spruce_tips

I've tried it, and wasn't really impressed. Curious if and how teams/individuals here are actually using it, and why!

Like many of you, I'm streaming responses token by token from LLMs using server sent events (SSEs). What's the best way to maintain the SSE connection through a page refresh?

I haven't seen a lot of documentation or examples covering this. In most LLM enabled apps I've used, if tokens are currently streaming and the page refreshes/changes, the stream gets interrupted.

One idea I had was writing the streamed tokens into some sort of queue or kafka topic, then connecting my UI to the queue and streaming tokens from there instead. But that seems like a lot of work.

How are most folks doing this?

Like many of you, I'm streaming responses token by token from LLMs using server sent events (SSEs).

What's the best way to maintain the SSE connection through a page refresh?

I haven't seen a lot of documentation or examples covering this. In most LLM enabled apps I've used, if tokens are currently streaming and the page refreshes/changes, the stream gets interrupted.

One idea I had was writing the streamed tokens into some sort of queue or kafka topic, then connecting my UI to the queue and streaming tokens from there instead. But that seems like a lot of work.

How are most folks doing this?