Now, ChatGPT can see my screen. Should I build this out? [video] (opens in new tab)

(youtube.com)

3 pointskrschacht2y ago3 comments

3 comments

3 comments · 1 top-level

krschachtOP2y ago· 2 in thread

I'm trying to decide if it's worth cleaning up the code and publishing this. Is it something that would be interesting to others? It's still pretty buggy, in part because ChatGPT itself is buggy, particularly these past few days! But also, I have to regularly refresh the chrome tab when mic access or audio playback stops working. It's going to take some effort to make this thing more reliable. I need to better understand how the background page & service workers works with Chrome extensions.

This project started after I tried a bunch of Chrome plugins that let you speak to Chat GPT. The one I liked best was this one: https://github.com/C-Nedelcu/talk-to-chatgpt

I forked the code and refactored it quite a bit in order to improve the voice recognition, improve the quality of text-to-speech (using WellSaid) and then I added the screen capture capabilities. That's when it started feeling truly magical and useful to me.

The biggest issue is that ChatGPT is still too slow, but Sam Altman claimed during devday that improving the speed is the next biggest priority for them.

jzombie2y ago

I feel that the latest ChatGPT web UI updates were a bit rushed and are buggy, especially on mobile (I'm not using their mobile app). I hope they fix this before making any new UI features.

krschachtOP2y ago

I like the improvements they've made. They polished things up a bit. But it's hard to know if it's the UI that's buggy or if it's the server infrastructure. I've been getting lots of cloudflare errors and seeing random failed HTTP responses in console so I suspect it's just growing pains with the infrastructure.

j / k navigate · click thread line to collapse