Show HN: Tine – Drive Wayland Around with Agents (opens in new tab)

(github.com)

4 pointstarboreus25d ago4 comments

So recently Anthropic came out with tools that let you drive Windows and Mac OS around. But I don't use Windows or MacOS, and was curious about some Wayland internals stuff and whether it would be possible to build something usable on a relatively restrictive platform like Wayland. So I made Tine.

Tine is a GNOME extension and CLI that lets an agent (I have used Claude but in theory any agent that can access the CLI) drive the desktop around using SPI trees (AT-SPI2), OCR, and visual fallbacks. Agent can do work with the a11y (AT-SPI2) trees, take screenshots, zoom in on a grid, click, enter text using a uinput device, and generally bumble their way around a Wayland Linux desktop.

This project would probably have been way easier in x11 but Wayland is teh future!!!111 Thanks for any thoughts and feedback and feels good to release something here after a decade of lurking. Decade plus but who's counting / I'm not old.

4 comments

aayushkumar12125d ago

Nice approach combining AT-SPI2 + grid fallback.

Have you run into issues where the a11y tree is incomplete (e.g. Electron apps)? Wondering how often the grid/OCR path becomes the primary path.

tarboreusOP25d ago

Yes. It works really well with Firefox. Chrome is basically a big blank, could expose the Chrome a11y tree but would have to turn on developer / debug mode or whatever which has downsides. It's really nice for Firefox and GTK apps and it does OCR + grid stuff for everything else.

aayushkumar12125d ago

That makes sense. Feels like the fallback path might end up being the default for a lot of real-world apps.

Have you thought about combining weak a11y signals + OCR to build more stable refs over time, or is that too brittle in practice?

1 more reply

j / k navigate · click thread line to collapse

4 comments

aayushkumar12125d ago

Nice approach combining AT-SPI2 + grid fallback.

Have you run into issues where the a11y tree is incomplete (e.g. Electron apps)? Wondering how often the grid/OCR path becomes the primary path.

tarboreusOP25d ago

aayushkumar12125d ago

That makes sense. Feels like the fallback path might end up being the default for a lot of real-world apps.

Have you thought about combining weak a11y signals + OCR to build more stable refs over time, or is that too brittle in practice?

1 more reply

j / k navigate · click thread line to collapse