jonasnelle

Hey HN, we're Alexi and Jonas the co-founders of Autotab (https://autotab.com). Autotab is a chrome-based browser you can teach to do complex tasks, with a simple API for running them from your app or backend.

Here is a walkthrough of how it works: https://youtu.be/63co74JHy1k, and you can try it for free at https://autotab.com by downloading the app.

Why a dedicated editor?

The number one blocker we've found in building more flexible, agentic automations is performance quality BY FAR (https://www.langchain.com/stateofaiagents#barriers-and-chall...). For all the talk of cost, latency, and safety, the fact is most people are still just struggling to get agents to work. The keys to solving reliability are better models, yes, but also intent specification. Even humans don't zero-shot these tasks from a prompt. They need to be shown how to perform them, and then refined with question-asking + feedback over time. It is also quite difficult to formulate complete requirements on the spot from memory.

The editor makes it easy to build the specification up as you step through your workflow, while generating successful task trajectories for the model. This is the only way we've been able to get the reliability we need for production use cases.

But why build a browser?

Autotab started as a Chrome extension (with a Show HN post! https://news.ycombinator.com/item?id=37943931). As we iterated with users, we realized that we needed to focus on creating the control surface for intent specification, and that being stuck in a chrome sidepanel wasn't going to work. We also knew that we needed a level of control for the model that we couldn't get without owning the browser. In Autotab, the browser becomes a canvas on which the user and the model are taking turns showing and explaining the task.

Key features:

1. Self-healing automations that don't break when sites change

2. Dedicated authoring tool that builds memory for the model while defining steps for the automation

3. Control flows and deep configurability to keep automations on track, even when navigating complex reasoning tasks

4. Works with any website (no site-specific APIs needed)

5. Runs securely in the cloud or locally

6. Simple REST API + client libraries for Python, Node

We'd love to get any early feedback from the HN community, ideas for where you'd like the product to go, or experiences in this space. We will be in the comments for the next few hours to respond!

Hi HN,

We’ve built a desktop app to create highly reliable AI agents that use a computer with mouse and keyboard.

Until last week, we had tried many different approaches to open-ended agentic features but none of them had met our reliability bar.

With Anthropic’s Computer Use this finally changed, and we just shipped a feature we’re calling Instruct. Instruct allows users to create agentic blocks as part of a larger Autotab skill that provides the structured logical flow to keep the automation on track.

If you haven’t had a chance to try Computer Use yet, it is an impressive leap from the last generation of vision models (e.g. gpt4o struggles with relative positions, let alone coordinates). At the same time, it is still not good enough to be given a prompt and let loose.

One of the big surprises to us early on was just how much intent specification is required for most real world workflows to run reliably. What looks at first like a simple form filling task usually turns out to have dozens of edge cases and super specific, hidden rules. Even human employees need to be shown how to perform these tasks, and then refined with question-asking + feedback over time.

We wanted to build a tool for specifying intent, and iterating with the model to make it reliable enough for real work.

- Automations run on top of an action scaffold, which works kind of like a very fuzzy programming language with strict types. This gives the model a high level plan that guides execution, and makes it easy to break out discrete steps to get the reliability you need. (Interestingly, this has also proven useful not just for the agent, but also for the human trying to create, verify and edit the automation.) - When the model is unsure it asks for clarification. For example, if you are in editing mode and the model thinks that an element looks meaningfully different than before, it will ask you to verify that it is the same element. - The agent has access to a memory system that lets it recall information from past runs as well as instructions and feedback from the user.

Here's a short video of Autotab Instruct in action: https://www.loom.com/share/ccf4e9d8c798450da3324a6cff024971?.... There are a few more demos at https://twitter.com/autotabai/status/1852393973165199425 a75f06f82cab521bc78672ed35d85e8a.

We’d love to hear what you think!