tmcneal

Three years ago we launched Reflect (https://reflect.run), our no-code end-to-end testing platform, on HN. We're back to show you some new AI-powered features that we believe are a big step forward in the evolution in automated end-to-end testing. Specifically, these features raise the level of abstraction for test creation and maintenance.

One of our new AI-powered features is something we call Prompt Steps. Normally in Reflect you create a test by recording your actions as you use your application, but with Prompt steps you define what you want tested by describing it in plain text, and Reflect executes those actions on your behalf. The other feature we're launching is a fallback to using AI to find an element when the selectors we generated are no longer valid. We're making both of these features publicly available so that you can sign up for a free account and try it for yourself. Here's a link to our docs which contains a video demonstrating this feature:

https://reflect.run/docs/recording-tests/testing-with-ai/

Our goal with Reflect is to make end-to-end tests fast to create and easy to maintain. A lot of teams face issues with end-to-end tests being flaky and just generally not providing a lot of value. We faced that ourselves at our last startup, and it was the impetus for us to create this product. Since our launch, we've improved the product by making tests execute much faster, reducing VM startup times, adding support for API testing, cross-browser testing etc, and doing a lot of things to reduce flakiness, including some novel stuff like automatically detecting and waiting on asynchronous actions like XHRs and fetches.

Although Reflect is used by developers, our primary user is non-technical - someone like a manual tester, or a business analyst at a large company. This means it's important for us to provide ways for these users to express what they want tested without requiring them to write code. We think LLMs can be used to solve some foundational problems these users experience when trying to do automated testing. By letting users express what they want tested in plain English, and having the automation automatically perform those actions, we can provide non-technical users with something very close to the expressivity of code in a workflow that feels very familiar to them.

In the testing world there's something called BDD, which stands for Behavior-Driven Development. It's an existing way to express automated tests in plain English. With BDD, a tester or business analyst typically defines how the system should function using an English-language DSL called "Gherkin", and then that specification is turned into an automated test later using a framework called Cucumber. There are two main issues that we've heard a lot when talking to users practicing BDD:

1. They find the Gherkin syntax to be overly restrictive.

2. Because you have to write a whole bunch of code in the DSL translation layer to get the automation to work, non-technical users who are writing the specs have to rely heavily on the developers writing the DSL translation layer.

We think our approach solves for these two main issues. Reflect's prompt steps have no predefined DSL. You can write whatever you want, including something that could result in multiple actions (e.g. "Fill out all the form fields with realistic values"). Reflect takes this prompt, analyzes the current state of the DOM, and queries OpenAI to determine what action or set of actions to take to fulfill that instruction. This means that non-technical users who practice BDD can create automated tests without developers having to build any sort of framework under the covers.

It's still early days for this technology, but we think our coverage of use cases is wide enough that this is now ready for real-world use. We're excited to launch this publicly, and would love to hear any feedback. Thanks for reading!