If You've ever tried to create tests that run with an LLM on mobile, probably encountered context problems due to the accessibility view being too long or just sending a screenshot to the LLM, which has low accuracy.
Cognisim:
1. Turns the accessibility into a more LLM parseable form for your agents, and uses set of mark prompting to combine the screenshot and text for more accuracy. We create a grid of the screen so the text representation also has some form of visual structure to it.
2. Implements a simple API to do interactions on IOS and Android
3. Maps LLM responses back to accessibility components via a dictionary that maps id -> accessibility component
Pretty simple, but we hope you get some value from it :)
We run CogniSim in production for thousands of mobile UI tests a day at Revyl. If you are interested in proactive observability - combining resilient end to end tests with open telemetry traces, give us a shout - revyl.ai
No comments yet.
