MobileMunchkin | Devpost

Inspiration

I used to be an iOS dev, unit test is easy but Mobile UI flow testing is hard. So many features are buried deep in the many layers of UI interaction. We wanted to create an automated way for such tests to be generated and applied to your mobile app.

What it does

Take an app codebase + an input text on what you want to test. It generates a series of commands to be passed in the mobile testing tool, you can optionally use multiOn to navigate the testing tool to see each test step being applied.

How we built it

We used three tools: GPT vision Maestro MultiOn Given a mobile app codebase, given a goal provided by the eng of the mobile app, we pass the screenshot of the mobile app and the goal into GPTVision to generate a list of steps; We then pass these steps into Maestro AI to generate executable yaml UI flows through multiOn. We see on the simulator that the test gets executed.

Challenges we ran into

Running into rate limiting issue with Maestro AI
Interfaces with the unfamiliar gradle build system to build Maestro’s UI client locally. (Helped people online do this too)
Prompting with GPTVision to get it to output the right commands
Multion was great some of the time but it was not being reliable; Reproducibility burned us a lot of time. Spent a lot of time iterating on the prompt.

Accomplishments that we're proud of

Using gradle Hacked the dom to make it be able to run window level actions Got multiOn to work with our system end to end through iterating on the prompts a lot