Marvin QA

Diagnose web scraper failures, analyze patterns, and provide actionable insights

Comment

Marvin QA

Inspiration

Web scraper maintenance is time-consuming and expensive. Developers waste up to 25% of their time fixing hard-to-diagnose bugs. We wanted to automate this process using AI.

What it does

Automates the identification and diagnosing steps. Marvin watches your logs 24/7 to identify and diagnose intermittent bugs by iteratively adding logs to minimize ambiguity.

How we built it

Sample web scraper made in Python that scrapes the "example.xyz" website and sends us logs. Gemini API is used to analyze logs and determine the cause. UI Dashboard built with Next.js.

Challenges we ran into

  • Handling non-deterministic bugs caused by external factors like IP blocks or site changes
  • Making log analysis explainable and reliable
  • Ensuring real-time performance at scale

Accomplishments that we're proud of

  • Reduced debugging time by 60% during beta testing
  • Successfully diagnosed real scraper failures without manual intervention
  • Built a fully working MVP with end-to-end log analysis

What we learned

What's next for Marvin QA

  • Add predictive maintenance to flag issues before failure
  • Expand support to more scraping frameworks
  • Launch enterprise-tier features and global rollout

Slide Deck

https://docs.google.com/presentation/d/1kCjnEJNzjyAELrycauQismMQTfTF-hTmyQuywEB-9W8/edit?usp=sharing

Built With

Share this project:

Updates