Introducing the FrontierCyber benchmark: Irregular’s new approach to advanced offensive-cyber evaluations. It measures AI models’ offensive skills on real systems, including mobile devices, hosted software services, databases, and networks.
We evaluated @AIatMeta's Muse Spark 1.1 across our private benchmark suites, Atomic Tasks for discrete technical skills and CyScenarioBench for end-to-end operations. The model showed clear gains over Muse Spark 1.0, with strong results on bounded tasks and its first
We ran preliminary evaluations of GLM-5.2, an open-weight model released in June 2026, on a limited, internal suite of vulnerability research tasks.
Early results indicate performance comparable to GPT-5.4 and Claude Opus 4.6, released roughly four months earlier, on the subset
We worked with @OpenAI to evaluate GPT-5.6 Sol, including the first deployment of FrontierCyber as part of a frontier model assessment with a partner. FrontierCyber measures offensive-cyber capability on real, off-the-shelf systems, with no planted vulnerabilities and no
GPT-5.6 Sol demonstrated capability slightly stronger than GPT-5.5. It discovered vulnerabilities more consistently than it could compose them into reliable attack paths under production defenses, with clear limitations against hardened targets and over long horizons.
Initial evaluations are already surfacing previously unknown vulnerabilities, now moving through responsible disclosure. For example, a model built a novel multi-vulnerability chain to gain unauthorized access to private information on a widely used mobile device.
At @ManGroup's Technology Offsite this week, our CEO @dan_lahav gave the keynote on frontier AI security risk as a category of its own, alongside classical cybersecurity.
The tools we defend networks with were built for systems that follow rules. AI systems reason toward a goal,
Most 'world-changing' AI ideas are about what these systems can do. Ours is about whether you can trust them to do it. We’re proud to be a winner on @FastCompany’s 2026 World Changing Ideas list, alongside the labs and teams betting on getting this right.
We’re happy to share that CyScenarioBench, our benchmark for offensive cyber operations, was used by @AnthropicAI to test Claude Mythos 5 and Claude Fable 5.
Most current cybersecurity evaluations check isolated skills, such as vulnerability research or exploitation.
The New York Times covered new research from the University of Toronto on AI-powered worms.
Speaking to @nytimes, our CEO @dan_lahav highlighted the gap between lab demonstrations and real-world cyber impact: reliability, complexity, and defenses.
At Irregular, we work on
