"The two hardest problems in computer science are: (i) people, (ii), convincing computer scientists that the hardest problem in computer science is people, and, (iii) off by one errors." - @jeffbigham
Massive shoutout to @hjalexnam for driving this from day one and is an absolute rockstar. She had this brilliant idea of self-evolving benchmarks, and we show multi-agent evolutionary search can auto-generate challenging new tasks and diagnose failures at scale.
Evaluating tool-use capabilities is expensive and domain-dependent. Once existing test data become saturated, expert humans need to design new tasks. Alternatively, LLMs can auto-generate challenging tasks from existing toolsets and task primitives through evolutionary search
So basically the ultimate goal of RSI is to build a Schmidhuber agent that auto-discovers every algorithm first, just so it can quote-tweet other agents' breakthroughs with 'I actually did this in 2027'?
