agreed! when it comes to ultra-long horizon execution, there is definitely a gap between model capabilities (in terms of context management and memory augmentation) that is currently buried by leaderboard scores
This paper shows why I am skeptical of the idea that frontier models can be easily replaced by cheaper ones (and why you should use the best models as orchestrators): Small improvements in accuracy & judgement compound over long-horizon tasks, creating huge gains in outcomes.
