Is cost per token even a relevant metric?
Not really.
What actually matters is cost per task:
cost per task = tokens used × price per token
A model cheaper per token can cost more if it uses more tokens or needs retries.
Unfortunately this is not tracked by most benchmarks.
