Odysseys2: a long-horizon web agent benchmark

Carnegie Mellon University and Browser Use released Odysseys2, a benchmark of 100 long-horizon tasks on the live web, each graded on 8 to 12 pass/fail checks. The best agent, GPT-6 Astra, fully completes 47 of 100 tasks with a mean rubric score of 84.6, while no other agent reaches 20 and five complete none. The authors note the LLM judge has not been validated against a human reference, reward-hacking flags are model judgments, and the live web changes.