WebArena and Web Bench get compared because both evaluate browser agents on “realistic” websites. The resemblance ends there.
WebArena asks: Can an agent complete long-horizon intents in a frozen, self-hosted world that looks like shopping, Reddit, GitLab, and maps?
Web Bench asks: Can that same class of agent read and write on today’s public internet—through logins, captchas, proxies, and UI churn?
Those are both important. Treating a WebArena win as a live-web win is how teams ship demos that die in production.
The numbers people look up
| WebArena | Web Bench | |
|---|---|---|
| Sites | Self-hosted clones of a few site families | 452 live websites in the open set |
| Tasks | 812 long-horizon intents | ~2,454 open tasks (from ~5,750 feasible) |
| Grading | Programmatic functional checks | Human trajectory review in the launch study |
| Strength | Reproducibility and paper comparison | Real-web difficulty, including writes |
| Weakness | Misses captcha, proxy, and day-to-day drift | Sites change; some tasks go stale |
WebArena task count: 812. That is the usual answer to “how many tasks does WebArena have?”
What WebArena got right
Before WebArena, web-agent papers were hard to compare. Everyone had a different toy site or a different scrape of the live web. Zhou et al. shipped Dockerized environments with real open-source software underneath, authentic-scale content, and outcome-based grading: did the repository update, did the cart contain the right item—not “did the agent emit the golden click path.”
That design is why WebArena became the default academic yardstick. If you are ablating memory, observation format, or planner structure, you want that kind of control.
Where controlled clones stop being enough
When we ran agents on live sites for Web Bench, a different failure distribution showed up:
- Proxy and geo blocks
- Captcha / bot detection mid-trajectory
- Login and authentication friction that does not exist in a lab account model
- UI and content changes that invalidate tasks between dataset creation and eval
None of that is WebArena’s fault. It is simply out of scope. A benchmark that freezes the world cannot measure hostility from the world.
We also care about writes. Web Bench tags tasks as READ, CREATE, UPDATE, DELETE, and file manipulation. Launch results were lopsided: strong agents often cleared >70% of READ tasks, while the best fully automated NON-READ score was Skyvern 2.0 at 46.6%. Overall fully automated SOTA was Anthropic Computer Use at 66.0%. Extraction looking “solved” while form-heavy workflows remain half-broken is exactly the gap product teams feel—and exactly what a read-skewed live suite can hide.
How to use both without lying to yourself
A pattern that works:
- Develop and ablate on WebArena (or BrowserGym / WorkArena for enterprise UI).
- Train in simulators when you need RL volume—Westworld exists for that reason.
- Gate releases on Web Bench if the agent will touch the public web.
- If live scores collapse, run BrowserBench before you rewrite the planner. Infrastructure alone moved Web Bench accuracy by large margins in our study.
WebArena success means the agent can think through realistic UIs. Web Bench success means that skill still works when the website is allowed to fight back.
For the broader map of evals, see Browser agent benchmarks in 2026. Full Web Bench methodology and charts live in the launch post.