Benchmarking Web Agents
in Live Environments

We embrace evaluation in live web environment by evaluating key node completion of each task. We are running an end-to-end system to constantly evaluate the accessibility and reliance of the workflow with regard to both the action execution and the reward signal verification in an optimized cost.

We develop a user-friendly web plugin and platform that simplifies the creation, testing, and sharing of custom challenges. This tool allows users to construct new datasets by capturing web-based human action sequences, store this data, and annotate it with reward signals for evaluating task completion.

We hold that clearer data visualization enhances comprehension and uptake of datasets and benchmarks. Our approach includes using screenshots and recordings to depict workflow sequences and sharing in-depth analytics with the community, such as task difficulty ratings based on collective capability, as well as comprehensive metrics like task completion, efficiency, and overall scores for comparing agents.

We welcome all forms of community involvement, from submitting datasets and challenges to reporting bugs related to task or reward validation. Additionally, we are open to suggestions for new reward functions, wider data adoption, enhancements to our recording tools, and any other constructive feedback that benefits the community.