WebSP-Eval: Evaluating Web Agents on Website Security and Privacy Tasks
Authors: Guruprasad Viswanathan Ramesh (University of Wisconsin-Madison), Asmit Nayak (University of Wisconsin-Madison), Basieem Siddique (University of Wisconsin-Madison), Kassem Fawaz (University of Wisconsin-Madison)
Volume: 2026
Issue: 4
Pages: 666–702
DOI: https://doi.org/10.56553/popets-2026-0140
Artifact: Available, Functional
Abstract: Web agents automate browser tasks, ranging from simple form completion to complex workflows like ordering groceries. While current benchmarks evaluate general-purpose performance~(e.g., WebArena) or safety against malicious actions~(e.g., SafeArena), no existing framework assesses an agent's ability to successfully execute user-facing website security and privacy tasks, such as managing cookie preferences, configuring privacy-sensitive account settings, or revoking inactive sessions.
To address this gap, we introduce WebSP-Eval, an evaluation framework for measuring web agent performance on website security and privacy tasks. WebSP-Eval comprises 1) a manually crafted task dataset of 200 task instances across 28 websites; 2) a robust agentic system supporting account and initial state management across runs using a custom Google Chrome extension; and 3) an automated evaluator. We evaluate a total of 8 web agent instantiations using state-of-the-art multimodal large language models, conducting a fine-grained analysis across websites, task categories, and UI elements. Our evaluation reveals that current models suffer from limited autonomous exploration capabilities to reliably solve website security and privacy tasks, and struggle with specific task categories and websites. Crucially, we identify stateful UI elements are a primary reason for agent failure, with toggles causing more than 45% task failure across many models.
Keywords: web security and privacy, web agents, web agent frameworks
Copyright in PoPETs articles are held by their authors. This article is published under a Creative Commons Attribution 4.0 license.
