PrimBench: Diagnosing Cognitive Primitives of Web Agents via Environment Interventions

A diagnostic benchmark that decomposes web-agent competence into seven cognitive primitives and stresses each one through controlled environment interventions. Seven interactive environments, 519 base tasks, 530 intervention variants, deterministic scoring against server state.

1. Motivation

Existing web-agent benchmarks score whole-task completion. A run either succeeds or fails, and a single scalar gets reported. The scalar is useful for ranking models, yet it carries almost no diagnostic information. When an agent finishes a 15-step checkout with score 0, the log does not say whether the agent misread a product card, accepted a forged "Saved" toast, abandoned a retry after a transient 503, or walked past the only remaining affordance. The benchmark does not know, and the training signal that follows does not know either.

What we actually want to measure is a set of cognitive primitives: the atomic capabilities a web user exercises across every domain. A competent user grounds an observation, plans sub-goals, tracks state, backtracks on a bad decision, waits out a flaky request, explores when the obvious path disappears, and verifies that an action took effect. A benchmark that exposes each primitive independently lets a developer read a model profile, identify the weakest axis, and target training at that axis. Whole-task benchmarks flatten this profile into a single number and obscure the gradient.

PrimBench addresses the gap through environment interventions. We take a well-formed base task, apply a controlled perturbation that stresses exactly one primitive, and grade the outcome against the same success criterion as the baseline. A drop between baseline and intervention isolates the failing primitive, because everything else in the environment is held fixed. The intervention is the experimental manipulation; the primitive is the dependent variable.

2. The Seven Cognitive Primitives

We settle on seven primitives. Two constraints guided the choice: the set should cover every failure mode reported in recent agent studies, and the primitives should not overlap semantically, so that a failure on a targeted intervention cannot be attributed to more than one primitive.

Coverage and Non-Overlap

Each variant names exactly one target_primitive. The catalog enforces |task.primitives ∪ {variant.target_primitive}| ≤ 2, which caps confounding at the (task, variant) level. A failure on a verification-targeted variant therefore reflects verification, holding the grounding demand of the base task fixed. The table below maps each primitive to the primary action classes that stress it and shows that each action class targets a disjoint primitive.

3. How We Intervene: Four Layers

An intervention is not a single mechanism. A phishing email and a 503 retry and a swallowed click are all interventions, but they live at different places in the web stack and fire at different times. Collapsing them into one hook would either paper over real mechanisms (DOM mocks of network failure miss real HTTP timing) or miss mechanisms entirely (a middleware has no view of DOM occlusion). We therefore split the perturbation space by where each action fires.

Why Four Layers Are Needed

When Each Layer Fires

Worked Example: A Patience Intervention

4. Catalog Map

The full catalog binds base tasks, intervention variants, and primitive coverage across seven environments.

Tasks and Variants per Environment

Intervention Families by Layer

We group dispatch branches into intervention families: each family captures one stressor mechanism, and the branches inside it are environment-specific specialisations that impose the same cognitive load. A family is the right unit for reasoning about primitive coverage; the dispatcher split matters only at the implementation level.

Seed Families injector/seed.py

Server Families injector/server.py

Network Families injector/middleware.py

Client Families BenchmarkToolbar.tsx

Families are a conceptual grouping. The underlying 48 dispatch branches exist for two reasons: (1) some families have environment-specific specialisations that share a mechanism but read from different state collections (add_decoy_notifications and add_noise_orders both inject noise, but one writes to the notification list and the other to the orders list); (2) the delay family collapses six behaviour modes into a single dispatch. One infrastructure branch, set_feature_flag, is excluded from both counts: it toggles a window.__wabFeatureFlags boolean and imposes no cognitive load on its own.

Primitive Coverage Across Environments

Every primitive has at least one variant in every environment. This lets a user compute a primitive-by-environment score matrix from a single evaluation run.

Design Invariants

5. Evaluation System

Scoring runs against live environment state rather than the DOM. The evaluator reads the Pydantic state objects the real API handlers mutate, so a silently-dropped write is caught even when the UI shows a success toast. Every task declares a canonical_diff block, and matching runs through webstress/eval_core/. (The legacy expression-based eval.checks path was removed in the 2026-04 refactor; all 519 tasks now carry a canonical_diff.)

Pipeline

# webstress/eval_core/orchestrator.py
def evaluate(task, server_state, targets, trajectory):
    canonical = task.canonical_diff              # missing_canonical_diff → score 0
    initial   = server_state._initial_state_copy  # deep-copy taken at session create

    agent_diff = compute_diff(initial, server_state)   # eval_core/diff.py
    report     = match_diff(agent_diff, canonical,        # eval_core/matcher.py
                            targets, initial, server_state, session_start)

    return {
        "score":       report.score,
        "success":     report.passed,
        "checks":      report.checks,             # positive match results
        "negative_checks": report.negative_checks, # invariants + constraints + collateral
        "failures":    report.failures,
        "collateral":  state.compute_collateral(initial),  # analytics only
        "bijection_graphs": report.bijection_graphs,
    }

Five Grammar Sections of canonical_diff

Two auxiliary structures extend the grammar: oneof wraps multiple acceptable alternatives at the block level and the matcher picks the best-scoring applicable alternative; named_invariants attach human-readable names and severities (critical / high / medium / low) to the invariant list.

# Task YAML (amazon_add_single_item.yaml, abridged)
canonical_diff:
  create:
  - entity: CartItem
    desc: Target product added to cart with quantity 1
    properties:
      product_id: {expr: "x == target['product_id']"}
      quantity:   {eq: 1}
      product_name: {any: true}
  invariant:
  - collection: state.cart_items
    filter: "a.product_id != target['product_id']"   # preserve everything but the target
    preserve: ALL
  named_invariants:
  - {name: Agent did not place orders, ref: invariant[4], severity: high}

Predicate Grammar (19 keys)

Properties of a create/update are matched by predicates, each a single-key mapping. eval_core/predicates.py

Automatic Collateral Sweep

After matching the explicit sections, the matcher scans every unmatched diff entry and penalises the agent for uncovered state mutations:

The agent is not required to enumerate every side-effect explicitly; a collection-level invariant blocks the sweep. The sweep exists to catch overreach: a successful "add to cart" that also silently writes a review or cancels an order loses score.

Score Formula

eval_core/matcher.py Severity table: critical 0.30, high 0.20, medium 0.15, low 0.10.

if total_weight > 0:                 # create/update/delete present
    raw_score = passed_weight / total_weight
elif constraints_total:              # constraint-only task
    raw_score = constraints_passed / constraints_total
else:
    raw_score = 1.0

penalty_total = sum(nc.penalty for nc in negative_checks if not nc.passed)
score         = max(0.0, min(1.0, raw_score - penalty_total))
success       = not failures     # any hard failure → not success

6. Evaluated Agents

We evaluate eight frontier models under two action surfaces: a vision harness (BrowserGym observations, screenshot plus accessibility tree) and a text harness (Browser Use, DOM-structured text actions). The same variant and seed is used for both surfaces, so the cross-surface gap isolates the contribution of visual grounding.

All models are called through their chat API. The vision and text harnesses differ only in how observations are presented and which action verbs are available; the variant catalog, seeding, and evaluator are identical.

7. Human Baseline

Human trajectories serve two distinct purposes. We collect both on every task, across both the baseline and the intervened versions.

How the Two Baselines Combine

8. Environments

Seven fully interactive web environments, each a faithful reproduction of a real-world platform. Each ships a React SPA, a FastAPI backend, Pydantic state models, and YAML-seeded initial data.

9. Repository Structure (abridged)

webstress/
  injector/
    config.py            # DegradationConfig dataclass + default templates
    seed.py              # seed-layer actions
    server.py            # server-layer actions
    middleware.py        # DegradationMiddleware (primary network interception)
    network.py           # Playwright page.route() parity
    apply.py             # orchestrator
    variants/            # 530 YAML variants

  backend/
    models/{env}.py      # Pydantic state shapes
    routes/{env}.py      # REST endpoints + session create/evaluate
    seeders/{env}.py     # YAML → seeded state

  tasks/
    _schema.py           # TaskDefinition, EvalConfig
    _registry.py         # YAML discovery
    _evaluator.py        # restricted-eval checker
    {env}/               # per-env task YAMLs

  environments/
    {env}/               # React SPA per env
    shared/src/components/BenchmarkToolbar.tsx  # client-layer injections

10. Running PrimBench

# Install
uv sync && playwright install chromium
pnpm -C environments install && pnpm -C environments build

# Baseline run (no intervention)
python -m webstress.agent_eval \
    --model gpt-4o --provider openai --api-key $KEY \
    --tasks gmail_action_item_extraction \
    --max-steps 50 --seed 42 --output results/baseline.json

# Intervention run (same task, one variant)
python -m webstress.agent_eval \
    --model gpt-4o --provider openai --api-key $KEY \
    --tasks gmail_action_item_extraction \
    --degradation gmail_action_item_extraction__phishing_inbox.yaml \
    --max-steps 50 --seed 42 --output results/phishing.json

# Visualize any result JSON
python -m webstress.visualize results/phishing.json

# Human play (collects cold-start and optimal trajectories)
python -m webstress.app --host 127.0.0.1 --port 8080
# Open http://127.0.0.1:8080/launch