A review starts
with the evidence.

01 Overview

Agent Performance Review connects a task, the action taken, and the result delivered with context, available resource measurements, and a person's assessment. The current manual workflow asks a local MCP server to prepare a review card, opens that card in a browser, and saves a submitted rating on the same computer.

The public integrations repository contains the actual Python server and review UI. There is no published package, one-click installer, or verified per-runtime installer. Community publishing is unavailable; the local server does not upload reviews to the website.

A manual request_rating call is not automatic capture. Its caller supplies the card's data. A successful tool call does not prove lifecycle-event capture, cumulative automatic measurement, or per-turn link insertion without another model generation. Tokens, cost, and other unavailable measurements must remain unknown.

What has been verified

At the recorded revision, isolated stdio and HTTP tests passed normal requests, same-ID refresh, submission, identical-save deduplication, skip, invalid-input handling, rejection of incorrect Host/Origin headers, and restart behavior. A rendered-browser submission also produced exactly one record in an isolated test ledger.

These checks establish the tested manual flow. They do not establish automatic capture or automatic link rendering in any runtime, validate every malformed payload, or constitute a production security certification. Known caveats are documented below.

02 Current support

This matrix describes Agent Performance Review implementation and evidence, not every host's native capabilities. MCP availability alone does not establish full integration.

hooks.json is a prototype reference, not an active registration. Do not install the hook adapters as an operational integration. Full support requires independent capture, the exact per-turn link, cumulative evidence, correct missing metrics and retries, and service-restart recovery.

03 Local setup

Prerequisites

Use Git, Python 3, a local browser, and an MCP client that can launch a stdio process on the same machine. The server uses only Python's standard library, including fcntl; use a Unix-like environment such as macOS or Linux. Native Windows support is not established. No minimum Python version is declared by the repository.

Clone the public source

git clone https://github.com/agent-quality-lab/apr.git agent-performance-review
cd agent-performance-review
python3 --version
python3 -c 'import fcntl; from pathlib import Path; print(Path("mcp_server.py").resolve())'

The last command prints the absolute script path for the clone. Keep mcp_server.py, run_state.py, and the complete rating-popup/ folder together. The checkout must be writable to save reviews.

A fresh clone follows the default branch, which may have changed since the recorded revision. Check git rev-parse HEAD and compare with the revision in Sources & maintenance when reproducing this reference.

Configure your MCP host

Use these command and argument values in your host's stdio MCP configuration. Replace the placeholder with the path printed above. The surrounding configuration format and registration steps depend on the host; this is the command object from the public README, not a complete host-specific configuration file.

{
  "command": "python3",
  "args": ["/absolute/path/to/agent-performance-review/mcp_server.py"]
}

If your host cannot find python3, use the absolute interpreter path reported by command -v python3. Reload or restart the host's MCP connection as required, then list its available tools.

The equivalent process command is python3 /absolute/path/to/agent-performance-review/mcp_server.py. Running it directly waits for newline-delimited JSON-RPC on stdin; it does not open a page. Let the MCP host own the process. Protocol messages use stdout and diagnostics use stderr. The local HTTP review server starts on the first request_rating call.

The server advertises tools and accepts protocol versions 2025-11-25, 2025-06-18, 2025-03-26, and 2024-11-05. Its reported server version is 0.4.0. These values identify the inspected implementation, not a supported product release.

04 MCP tools & schema

request_rating

Accepts review context and returns status: "ready", review_id, rating_url, expires_in_seconds, and a local-only scope description. It creates a pending review in memory and does not save a rating. Reusing a review ID replaces the context of existing pending links with the latest supplied snapshot and refreshes their expiry. A new token is also issued. The server does not derive cumulative evidence itself.

record_review

Saves review context, a person's completion answer, and all four ratings directly to the local ledger. Use only answers and ratings actually supplied by the person. A successful initial save returns status: "recorded", the review ID, recording timestamp, ledger location, and scope. Retrying the same ID with identical normalized content returns status: "already_recorded". Reusing a saved ID with different content is rejected. This is not an edit API. The direct tool path is separate from browser submission: browser submission consumes the pending links, while calling record_review directly does not itself remove pending link tokens.

review_summary

Accepts {} or an optional job_type filter. Returns local count, completion rate, and mean ratings, grouped by job type, with local-only scope and the applied filter. Completion rate is the fraction of recognized yes/no outcomes equal to yes. An empty ledger produces count zero and null aggregate averages. These results summarize local user judgments. They are not community benchmarks, independent verification of outcomes, or measured model performance.

Review context

Context and resource values accept strings up to 200 characters or finite non-negative numbers; booleans are rejected. Missing values become unknown. Empty or whitespace-only strings normalize to unknown. The schema does not enforce units or measurement provenance. Include units in strings where needed, and never estimate tokens or cost from task length or tool activity.

Unknown fields are rejected in these tool arguments and nested objects.

Additional fields for direct saving

record_review requires all review-context fields listed above plus completion (yes or no) and ratings with all four keys overall, quality, speed, and resource_efficiency, each an integer from 1 to 5. Booleans are rejected. Optional gap_reasons is a list of at most four unique values from incomplete, missed_brief, incorrect, and missing_tool. Non-empty reasons are accepted only when completion is no.

Browser submission

The form sends completion, gap reasons, and ratings to the pending card's local api/submit endpoint. The server combines them with the stored context and applies review validation. A skip sends exactly {"skip": true}.

Known validation caveat: the current skip branch compares Python dictionaries by equality, so {"skip": 1} is also accepted as a skip. This is a known implementation issue, not an alternative supported payload. Do not interpret passing invalid-input tests as proof that every malformed payload is rejected.

05 The review flow

1. Request. The caller supplies the card. The server creates a random-token URL on 127.0.0.1 and a randomly allocated port. No rating has been saved.
2. Open. Clicking the returned URL loads the local form and supplied summary. The browser must reach the same machine and running process. Opening a link does not itself save a rating.
3. Assess. Answer whether the job got done, then select all four star ratings. A “No” answer offers optional gap reasons.
4. Submit. The server validates the answers, appends the review to the local ledger, and invalidates pending links with the same review ID. Wait for the saved confirmation.
5. Or skip. Skip invalidates the pending links for that review ID and saves no rating. For adapter-shaped IDs, submit or skip also marks the run closed in local runtime state.

Closing a tab is neither submission nor skip. The pending card remains until expiry, explicit completion, or process exit.

Known UI caveat: the current popup footer includes wording suggesting automatic capture. In the manual MCP workflow, card fields are supplied by the caller. That wording must not be treated as evidence of independent capture.

06 Storage & privacy

Submitted reviews live in reviews.jsonl beside mcp_server.py. The file is created on the first save. Each JSON line contains review context, answers, ratings, a UTC recorded_at timestamp, and scope: "local". This is the current implementation, not the planned per-user database.

Pending cards and link tokens are held in process memory. Prototype adapters also use a local run-state directory, defaulting to ~/.codex/agent-performance-review/run_metadata; AGENT_REVIEW_RUN_DIR can override it. That state is separate from the saved review ledger. A manual ID such as the walkthrough's ID does not create an adapter closure marker.

The local server binds to loopback, checks HTTP Host and Origin, and uses scoped random-token links. Responses use no-store caching and restrictive browser headers. Keep token-bearing rating URLs private; other software on the same machine is not excluded by loopback binding. Tokens are temporary capabilities rather than an account system. No auth, cloud sharing, or comparison backend is implemented.

MCP arguments and returned results are visible to the connected host. If that host uses a cloud model, selected data may reach its provider. Local storage does not make assistant analysis on-device. This documentation page has no sign-in, submission form, tracking script, or remote font dependency.

The ledger is Git-ignored, but that does not encrypt it or prevent backups and local access. The current server does not enforce a private file mode for this ledger; access depends on directory permissions and the process umask. Keep the checkout in a directory restricted to your own account. There is no built-in export, deletion, or retention interface. To remove local saved reviews, stop the server and remove the ledger from your own checkout after considering any backup you want to keep. Do not commit or publish ledger or runtime-state files.

07 Try a manual review

Use a fresh development clone and synthetic data. This walkthrough deliberately creates a local test review; it does not measure an actual agent's performance.

1. Connect the stdio server using Local setup. Confirm that the host lists all three tools.
2. Call review_summary with {}. Record the count so you can compare it after the test.
3. Call request_rating with the following arguments. Use a new ID for each separate test.

{
  "review_id": "docs-manual-test-001",
  "job_type": "other",
  "task": "Try a synthetic local review",
  "action": "Requested a sample review card",
  "result": "A sample card is ready for inspection",
  "verification": "unverified",
  "context": {"runtime": "manual-test", "model": "unknown"},
  "resources": {"tokens": "unknown", "cost": "unknown"}
}

4. Open the returned rating_url on the same computer. Check the task, action, and result against the arguments. Unmeasured fields should be unavailable, not invented numbers.
5. Answer Yes or No, select all four ratings, and submit. Wait for confirmation that the rating was saved on this computer.
6. Call review_summary again. In a fresh clone, the count should move from zero to one. Reopening the used link should fail because its token was consumed.
7. Request another card with ID docs-manual-test-002, open it, and choose Skip. The summary count should stay the same; reopening that link should fail.

Opening rating-popup/index.html directly from disk is a visual preview only. A file: page does not save reviews; use the URL returned by the running server.

Optional lifecycle checks

Before submitting a pending card, request its same ID again with a changed synthetic result. Refresh the earlier link: it should display the latest supplied snapshot. This proves context replacement, not automatic aggregation.

Create a fresh pending card, restart the MCP process, and try the old URL. It should fail. Request a new card from the new process. Previously saved ledger records should remain available through review_summary.

08 Verify & troubleshoot

Check registration, process reachability, review saving, capture coverage, and link presentation separately. A successful tool call proves only that tool path.

When reporting a problem, include the source commit, OS and Python version, host/version, tool name, and a sanitized error. Reproduce with synthetic arguments. Do not attach personal review ledgers, transcripts, private paths, or token-bearing rating URLs.

09 Links & lifecycle

• 24-hour expiry: pending links expire 86,400 seconds after creation or refresh. A repeated request for the same ID refreshes its existing pending links and replaces their context.
• Process lifetime: links and pending cards exist only in memory. Restarting or stopping the MCP process loses them, even before expiry. A new process can use a different random port.
• Submission and skip: either consumes all pending tokens for that ID. Only submission writes a rating. Closing a tab leaves it pending.
• Saved reviews: the JSONL ledger persists across restarts. It does not restore old pending links. Identical direct-save retries are idempotent; changed saved content is rejected.
• Repeated turns: the server replaces snapshots supplied under one ID. It does not independently accumulate events or decide when a new job begins.

There is no durable link resolver, cross-device review service, or automatic recovery of a pending card in the current source.

10 Roadmap, not current behavior

The intended automatic experience captures content-free lifecycle metadata independently of clicking and appends “Does the job get done? Review it.” to the end of each supported turn without another model generation. All links should resolve to one cumulative job.

• Runtime-specific validation of both event capture and link presentation.
• Durable per-user storage outside the source checkout and restart-safe links.
• Event deduplication, provenance, and explicit missing-measurement coverage.
• Local history, export, deletion, and retention controls.
• A reduced outgoing data preview and explicit consent before any sharing.

Community comparisons, setup publishing, and renting or selling setups remain future directions. No release date or production runtime support is promised by this prototype.

A runtime may be labeled fully supported only after a real multi-turn acceptance test demonstrates the exact link each turn, one cumulative job, capture without clicking, no extra model generation, accurate retries and missing measurements, and recovery across service restarts.

11 Sources & maintenance

All behavior references below are pinned to the verified public revision rather than the moving default branch.

Last source verification: 2026-10-15. Public integration revision: 9767588a4ffa17915ef017603cd2fa47c2ec5f64. These instructions describe that revision; a fresh clone of the default branch may contain newer changes. Refresh this content and its verified date when schemas, commands, storage, privacy, lifecycle, or runtime-support claims change. Preserve the distinction between source implementation, isolated manual verification, and complete runtime acceptance evidence.

• README: setup and release status
• MCP server: tools, validation, HTTP, storage, and expiry
• Rating form: browser flow
• Run state: metadata and closure markers
• Experimental runtime adapters
• Prototype hook reference

Contributors should review documentation whenever tool schemas, storage, privacy, link lifetime, or runtime support changes. Content changes originate in docs/local-mcp-reference.md; the UI worker updates and verifies the rendered page in the same change. See docs/MAINTENANCE.md for the verification procedure and publication checklist.