Agent verification metadata store

The pycharter.verification_store package persists R1.7 preparations and R3 receipts. pycharter.agent.hosted_verification connects it to scoped preparation and exact verification. The HTTP routes are POST /api/v1/agent/preparations and POST /api/v1/agent/verify. They require a verified local JWT with a subject and concept-scheme workspace UUID; the namespace comes from that workspace. A receipt documents a check and grants no action permission. The shipped name-only validate_record and offline MCP paths retain their existing behavior.

Versioned DTOs

PreparationEvent (schema_version: "1.0") records an opaque preparation ID, principal and workspace, exact contract namespace/name/version and fingerprint, effective context fingerprint, policy projection label, creation time, and expiry. A JSON null namespace means the global namespace. Hosted preparation mints the ID and persists the event before returning the view. The lifetime is one hour, within the DTO's 24-hour maximum.

VerificationReceipt (schema_version: "1.0") records an opaque receipt ID, the same scope and exact contract/context identity, preparation ID, time, accepted/rejected verdict, normalization status, stable diagnostic codes and field paths, attempt number, optional correlation/attempt IDs, and validator version. It carries no diagnostic messages or values. A new DTO has recorded: false; a successful store write returns recorded: true. Storage failure cannot yield a stored receipt. In the default required mode, the hosted service returns a 503 with no accepted verdict when persistence fails. An explicitly selected optional mode may return recorded: false with reason: receipt_unavailable. A conflicting replay always fails closed.

Both DTOs reject unknown keys and require timezone-aware timestamps and sha256: fingerprints with 64 lowercase hexadecimal digits. They cannot accept a candidate record, normalized record, prompt, full schema/context, secret, or unrestricted error text. The SQL payload JSON and MongoDB documents contain only these allowlisted fields.

Backend and replay behavior

open_verification_store selects SQLite, PostgreSQL, or MongoDB. SQL stores use agent_preparations and agent_verification_receipts in the pycharter schema on PostgreSQL; MongoDB uses equally named collections. Alembic adds both SQL tables. The caller must supply a configured HMAC key ring and active key ID when opening the store. Keys are never persisted or logged.

save_receipt receives canonical request bytes only in memory. When both correlation and attempt IDs exist, the store computes and persists a server-keyed HMAC-SHA256 digest. The unique key is principal, workspace, correlation ID, and attempt ID. An identical retry returns the original receipt, including its ID; different canonical bytes raise ReceiptConflictError. If either caller ID is missing, every call creates a fresh receipt and stores no digest. The host must canonicalize the full verification request consistently before calling this store; the raw bytes are never written to either backend. Hosted canonicalization covers the exact contract/context identity, candidate, direction, normalization handoff, attempt number and labels, receipt mode, projection options, and validator version. The candidate exists only in request memory and this temporary digest input.

On key rotation, keep prior keys until the replay window has closed. The store verifies a replay with the key ID used for the original receipt. If that key is unavailable, it raises ReplayKeyUnavailableError and refuses to guess whether the request is identical. This failure is safe to surface as a stable conflict, without exposing canonical request bytes.

get_preparation and get_receipt require the original principal and workspace; an opaque ID alone does not grant access. The preparation read also enforces expiry. The hosted write service derives identity from its JWT, rechecks workspace namespace and the current prepared fingerprint before validating, and refuses a stale or foreign preparation. HTTP receipt reads apply current JWT scope and commit an audit event before querying metadata.

Environment provisioning

The HTTP API and the remote MCP command use these explicit settings:

Environment variable	Meaning
PYCHARTER_AGENT_VERIFICATION_URL	Required dedicated metadata-store URL: file-backed SQLite, PostgreSQL, or MongoDB.
PYCHARTER_AGENT_RECEIPT_KEYS	Required JSON object mapping key IDs to standard base64-encoded HMAC keys, each at least 32 bytes.
PYCHARTER_AGENT_RECEIPT_ACTIVE_KEY	Required ID of an entry in that key ring, used for new replay digests.
PYCHARTER_AGENT_RECEIPT_RETENTION_DAYS	Optional integer from 1 to 90; defaults to 90. Applies to receipts and receipt-read audit events.

Supply URL credentials and key material through the deployment's secret environment. For example, configure the ledger location explicitly:

export PYCHARTER_AGENT_VERIFICATION_URL=sqlite:////var/lib/pycharter/agent-metadata.db
export PYCHARTER_AGENT_RECEIPT_ACTIVE_KEY=receipt-current
# Inject PYCHARTER_AGENT_RECEIPT_KEYS from the secret store as
# {"receipt-current":"<base64 encoding of at least 32 random bytes>"}.
PYCHARTER_DATABASE_URL="$PYCHARTER_AGENT_VERIFICATION_URL" pycharter db upgrade

Run SQL migrations against the dedicated ledger URL, as well as against the contract database, before enabling receipt reads. The migration creates agent_receipt_read_events alongside preparation and receipt tables. MongoDB creates the corresponding collections and indexes in the URL's database (default pycharter) when opened. Hosted Mongo URLs must use acknowledged write concerns; w=0 is refused. The hosts never infer this URL from PYCHARTER_DATABASE_URL, accept in-memory SQLite, or reuse the JWT signing secret. These keys protect internal replay digests; the receipt itself is metadata, not a signed authorization token.

The per-command database setting above points the migration runner at the ledger even when the host also has a contract database URL configured. It does not change the host's contract database setting.

To rotate, first install the new key alongside all retained old keys on every host, then switch the active ID. Keep old keys through the entire retained receipt replay window. HTTP requests open and close their own ledger handle; the remote command holds its configured handle until shutdown and must be restarted when its key settings change. Invalid/missing host settings produce a generic HTTP 503 or stop remote startup without printing input secrets.

Hosted HTTP migration and deployment

The prepare body accepts name, optional exact version, profile, and budget_tokens. It deliberately has no principal, workspace, or namespace field. The response adds a preparation_id and recorded: true to the governed prepared view. The verify body requires that ID, an exact contract_ref such as orders@1.0.0, context_fingerprint, and a JSON object candidate. Optional fields are correlation_id, attempt_id, attempt_number, direction, profile, budget_tokens, and allow_normalized_record. A completed result includes receipt: {id, recorded, reason}. The receipt ID is the same on an identical retry with both caller labels; different content under that key yields 409. Receipt mode is a trusted host setting, never a caller request field. The HTTP dependency get_hosted_receipt_mode defaults to required; a deployment may explicitly inject optional when its policy permits unrecorded results.

The HTTP store dependency get_hosted_verification_store shares the explicitly configured ledger with receipt reads. Trusted deployment dependency overrides remain available. The application operator must provision the store, run SQL migrations where applicable, retain retired HMAC keys through the replay window, and schedule retention. External introspection tokens need a separately reviewed identity adapter; this route accepts verified local JWTs only.

The authenticated remote MCP server offers the same lifecycle when its host passes RemoteHostedVerification to create_app: remote preparation is recorded, and validate_record accepts only the exact hosted request. Its receipt mode is the same trusted host setting. See Authenticated remote MCP.

Existing offline verify_prepared_record callers need no migration. A hosted caller first POSTs a preparation, then POSTs verification with its returned ID and fingerprint. A caller-supplied fingerprint without a stored preparation does not authorize hosted validation.

Retention and remaining operations

Receipt retention defaults to 90 days. A tenant may configure a shorter window from 1 to 90 days. Reads hide receipts outside that window even before cleanup. sweep_retention(now=...) deletes expired preparations and receipts beyond the configured window; sweep_read_events(now=...) deletes audit events beyond the same window. The operator command runs both sweeps:

pycharter agent receipts sweep

It uses the same host environment and reports preparation, receipt, and audit deletion counts as JSON. Schedule it with the deployment's job runner; it is safe to repeat. An unavailable store returns a nonzero exit status with a generic message. No automatic scheduler is installed. A deleted receipt cannot be recovered from this store. Tenant deletion and export need a separate authenticated, auditable service operation before R3.2/R3.4 can be claimed complete. The current store has no public bulk export or deletion endpoint and does not expose raw request payloads.

Tenant deletion/export and deployment retention scheduling remain in the single BACKLOG.md R1/R2/R3 entries.

Audited receipt reads

GET /api/v1/agent/receipts, its /{receipt_id} detail route, and GET /api/v1/agent/receipt-conformance require a verified local JWT even when development authentication is disabled. Viewers see only their own receipts; reviewers may select a principal in their workspace. A superuser must name a workspace explicitly. An opaque receipt ID never grants access by itself.

Before querying, the service writes a ReceiptReadEvent (schema_version: "1.0") through record_read_event. Its strict metadata-only shape contains event ID, aware timestamp, operation (recent, detail, conformance, or trend), actor, workspace, optional principal filter, and optional target receipt ID. Detail events require the exact principal and receipt ID. No raw record or free-text diagnostic can be added. SQL commits and MongoDB acknowledged journaled writes complete before reads proceed; an audit failure returns generic 503 and prevents the receipt query. Events survive restart and can be inspected through the operator store's workspace-scoped get_read_event; there is no unauthenticated audit endpoint.

The original VerificationStore protocol remains the preparation/receipt interface for existing injected ledgers. ReceiptAuditStore is the additional capability for read audit operations. HTTP audit overrides and retention sweeps require that capability; remote prepare/verify custom stores retain their original interface.

Receipt-backed conformance reports

pycharter.agent.receipt_conformance.summarize_receipt_conformance reads retained receipt metadata through the same store on SQLite, PostgreSQL, and MongoDB. Its caller supplies an authenticated workspace scope, an optional agent principal, a half-open UTC window (since inclusive, until exclusive), and a fixed now. The public scan_receipts store operation returns at most scan_limit records, newest first, and flags whether more records existed. The default limit is 2,000.

The report groups by exact contract namespace, name, and version plus agent principal. Its denominator is persisted receipts in the requested scope, not all verification attempts. First-pass acceptance means an accepted receipt with attempt_number == 1; repaired acceptance means an accepted receipt with attempt_number > 1. Rejected receipts contribute one count per distinct diagnostic code and field path, showing the top 10 categories in each group; duplicate diagnostics in a single receipt do not inflate a category. The report keeps contract quality scores separate from agent-output conformance.

denominator_exact is false when the scan is truncated or the requested window extends before configured retention. Observed rates are then null; the visible counts are lower bounds. coverage_complete is always false in this receipt-only slice, because failed receipt writes and quarantine are not represented by a persisted receipt. Both fields are null, and incomplete_reasons names the missing sources. No total verification rate, receipt availability percentage, or quarantine count is inferred from these receipts. The HTTP operator path requires authenticated workspace scope and the environment-provisioned ledger and audit table before exposing reports or diagnostic paths.

Daily conformance trends

pycharter.agent.receipt_trends.summarize_receipt_trend counts the same persisted receipts per exact contract version (namespace, name, version) and UTC day, over days days ending on the day that contains now (1–366). Each series has one point per day, oldest first, with zero-filled days, so a chart can show gaps. A point reports first-pass acceptances, repaired acceptances, rejections, and observed rates over that day's persisted receipts. At exactly UTC midnight, a one-day window is empty; it never counts the previous day's receipts.

A day is complete only if every retained receipt from that day was read. A day that starts before the retention boundary, or that the bounded newest-first scan may have cut off, is incomplete. The report lists it in incomplete_days, and its rates are null; its counts are lower bounds. ReceiptScan.retained_since reports the lower bound the store actually read. It is the later of the requested start and retention; SQLite, PostgreSQL, and MongoDB set it. coverage_complete is always false, for the same reasons as the summary: write failures and quarantine are not persisted receipts.

The HTTP route is GET /api/v1/agent/receipt-conformance/trend with days (default 30), workspace, and principal. It uses the same verified-JWT scope rules as the other receipt reads. A viewer sees only their own receipts, and a superuser must name a workspace. It records a trend audit event before reading, and fails closed with a generic 503 when storage or auditing is unavailable.