Migration Guide for Existing Vocabulary Users¶

This guide documents every breaking change and deprecation introduced by the KB / Vocabulary workflow program (Phase 0 through Phase 6) and explains how to migrate an existing PyCharter deployment that was running an earlier vocabulary surface.

The program landed iteratively between 2026-03 and 2026-05; each change below is keyed to the kanban card that introduced it so operators can trace risk back to the source review notes in kb-ontology-workflow-kanban.md.

Audience: operators who maintain a deployed PyCharter instance, authors of HTTP clients (UIs, CLIs, agents), and integrators that embed pycharter as a library.

API namespace rename — Branch D Wave 1 (2026-05-20)¶

On 2026-05-20 the API surface was reorganised into a canonical namespace scheme. Every path under /api/v1/ was reviewed; many were renamed (e.g. /semantic/proposals/... → /proposals/..., /semantic/bindings/... → /contracts/{contract_id}/fields/{field_path}/binding/..., audit and templates re-homed under /ops/..., validation re-homed under /quality/...). A handful of paths now take their identifier as a path param rather than a body field. Two endpoints (GET /docs/playground-routes, GET /schema-import/plugins) were removed outright.

The authoritative table of every old URL → new URL, body-shape change, and dual-route registration is in url-migration-map.md. HTTP clients that have not been regenerated against the current OpenAPI schema should consult that map before upgrading.

Overview¶

Area	Change	Card	Action required
Database schema	11 new tables / columns	P1..P6	Run `alembic upgrade head`. One reversible data migration in `binding_events_actions`.
HTTP — `POST /bindings/remap`	Response shape changed from `{detach_event, attach_event}` to a single `BindingEventResponse`	P3-008-FU-B	Update client parsing (single event).
HTTP — `POST /proposals/link`	New endpoint replaces `payload["proposal_id"]`	P3-006-FU-A	Switch clients; legacy fallback emits `DeprecationWarning`.
HTTP — `POST /semantic/ingestion/kb-text` / `ingest-upload`	Returns `202 IngestAcceptedResponse` for payloads > 1 MiB or > 1000 rows	P6-003	Clients must poll `GET /semantic/ingestion/runs/{run_id}/status`.
HTTP — `GET /audit/export`	New admin-gated JSONL streaming endpoint	P6-004	None — additive.
HTTP — list routes	New `limit` / `offset` query params on bindings events, assets, ingest-runs, proposals	P6-002	Optional — params default to a paginated slice.
FSM	`[technical_only]` / `[annotation]` reason-prefix markers replaced by first-class FSM states / actions	P3-008-FU-A	Data migration rewrites legacy rows on upgrade.
FSM	REMAP events now fire both `on_concept_detached(from)` and `on_concept_attached(to)`	P3-008-FU-D	Optional projection backfill via the script below.
Defaults	Review-event repo defaults to SQLAlchemy when a session is provided	P3-009-FU-B	None unless you intend to keep the in-memory backend.
Defaults	Health snapshot `persist_history` default flipped from `True` to `False`	P4-001-FU-B	Opt in via `persist_history=True` if you rely on trend data.
Auth	`RequesterContext` resolution: header-only → JWT-first	P6-001	Migrate clients to JWT; flip `PYCHARTER_REQUESTER_CONTEXT_FROM_HEADERS=false` when ready.

Required first step — `alembic upgrade head`¶

Every schema change in the program is idempotent and reversible. Apply the full chain in one step:

pycharter db upgrade
# or, if you invoke Alembic directly:
alembic -c alembic.ini upgrade head

The chain shipped by this program (oldest → newest) is:

Revision	File	Purpose
`kb_management_tables`	`20260518000000_add_kb_management_tables.py`	KB versions / links / lineage tables (P1-001).
`semantic_asset_ingest_run_tables`	`20260518000001_add_semantic_asset_and_ingest_run_tables.py`	Semantic assets + ingest runs (P1-002).
`asset_ingest_run_link`	`20260518000002_add_asset_ingest_run_link.py`	Soft `ingest_run_id` on assets (P1-003).
`suggestion_source_traceability`	`20260518000003_add_suggestion_source_traceability.py`	`source_asset_id` + `ingest_run_id` on suggestions (P1-005).
`mapping_projections`	`20260518000004_add_mapping_projections.py`	Durable field-binding projection (P1-007-IMPL).
`suggestion_evidence_columns`	`20260519000000_add_suggestion_evidence_columns.py`	Evidence shape on suggestions (P2-002).
`suggestion_is_duplicate_of`	`20260519000001_add_suggestion_is_duplicate_of.py`	Duplicate-candidate link (P2-005).
`semantic_review_events`	`20260519000002_add_semantic_review_events.py`	Review-decision audit rows (P3-004).
`proposals_suggestion_id`	`20260519000003_add_proposals_suggestion_id.py`	FK from proposal to suggestion (P3-006-FU-A).
`semantic_health_issue_snapshots`	`20260519000004_add_semantic_health_issue_snapshots.py`	Health-issue snapshot history (P4-001-FU-A).
`binding_events_actions`	`20260519000005_extend_binding_events_for_actions.py`	Adds `action`, `from_concept_id`, `to_concept_id`. Includes one data migration — see below.
`ingest_run_progress`	`20260519000006_add_ingest_run_progress.py`	`progress` JSON column on ingest runs (P6-003).

Data migration in `binding_events_actions`¶

Revision binding_events_actions adds three columns to concept_binding_events and rewrites historical rows whose reason starts with [technical_only] or [annotation]:

Rows with reason starting with [technical_only] have their to_state flipped to "technical_only" and the marker stripped from reason.
Rows with reason starting with [annotation] get action="annotation" set and the marker stripped from reason.

The data migration is idempotent (re-running is a no-op) and the downgrade is reversible. It does not restore the marker prefixes on downgrade — every value remains recoverable from the new action and to_state columns.

If you have automation that scrapes the reason column for those markers, retire that code; the discriminator now lives in the action column.

API consumer migration¶

`POST /api/v1/contracts/{contract_id}/fields/{field_path}/binding/remap` (P3-008-FU-B)¶

The remap route used to perform a detach followed by an attach and returned both events:

// Before — two events
{
  "detach_event": { "id": "...", "to_state": "unmapped", ... },
  "attach_event": { "id": "...", "to_state": "mapped",   ... }
}

It is now atomic — a single BindingEvent row is written with action="remap", from_concept_id set to the prior concept, and to_concept_id / concept_id set to the new concept. The response is a single BindingEventResponse:

// After — one event
{
  "id": "...",
  "contract_id": "orders:1.0.0",
  "field_path": "header.customer_id",
  "concept_id": "<new_concept_id>",
  "from_state": "mapped",
  "to_state": "mapped",
  "action": "remap",
  "from_concept_id": "<prior_concept_id>",
  "to_concept_id": "<new_concept_id>",
  "actor": "alice",
  "reason": "",
  "timestamp": "2026-05-19T10:00:00Z"
}

Client-side migration:

- const { detach_event, attach_event } = await api.remapBinding(payload);
+ const event = await api.remapBinding(payload);

The bundled UI client in src/pycharter/ui/lib/api/semanticBindings.ts was updated in the same release.

Proposal linkage — `POST /api/v1/proposals/link` (P3-006-FU-A)¶

Domain-tier and core-tier suggestions cannot be materialised until a proposal points at them. The previous mechanism set payload["proposal_id"] on the suggestion before calling /accept. That path now emits a DeprecationWarning and is scheduled for removal in card P3-006-FU-B (the next minor release).

Switch to the link endpoint, which sets proposals.suggestion_id in one round-trip:

curl -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "suggestion_id": "11111111-1111-1111-1111-111111111111",
    "actor": "alice",
    "summary": "Promote customer_id to a core concept",
    "details": "Optional longer rationale"
  }' \
  http://example.invalid/api/v1/proposals/link

Response shape (LinkProposalResponse):

{
  "id": "<proposal_uuid>",
  "suggestion_id": "<suggestion_uuid>",
  "summary": "Promote customer_id to a core concept",
  "status": "open",
  "actor": "alice",
  "created_at": "2026-05-19T10:00:00Z"
}

The endpoint requires Role.ADMIN or higher (see role-permissions-matrix.md).

To find suggestions that still carry the legacy key in their payload:

SELECT id, payload->>'proposal_id' AS legacy_proposal_id
FROM suggestions
WHERE payload ? 'proposal_id'
  AND status = 'pending';

For each row, call POST /proposals/link with the suggestion_id to populate proposals.suggestion_id. After the next minor release the legacy fallback will be deleted and any remaining payload["proposal_id"] markers will be ignored.

Large imports — async path on ingest endpoints (P6-003)¶

POST /api/v1/semantic/ingestion/kb-text and POST /api/v1/semantic/ingestion/kb-upload now dispatch to a background worker when the request body exceeds either threshold:

1 MiB body size (IMPORT_SYNC_THRESHOLD_BYTES = 1_048_576).
~1000 rows estimated from the payload (IMPORT_SYNC_THRESHOLD_ROWS = 1_000).

Small payloads keep their previous synchronous shape (IngestReportResponse). Large payloads now return 202 Accepted with IngestAcceptedResponse:

{
  "ingest_run_id": "abcd1234-...",
  "status": "pending",
  "poll_url": "/api/v1/semantic/ingestion/runs/abcd1234-.../status"
}

Clients must follow up by polling the status endpoint until the run reaches a terminal state (succeeded or failed):

curl -H "Authorization: Bearer $TOKEN" \
  http://example.invalid/api/v1/semantic/ingestion/runs/abcd1234-.../status

Response (IngestRunStatusResponse):

{
  "ingest_run_id": "abcd1234-...",
  "status": "processing",
  "progress": {
    "rows_processed": 250,
    "rows_total": 1000,
    "message": "Importing concepts..."
  },
  "started_at": "2026-05-19T10:00:00Z",
  "finished_at": null,
  "error_summary": null,
  "result_summary": null
}

The terminal succeeded response carries result_summary (which mirrors IngestReportResponse); failed carries error_summary.

To make a large import idempotent, send ingest_run_id in the request body — the server will return the existing run's poll URL instead of scheduling a second pipeline.

Audit export — `GET /api/v1/ops/audit-export` (P6-004)¶

This is a new admin-gated endpoint with no legacy equivalent. It streams chronological newline-delimited JSON (application/x-ndjson) covering five event types (review, binding, accepted_suggestion, proposal_link, health_snapshot):

curl -H "Authorization: Bearer $ADMIN_TOKEN" \
  "http://example.invalid/api/v1/ops/audit-export?since=2026-05-01T00:00:00Z&event_types=review,binding"

Each line is one event:

{"event_type": "review", "event_id": "...", "timestamp": "...", "actor": "alice", "workspace_id": "...", "payload": {"decision": "accept", ...}}

Workspace scoping: superuser sees all rows; admin sees only the workspace-scoped streams (review, accepted_suggestion) filtered by their workspace. The other three streams currently lack scheme_id columns and pass through for any admin (see the residual risk in P6-004).

Pagination on list routes (P6-002)¶

The following routes gained limit and offset query parameters. Existing callers that do not pass them keep working — the server applies the default slice (limit=50, MAX_LIMIT=500):

GET /api/v1/contracts/{contract_id}/fields/{field_path}/binding/events
GET /api/v1/semantic/ingestion/assets
GET /api/v1/semantic/ingestion/runs
GET /api/v1/proposals/

Already-paginated routes (/suggestions, /health, /context/query) are unchanged.

The new envelope on these routes is PaginatedResponse[T] { items, total, next_offset, limit, offset } where total and next_offset are populated when cheap to compute. See pagination-and-filtering.md for the full convention.

Configuration migration¶

Set or review the following environment variables before flipping new defaults in production.

Authentication / authorisation (P6-001)¶

# Default is "true" — keep this on while clients still send X-Role / X-Workspace.
PYCHARTER_REQUESTER_CONTEXT_FROM_HEADERS=true

# Bootstrap users; at least one superuser is required for cross-workspace work.
PYCHARTER_AUTH_USERS='[
  {"username": "root", "password": "<dev-only>", "role": "superuser"}
]'

The four-step migration horizon for header → JWT is documented in role-permissions-matrix.md.

Semantic review-event backend (P3-009-FU-B)¶

The resolver now picks "sqlalchemy" automatically when a session is supplied, so most deployments need no change. To force the in-memory backend (tests, ephemeral CLI runs):

PYCHARTER_SEMANTIC_REVIEW_EVENT_BACKEND=memory

If you were already setting this env var to "sqlalchemy" to opt-in, you can drop it — the new default covers that case.

Semantic health backend¶

# Optional — pick "sqlalchemy" or "in_memory". The factory defaults
# to SQLAlchemy when a session is provided.
PYCHARTER_SEMANTIC_HEALTH_BACKEND=sqlalchemy

Health snapshot retention (P4-001-FU-B)¶

SqlAlchemySemanticHealthService now constructs with persist_history=False by default. Operators that rely on the semantic_health_issue_snapshots table for trend data must opt in:

service = SqlAlchemySemanticHealthService(
    session_or_factory=session,
    persist_history=True,
)

A TTL / compaction strategy is deferred to card P4-001-FU-D — opting in trades unbounded growth for trend tracking until retention lands.

Frontend client migration¶

The bundled UI in src/pycharter/ui/ was migrated in lockstep with the backend changes. If you maintain your own front end, mirror these updates:

lib/api/semanticBindings.ts — remapBinding now returns a single BindingEvent rather than {detach_event, attach_event}.
lib/api/ingest.ts — split into synchronous (ingestText, ingestUpload) and async-aware (ingestTextSubmit, ingestUploadSubmit, pollIngestRunStatus) helpers. The ImportWizard switches to a polling view on a 202 response.
lib/api/bindingEvents.ts — getReviewEvents now hits the real backend route (GET /suggestions/{id}/review-events).
Role enforcement — all write actions that gained require_role(...) (see the matrix) surface 403 errors when the JWT role rank is below the minimum. Inline guards in the review and mapping panels already render the read-only state.

Database changes — new tables and columns¶

Table	Source migration	Notes
`kb_versions` / `kb_links` / `kb_lineage_records`	`kb_management_tables`	KB management surface (P1-001).
`semantic_assets`	`semantic_asset_ingest_run_tables`	+ `ingest_run_id` column added by `asset_ingest_run_link`.
`semantic_ingest_runs`	`semantic_asset_ingest_run_tables`	+ `progress` JSON column added by `ingest_run_progress`.
`suggestions.source_asset_id`, `suggestions.ingest_run_id`	`suggestion_source_traceability`	Soft refs (no hard FK), matches the program convention.
`mapping_projections`	`mapping_projections`	Row-per-binding projection (P1-007-IMPL).
`suggestions.source_span`, `suggestions.rationale`, `suggestions.proposed_action`	`suggestion_evidence_columns`	Structured evidence (P2-002).
`suggestions.is_duplicate_of`	`suggestion_is_duplicate_of`	Duplicate-candidate link (P2-005).
`semantic_review_events`	`semantic_review_events`	Review-decision audit rows (P3-004).
`proposals.suggestion_id`	`proposals_suggestion_id`	FK to suggestion (P3-006-FU-A).
`semantic_health_issue_snapshots`	`semantic_health_issue_snapshots`	Health-issue snapshot history (P4-001-FU-A).
`concept_binding_events.action`, `from_concept_id`, `to_concept_id`	`binding_events_actions`	FSM extension + data migration (P3-008-FU-A/B).

See database-erd.md for the full schema diagram.

Optional backfill — REMAP projection retirement¶

P3-008-FU-D extended the lifecycle dispatcher so that REMAP events fire both on_concept_detached(from_concept) and on_concept_attached(to_concept). REMAP rows persisted before that card landed never fired the detach half, so the mapping_projections table may still claim the prior concept is bound for those legacy events.

The script service-level remap reconciliation workflow is implemented through service-level lifecycle replay in the semantic services layer.

The script is idempotent. Rows that already have state="unresolved" for the (contract_id, field_path, from_concept_id) tuple are left alone.

Rollback considerations¶

Every migration in this program is reversible via alembic downgrade -1 (or to a named revision). The recommended rollback order is reverse of the upgrade chain — start with the newest revision and walk back. The binding_events_actions downgrade drops the three new columns but does not restore the [technical_only] / [annotation] reason-prefix markers because the new columns are the canonical source.

Before rolling back a production database, confirm that:

No client is depending on the new HTTP responses (BindingEventResponse-shape remap, LinkProposalResponse, async ingest 202s, /audit/export).
The semantic_review_events, semantic_health_issue_snapshots, and mapping_projections tables are not feeding any downstream reporting pipeline you still need.
Any custom clients have been reverted to the legacy header / payload paths.

Per-revision rollback is documented in each migration file's downgrade() block.

Cross-references¶

User guide for the end-to-end workflow (P6-006).
Admin guide for governance policy and permissions (P6-007).
Role / permissions matrix (P6-001).
Pagination and filtering convention (P6-002).
KB / Ontology workflow kanban — full card history with residual risks per change.
Database ERD — schema diagram.