Skip to content

PyCharter — Immutable Contract Versions (edit = new version)

Status: implemented (cutover #279–#294); CIV-FOLLOWUP-1…4 are tracked in BACKLOG.md · Created: 2026-06-08 · Audience: maintainers · Supersedes: contract-artifact-ownership.md · Resolves: PAC-2b in python-api-consolidation.md

Why

A data contract is a composition of independently-versioned artifacts (json_schema, coercion_rules, validation_rules, metadata, field_bindings), and the contract (name, version) is itself a versioned tuple of pointers to those artifact versions. The intended model — confirmed by the maintainer — is:

Artifacts are immutable per version. A contract version is a frozen, self-contained tuple of pointers to immutable artifact versions. Different contract versions may reuse the same artifact version for the parts that didn't change. Any content change produces a new artifact version and a new contract version. Nothing ever saves over an existing version.

This is a clean, content-addressable model (artifacts ≈ git blobs, contracts ≈ git commits). The code almost enforces it: the per-artifact create-writers reject overwriting an existing (title, version) — so schema / coercion / validation / metadata are genuinely immutable. One hole breaks it:

  • upsert_field_bindings (contract_store/_sqlalchemy_artifacts.py:458-459) edits the field-bindings artifact in place (overwrites mapping_data, bumps a contract revision counter, no new version). It is the inspector's "retag a field" path. Because field bindings can also be shared by reference (create_mixed accepts field_bindings_title/field_bindings_version), an in-place edit on a shared field-bindings artifact silently mutates every contract version pointing at it — violating immutability.

There is also no coherent edit story for the immutable artifacts: editing coercion rules on an existing version hits the "already exists" rejection, and there is no version-bump-on-edit machinery — so the editor's per-artifact "save" paths are inconsistent (field-bindings mutate in place; everything else has no in-place path at all).

This proposal makes the model real and uniform: every edit in the contract editor creates a new contract version. It also resolves PAC-2b: once immutability is enforced, artifact reuse is provably safe, the per-artifact writers are correctly understood as the composition API (not debt to internalize), and the de-sharing initiative (contract-artifact-ownership.md) is withdrawn — de-sharing would destroy a designed, now-safe feature.

The model

  • Artifacts are immutable per (title, version). No writer ever overwrites; upsert_field_bindings (the lone exception) is removed.
  • A contract version is immutable. Opening a contract in the editor loads a working copy (UI state); the original is never mutated.
  • Edit = new version. Saving the editor creates a new contract version, named by the user (the save dialog pre-fills a suggested patch bump). The new version reuses unchanged artifacts by reference and creates new artifact versions only for what changed — the existing create_mixed "new-payload XOR reuse-existing" shape.
  • Batch save (decided). The editor holds all edits across all sections + the field-bindings inspector; one "Save as new version …" bundles every dirty artifact into a single new contract version (no per-section version explosion).
  • Concurrency = the (name, version) unique constraint. Two editors that fork the same base to the same new version collide → 409 "version already exists". The revision counter and expected_revision optimistic lock become vestigial and are removed from the edit path.
  • Semantic projection rows (_record_projection_rows, keyed by (contract_id, contract_version, …)) re-materialize for the new version; the old version's rows are untouched (it is frozen).

Design

Backend — "fork a new version"

Prefer a fork-from-base path over making the UI enumerate every unchanged artifact: the editor sends {base_version, new_version, changed_artifacts}; the service reads the base contract version's artifact refs, creates new artifact versions for the changed artifacts (via the existing create-writers), reuses the base's refs for the rest, and links a new DataContractModel row — all in one transaction (PAC-2a's atomic machinery). create_mixed is the underlying mechanism (and the fallback when the caller enumerates explicitly). The per-artifact writers stay — they are how new artifact versions get created.

  • Remove upsert_field_bindings and the field-bindings PATCH in-place path; a field-bindings edit becomes a fork with field_bindings as the changed artifact. The field-bindings GET (read current bindings) stays.
  • Remove revision / expected_revision checks from the edit path. Keep the revision column initially (unused; drop in a later migration) to avoid a schema change in the first cut.
  • Projection rows: materialize for new_version; leave the base's rows.

UI — "Save as new version"

  • The editor's Save becomes "Save as new version …" with a version field pre-filled with a suggested patch bump of the loaded version; collision → a clear "version already exists" error.
  • Track dirty-per-artifact so the fork request marks exactly what changed.
  • After save, the editor switches to the new version.
  • The field-bindings inspector folds into the same save (its edits are part of the working copy); remove the standalone revision display + expected_revision.
  • Config sections (CoercionRulesSection, ValidationRulesSection, MetadataConfigSection) stop POSTing per-section; they mark the working copy dirty and the unified save commits.

What this does NOT change

  • Per-artifact getters, upsert_field_bindings's GET counterpart, and the contract store's whole-contract save/get/update (PAC-2a) stay.
  • Artifact reuse/sharing stays — it is the point, and now provably safe.
  • No contract data migration — existing contracts and their per-version artifacts are already valid under the model.

Phasing

  • ✅ Phase 1 (backend, ADDITIVE): fork-from-base service + endpoint (atomic). Shipped — PRs #279–#283.
  • ✅ Phase 2 (UI + cutover): "Save as new version" fork + dirty tracking + switch-to-new-version; config sections + bindings inspector feed the working copy and persist via the fork; upsert_field_bindings, the field-bindings PATCH, and revision/expected_revision plumbing removed (backend + UI). Shipped — staged across 5a (#284), 5b-1 (#291), 5b-2 (#293), plus the seed-from-working-copy stickiness fixes (#291 bindings, #294 config sections) and the binding-flow enablers/bug-fixes #285–#290, #292.
  • 🔻 Phase 3 (cleanup) — partially deferred: contract-artifact-ownership.md marked withdrawn ✅; PAC-2b closed ✅; this doc + the PAC backlog updated ✅. The revision column drop is deferred (see CIV-FOLLOWUP-4) — the column is unused but is still woven through publish_contract_event + two store serializers, so the drop is a multi-system change for cosmetic benefit and the original proposal sanctioned deferring it. Removing the per-section /metadata POST routes is "5c", also deferred (the routes are still used by the per-section Save buttons in the config panel / upload modal / field-rules inspectors).

Resolved decision

  • Fork endpoint shape: fork-from-base (decided). A dedicated POST .../contracts/{name}/{base_version}/versions taking {new_version, changed_artifacts}; the backend reads the base version's artifact refs, creates new artifact versions for the changed artifacts, reuses the base's refs for the rest, and links the new contract in one transaction. Chosen over enumerating reuse in create_mixed from the UI — less client bookkeeping, harder to mis-wire reuse.

Risks / danger flags

  1. Live editor behavior change — the contract editor's save semantics change (now version-creating). Backend + UI must land together (or feature-flag).
  2. Version growth — every save is a new version (intended; mitigated by batch save so an editing session = one version, like a git commit).
  3. The field-bindings PATCH endpoint is UI-consumed — coordinate its change to the fork flow; keep the GET.
  4. Projection rows must target the new version without clobbering the base's.
  5. Concurrent forks to the same version → 409; the UI must surface it and offer a different version.
  6. Audit other consumers of in-place field-binding edits / revision (MCP, streaming editor, importers) before removing them.

Phase 2 — implementation plan (from the editor audit)

Key finding: the editor has no unified "save the contract" — there are three independent save paths with no shared working-copy or cross-artifact dirty model:

Path Artifact Mechanism State
Canvas/schema json_schema toolbar Save → saveContractLayer → createMixed workspace.dirtyLayers
Config sections coercion / validation / metadata / data-quality per-section POST /metadata/* (in place) each section's own local useState
Field-bindings inspector field_bindings PATCH /field_bindings + expected_revision (in place) its own dirty/revision

So "one Save as new version batching all dirty artifacts" requires unifying these three into one fork request. Decision: full unification — lift schema + all config sections + field bindings into one contract working-copy with per-artifact dirty tracking and a single save.

Phase 2a — backend completeness (additive; this change set)

  • The fork endpoint materializes mapping-projection rows for the new version's field bindings (materialize_projection_for_version in _contract_field_bindings.py), at parity with the in-place PATCH. Best-effort — the field_bindings artifact JSON is the source of truth; projection rows are a secondary index (and the default lineage path scans the JSON anyway).
  • UI API client gains api.contracts.fork(...).
  • No behavior change; ships independently.

Phase 2b — UI cutover (the core; backend + UI in lock-step)

  • Unified working copy: one editor contract document holding schema + the four config sections + field bindings, with per-artifact dirty tracking. Config sections stop owning local state / POSTing per-section; the inspector's edits feed the same working copy.
  • Save as new version: a forkContractLayer workspace action + a version-prompt dialog (reuse SaveContractModal's version input; add a suggested patch bump). It assembles the fork request from the dirty artifacts, calls api.contracts.fork, then updates selectedContract and router.push to /editor/contracts/{name}/{newVersion}.
  • Remove the in-place paths: upsert_field_bindings, the field-bindings PATCH in-place body, revision/expected_revision (field-bindings service
  • routes + the PUT-metadata check), FieldBindingsRevisionConflict, and the 6 UI revision sites. The field-bindings GET stays. Concurrency becomes the (name, version) unique constraint → 409, surfaced as "version exists".

Phase 2c / Phase 3 — cleanup

  • Drop the revision column (Alembic); remove now-dead per-section /metadata/* POST routes if fully unused; docs + changelog; close PAC-2b; mark contract-artifact-ownership.md withdrawn in the PAC backlog.

Phase 2 risks

  • The three-save-path unification is the bulk of the effort (config sections must stop owning state). Largest single editor-state change.
  • Projection parity must be correct (lineage "where-used" + the deprecated/missing-concept health checks read projections when the SQLAlchemy backend is enabled).
  • Version routing/state sync — after a fork, the URL + selectedContract must move to the new version or the editor shows a stale version.

Tracked follow-ups (MUST ADDRESS)

CIV-FOLLOWUP-1 — Unify the canvas and field-bindings workbench onto one binding state (MUST address)

Status: open · Owner decision (2026-06-08): proceed with the pragmatic "workbench-authoritative" resolution in increment 5a, but this unification is required follow-up — not optional.

Problem. The contract editor has two editors over the same field_bindings artifact that are not in sync: 1. No live coupling — useContractFieldBindings (the workbench/inspector, shared by the inspector, form view, and legacy modal) is a standalone hook that GET/PATCHes bindings against the server and never touches the canvas/workspace store. The canvas derives field_bindings from its mapping edges (buildFieldMappingJSON). Edits in one do not propagate to the other; they reconcile only on save + reload. 2. Lossy canvas representation — a canvas mapping-edge projects to { concept, concept_scheme: '', tags: [] } with no confidence (json-builders.ts). It hardcodes concept_scheme (the source ontology / domain_schema), tags, and omits confidence — exactly the fields the workbench owns. So the canvas cannot round-trip the workbench's data even with perfect live sync.

Interim resolution (increment 5a). Workbench is authoritative: binding edits feed the contract working copy and forkContractLayer prefers the working-copy field_bindings over the canvas projection (canvas projection is the fallback when the workbench was not used). No workbench data is lost. Residual gap: editing bindings on the canvas and in the workbench in the same session before forking — the workbench copy wins and canvas-only edge edits since load can be dropped.

Required end state. Enrich the canvas mapping-edge model to carry concept_scheme / tags / confidence, and unify both surfaces onto one shared store-backed binding state (single source of truth) so the canvas and the workbench are always in sync and the residual gap disappears. Until this lands, the editor can silently drop cross-surface binding edits — track and prioritise it.

CIV-FOLLOWUP-2 — Reconcile domain_schema vs concept_scheme (model drift vs terminology.md)

Status: open · Owner decision (2026-06-08): document as tracked debt now; reconcile deliberately later (no behaviour change — the binding flow works as-is).

Problem. docs/proposals/terminology.md (accepted) treats two things as orthogonal: a Domain schema reference (domain_schema) is a contract's pointer to a LinkML schema (shape); a concept scheme ref is what a binding carries when it targets a concept (meaning). The editor conflates them: - the field-bindings hook uses the contract's domain_schema value as the concept_scheme on primary bindings (PrimaryBindingPicker sets concept_scheme: sourceOntology, sourceOntology = domainSchema); - the inspector concept-scheme picker lists concept schemes (/semantic/schemes = ConceptSchemeModel) but stores the choice into domain_schema.

So today domain_schema is effectively a concept-scheme ref, not a LinkML-schema ref. Reconcile the data model + UI to terminology.md (separate the shape pointer domain_schema → LinkML from the meaning pointer concept_scheme), or amend terminology.md if the conflation is intended. Decide before any UI copy sweep settles on a binding-context label, so the label matches the resolved model.

CIV-FOLLOWUP-3 — Editor UI terminology compliance with terminology.md

Status: open. The accepted glossary retires "ontology" (forbidden) and "source ontology", and fixes Vocabulary = the pillar/system while a concept scheme = one named, versioned collection of concepts ("Vocabulary instance" is a retired synonym for concept scheme). The editor UI is non-compliant: ~41 component files still say "ontology". The binding concept-scheme picker has been brought to "concept scheme" already; the remaining sweep (purge "ontology"/"source ontology" elsewhere) is incremental — it touches copy + snapshot/text assertions — and the binding-context label is gated on CIV-FOLLOWUP-2.

CIV-FOLLOWUP-4 — Drop the unused revision column + serve-side plumbing (deferred)

Status: open (deferred from Phase 3). After the cutover the revision column on data_contracts is unused (no concurrency), but it is still threaded through: the SQLAlchemy + Mongo store serializers (_sqlalchemy_connection.py, _mongodb_queries.py), contract_parse_service serialization, ContractListItem. revision, the four item["revision"] response sites + revision=0 in contracts.py, and publish_contract_event (the contract-change event stream carries it). Dropping it is therefore a multi-system change (model + 2 serializers + service + event bus + responses + an Alembic batch_alter_table drop on pycharter.data_contracts) for purely cosmetic benefit. The original proposal sanctioned deferring it ("keep the column initially; drop in a later migration"). Do it as a focused, well-tested PR when convenient; verify the contract event stream + ContractListItem consumers first.