Artifact Format Versioning

PyCharter persists several artifact documents — data contracts, concept schemes, lineage records — that downstream consumers read and write. As pycharter evolves, the shape of those documents can change. DG-2 adds a format_version header to each persisted artifact plus a read-time registry so only the current document shape is accepted, or a clear versioned error is raised for anything else.

This is distinct from two existing mechanisms:

• Database schema migrations (alembic, under db/migrations/) reshape tables. Format versioning validates documents. Alembic covers the SQL backends only — see Store backend support tiers for what that means on MongoDB / Redis.
• Content versions (scheme.version, field_bindings.version, KBVersion.version) track what changed in the data. format_version tracks how the document is serialised.

The manifest envelope (current)

Every pycharter artifact file carries the unified manifest envelope (KM, 2026-07-12):

api_version: pycharter.io/v2   # the format marker (replaces format_version)
kind: StreamingWorker          # DataContract | Pipeline | PipelinePlan | PipelineBundle |
                               #   StreamingWorker | Bridge | ConceptScheme
metadata:
  name: orders                 # identity slug
  namespace: example           # identity scope
  version: "1.0.0"             # content version
spec:                          # the kind body
  streams: [...]

Identity is (metadata.namespace, metadata.name, metadata.version) (KM-1, NS-1). For data contracts this is enforced end-to-end: the store keys rows on it (uq_data_contracts_ns_name_version), the API accepts a namespace parameter and returns 409 with the candidate namespaces when a name-only lookup is ambiguous, and the editor threads the namespace through load and save. A null/absent namespace means the global namespace.

The artifact layer (NS-9). A contract composes five separately stored artifacts. metadata_records is the contract's identity record — keyed (namespace, name, version), in lockstep with the contract row. The other four (json_schemas, coercion_rules, validation_rules, field_bindings) form an immutable global pool keyed (name, version) with no namespace: any contract may reuse them by reference (*_name + *_version request fields; the retired *_title spellings are rejected with a 422). Writing an existing (name, version) with identical content is a no-op; different content is rejected with a 409 (ArtifactContentConflictError) — bump the version or pick another name. Inline-authored artifacts derive namespace-prefixed pool names (logistics_order_coercion_rules) so contract twins never collide by default. Pool artifacts record their creator as nullable provenance (SET NULL on contract delete) — deleting a contract never destroys artifacts other contracts or versions reference; orphaned-artifact GC is an explicit non-goal.

Verdict-deciding sections. A contract version's data_quality.rules and strict_types also decide whether data passes, so they follow the same rule: they may be written once (a contract saved without rules can gain them later), and a later write that changes them is rejected with the same 409 (ArtifactContentConflictError); bump the version instead. An update whose metadata omits strict_types keeps the stored value. data_quality.targets (edited by the quality-threshold writeback), the SLA sections, the policy sections and the rest of the metadata stay editable at a version.

Serialization preserves supplied extraction, AI-usage, and KB-capture policy mappings, including {}, without expanding their defaults. Absent and null policies are omitted from canonical exports. Preserving an existing declaration does not change the document format or any artifact version; no format migration is required. Earlier parsed-contract exports could lose empty declarations, which cannot be recovered from an already incomplete stored record. Re-export the original reviewed configuration under the existing version/update rules; see Retained pipeline components.

Current authored manifests use metadata.name as both identity and display label; metadata.title is retired. Use the explicit migration command for historical documents rather than authoring a second display/identity field.

PipelinePlan is a generated offline inspection artifact with the same envelope. Its owning definition's metadata.version is distinct from the format version. It has no historical format, store representation, or migration chain. The migration CLI skips generated plans. Its scoped structural digest does not identify executable content; see Offline pipeline plans for the separate definition, bundle, deployment, and invocation identities.

PipelineBundle is an additive generated artifact for retained native components. It shares the current envelope and likewise has no historical format, store representation, or migration chain. Its reader requires explicit api_version and kind; the migration CLI skips it, including explicit kind overrides. Scoped component checksums identify admitted retained behavior, while the full bundle_digest remains null and frozen/verified/executable claims remain false. Generated bundles declare a closed inner capability scope. The original supplied_native_components_v1 representation remains unchanged. Explicit contract-file capture uses resolved_file_components_v1 and a required capture: supplied_file_contract_v1 attachment variant, keeping authored unknown pins separate from caller-expected and resolved identities. New readers accept both scopes; older readers reject the new one. This additive inner mode does not change the authored manifest format or artifact/constituent versions. See Retained pipeline components for supported rules, credentials, unresolved bindings, and execution limits. Pipeline, contract, and constituent artifact versions retain their existing meanings.

The internal registry token for pycharter.io/v2 is km2. Per D8, the runtime carries no backward-compatibility chains: an authored file declares api_version: pycharter.io/v2 or loading fails with UnsupportedFormatVersion. Historical formats and where they convert:

Kind	Historical formats	How they reach km2
Data contract	"1" (canonical sections)	pycharter migrate only
Concept scheme	"1" (scheme: header + concepts:)	pycharter migrate only
Pipeline	"1", "2", "3" (TH envelope)	pycharter migrate only
Streaming config	"1", "2" (TH envelope)	pycharter migrate only
Bridge config	"1", "2" (TH envelope)	pycharter migrate only
Lineage record	"1" (stamped on write only)	—

(The former ConceptSchemeTemplate kind was retired; concept-scheme templates are ordinary ConceptScheme manifests.)

Stores are envelope-free (KM-6): persisted store documents keep the canonical section shape stamped format_version: "1"; the manifest is the file format. The single runtime step per kind is the store-document → manifest wrap — not a legacy upgrade. Loaders normalize everything through upgrade(kind, doc) followed by unwrap_manifest(kind, doc).

A camelCase apiVersion + kind: DataContract document is a foreign ODCS document, not a pycharter manifest — it loads through the ODCS codec (contract_io.from_odcs) anywhere a contract is accepted.

The TH-1 harmonized envelope (top-level format_version/id/version, body at the root) is historical: the runtime rejects it, and pycharter migrate rewrites files in place.

Read-time validation

pycharter._format_migrations holds the registry. Each artifact kind declares a current version with read-time upgrade steps:

from pycharter._format_migrations import upgrade, current_version
from pycharter._format_migrations.manifest import unwrap_manifest

doc = upgrade("contract", raw_dict)  # -> manifest (api_version stamped)
sections = unwrap_manifest("contract", doc)  # -> canonical sections
current_version("contract")  # -> "km2"

upgrade:

1. reads api_version when present (else format_version; a document with neither is treated as the newest pre-manifest format, since manifests always declare api_version),
2. wraps store-shape documents and stamps api_version: pycharter.io/v2,
3. raises UnsupportedFormatVersion(kind, saw, expected) for unknown versions or groups.

Loaders that consume the body models call unwrap_manifest after the upgrade — the Pydantic body models validate the internal section shape verbatim (KM-6), and unwrap(wrap(doc)) == doc is pinned per kind in tests/unit/test_manifest_envelope.py.

Changing the document shape

When an artifact's body shape changes:

1. Update templates, seeds, and serializers to the new shape.
2. Bump the envelope group version for the kind (pycharter.io/v2) and register the read-time step in pycharter/_format_migrations/registry.py + the converter chain in steps.py.
3. Rewrite repo artifacts with pycharter migrate — the CI format ratchet (tests/unit/test_artifact_format_ratchet.py) fails until every file is current.

Breaking release: field bindings terminology (2026-05)

The following renames apply after pycharter db upgrade (revision rename_field_bindings):

Before	After
Table field_mappings	field_bindings
Column data_contracts.field_mapping_id	field_bindings_id
Contract YAML field_mapping / ontology artifact	field_bindings
store.get_field_mapping / store_field_mapping	get_field_bindings / store_field_bindings
versions.field_mapping	versions.field_bindings

Legacy contract keys field_mapping and ontology at the top level are rejected at parse time.

Stability commitment

The format_version header is part of pycharter's stable persisted contract (see the Stability Policy). Within a major version, artifacts must be written at the current format version.