Skip to content

PyCharter — Configuration Template Harmonization

Status: implemented (2026-07-12; TH-1..9 shipped, TH-3 superseded by kubernetes-style-manifests.md) · Created: 2026-07-10 · Audience: every contributor · Builds on: contract-foundation-unification.md, ref-unification.md, Artifact format versioning

PyCharter's thesis is configuration-based engineering: the YAML artifact is the product surface. Today that surface speaks four envelope dialects. The contract puts identity inside metadata; the pipeline puts id/version at the top level; the streaming worker and bridge nest everything under a streaming:/bridge: wrapper key; the concept-scheme templates nest under a template: key with a fifth metadata shape and no format_version at all. The central format gate (_format_migrations) covers two of five artifact kinds.

This proposal collapses all of it into one envelope. Nine TH items, hard cut, no dual-accept code paths — the same principle as the T-series (names) and the CF-series (contract shapes), applied to the artifact envelope. A one-shot converter (pycharter migrate) ships before any break so users and CI rewrite old files mechanically.

Why

Concrete costs of the current four-dialect situation:

  1. Per-kind glue everywhere. The UI editor needs envelope-aware loaders (src/pycharter/ui/src/lib/editor/bridge-config-loader.ts); the stores unwrap/rewrap per kind (streaming/_config_io.py::streaming_dict_from_envelope, streaming/_bridge_config_io.py::bridge_dict_from_envelope); no generic tooling (lint, diff, catalogue) can be written over "a pycharter YAML artifact".
  2. Identity is spelled three ways. Top-level id + version (pipeline), nested id + version (streaming/bridge), and metadata.title + metadata.version (contract). The same concept — "which stored artifact is this, at which content version" — cannot be read uniformly. This is the envelope-level twin of ref-unification.md.
  3. The format gate has holes. _format_migrations/registry.py registers only contract and concept_scheme. The pipeline gate is a hardcoded constant in semantic/pipelines/loader.py (PIPELINE_FORMAT_VERSION = "1"); streaming and bridge rely on a Pydantic default; concept-scheme templates, the pipeline snippets/settings.yaml, and the standalone data-quality template carry no version key at all. A future shape change has no uniform upgrade or rejection story.
  4. metadata.version duplicates top-level version in pipeline, streaming, and bridge configs (and is required by StreamingMetadataBlock in streaming/config.py). Two fields, one meaning, guaranteed drift.

Current state (verified)

Kind Wrapper key format_version Identity Metadata block
Contract none yes ("1") metadata.title + metadata.version contract-specific (status, type, ownership.roles, …)
Pipeline none yes ("1", loader-local gate) top-level id + version title, version, description, owner, tags, namespace
Streaming worker streaming: yes (nested) nested id + version same as pipeline (nested)
Bridge bridge: yes (nested) nested id + version same as pipeline (nested)
Concept-scheme template template: no flat id + name none — flat id/name/description/category/tags
Pipeline snippets/settings.yaml none no free name/version keys none
Standalone template_data_quality.yaml none no none none

Two clarifications that shrink the blast radius:

  • Store documents are already wrapper-free. The streaming: / bridge: wrappers exist only in YAML files; persisted documents in streaming_store/ / bridge_store/ are stored unwrapped. The store migration is a stamp-and-field-drop, not a structural rewrite.
  • Variable substitution is one engine, not three. The ${VAR} / ${VAR:-default} / ${VAR:?required} resolver lives in pipeline_generator/context.py; streaming/_variables.py and the two interpolate_env helpers are thin wrappers. TH-7 is a relocation and wrapper collapse, not a reimplementation.

Scope

Nine items. Each phase lands as its own atomic change set.

ID Title Touches
TH-1 One artifact envelope Every artifact template, loader, and schema model
TH-2 Artifact vs fragment taxonomy Template directory contracts, converter classification
TH-3 Contract identity moves to top-level id + version DataContractSchema, contract_parser, contract_store, ODCS I/O
TH-4 Per-kind envelope rewrites Templates, seeds, examples, docs per kind
TH-5 Full format-registry coverage, hard break _format_migrations/registry.py + every loader
TH-6 pycharter migrate one-shot converter New CLI subcommand + _format_migrations/steps.py
TH-7 One variables engine in shared/ pipeline_generator/context.py, streaming/_variables.py, _config_io
TH-8 Shared retry/timeout/observability policy blocks shared/policies.py, batch settings model
TH-9 Data-quality scheduler fields stay informational Doc note only

Principles

  1. One envelope per artifact. Same top-level keys, same order, same semantics, for every loadable kind.
  2. Hard cut, converter first. Old shapes are rejected at load time with an error that names pycharter migrate. No dual-accept windows, no read-time upgrade steps in the runtime — pre-1.0 stability policy applies, and this repo owns every consumer.
  3. Fragments are not artifacts. Copy-paste snippets deliberately carry no envelope, and the converter must never touch them.
  4. The envelope carries identity and cataloguing; the body carries behavior. Nothing in metadata may change runtime semantics.

Decisions

TH-1 — One artifact envelope

Normative shape for every loadable artifact:

format_version: "2"        # required; string; per-kind current version
id: <namespace>/<name>     # required; slash-qualified; name segment
                           #   matches ^[a-z0-9_-]+$
version: "1.0.0"           # required; the artifact's content version
metadata:                  # optional cataloguing block
  title: Orders stream     # display string, free-form
  description: ...
  owner: data-platform
  namespace: <namespace>   # MUST equal the id prefix when present
  tags: [streaming, orders]
  # kind-specific extensions allowed; the contract keeps status, type,
  # ownership.roles, lifecycle, governance_rules, lineage, runtime, …
variables: {...}           # optional; runnable kinds only
<kind body at top level>   # steps: | streams: | bridges: |
                           # json_schema: + rules | concepts:

Rules:

  1. Wrapper keys are deleted: streaming:, bridge:, template:.
  2. metadata.version is deleted everywhere. It duplicated the top-level version. StreamingMetadataBlock.version (currently required) is removed; same for the pipeline and bridge metadata models.
  3. metadata.namespace stays but is validated equal to the id prefix. (The current bridge template's namespace: example/firehose-bridge — namespace including the name — is exactly the drift this rule prevents.)

    Amended at Phase 3b (2026-07-11, approved): the streaming/bridge config store and the editor deliberately use metadata.namespace as a free-form virtual browse folder decoupled from the id (regression tests guard reorganizing the browse tree without renaming ids), so the equality rule is not enforced by the runtime models. Repo-owned YAML (templates, seeds, examples) still follows the prefix convention by hand; the bridge template's name-including namespace drift was fixed.

  4. format_version values are strings, matching the registry's FORMAT_KEY semantics.
  5. variables: is allowed only on runnable kinds (pipeline, streaming worker, bridge). Contracts and concept schemes are declarative artifacts; they do not interpolate environment state.

The two version axes stay deliberately separate, as documented in Artifact format versioning: format_version = how the document is serialized; version = what the content is.

TH-2 — Artifact vs fragment taxonomy

Two tiers, stated in each template directory's README:

  • Artifacts — loadable, storable, format-versioned. Get the TH-1 envelope: contract, pipeline, streaming worker, bridge, concept-scheme template, persisted concept scheme.
  • Fragments — copy-paste bodies that are assembled into an artifact. Carry no envelope: everything under data/templates/pipeline/snippets/, and the per-section contract templates (template_json_schema.yaml, template_coercion_rules.yaml, template_validation_rules.yaml, template_field_bindings.yaml, template_data_quality.yaml, template_sla.yaml, template_metadata.yaml).

This resolves the "no format_version" rows in the current-state table by design rather than by stamping: fragments are exempt, and the converter classifies them as skipped: fragment (TH-6). Side fix: snippets/settings.yaml currently carries free name / description / version keys — it becomes a pure fragment with a comment-only header.

TH-3 — Contract identity moves to top-level id + version

VETOED (2026-07-11). The maintainer ruled against moving contract identity to the root: identity should live in metadata, following Kubernetes manifest design (metadata.name + metadata.namespace) rather than this proposal's top-level id. TH-3 / Phase 3d will not be implemented in this shape. The direction is superseded by kubernetes-style-manifests.md, which applies the K8s envelope (apiVersion / kind / metadata-identity / spec:) uniformly to all artifact kinds — including re-migrating the kinds that adopted this ADR's top-level id/version envelope in Phases 3a–3c. Everything else in this ADR (wrapper-key deletion, metadata.version de-duplication at the time, the converter, the registry, the CI ratchet) stands and carries forward as the migration machinery for the successor proposal.

The hard call. Today contract identity is metadata.title (store key, pattern ^[a-z0-9_]+$, doing double duty as display text) plus metadata.version. Decision: top-level id and version become contract identity; metadata.title becomes pure display; the store keys on the id tail.

# Before                          # After
format_version: "1"               format_version: "2"
json_schema: {...}                id: examples/intraday_trade_order
metadata:                         version: "1.0.0"
  title: intraday_trade_order     json_schema: {...}
  version: "1.0.0"                metadata:
  status: active                    title: Intraday trade order
  ...                               status: active
                                    ...

Consequences and why this is the right cut:

  • Strengthens contract-immutable-versions.md. The contract version is the identity of a frozen tuple of artifact pins; the versions: pin block stays and is now visibly distinct from contract identity instead of sharing a metadata block with it.
  • metadata.title is freed for humans. The ^[a-z0-9_]+$ pattern moves to the id tail; the title becomes display text, ending its dual identity/display duty.
  • ODCS round-trip improves. ODCS v3 has top-level id / name / version; contract_io/odcs.py and contract_io/from_odcs.py map them 1:1 instead of digging into metadata.
  • contract_store keeps its columns. Store records already key on (name, version); only the source of truth in the document changes (id tail → name column).

Tradeoffs recorded:

  • Largest blast radius of any kind: DataContractSchema (db/schemas/_data_contract_schema.py, extra="forbid"), contract_parser/parser.py, store read paths, every seed / template / test contract, API create paths, and the UI contract editor. Hence it is sequenced last (Phase 3d).
  • Interlocks with in-flight CF-1 (one canonical contract model). If CF-1 lands first, TH-3 is one model edit; if TH-3 lands first, CF-1 rebases trivially. Whoever lands second coordinates.

Rejected alternatives:

  • Option A — identity stays in metadata. Forfeits harmonization for the flagship kind; every generic tool still needs a contract special case.
  • Option C — top-level id, version stays in metadata. Splits one identity across two levels; worse than either consistent choice.

TH-4 — Per-kind envelope rewrites

Templates rewritten by hand (comments curated), in src/pycharter/data/templates/:

  • Streaming (streaming/worker.yaml): unwrap streaming:, drop metadata.version → format_version: "2". id and version become required (StreamingConfig currently types them str | None).
  • Bridge (bridge/bridge.yaml): same → "2". Fix the namespace drift noted in TH-1.
  • Pipeline (pipeline/template_pipeline.yaml, template_pipeline_minimal.yaml): already unwrapped; drop metadata.version, apply the namespace rule → "2". The bump is deliberate even though the change is small: a hard-break signal beats silently rejecting a field that used to be required.
  • Contract (contract/template_contract.yaml, template_contract_minimal.yaml; template_metadata.yaml fragment updated to match): TH-3 → "2".
  • Concept-scheme templates (concept_schemes/*.yaml, 6 files): unwrap template:; name → metadata.title; description / tags / category move into metadata; add id (builtins are namespace-qualified as pycharter/<name>, e.g. pycharter/ecommerce) and version: "1.0.0"; stamp format_version: "1" under the new registry kind concept_scheme_template — "1" because this kind never had a versioned shape before, and a distinct kind name because bare "template" is overloaded.
  • Persisted concept scheme (registry kind concept_scheme, current "1"): audit ConceptScheme.to_dict() (semantic/concepts/models.py) for envelope conformance. Expected outcome: already flat, stays "1"; if metadata.version-style duplication is found, bump alongside. Open verification item for Phase 3a.

TH-5 — Full registry coverage, hard break

src/pycharter/_format_migrations/registry.py registers all six artifact kinds:

Kind Current version after this proposal
contract "2"
pipeline "2"
streaming "2"
bridge "2"
concept_scheme "1"
concept_scheme_template "1"

The default (runtime) migrator carries no upgrade steps. An old shape fails with a clear error naming the converter:

UnsupportedFormatVersion: streaming config uses format_version "1"
(current: "2"). Run `pycharter migrate <path>` to rewrite it.

Loader changes, all routed through upgrade(kind, ...):

  • semantic/pipelines/loader.py — delete the local PIPELINE_FORMAT_VERSION constant; delegate to the registry.
  • streaming/_config_io.py::load_streaming_config_dict — delete the lenient dual-accept of {streaming: {...}}; a wrapper key is a hard error pointing at the converter.
  • streaming/_bridge_config_io.py — same for {bridge: {...}}.
  • semantic/templates/engine.py::load_template — delete the data.get("template", data) fallback.
  • contract_parser/parser.py — already wired; version bump only.

Subtlety: registry semantics treat an absent format_version as current. For kinds whose old shape lacked the key entirely (concept-scheme templates), the wrapper key is the discriminator — wrapper-key detection runs before upgrade() so an old file cannot be mistaken for a current one.

TH-6 — pycharter migrate: the one-shot converter

New subcommand in src/pycharter/cli.py. Ships in Phase 2, before any shape breaks.

UX:

pycharter migrate PATH...            # files, dirs (recursed *.yaml|*.yml), globs
pycharter migrate --write PATH...    # apply; default is dry-run
pycharter migrate --kind KIND PATH   # override detection
pycharter migrate --stores [--database-url ...]
  • Dry-run by default: per-file classification (streaming v1 → v2, fragment — skipped, already current) plus a unified diff. Exits non-zero when changes are pending, so CI can enforce "no stale configs".
  • Detection heuristics (order matters; wrapper keys first): top-level streaming: → streaming v1; bridge: → bridge v1; template: with concepts → concept_scheme_template v0; steps: → pipeline v1; json_schema: + metadata: → contract v1; rules/targets-only or step-body-only documents → fragment, skipped.
  • Engine reuse without weakening the hard break: the conversion functions are plain dict→dict FormatSteps in a new _format_migrations/steps.py, registered on a separate FormatMigrator instance owned by the CLI with chains ("1", "2", fn) / ("0", "1", fn). The default runtime migrator stays step-free; the steps are unit-testable alongside tests/unit/test_format_migrations.py.
  • Comment fidelity: yaml.safe_dump loses comments. Repo-owned YAML (templates, seeds, examples) is migrated by hand in the phase PRs so comments stay curated; for user files the CLI prints a "comments are not preserved" warning. ruamel.yaml was considered and rejected — a permanent dependency is not worth a one-shot conversion's comment fidelity.
  • --stores mode: iterates documents in contract_store, pipeline_store, streaming_store, and bridge_store, applies the same step functions to each document payload, rewrites in place; dry-run prints per-kind counts. Streaming/bridge store documents are already wrapper-free (see Current state), so their migration is stamp + metadata.version drop; the contract store additionally gets the TH-3 identity edits.

TH-7 — One variables engine in shared/

Move the ${VAR} / ${VAR:-default} / ${VAR:?message} resolution engine (resolve_declared_variables + substitution) from pipeline_generator/context.py to src/pycharter/shared/variables.py (precedent: shared/refs.py from ref-unification, shared/enums.py from CF-2). Then:

  • pipeline_generator/context.py re-exports for the batch pipeline.
  • streaming/_variables.py is deleted.
  • The two near-identical interpolate_env merge helpers in streaming/_config_io.py and streaming/_bridge_config_io.py collapse into one shared resolve_variables(doc, overrides=None).

This also fixes a layering inversion: streaming currently imports from pipeline_generator. Semantics are unchanged — this is a relocation. Per TH-1 rule 5, contracts and concept schemes do not gain variables:.

TH-8 — Shared policy blocks (scoped)

In this proposal (additive, no behavior change to existing configs):

  • Move RetryConfig from pipeline_generator/_config_extract.py to shared/policies.py as RetryPolicy and add a TimeoutPolicy sibling. Batch HTTP/GraphQL sources keep their behavior; the shapes become reusable by any future source.
  • Move ObservabilityConfig from streaming/config.py to shared/ and add an optional settings.observability block to the batch CanonicalPipelineConfig (semantic/pipelines/models.py) — closing the gap where only streaming/bridge workers have health_port / metrics_port / log_format.

Explicitly deferred (see Out of scope): executing retry/backoff on streaming sources, typing StreamConfig.source/load/dlq, resource limits, secret backends.

TH-9 — Data-quality scheduler fields stay informational

data_quality.default_scheduler / default_schedule remain informational for external schedulers — pycharter does not execute schedules, and the data-quality template is a fragment (TH-2). Noted here so the question is settled, not re-litigated per review.

Execution

Phases are sequential; each is one coherent change set with its own tests green.

Phase 1 — shared substrate (no shape change)

TH-7 + TH-8 relocations. Verify: existing suites green (tests/unit/streaming/, tests/unit/test_pipeline_config_loader.py, tests/test_extractors.py, tests/test_import_smoke.py).

Phase 2 — converter first

TH-6: _format_migrations/steps.py, the pycharter migrate subcommand (files mode), and fixtures covering every old shape. Verify: new tests/unit/test_migrate_cli.py; extended tests/unit/test_format_migrations.py. Nothing breaks yet.

Phase 3 — per-kind hard breaks, smallest blast radius first

Each sub-phase is atomic: model + loader + registry + templates + seeds + examples + UI + docs in one change set.

  • 3a — concept-scheme templates. semantic/templates/engine.py, semantic/concepts/schema_loader.py, the 6 template files, registry entry concept_scheme_template, the TH-4 concept_scheme conformance audit. Tests: tests/unit/test_domain_templates.py.
  • 3b — streaming + bridge together. streaming/config.py, streaming/bridge_config.py, both _config_io modules, api/models/streaming_configs.py, api/models/bridge_configs.py, api/routes/v1/streaming_configs.py, api/routes/v1/bridge_configs.py, UI (src/pycharter/ui/src/lib/editor/bridge-config-loader.ts, stores/file-browser.ts, lib/api/index.ts, components/pipelines/NewStreamingConfigDialog.tsx, NewBridgeConfigDialog.tsx, and the streaming UI tests), examples/streaming/orders/stream-worker.yaml, docs/reference/streaming-config-schema.md. Tests: tests/unit/streaming/, tests/unit/bridge_store/, tests/test_seed_bridge_configs.py, tests/test_local_kafka_bridge_example.py.
  • 3c — pipeline. semantic/pipelines/loader.py, models.py, serializer.py, pipeline templates, seed pipeline configs. Tests: tests/test_seed_pipeline_configs.py, tests/test_pipeline_config_models.py.
  • 3d — contract (TH-3). db/schemas/_data_contract_schema.py, contract_parser/parser.py, contract_io/odcs.py, contract_io/from_odcs.py, semantic/contracts/serializer.py, contract templates, src/pycharter/data/seed/examples/ecommerce/contracts/, files served by api/services/template_service.py, UI contract editor. Tests: tests/test_contract_parser.py, DQC suites, ODCS round-trip. Gated on CF-1 coordination at land time.

Phase 4 — stores + CI ratchet

pycharter migrate --stores; a CI step that round-trips every YAML under data/templates/, data/seed/, examples/, and tests/fixtures/ through upgrade() (the CF risk-register mitigation, borrowed verbatim); update the current-version table in Artifact format versioning.

Risk register

Risk Mitigation
Comment loss on user YAML during conversion CLI warns loudly; repo-owned YAML hand-curated in phase PRs; ruamel.yaml rejected as a permanent dep for a one-shot need
Collision with in-flight CF-1 on the contract model 3d sequenced last and explicitly gated on CF-1 status; whichever lands second coordinates
UI/backend lockstep for the streaming/bridge editor 3b is atomic — UI files and their tests land in the same change set
External consumers of REST response shapes Store envelopes ({name, version, document}) are unchanged; only YAML file shape changes. Stated explicitly per python-api-consolidation.md's wire-format stance
Old YAML embedded in notebooks / archived docs Grep sweep per phase; docs/proposals/archived/ and historical guides exempt
pyintake seam DTO-mediated (docs/reference/integration-seam.md); unaffected — verify once in Phase 3b

Out of scope / deferred

  • Reintroducing apiVersion / kind. Considered and rejected: the K8s envelope adds dispatch machinery pycharter does not need — kind is implied by the loader entry point and store, and a uniform envelope delivers the tooling benefits without a new required key.
  • Streaming-source retry/backoff execution semantics (consumer reconnect behavior) — feature work belonging with streaming-first-class.md.
  • Typed StreamConfig.source/load/dlq (today dict[str, Any], validated by builders) — large, and interacts with ref-unification.
  • Resource limits (memory/CPU/queue depth) — pycharter has no executor to enforce them; revisit if a scheduler/executor ever lands.
  • Secret resolution beyond ${VAR} env interpolation (vault/secret- manager references).
  • variables: on declarative kinds (contracts, concept schemes).

Acceptance

  • One envelope shape (TH-1) in every artifact under data/templates/; fragments carry none (TH-2).
  • All six artifact kinds registered in _format_migrations/registry.py; every loader routes through upgrade(); lenient fallbacks deleted.
  • Zero wrapper keys and zero metadata.version occurrences in repo YAML (templates, seeds, examples, fixtures).
  • pycharter migrate converts every fixture of every old shape; dry-run/diff/exit-code contract covered by tests.
  • Gate errors for old shapes name pycharter migrate.
  • CI round-trip step green over data/templates/, data/seed/, examples/, tests/fixtures/.
  • Current-version table in Artifact format versioning updated.