PyCharter — Kubernetes-Style Manifests (KM)

Status: APPROVED 2026-07-12 (templates as written are the acceptance artifacts) · IMPLEMENTED 2026-07-12 — KM Phase 1 (envelope core), Phases 2a–2d (streaming+bridge, pipeline, concept schemes, contracts), and Phase 4 (docs + acceptance) shipped; KM Phase 3 (contract ODCS codec) shipped earlier. The normative format reference is artifact-format-versioning.md.

Supersedes the vetoed TH-3 (contract identity to top-level id) in config-template-harmonization.md and revises that ADR's TH-1 envelope for every kind. Maintainer direction (2026-07-11): artifact identity belongs inside metadata, following Kubernetes manifest design, uniformly across all kinds — not at the document root.

Why

The TH-1 "harmonize-only" envelope (top-level format_version / id / version, kind body at the top level) shipped for concept-scheme templates, streaming, bridge, and pipeline in TH Phases 3a–3c. Contracts never moved (TH-3 vetoed). Two problems remain:

1. Contracts diverge forever under the harmonize-only plan — the flagship kind keeps identity in metadata while every other kind carries it at the root, so generic tooling still needs a contract special case. (Resolved here: KM moves identity into metadata for every kind — the contract's existing shape generalizes instead of deviating — and per 2026-07-12 maintainer direction contracts wear the same envelope as everything else.)
2. The envelope is bespoke. Everyone who has touched Kubernetes already knows apiVersion / kind / metadata / spec (pycharter spells the first api_version — keys follow the repo's snake_case YAML convention; the structure is what is adopted). Adopting it buys instant familiarity, a natural home for identity (metadata.name, scoped by metadata.namespace), a clean display/identity split (metadata.title for humans), and a future-proof extension surface (metadata.labels / metadata.annotations) that TH-1 lacked.

The migration machinery built for TH makes this cheap now: the format registry with read-time upgrade steps, pycharter migrate (files and --stores), and the repo-wide CI format ratchet all carry over unchanged in mechanism.

The envelope

api_version: pycharter.io/v1       # group/version — see KM-2
kind: StreamingWorker              # artifact kind — see KM-3
metadata:
  name: orders                     # identity slug: ^[a-z0-9_-]+$
  namespace: example               # identity scope (replaces id prefix)
  version: "1.0.0"                 # content version; store keys on
                                   #   (namespace, name, version)
  title: Orders stream             # display text, free-form
  description: ...
  owner: data-platform
  labels: {}                       # optional selector-style key/values
  annotations: {}                  # optional free-form extension data
spec:                              # the kind body, nested (K8s-style)
  variables: {...}                 # runnable kinds only
  streams: [...]                   # per-kind: steps: | bridges: |
                                   #   json_schema+rules | concepts:

Scope note (revised 2026-07-12, maintainer direction — one unified theme): KM-1..KM-6 apply to every kind, data contracts included. KM-7 governs the contract's at-rest content (canonical sections, ODCS via guaranteed codec) — that content nests under spec: unchanged, wearing the same envelope as every other kind. Foreign ODCS documents keep their own envelope and are detected by ODCS's camelCase apiVersion key.

Decisions

KM-1 — Identity lives in metadata

(metadata.namespace, metadata.name, metadata.version) is the artifact identity; stores key on it (existing (name, version) columns map from namespace/name + version — no schema migration; the document is the source of truth, exactly as TH-3 planned for its direction). metadata.title is pure display. This reintroduces metadata.version deliberately: with no top-level version there is no duplication — the TH-1 rationale for deleting it (two version fields) no longer applies, because there is now exactly one.

KM-2 — api_version carries format evolution

api_version: pycharter.io/v1 replaces the top-level format_version. The key is snake_case, deviating from Kubernetes' literal apiVersion spelling deliberately: every key in pycharter's YAML surface is snake_case (repo convention), and KM adopts the K8s structure, not its casing. A side benefit: a camelCase apiVersion + kind: DataContract document is unambiguously a foreign ODCS document (ingested via the KM-7 codec), while api_version + kind marks a pycharter manifest. The format registry keys on (kind, version-tail); read-time upgrade steps and the converter work exactly as today. Unknown group or version → the existing UnsupportedFormatVersion error. Per-kind format versions collapse into one envelope version per group — a kind that needs a breaking body change bumps the group version for its kind (pycharter.io/v2), same registry mechanics.

KM-3 — kind is explicit

DataContract, Pipeline, StreamingWorker, Bridge, ConceptScheme, ConceptSchemeTemplate — PascalCase values (the snake_case convention governs keys, not values, and ODCS's own kind value is PascalCase DataContract). EDIT POINT: switching to snake_case registry slugs (streaming, bridge, pipeline, concept_scheme, concept_scheme_template) is a one-line edit here and removes the kind-value → registry-key mapping table. Detection heuristics (detect_artifact's body-shape sniffing) become a fallback for legacy documents only; new documents declare their kind.

KM-4 — Kind body nests under spec:

The TH-1 top-level body (streams:, steps:, json_schema: + rules, concepts:) moves under spec:. variables: moves into spec for runnable kinds. Loaders validate spec with the existing Pydantic models — the models themselves do not change shape, only the document nesting above them.

KM-5 — One coordinated migration, existing machinery

Kind	Today	KM target
streaming	format 2 (TH-3b)	pycharter.io/v1 + spec nesting
bridge	format 2 (TH-3b)	same
pipeline	format 2 (TH-3c)	same
concept_scheme_template	format 1 (TH-3a)	same
concept_scheme (persisted)	format 1	same
contract	format 1 (TH-3 vetoed)	pycharter.io/v1 + spec nesting, same as every kind; canonical content and ODCS codec unchanged (KM-7)

Per kind: a converter step (file-level, comment-curated for templates; mechanical for seeds), a read-time upgrade step for stored documents, pycharter migrate --stores coverage, and the CI ratchet auto-enforces the moment the registry bumps (the contract-kind skip that has been in place since Phase 4 activates the same way).

KM-6 — What deliberately does not change

• Store schemas (columns stay; only the document shape moves).
• The Pydantic body models (StreamingConfig.streams, CanonicalPipelineConfig.steps, …) — they validate spec verbatim.
• REST path segments (3a's flat-id decision for template routes holds: metadata.name is the path segment).
• The suggestion queue, governance, and versioning systems.
• The in-memory contract runtime model — validation and DQ engines keep consuming the canonical sections (json_schema + rules); parse_contract unwraps spec: exactly as the other loaders do.

KM-7 — Contracts: canonical at rest, ODCS as a guaranteed codec (APPROVED)

Maintainer requirements (2026-07-11): the data contract artifact must be highly compatible with ODCS, remain extensible (coercion rules, validation rules, data-quality expectations), and keep powering pycharter's validation and DQ runtimes.

Decision (maintainer-approved 2026-07-11): the canonical pycharter contract format stays the at-rest and authored form. ODCS compatibility is delivered as a guaranteed codec, not a format replacement — three commitments:

1. Codec completion. Close the known lossiness gaps and promote pycharter concepts to native ODCS fields where the standard has them: nested object schemas emit full ODCS nested properties[]; x-database.sql_type → physicalType; primary_key → primaryKey; table name → physicalName; PII governance marks → per-property classification.
2. Lossless round-trip as a CI gate. canonical → to_odcs → from_odcs → canonical must be equivalent for every contract in the repo (templates, seeds, examples), enforced in CI alongside the format ratchet — the contract twin of VC-14's SKOS guarantee. The export can never rot again.
3. Native ODCS ingestion. .odcs.yaml / ODCS documents are accepted everywhere a contract is accepted — the validate CLI, pipeline and streaming contract: blocks, the contract store, and the API — auto-detected by kind: DataContract + apiVersion and loaded through from_odcs.

Why this over ODCS-native at rest (the previously drafted direction): hand-authoring stays ergonomic (2-line rules vs 10-line quality blocks; no enum-in-customProperties duplication); engine sections keep first-class, extra="forbid"-validated schema instead of unvalidated customProperties pair-lists; zero migration of the contract corpus; and the artifact's lifecycle is not coupled to the ODCS release treadmill (v3.1 deprecations / v4.0 removals). The interop delta is small because the gate keeps import/export provably fresh — roughly 80% of the interchange value at a fraction of the cost.

Deferred decision — ODCS-native at rest. Recorded, not rejected. Trigger condition: contracts become pycharter's public interchange artifact — routinely exchanged across organizational and tool boundaries as the primary product surface, where "the file IS the standard" is a position worth its costs. If triggered, the flip is a mechanical migration over the then-proven codec (the round-trip gate must have been green for at least one release cycle first). Until then, the contract's canonical content nests under the unified KM envelope like every other kind (2026-07-12 unification — fully consistent with the TH-3 veto, since identity stays in metadata). If the flip ever triggers, it replaces the whole document with ODCS's own envelope; pycharter.io/v1 is never an ODCS carrier.

Detection stays unambiguous (and the shipped native-ODCS ingestion keeps working): a foreign ODCS document spells its key camelCase (apiVersion: v3.x + kind: DataContract) and loads through from_odcs; a pycharter manifest spells it api_version: pycharter.io/v1 and loads through the KM loader.

Templates by kind (normative — edit here, execution follows)

These manifests are the acceptance artifacts for KM Phases 1–2: approved as written, data/templates/** and every seed/example is rewritten to match them, and the loaders/converters implement exactly this shape. Edit these blocks to change the design — comments marked # EDIT POINT flag the open choices.

StreamingWorker

api_version: pycharter.io/v1
kind: StreamingWorker
metadata:
  name: orders                    # identity slug ^[a-z0-9_-]+$
  namespace: example              # identity scope (was the id prefix)
  version: "1.0.0"                # content version; stores key on
                                  #   (namespace, name, version)
  title: Orders stream            # display, free-form
  description: Validate the orders.raw topic against the orders contract.
  owner: ${TEAM:-data-platform}
  tags: [streaming, orders]
  labels: {}                      # optional selector-style key/values
  annotations: {}                 # optional free-form extension data
spec:
  variables:                      # ${VAR} defaults; env still wins on
    KAFKA_BROKERS: localhost:9092 #   ${VAR:?required} / ${VAR:-default}
  streams:
    - name: orders-validator
      source:
        type: kafka               # kafka | websocket | sse | rabbitmq | sqs
        topic: orders.raw
        bootstrap_servers: ${KAFKA_BROKERS:-localhost:9092}
        consumer_group: pycharter-orders
        auto_offset_reset: latest
      contract:                   # same block as a pipeline step
        ref:
          type: file              # file | store
          path: ./contracts/orders
      sinks:
        valid:
          type: kafka
          topic: orders.valid
        dlq:
          type: kafka
          topic: orders.dlq

Bridge

api_version: pycharter.io/v1
kind: Bridge
metadata:
  name: firehose-bridge
  namespace: example
  version: "1.0.0"
  title: Example firehose bridge
  description: Republish a WebSocket firehose to a Kafka topic.
  owner: ${TEAM:-data-platform}
  tags: [bridge, websocket, kafka]
spec:
  variables:
    WS_URL: wss://example.com/stream
  bridges:
    - name: firehose
      source:
        type: websocket
        url: ${WS_URL:?required}
      target:
        type: kafka
        topic: firehose.raw
        bootstrap_servers: ${KAFKA_BROKERS:-localhost:9092}

Pipeline

api_version: pycharter.io/v1
kind: Pipeline
metadata:
  name: file_to_file
  namespace: examples
  version: "1.0.0"
  title: File To File
  description: Copy records from a CSV file to a JSONL export
spec:
  settings:                       # settings joins the body under spec
    lineage:
      enabled: false
  variables:
    INPUT_PATH: data/input/sample.csv
    OUTPUT_PATH: data/output/sample.jsonl
  steps:
    - id: extract_input
      type: extract
      source:
        type: file
        path: ${INPUT_PATH}
        format: csv
    - id: load_output
      type: load
      input: extract_input
      target:
        type: file
        path: ${OUTPUT_PATH}
        format: jsonl
        write_mode: overwrite

Step snippets (data/templates/pipeline/snippets/*.yaml) stay envelope-free fragments (TH-2) — they are pasted into spec.steps and are never loadable artifacts.

ConceptSchemeTemplate

api_version: pycharter.io/v1
kind: ConceptSchemeTemplate
metadata:
  name: party                     # builtin template ids stay flat —
                                  #   they are REST path segments (3a
                                  #   decision, KM-6); namespace omitted
  version: "1.0.0"
  title: Party Pattern
  description: >-
    The Party pattern models people and organizations that play roles
    in relationships.
  category: pattern               # kind-specific metadata extensions
  tags: [crm, contacts, organizations]
spec:
  concepts:
    - name: Party
      concept_type: entity
      definition: A person or organization that participates in business activities.
      alt_labels: [Actor, Participant]
    - name: Person
      concept_type: entity
      definition: An individual human being.
      broader: Party

ConceptScheme (persisted scheme document)

api_version: pycharter.io/v1
kind: ConceptScheme
metadata:
  name: EcommerceOntology         # EDIT POINT: scheme keys are
                                  #   PascalCase concept-scheme keys
                                  #   today. Either the name regex is
                                  #   relaxed for this kind (keeps every
                                  #   existing key + binding ref
                                  #   working), or schemes migrate to
                                  #   slugs with a key-mapping step.
                                  #   Default if unedited: relax the
                                  #   regex for ConceptScheme only.
  version: "2.0.0"
  title: Ecommerce Platform Ontology
  description: >-
    Full vocabulary covering identity, catalog, pricing, checkout,
    fulfillment, and returns.
spec:
  base_uri: https://vocab.example.com/ecommerce/   # optional (VC-14)
  concepts:
    - id: User
      label: User
      definition: A registered storefront account.
    - id: UserEmail
      label: User Email
      broader: User
      definition: Primary contact email for a user (PII).
      tags: [pii]

DataContract

Same envelope as every other kind (unified 2026-07-12). The canonical sections — the KM-7-approved at-rest content — nest under spec: unchanged; the rich governance blocks stay in metadata as kind-specific extensions (they are cataloguing/governance, exactly where contracts have always kept them):

api_version: pycharter.io/v1
kind: DataContract                # EDIT POINT: rename to `Contract` if
                                  #   you want zero overlap with ODCS's
                                  #   kind value. Detection is unambiguous
                                  #   either way — ODCS spells its key
                                  #   camelCase `apiVersion`.
metadata:
  name: example_record            # identity slug ^[a-z0-9_-]+$
  namespace: examples             # identity scope
  version: "1.0.0"                # content version
  title: Example Record           # display, free-form
  description: Minimal example contract
  status: active                  # contract-specific extensions —
  type: domain_entity             #   status, type, ownership,
  ownership:                      #   governance_rules, lifecycle stay
    roles:                        #   in metadata (KM-1 allowance)
      business_owner: [example-team]
  governance_rules:
    pii_fields:
      fields: []
spec:
  json_schema:
    type: object
    title: example_record
    additionalProperties: false
    properties:
      id: {type: string, format: uuid}
      name: {type: string, minLength: 1}
    required: [id, name]
  coercion_rules:
    rules:
      id: coerce_to_uuid
  validation_rules:
    rules:
      name:
        matches_regex: {pattern: "^\\S.*$"}
  data_quality:
    rules: []
    targets: {}
  field_bindings: {}

Sub-artifact fragment templates (template_json_schema.yaml, template_coercion_rules.yaml, …) stay envelope-free (TH-2), like pipeline snippets; per-section version keys used by the store's artifact versioning remain supported inside spec.

A foreign ODCS document (camelCase apiVersion: v3.x + kind: DataContract) is also accepted anywhere a contract is accepted (KM-7 commitment 3, shipped) — it keeps ODCS's own envelope and converts on read.

Execution sketch (post-approval)

1. KM Phase 1 — envelope core: registry group/version support, kind dispatch, spec unwrapping in one shared loader helper.
2. KM Phase 2 — converter + kinds: per-kind converter/upgrade steps
3. template/seed/example rewrites (streaming, bridge, pipeline, concept-scheme template, persisted schemes, contracts — the contract corpus takes the envelope wrapper only; its canonical sections move under spec: verbatim, and the ODCS round-trip gate must stay green through the move).
4. KM Phase 3 — contract ODCS codec (approved, SHIPPED 2026-07-12): codec completion + lossless round-trip CI gate + native ODCS ingestion per KM-7 landed ahead of KM 1–2 as planned.
5. KM Phase 4 — ratchet + stores: registry bumps activate the CI ratchet per kind; migrate --stores sweep; docs.

Risk register

Risk	Mitigation
Third breaking format change for streaming/bridge/pipeline in quick succession	Read-time upgrades keep every stored artifact loading; pycharter migrate rewrites files in one command; sequence after the VC track completes so the churn is one window, not three
UI editors assume top-level id/version (streaming/pipeline save paths)	Each kind's UI lands atomically with its converter step (the TH-3b lockstep pattern)
api_version group naming bikeshed	One decision, recorded here: pycharter.io/v1 under a snake_case key; alternatives considered: bare v1 (collides with format_version semantics), charter.optophi.io/v1 (verbose), literal K8s apiVersion (breaks the repo-wide snake_case key convention)
CF-1 (one canonical contract model) interlock	Unchanged from TH-3: whoever lands second coordinates; KM Phase 3 is the contract touchpoint
ODCS version drift (v3.1 deprecations, v4.0 removals)	Codec targets one ODCS version per release (v3.1.0 today); the round-trip gate makes any bump's blast radius visible before it ships; the at-rest format is insulated by design (KM-7)

Acceptance

• Every repo artifact under data/templates/, data/seed/, examples/ carries the K8s envelope; the CI ratchet enforces it.
• pycharter migrate (files + --stores) converts any pre-KM document.
• One documented envelope in docs/reference/artifact-format-versioning.md; the TH-1 shape is marked historical.
• Generic tooling needs zero per-kind special cases for identity, display, or format detection.