Skip to content

PyCharter — Contract Foundation Unification

Status: implemented (CF-1…CF-7, CF-3b, CF-7b); the loader-entry-point CF-5 is folded into ARCH-1 in BACKLOG.md · Created: 2026-05-29 · Audience: every contributor

The contract subsystem currently carries two parallel models, three field-binding shapes, two relationship enums, four ownership representations, three field-path formats, two severity vocabularies, and two locations for "data quality." This is the result of accreting features (DG-1 bindings, DQC, ODCS export, LinkML loop) without periodic consolidation. The result: new users hit shape mismatches on first touch, and contributors cannot tell which form is canonical.

This proposal collapses all of it. Six CF items, no aliases, no deprecation shims, no transitional accept-both code paths. Old shapes are deleted. Migration of contract artifacts already on disk is handled by _format_migrations (read-time upgrade); no user-facing back-compat is required because the system owns every consumer in this repo and migrates them atomically with the model.

This is the contract-side analogue of the T1–T10 terminology overhaul: same principle, applied to data shapes instead of names.

Scope

Six items. Each lands as its own atomic change set with its own tests green; phasing below sequences them by dependency.

ID Title Touches
CF-1 One canonical contract model The biggest. Collapses semantic.contracts.ContractConfig into db.schemas.DataContractSchema.
CF-2 One severity + one quality-dimension enum Every rule type (governance, coercion, validation, quality, extraction).
CF-3 One ownership model with named roles OwnershipBlock, OwnershipComponent, the 6-field ownership metadata block.
CF-4 One field-path format (JSONPath) Every artifact that references a field — coercion, validation, bindings, quality rules.
CF-6 One relationship enum with context subsets BindingRelationship + RelationshipType → one Relationship.
CF-7 Delete legacy DQ locations metadata.data_quality (legacy block) + metadata.governance_rules.business_rules removed from schema; migrator stays.

(CF-5, "single contract-loader entry point", is deferred. The user elected to skip it; loader sprawl is real but does not block the foundation work.)

Why now

Three forcing functions converged:

  1. DQC just landed. Phases 0–6 added a top-level data_quality section + ODCS sla_properties. The legacy metadata.data_quality block + metadata.governance_rules.business_rules block stayed accepted "for now." This proposal closes that loop (CF-7).
  2. Templates ship to users. The session that produced this proposal found the bundled templates fail parse_contract on first touch because top-level identity (name/status/description) was rejected. The two-model split (DataContractSchema vs ContractConfig) was the root cause — templates were authored against ContractConfig conventions but parsed by DataContractSchema.
  3. External integrations expanding. Pyintake handshake (CR-1 through CR-5), MCP surface (P5-007), Theme D wiring all consume the contract model. Every consumer added against a split foundation multiplies the cost of unification later.

Principles

These hold across all six items:

  1. One name per concept. No concept / concept_id / concept_key triplets; no target / target_field / column triplets; no team / steward / owner triplets.
  2. One shape per artifact. No "the YAML may be either form X or form Y." The migrator upgrades on read; the validator rejects anything not in canonical form.
  3. Typed enums, not free strings. Severity, dimension, relationship, role — all named enums with extra="forbid"-style strictness. Free strings allowed only inside customProperties bags (ODCS forward-compat).
  4. JSONPath is the canonical address. A field is referenced one way in every artifact section: $.field (top-level) or $.parent.child (nested). No dotted (orders.field), no bare (field). The normalizer becomes a one-shot migrator, not a permanent input mode.
  5. No back-compat code in the runtime. Anything the old shape could express either has a canonical equivalent in the new shape or is deleted. Format-version migrations (DG-2) handle artifacts on disk; the runtime only sees the new shape.

Decisions

CF-1 — One canonical contract model

Decision: pycharter.db.schemas._data_contract_schema.DataContractSchema is the single canonical contract Pydantic model. pycharter.semantic.contracts.models.ContractConfig and its supporting models (OwnershipBlock, GovernanceRule, CoercionRule, ValidationRule, ContractMetadataBlock, AIUsagePolicy) are deleted as parallel definitions.

The capabilities ContractConfig added (frozen typed objects, ai_usage_policy, extraction_policy, domain_schema, bound_field_count / referenced_ontologies accessors) move onto the canonical model:

  • ai_usage_policy and extraction_policy become typed fields on MetadataComponent (or new top-level if the semantic split matters).
  • domain_schema becomes a typed top-level field on DataContractSchema (currently only carried inside ContractConfig).
  • Accessor methods (bound_field_count, referenced_ontologies) become functions in pycharter.contract_parser or class methods on ContractMetadata (the runtime view object).
  • Sub-models that were richer in ContractConfig (PrimaryBinding, AnnotationBinding, FieldBinding) move into db/schemas/_data_contract_validators.py and replace the existing FieldBindingValidator — see CF-4 for the field-binding shape decision.

The rule sub-shapes consolidate to the DataContractSchema form (dict-of-field-keyed-rules), not the ContractConfig form (list-of-rule-objects). Rationale: the dict form is denser, direct-keyed by field, and is what every template + every seed file already uses. Migrating from list-of-CoercionRule to dict-of-rules is purely internal — no artifact change.

After CF-1:

  • One Python module declares the contract: pycharter.db.schemas.
  • One runtime view object: ContractMetadata (returned by parse_contract).
  • semantic/contracts/ shrinks to thin functions that operate on ContractMetadata (validation against a vocabulary, ontology resolution, etc.) — no parallel model.

CF-2 — One severity, one quality-dimension enum

Decision: Two new public enums in pycharter.shared.enums:

class Severity(StrEnum):
    INFO = "info"
    WARNING = "warning"
    ERROR = "error"
    FATAL = "fatal"


class QualityDimension(StrEnum):
    ACCURACY = "accuracy"
    COMPLETENESS = "completeness"
    CONFORMITY = "conformity"
    CONSISTENCY = "consistency"
    COVERAGE = "coverage"
    TIMELINESS = "timeliness"
    UNIQUENESS = "uniqueness"

Every severity field across the codebase (governance, coercion, validation, quality, extraction, drift detector) uses Severity verbatim. No warn (use warning), no fail (use fatal). The existing QualityDimension in db.schemas._quality_models moves to shared.enums and is the only definition.

Touchpoints (cited so the work is fully scoped):

  • db.schemas._quality_models.QualitySeverity — replace with re-export.
  • semantic.contracts.models (governance/coercion/validation severity) — deleted by CF-1; new model uses Severity.
  • semantic.extraction_policy.ExtractionPolicy.severity — change to Severity (was Literal["info", "warn", "error"]).
  • quality.profile_drift.evaluate_drift (drift severity) — typed field uses Severity.

CF-3 — One ownership model with named roles

Decision: Replace the 6-key flat ownership block + the two parallel Pydantic models with a single Ownership model:

class OwnershipRole(StrEnum):
    BUSINESS_OWNER = "business_owner"
    BUSINESS_SME = "business_sme"
    TECHNICAL_OWNER = "technical_owner"
    TECHNICAL_SME = "technical_sme"
    SUPPORT_LEAD = "support_lead"
    CONTRACT_OWNER = "contract_owner"
    DATA_STEWARD = "data_steward"


class Ownership(BaseModel):
    roles: dict[OwnershipRole, list[str]] = Field(default_factory=dict)
    additional_info: dict[str, Any] = Field(default_factory=dict)

YAML form:

ownership:
  roles:
    business_owner: [quant-desk]
    business_sme: [portfolio-management]
    technical_owner: [data-engineering, platform-team]
    technical_sme: [risk-team]
    support_lead: [compliance-ops]
    contract_owner: [data-engineering, quant-desk]
    data_steward: [governance-team]

Maps cleanly to the existing owners table (each entry is an owner id; the role becomes a column on a join table contract_ownership_assignments). The migrator turns the legacy six-key flat block into the new roles: map in one read-time upgrade step.

OwnershipBlock.team / OwnershipBlock.steward become entries in the roles map (technical_owner and data_steward respectively). Namespace.owner (semantic) is out of scope — it's an unrelated single-owner identifier and stays as-is.

CF-4 — One field-path format (JSONPath)

Decision: Every field reference in every contract artifact is a JSONPath expression rooted at $. There is no shorthand, no dotted form, no bare-name form.

Affected:

Section Before After
coercion_rules.rules keys order_id: coerce_to_string $.order_id: coerce_to_string
validation_rules.rules keys order_id: {...} $.order_id: {...}
data_quality.rules[].column column: order_id column: $.order_id
data_quality.rules[].columns [a, b] [$.a, $.b]
field_bindings keys $.order_id (already correct) $.order_id (unchanged)
sla_properties[].element orders.created_at $.created_at
Field-binding relationships[].target_field $.field (already correct) unchanged

pycharter.semantic.concepts.normalizer.to_jsonpath and normalize_field_bindings become the one-shot upgrade step inside _format_migrations. After upgrade, the parser refuses non-JSONPath references with a clear error ("field path must be a JSONPath expression starting with $").

This makes the artifact uniform with the field-bindings convention already enforced by FieldBindingsComponent.validate_jsonpath_keys.

CF-6 — One relationship enum with context subsets

Decision: Merge BindingRelationship and RelationshipType into a single Relationship enum. Add a RelationshipContext policy that declares which subset is valid where. Duplicate-meaning members collapse to one canonical name.

class Relationship(StrEnum):
    # SKOS / mapping family
    EQUIVALENT_TO = "equivalent_to"  # collapses BindingRelationship.EXACT_MATCH
    CLOSE_MATCH = "close_match"
    RELATED_TO = "related_to"
    BROADER = "broader"
    NARROWER = "narrower"

    # Identity / reference family
    SAME_AS = "same_as"
    REFERENCES = "references"

    # Derivation family
    DERIVED_FROM = "derived_from"
    IMPLEMENTS = "implements"
    DESCRIBES = "describes"

    # Structural family
    IS_A = "is_a"
    HAS = "has"
    DEPENDS_ON = "depends_on"
    PRECEDES = "precedes"
    CAUSES = "causes"
    GOVERNS = "governs"


class RelationshipContext(StrEnum):
    FIELD_BINDING = "field_binding"  # subset valid on binding annotations
    CONCEPT_GRAPH = "concept_graph"  # subset valid on graph edges


_CONTEXT_ALLOWED = {
    RelationshipContext.FIELD_BINDING: {
        Relationship.EQUIVALENT_TO,
        Relationship.CLOSE_MATCH,
        Relationship.RELATED_TO,
        Relationship.BROADER,
        Relationship.NARROWER,
        Relationship.DERIVED_FROM,
        Relationship.SAME_AS,
    },
    RelationshipContext.CONCEPT_GRAPH: {
        Relationship.IS_A,
        Relationship.HAS,
        Relationship.DEPENDS_ON,
        Relationship.PRECEDES,
        Relationship.CAUSES,
        Relationship.GOVERNS,
        Relationship.DERIVED_FROM,
        Relationship.SAME_AS,
        Relationship.RELATED_TO,
        Relationship.EQUIVALENT_TO,
        Relationship.REFERENCES,
        Relationship.IMPLEMENTS,
        Relationship.DESCRIBES,
    },
}

exact_match → equivalent_to is the one collapse. Rationale: equivalent_to is the OWL-canonical name and the one that survives in the merged graph use case. Rename only — same semantics.

Validators (FieldBindingRelationshipValidator.validate_type, schema_loader.validate_relationship_type) consult the relevant RelationshipContext allowlist instead of two separate enums.

CF-7 — Delete legacy data-quality locations

Decision: Hard-delete the legacy slots from the model:

  • Remove the data_quality field from MetadataComponent (the old DataQualityComponent with accuracy / sla / total_records).
  • Remove DataQualityComponent, SLAThresholdsComponent, MonitoringScheduleComponent, AlertingComponent from db/schemas/_data_contract_schema.py. The top-level DataQualitySection (in _quality_models.py) is the only data-quality model.
  • Change GovernanceRulesComponent from extra="allow" to extra="forbid" with an explicit field list (data_retention, pii_fields, sensitivity, access_control, dependencies, data_feed_association, indexes). business_rules is no longer accepted.
  • The pycharter.contract_migrations.v1.migrate_contract_v0_to_v1 helper stays — it's the read-time migration path for any contract that still has the legacy block. It is wired into _format_migrations so every read upgrades transparently.

Templates already shipped in the previous session. CF-7 closes the loop on the model itself.

Phasing

Three phases. CF-1 ships first because every other CF either sits on top of it (CF-2, CF-3, CF-4, CF-7) or is independent (CF-6).

Phase 1 (atomic) — CF-1 + CF-7

Land together. Both touch the same Pydantic models; landing them in one PR keeps the model file in a coherent state. Migration helper ensures every read still works.

Acceptance: - pycharter.semantic.contracts.models no longer declares a ContractConfig; the import either disappears or re-exports the canonical model. - DataContractSchema is the only contract-shape Pydantic model in the repo. - metadata.data_quality and metadata.governance_rules.business_rules are rejected by the validator. - migrate_contract_v0_to_v1 is wired into _format_migrations and runs on every parse_contract read. - All existing tests pass (DQC, contract-store, semantic, runtime).

Phase 2 (parallel) — CF-3, CF-4

Both are shape changes on top of CF-1. They do not interact (one is ownership; the other is field-path format). Land in either order.

Acceptance: - One Ownership model with roles: dict[OwnershipRole, list[str]] is the only ownership representation. The six-key flat block is a one-shot read-time upgrade step. - Every field reference in every artifact section is JSONPath rooted at $. Templates, seed YAMLs, and parser-side acceptance all enforce this; the dotted/bare normalizer is a migration-only helper.

Phase 3 (parallel) — CF-2, CF-6

Vocabulary unification. Independent of each other and of Phases ½ in the model sense, but conceptually finish the foundation.

Acceptance: - Every severity field uses the Severity enum from pycharter.shared.enums. - Every quality-dimension field uses QualityDimension from pycharter.shared.enums. - One Relationship enum exists. BindingRelationship and RelationshipType are deleted. Context allowlists enforce subset validity.

Out of scope

  • CF-5 (single contract-loader entry point). Deferred.
  • Exception hierarchy unification (PyCharterError vs PyCharterSemanticError). Not part of the contract foundation — separate sweep if undertaken.
  • Namespace.owner collapse into Ownership.roles. Different concept (namespace authority vs contract role); keep separate.
  • OpenAPI / route-path renames. The HTTP surface is governed by the DG-5 stability policy; renames there pay a heavier breakage cost without contract-side benefit.
  • External-system ID formats (FIX feed, OMS, etc.). Out of scope — these are inputs to the system, not part of the contract foundation.

Risk register

Risk Likelihood Mitigation
CF-1 PR too big to review Medium Split into "delete ContractConfig" + "move policy fields to canonical" if needed; both still in one phase.
Migration miss on some on-disk contract Low _format_migrations chain is well-trodden (DG-2 pattern). Add a one-time CI step that round-trips every fixture YAML in tests/ and data/seed/ through the migrator.
Downstream consumer pinned to an old shape Low Repo-internal consumers only at this point; pyintake handshake is DTO-mediated (integration-dtos.linkml.yaml) and unaffected.
Field-path migration noise Medium The migrator logs one WARNING per applied upgrade rule; collect a count after the merge to see if any artifact in the wild still trips it.

Acceptance for the program (CF complete when)

  1. pycharter.db.schemas._data_contract_schema.DataContractSchema is the only Pydantic model defining a contract shape in the repo. No alternative module exports a class fitting that role.
  2. Every YAML in data/templates/contract/, data/seed/contracts.yaml, samples/, examples/, and tests/ parses cleanly via parse_contract after the migrator runs.
  3. Every severity / quality-dimension / relationship / ownership-role string in the codebase is a typed enum member, not a free string.
  4. Every field reference in every artifact section is JSONPath.
  5. metadata.data_quality and metadata.governance_rules.business_rules are no longer accepted by the validator.
  6. The BACKLOG.md Open Items index shows all CF rows checked.

Addendum (2026-06) — Pipeline canonical unification

Status: landed (separate from the CF-1–CF-7 contract program above).

Pipeline authoring now uses a single pipeline.yaml envelope:

  • format_version, slash-qualified id, metadata.namespace, settings, variables, and a steps: list only.
  • Section-based configs (extract / transform / load files) and legacy editor_folders browse tables are removed.
  • Step-body fragments live under data/templates/pipeline/snippets/ for copy-paste into canonical pipelines.

See the consolidated plan at docs_pipeline_dqc_unify and Pipeline templates.

Addendum — CF-5 JSONata field references (separate epic)

This proposal originally deferred CF-5 as “single contract-loader entry point.” A later program (CF-5 JSONata) targets a different problem: one field-reference language across coercion, validation, bindings, DQC, and SLA artifacts.

Artifact Today (CF-4) CF-5 JSONata target
Rule keys (column, columns, SLA element) JSONPath $.field JSONata $field
type: expression bodies Informal bare names (not executed) JSONata predicates

Status: deferred. Interim docs in Data quality and contract template README; expression examples carry # CF-5 pending comments until the executor and format migration (contract v6 → v7) land.