Skip to content

Data Quality in Contracts (DQC)

Status: implemented (DQC-0…DQC-6, DQC-4b) · Created: 2026-05-28 · Audience: pycharter maintainers · Branch: refactor/contract-template-strip-inline-rules (Phase 0 work in progress)

User-facing docs

Open work: DQC-3b and DQC-DEFER-1 are the only open items and live in BACKLOG.md. This document is the design record.

1. Why this initiative

Pycharter ships data contracts that decompose into five artifacts (JSON schema, coercion rules, validation rules, metadata, field bindings). Quality concerns are currently spread across three places:

Today's location Content
validation_rules (top-level) Per-field, per-record library assertions; gate at ingest with DLQ on fail
metadata.governance_rules.business_rules Cross-field rules with description + enforced flags
metadata.data_quality Aggregate SLA thresholds (min_completeness, min_accuracy, …) and monitoring_enabled

This split is invisible to contract authors and disagrees with how ODCS v3 — the emerging cross-tool standard — expresses quality (one root-level quality: block plus operational slaProperties:). It also predates pycharter's existing always-on profiler (quality/profiling.py) and pipeline_run_field_profiles history.

The goal: consolidate the contract's quality surface into the ODCS-aligned shape, position pycharter as a superset of ODCS (every valid ODCS contract loads; pycharter adds extensions for coercion, FSM scoping, ontology scoping, and OpenLineage facet emission), and wire it cleanly into the existing pipeline runner, profiler, and scorer.

2. Design summary

2.1 Three-layer quality model

The codebase audit (during the design conversation that produced this doc) confirmed that quality belongs in three distinct layers, not one or two. Trying to collapse them into a single section produces ambiguity and round-trip drift. The layers:

Layer Scope What it produces Trigger
validation_rules per-record DLQ outcome on fail step's validate: true (existing)
data_quality.rules aggregate per contract per step violation reports / alerts step's quality: true ★ NEW
Profiler + scorer pipeline-wide rows in pipeline_run_field_profiles + roll-up reports always-on

validation_rules is the synchronous record gate — what happens on fail is DLQ. data_quality.rules is the aggregate threshold check — what happens on fail is a violation report. The profiler is the telemetry layer — unconditional metric collection — and feeds both data_quality.targets (aggregate DQ commitments) and sla_properties (operational SLAs) via the existing scorer at quality/scoring.py.

2.2 Final contract shape

The post-DQC contract carries the following top-level sections (unchanged sections marked italic):

  1. Identity — name, version, status, description
  2. json_schema — raw JSON Schema; coercion/validation are NOT inlined here (cleaned up in Phase 0)
  3. coercion_rules — input normalization (unchanged)
  4. validation_rules — per-record gate (unchanged shape; the DLQ path)
  5. data_quality ★ NEW — version, monitoring_enabled, default_severity, default_scheduler, default_schedule, targets: { min_overall_score, max_violation_rate, min_completeness, min_accuracy }, and rules: [ODCS-shaped quality rule list]
  6. sla_default_element ★ NEW (top-level) — default column for SLA element references (kept ODCS-compatible)
  7. sla_properties ★ NEW — ODCS-shaped operational SLAs (property, value, unit, element, driver)
  8. metadata — slimmed: business_rules and data_quality.* move out; governance_rules.indexes removed in Phase 0
  9. field_bindings — unchanged
  10. versions — extended with data_quality and sla_properties entries

A worked example is canonical in src/pycharter/data/templates/contract/template_contract.yaml once Phases 0 + 1 land.

2.3 Step contract block extension

The step contract block in pipeline configs already exposes coerce, validate, annotate, on_error toggles (semantic/pipelines/models.py:103). This initiative adds one more flag:

contract:
  ref: { name: ..., version: ... }
  coerce: true
  validate: true       # validation_rules → sync gate (existing)
  annotate: true       # field_bindings → record metadata (existing)
  quality: true        # ★ NEW: data_quality.rules at this step
  on_error: quarantine

When quality: true, the step runner invokes QualityCheckEngine against the active contract's data_quality.rules, profiles the in-flight dataframe, and emits a profile row to pipeline_run_field_profiles. When quality: false (default), nothing changes from today.

2.4 schedule: / scheduler: semantics

ODCS rules can carry schedule: (cron / @hourly) and scheduler: (airflow, soda, dbt, …). Per design decision, pycharter stores these but does not execute them. They are purely informational and exported via to_odcs() for external orchestrators to consume. There is no in-process scheduler. This keeps pycharter focused on synchronous, pipeline-driven evaluation while preserving ODCS fidelity.

2.5 Multi-store / backend-agnostic rules

DQ rules can take four shapes, in increasing order of backend coupling:

type: What it is Portability
library (default) Pycharter's 16 built-in validators (null_rate, unique_check, row_count, matches_regex, …) — semantic, not engine-specific Engine-agnostic by construction
expression Backend-agnostic predicate compiled per-engine (SQL ⇄ MongoDB MQL ⇄ Spark) Engine-agnostic if executor exists
sql Raw SQL query: body SQL stores only
custom (engine: soda, …) Engine-native body Whatever the engine supports

Authoring guidance: reach for library first; expression for cross-field; sql/custom as last resorts. MongoDB and other non-SQL backends remain supported as long as the executor exists for the rule type used.

2.6 Superset relationship with ODCS

Feature ODCS v3 Pycharter (post-DQC)
Top-level quality: rules list ✅ ✅ (as data_quality.rules)
Dimensions (accuracy/completeness/…) ✅ ✅ same vocabulary
Library / SQL / custom rule types ✅ ✅ + type: expression
severity, mustBe*, unit, dimension ✅ ✅
schedule, scheduler (per-rule) ✅ ✅ (informational only)
slaProperties[] ✅ ✅ as sla_properties[]
slaDefaultElement ✅ ✅ as sla_default_element
Pycharter engine: pycharter library rules — ✅
Pycharter phase: pre_coercion / post_coercion — ✅
Pycharter fsm_state: scoping — ✅
Pycharter concept: / concept_scheme: scoping — ✅
Pycharter emit_facet: true → OpenLineage facet — ✅
coercion_rules (input normalization) — ✅ (via customProperties on ODCS export)
field_bindings (ontology) — ✅ (via customProperties on ODCS export)
validation_rules (per-record gate) — (folded into quality: with customProperties.enforcement: gate on export) ✅ first-class

Net effect: any valid ODCS document loads as a pycharter contract; pycharter-specific extensions degrade to customProperties on to_odcs() export.

3. Phased rollout

Each phase is one PR designed to keep the codebase green between landings. Tracked in BACKLOG.md as DQC-0 … DQC-6.

Phase 0 — Template cleanup (DQC-0, in progress)

Branch: refactor/contract-template-strip-inline-rules

  • Strip the inline coercion: / validations: keys from json_schema.properties.<f> in template_contract.yaml — they duplicate the canonical sections and contradict the runtime merger at runtime_validator/utils.py:38 (done).
  • Remove metadata.governance_rules.indexes — confirmed unread by any consumer; the DDL generator reads x-database-table.indexes instead (done).
  • Update src/pycharter/data/templates/contract/README.md to remove the misleading inline form from the structure example (done).
  • Note x-database-table.indexes as the correct home for index hints (done in the comment beside the removed block).

Risk: zero — no consumers broken (verified by grep + a round-trip parse test on the modified template).

Phase 1 — Schema + parser additive (DQC-1)

  • Pydantic models for DataQualitySection (with targets, default_*, rules: list[QualityRule]) and SLAProperty / SLAPropertiesSection. Live under src/pycharter/contract_parser/_quality_models.py (new file).
  • Extend contract_parser/parser.py to read data_quality, sla_default_element, sla_properties, versions.data_quality, versions.sla_properties.
  • Round-trip support in ContractMetadata.to_dict().
  • Validation rules: structural only at this phase (rule type is one of library/expression/sql/custom; required fields per type; dimension and severity enums; phase/fsm_state/concept optional).
  • No runtime behavior change. Sections parse and round-trip but no executor consumes them yet.

Risk: low. Existing contracts (no new sections) are unaffected; new sections are optional.

Phase 2 — Per-step quality wiring (DQC-2)

  • Add quality: bool = False (and optional quality_options: { phases, include_profiling }) to StepContractBlock in semantic/pipelines/models.py:103.
  • In pipeline_generator/step_contract.py, when quality: true, invoke QualityCheckEngine.run(contract=..., data=batch, options=...) after the existing validate/coerce/annotate pipeline and surface the report (logs + return value).
  • Step runners (extract.py, transform.py, load.py) pass the invocation context (contract_store, pipeline_name, step_id) to the executor.
  • Failure mode: data_quality.rules failures never affect record flow (no DLQ; that is validation_rules's job). Hard failures marked severity: error raise a step-level error captured in the run summary; warnings log only.

Risk: medium. New code path in step runners. Mitigations: opt-in flag (default off), separate from validate: so behavior is explicit, comprehensive tests against the existing seed pipelines.

Phase 3 — Always-on profiler + scorer integration (DQC-3)

  • Step runner emits one pipeline_run_field_profiles row per (step, field) on every run when a contract is present. This may already happen via QualityCheckEngine when quality: true; ensure the unconditional path is independent.
  • Extend quality/scoring.py to read contract.data_quality.targets and contract.sla_properties and compare aggregated profile metrics against them. Output: structured roll-up report with per-target observed value, threshold, breach status.
  • New DB table OR extension of pipeline_run_summaries — decision DQC-DECISION-2.
  • Retention via existing profile_retention.py.

Risk: medium. New DB rows on every pipeline run; needs retention policy validated for sizing. No regression on existing scorer output.

Phase 4 — Deprecate legacy quality_checks: (DQC-4)

  • The load step's quality_checks: list[dict] field (steps/_base.py:203) is parsed and translated to data_quality.rules shape via a 4-type translator (covers row_count, null_rate, uniqueness, expression — the entire _CHECK_REGISTRY).
  • Translator location: new src/pycharter/pipeline_generator/_legacy_quality_checks.py.
  • Emit DeprecationWarning at parse time. One release later, remove the field and the legacy code path entirely.
  • Existing pipelines using quality_checks: continue to work unchanged until the removal release.

Risk: low. 1:1 mapping, deterministic translation, no behavioral change for users.

Phase 5 — ODCS exporter (DQC-5)

  • to_odcs(contract: dict | DataContract) -> dict in new src/pycharter/contract_io/odcs.py.
  • Maps pycharter sections to ODCS v3 shape per the table in §2.6. Pycharter-specific fields ride along in customProperties.
  • CLI surface: pycharter contract export --format odcs CONTRACT.
  • from_odcs() deferred — not in DQC-5 scope. Tracked as future work if a real consumer needs round-trip import.

Risk: zero. Read-only exporter, no internal behavior change.

Phase 6 — Migration helper (DQC-6)

  • migrate_contract_v0_to_v1(contract: dict) -> dict in new src/pycharter/contract_migrations/v1.py. Deterministic rewrite:
  • metadata.governance_rules.business_rules → data_quality.rules (entries with type: sql or type: expression)
  • metadata.data_quality.sla.* → data_quality.targets.* (DQ aggregates) and sla_properties[...] (operational SLAs)
  • metadata.governance_rules.data_retention.days → sla_properties[{ property: retention, value: N/365, unit: y }]
  • metadata.governance_rules.indexes → DROP (already removed in Phase 0)
  • CLI: pycharter contract migrate CONTRACT_PATH [--in-place].
  • Golden-file tests with old/new pairs.

Risk: low. Deterministic transformation; bounded scope.

4. Out of scope / deferred

ID Item Reason
DQC-DEFER-1 Post-load DQ executor that queries the destination table (warehouse) and runs DQ against materialized data Real gap relative to ODCS / Soda / dbt-tests, but a substantial new code path. Deferred until DQC-2/DQC-3 land and there is a concrete consumer ask. Captured in §2.5 as phase: post_load rules that the contract can author but pycharter will not execute today.
from_odcs() import Round-trip ODCS import into pycharter shape Deferred until a real consumer needs it. to_odcs() (Phase 5) is sufficient for one-way interop.
Validation_rules cross-field shape Allow validation_rules to express cross-field assertions (e.g. fill_quantity_leq_order) directly, instead of routing through data_quality.rules Deferred — out of scope for DQC. Today's workaround: author cross-field as data_quality.rules with type: expression.
Profile drift contract awareness Allow contracts to declare which drift signals matter and their thresholds Deferred — today's drift detector uses hard-coded thresholds. Worth revisiting after DQC-3.

5. Resolved decisions

All five accepted 2026-05-28 as recommended. Listed here for the record; alternatives kept for context if future use forces a revisit.

DQC-DECISION-1 — Step scope for data_quality.rules ✅ all steps, per-rule phase:

data_quality.rules evaluates at every step where the step's contract block sets quality: true (extract, transform, load). Each rule carries a phase: field (default inline); phase: post_load rules are stored and round-tripped but not executed today (see DQC-DEFER-1). Alternative considered: load-only — rejected because inconsistency with the other step contract toggles (validate/coerce/annotate) outweighs the cost of additional executor invocations (which are opt-in anyway).

DQC-DECISION-2 — Roll-up report storage ✅ new contract_quality_reports table

Dedicated table indexed on (contract_name, contract_version, computed_at). Alembic migration ships in the Phase 3 PR. Alternatives considered: extending pipeline_run_summaries (couples report lifetime to pipeline-run lifetime, which is wrong because the report rolls up across runs), and JSON-only artifacts (no history, no UI surfacing).

DQC-DECISION-3 — Default state of quality: ✅ opt-in (default off)

The new flag must be set explicitly per step. Alternative considered: default-on when a data_quality section exists — rejected because surprise performance hits in existing pipelines erode trust more than docs friction slows adoption.

DQC-DECISION-4 — Coupling of quality + profiling ✅ coupled (one toggle)

quality: true implies the profiler runs at that step. No separate profile: flag. Alternative considered: two independent toggles — rejected because the "metrics-only without rule eval" use case has no concrete asker yet, and profile output is what the rule executor reads from. Revisit if a real consumer surfaces.

DQC-DECISION-5 — Section naming ✅ data_quality:

Top-level section name is data_quality:, not literal ODCS quality:. Alternative considered: literal ODCS naming — rejected because data_quality: disambiguates from "code quality" in docstrings, greps distinctively, and aligns with the (now-removed) metadata.data_quality block authors are already familiar with. The ODCS exporter (Phase 5) handles the literal rename during round-trip.

6. Discovered bugs / issues

Surfaced during the Phase 0 audit. Each is tracked in BACKLOG.md under the DQC-BUG-N IDs and is not strictly required for the DQC initiative — they are pre-existing pycharter issues uncovered by this work and should be triaged independently.

DQC-BUG-1 — Strict validator rejects top-level identity fields

DataContractSchema.model_validate(contract_data) raises extra_forbidden on name, version, status, description when called against the canonical template_contract.yaml shape. Reproduces on develop. The template stores identity at the top level (which is the documented authoring shape) but the strict schema doesn't allow it. Either the schema needs extra: allow for identity, or the template authoring shape needs updating.

Evidence: contract_parser/parser.py:120 (the validator) vs. data/templates/contract/template_contract.yaml (the template).

DQC-BUG-2 — Canonical rules format mismatch (dict vs. list)

merge_rules_into_schema() ultimately calls to_canonical_list() which raises ValueError: coercion_rules.rules must be a list, got dict when fed the template's dict shape. Reproduces on develop. Either the canonical converter needs to accept the dict shorthand, or the template needs to use list form.

Evidence: semantic/contracts/rules_canonical.py:146 vs. the template's coercion_rules.rules: {field: name} mapping.

DQC-BUG-3 — Dead metadata.governance_rules.indexes (now fixed)

The template authored an indexes: block under governance_rules that was never read. The DDL generator reads x-database-table.indexes instead. Removed in Phase 0.

DQC-BUG-4 — Misleading inline coercion: / validations: in template (now fixed)

The template inlined coercion: and validations: keys under json_schema.properties.<f> alongside the canonical coercion_rules / validation_rules sections. The inline form is authored output from merge_rules_into_schema(), not a source of truth — confusing for authors. Removed in Phase 0.

7. References