Skip to content

Pipeline quality and execution reports

Step contracts support record validation and aggregate quality evaluation. Record validation decides whether to retain, skip, reject, or quarantine each record. Aggregate quality evaluates rules against a batch and reports their outcomes separately. Quality failures do not send records to the DLQ.

Contract quality before loading

Attach a contract to the relevant step and enable quality. On a load step, this evaluation runs before the destination write:

- id: load_output
  type: load
  input: transform_rows
  contract:
    ref: {type: file, path: contracts/orders.yaml}
    validate: true
    quality: true
    on_error: fail
  target:
    type: file
    path: output/orders.json
    format: json

The referenced contract's data_quality.rules define aggregate checks such as row count, null rate, and uniqueness. An inline rule failure with severity: error fails the step; warning and info outcomes remain visible without failing it. Unsupported or unevaluated rules must be read through their explicit skipped/unprocessed status, rather than assumed to have passed. The engine does not execute warehouse phase: post_load rules.

step.contract.quality: true enables these inline evaluations. Contract monitoring enablement, schedules, and SLAs are separate declarations; setting monitoring_enabled: false does not disable an explicitly enabled inline check.

Numeric assertions include equality, inequality, strict/inclusive bounds, and inclusive two-endpoint ranges. mustNotBe: 0 on row_count requires a nonempty batch. Every distinct non-null assertion on a native metric rule must pass; the metric is measured once and the rule retains one outcome/severity. Malformed operands fail validation; booleans are not numeric thresholds. See the numeric assertion reference for aliases, defaults, compatibility, and ODCS representation.

Custom quality callables can return a plain boolean or a valid CustomRuleResult: passed is a boolean, an optional observed metric is finite and numeric, and failure_count is a nonnegative integer. A malformed result fails the rule with a fixed diagnostic rather than being interpreted by truthiness. Arbitrary custom diagnostics remain outside the safe serialized execution report.

Reading execution evidence

Each StepResult.quality_reports contains evaluations tagged with a scope: contract for inline contract quality, or load_checks for the deprecated post-load checks below. Reports retain evaluated, passed, failed, error, warning, and skipped rule counts, individual outcomes, and completion state. An interrupted evaluation retains the outcomes known before the failure and reports the remaining rules as unprocessed.

Serialized reports omit arbitrary custom-rule message text and exception details so those diagnostics cannot copy restricted source values into API responses. Numeric observations, counts, rule identities, and completion state remain available.

PipelineResult.quality_summary aggregates rule events across steps. A rule evaluated at two steps contributes twice. record_outcomes separately reports record validation and routing; rows_failed retains its loader-failure meaning. See record outcome accounting for cardinality and partial-failure semantics.

from pycharter import Pipeline

pipeline = Pipeline.from_file("pipeline.yaml")
result = await pipeline.arun()
report = result.to_dict()

print(report["quality_summary"])
for step in report["step_results"]:
    print(step["step_id"], step["record_outcomes"], step["quality_reports"])

The native pipeline run responses expose the same summary and per-step evidence. These are execution reports; collecting durable observations for quality history, drift, and scoring requires the separate persistence flow. An empty quality summary does not prove a dataset meets any quality standard.

Deprecated load-step checks

Existing quality_checks configurations remain supported through translation to the inline quality evaluator and emit a deprecation warning. Put this field on the load step, alongside target:

- id: load_output
  type: load
  input: transform_rows
  target:
    type: file
    path: output/data.csv
    format: csv
  quality_checks:
    - type: row_count
      min: 1
      severity: fail
    - type: null_rate
      fields: [id, name]
      max_null_percent: 0
      severity: fail
    - type: uniqueness
      fields: [id]
      severity: warn

These checks run after a successful loader call. A failing check can fail the pipeline after rows have been written; it does not roll back those rows. The result retains the loader's acknowledged row counts and the failing quality report. During an extraction preview, these checks are marked skipped with a reason because no destination load occurs. Inline contract quality still runs during the preview.

Check type Configuration
row_count min, max, exact, expected, tolerance_percent, severity
null_rate fields, max_null_percent, severity
uniqueness fields, severity
expression expression, description, severity

Native DAG authoring rejects unknown check types, misspelled severity values, and malformed check configuration, including unknown entries mixed with valid checks. Earlier versions could drop those entries or treat a failed gate as passing. Migrate to one of the four types above. Severity aliases fail / error and warn / warning remain supported, as does info; omitted severity retains its existing default. This does not change the separate standalone quality-checker API.

Row-count checks require at least one bound; use min instead of the previously ignored min_rows. tolerance_percent requires expected. Null-rate and uniqueness checks require nonempty fields; null percentages must be between zero and 100. Expression checks require a nonempty expression.

Known column / field metadata can identify fields for privacy preflight diagnostics. It does not replace executable fields on null-rate or uniqueness checks; migrate a singular declaration to fields: [name].

The legacy QualityChecker / PostLoadChecker API remains available for callers using its standalone report type. Native DAG execution uses the scoped reports above; do not infer that inline quality populated a legacy pipeline_quality_report field.

See also