Skip to content

Periodic quality monitoring

PyCharter evaluates data quality when you trigger a check — via Python, CLI, REST API, pipeline run, or validation worker. It does not ship an in-process cron scheduler.

Contract fields such as data_quality.default_schedule, per-rule schedule, and scheduler are metadata for your orchestrator (Airflow, cron, Prefect, etc.). PyCharter stores and can export them (ODCS); it does not execute them automatically.

See also: Contract vs pipeline quality, Data quality monitoring tutorial, Validation worker.

Three layers (what runs when)

Layer Scope How it runs on a schedule
validation_rules Per-record gate Pipeline step validate: true, Validator, or QualityCheck
data_quality.rules Aggregate per batch Pipeline step quality: true on each scheduled pipeline run
QualityCheck + worker Row metrics, thresholds, violations Cron/Airflow calls API or submit_job_sync

data_quality.monitoring_enabled in the contract documents intent; runtime evaluation for aggregate rules is opt-in via the step flag quality: true, not a background monitor service.

Pattern 1: Cron + synchronous check (simplest)

Best for: a file or table snapshot you can read each hour or day.

from pycharter import check_quality, QualityCheckOptions

report = check_quality(
    contract="contracts/order.yaml",
    data="/data/exports/orders.jsonl",
    options=QualityCheckOptions.monitoring(),
)

if not report.passed:
    raise SystemExit(f"DQ failed: {report.threshold_breaches}")

QualityCheckOptions.monitoring() enables thresholds, profiling, deduplicated violations, and skip-if-unchanged (when using QualityCheck with a database session).

CLI (crontab):

0 * * * * PYCHARTER_DATABASE_URL=postgresql://... \
  pycharter quality check \
    --contract-name order \
    --contract-version 1.0.0 \
    --data /data/exports/orders.jsonl \
    --record-violations \
    --check-thresholds

Pattern 2: Cron + validation worker (non-blocking)

Best for: large files, many contracts, or not blocking the scheduler process.

  1. Run a long-lived worker: pycharter worker --mode validation
  2. Each tick, enqueue a job:
from pycharter.worker import submit_job_sync

job_id = submit_job_sync(
    contract_name="order",
    contract_version="1.0.0",
    data_source="s3://bucket/daily/orders.parquet",
    options={
        "check_thresholds": True,
        "include_profiling": True,
        "skip_if_unchanged": True,
        "record_violations": True,
    },
    idempotency_key="order-daily-2026-06-04",
)

REST: POST /api/v1/quality/jobs with the same fields. Poll GET /api/v1/quality/jobs/{job_id} or use the Pipelines/Quality UI.

See Validation worker for deployment and configuration.

Pattern 3: Scheduled ETL + inline contract quality

Best for: quality tied to ingest — every pipeline run checks the batch.

In pipeline config, on each step that should run aggregate rules:

contract:
  ref: { type: store, name: order, version: "1.0.0" }
  validate: true   # per-record validation_rules
  quality: true    # data_quality.rules on the in-flight batch
  coerce: true
  on_error: fail

Schedule the pipeline with Airflow/cron (POST /api/v1/pipeline/jobs + pycharter worker --mode pipeline). Example config in the repo: src/pycharter/data/seed/examples/ecommerce/pipeline_config.yaml (see Seed data & samples).

Separate knob: quality_checks on the load step are pipeline (column/dataset) checks — complementary to contract data_quality.

Document schedules in the contract

data_quality:
  monitoring_enabled: true
  default_schedule: "@daily"
  default_scheduler: airflow
  rules:
    - name: rowcount_floor
      type: library
      rule: row_count
      mustBeGreaterThan: 100
      schedule: "@daily"
      scheduler: airflow

Implement @daily in your Airflow DAG (or cron), not inside PyCharter:

@dag(schedule="0 3 * * *", ...)
def daily_order_dq():
    run_order_quality()  # Pattern 1 or 2 above

Alerts and metrics

After each run:

  • Fail the job — non-zero exit or if not report.passed
  • Prometheus — Emit quality to Prometheus
  • Slack / PagerDuty — notify_on_breach from pycharter.quality.observability (requires pip install pycharter[server])

Limitations to plan around

  • No built-in scheduler — use cron, Airflow, Prefect, or similar.
  • Inline step quality today executes type: library rules reliably; sql, expression, and phase: post_load rules may be stored but skipped until fully implemented.
  • Great Expectations Cloud can schedule in-product for some connectors; PyCharter expects your orchestrator — see Comparisons.