Skip to content

Core Concepts

Understanding PyCharter's key concepts will help you use it effectively.

New here? See Start here for a runnable example, or PyCharter vs. the alternatives if you are choosing between data quality tools. Unfamiliar terms? Check the glossary.

Why configuration-driven? Schema in YAML, not code

Most validation libraries embed shape and rules in code — Pydantic models, Pandera schemas, hand-written if checks. PyCharter inverts this: contracts live in versioned YAML/JSON files, and the same definition drives your APIs, pipelines, workers, and governance tooling.

This makes three things easier:

  • Single source of truth. One contract file feeds the FastAPI gate, the batch pipeline, the streaming worker, and the quality monitor. Pin the reviewed version at each enforcement point and promote changes explicitly. Updating a file does not automatically update a deployed consumer.
  • Safe evolution. Contracts are plain files, so they diff, review, and version like code: pycharter contract diff old.yaml new.yaml --fail-on-breaking runs in CI and blocks breaking changes before they ship. A contract store adds named, versioned history on top.
  • Language-agnostic. A JavaScript client, a dbt model, and a Python worker can read a shared YAML/JSON artifact. Execution support still depends on the consumer and the supported schema/rule subset.

The trade-off is one level of indirection: instead of writing a class, you declare a schema and PyCharter compiles it to a Pydantic model at load time (see benchmarks for measured scenarios). Start with a local file and in-process validation; adopt a shared contract store only when several teams or services need the same contracts.

Three domains and storage

PyCharter divides functionality into three domains. Each domain has a dedicated store (persistence layer) and is keyed by its own identifier and version.

Domain What it manages Store Keyed by
Pipelines ETL pipeline definitions (extract/transform/load steps, settings, variables) Pipeline Store (pycharter.pipeline_store) pipeline_name, pipeline_version
Contracts Data contracts: JSON Schema, coercion rules, validation rules, metadata, optional field mapping Contract Store (pycharter.contract_store) contract_name, contract_version
Ontology Concept schemes, concepts, relationships, and versioned ontology configs (e.g. LinkML) Semantic Store (pycharter.semantic_store) Schemes by scheme_key; concepts by name; configs by schema_name, schema_version
  • Contract Store — The schema registry. Use it when you need to store and retrieve contracts by name/version for validation, quality checks, or the API. Implementations: InMemoryContractStore, SQLiteContractStore, PostgresContractStore, MongoDBContractStore, RedisContractStore. See Contract Store.
  • Pipeline Store — The pipeline registry. Use it when you want to persist pipeline configurations (e.g. from the UI or API) and run them by name/version. Implementations: InMemoryPipelineStore, SQLitePipelineStore, PostgresPipelineStore, MongoDBPipelineStore, RedisPipelineStore. See Pipeline Store.
  • Semantic Store — The ontology registry. Use it for concept schemes, concepts, relationships, and versioned ontology artifacts. Implementations: InMemorySemanticStore, SQLiteSemanticStore, PostgresSemanticStore. See Semantic Store.

The REST API and Web UI use all three stores: contracts for validation and quality, ETL store for pipeline management, and semantic store for the Ontology workspace.

Bundled YAML templates for contracts, batch pipelines, streaming worker configs, and vocabulary seeds are introduced in Templates overview.

For visual workflows (pipeline step contract: block, row vs batch quality), see Contract → ETL → quality.

Data Contracts

A data contract is a formal specification that defines:

  • Schema: The structure and types of your data (JSON Schema)
  • Coercion Rules: Pre-validation transformations (e.g., string → integer)
  • Validation Rules: Post-validation business constraints
  • Metadata: Description, ownership, governance information
  • Ontology (optional): Semantic field annotations—what each field means in the business domain (concepts, definitions, relationships)

Contract Structure

# user_contract.yaml
api_version: pycharter.io/v2
kind: DataContract
metadata:
  name: user_contract              # identity slug ^[a-z0-9_-]+$
  version: "1.0.0"                 # the contract version
  description: Defines the structure of user records
  ownership:
    roles:
      business_owner: [data-team]
      data_steward: [alice@example.com]
spec:
  json_schema:
    type: object
    version: "1.0.0"
    properties:
      name:
        type: string
        minLength: 1
      email:
        type: string
        format: email
      age:
        type: integer
        coercion: coerce_to_integer  # Pre-validation coercion
        validations:
          is_positive: {}            # Post-validation check
    required:
      - name
      - email
  # Optional: field_bindings (semantic field annotations; see Ontology / Concepts)
  # field_bindings:
  #   $.email:
  #     primary:
  #       concept: { name: user_email, definition: "Primary email address" }

When you use build_contract() or load a contract from the store, you get a DataContract (or its dict form) with schema, coercion_rules, and validation_rules as separate keys (rules are not merged into the schema). The Validator merges rules internally when validating. See the Contract-First Workflow for a full define → parse/build → store → validate flow.

Contract Lifecycle

graph LR
    A[Define Contract] --> B[Parse]
    B --> C[Store in Registry]
    C --> D[Generate Validators]
    D --> E[Validate Data]
    E --> F[Monitor Quality]
    F --> A

Validation Pipeline

When data is validated, it goes through three stages:

1. Coercion (Pre-validation)

Transforms input data before validation:

# Input: {"age": "25"}
# After coercion: {"age": 25}  (string → integer)

Built-in coercions:

Coercion Description
coerce_to_string Convert to string
coerce_to_integer Convert to integer
coerce_to_float Convert to float
coerce_to_boolean Convert to boolean
coerce_to_datetime Parse ISO datetime
coerce_to_date Parse date only
coerce_to_lowercase Lowercase string
coerce_to_uppercase Uppercase string

2. Schema Validation

Validates against JSON Schema (Draft 2020-12):

# Checks: type, required, minLength, pattern, enum, etc.

3. Custom Validation (Post-validation)

Applies business rules after schema validation:

# Checks: is_positive, is_email, matches_regex, etc.

Built-in validations:

Validation Description Config
min_length Minimum string/array length {"threshold": N}
max_length Maximum string/array length {"threshold": N}
is_positive Value > 0 {}
is_email Valid email format {}
matches_regex Match pattern {"pattern": "..."}
only_allow Whitelist values {"allowed_values": [...]}

Warning-severity rules

By default a failed validation rule invalidates the record. Add severity: warning to a rule to make it advisory — the violation is reported (as a yellow ⚠ diagnostic, in result.warnings) but the record stays valid:

validation_rules:
  rules:
    $symbol:
      matches_regex:
        pattern: "^[A-Z]{1,5}$"
        severity: warning     # report, don't fail
      max_length:
        threshold: 5          # error (default) — fails the record

Type and required checks in json_schema are always hard failures; severity applies only to these business rules. On the CLI, pycharter validate --strict-warnings promotes warnings back to a non-zero exit for CI gates.

ETL Pipeline Architecture

PyCharter's ETL system follows the Extract-Transform-Load pattern:

graph TB
    subgraph Extract
        E1[HTTPExtractor]
        E2[FileExtractor]
        E3[DatabaseExtractor]
        E4[CloudStorageExtractor]
    end

    subgraph Transform
        T1[Rename]
        T2[Filter]
        T3[AddField]
        T4[Convert]
        T5[CustomFunction]
    end

    subgraph Load
        L1[PostgresLoader]
        L2[FileLoader]
        L3[CloudStorageLoader]
    end

    E1 --> T1
    E2 --> T1
    E3 --> T1
    E4 --> T1

    T1 --> T2 --> T3 --> T4 --> T5

    T5 --> L1
    T5 --> L2
    T5 --> L3

Pipeline Composition

Pipelines are built using the | (pipe) operator:

pipeline = Pipeline(extractor) | transformer1 | transformer2 | loader

Async Execution

All pipeline operations are async for better performance:

import asyncio

# From a script
result = asyncio.run(pipeline.run())

# From async code
result = await pipeline.arun()

Contract Store

The contract store is a centralized registry for:

  • Schemas and their versions
  • Coercion rules
  • Validation rules
  • Metadata (ownership, description, governance)
  • Ontology (when using an ontology-enabled store)
  • Quality metrics history

Store Backends

Backend Use Case
InMemoryContractStore Testing, development
SQLiteContractStore Single-user, local development
PostgresContractStore Production, multi-user
MongoDBContractStore Document-oriented workloads
RedisContractStore High-performance caching

Schema Versioning

Schemas are versioned to track changes:

# Store a new version
store.store_json_schema("user", schema_v1, version="1.0.0")
store.store_json_schema("user", schema_v2, version="2.0.0")

# Get specific version
schema = store.get_json_schema("user", version="1.0.0")

# Get latest version
schema = store.get_json_schema("user")  # Returns 2.0.0

Quality Assurance

PyCharter supports two types of quality checks: contract (row-based) and pipeline (column/dataset-based). See Data quality: contract vs pipeline for an overview and when to use each.

Contract quality metrics (row-based)

Metric Description
overall_score Overall quality (0-100)
violation_rate % of records with errors
completeness % of non-null required fields
accuracy % of valid values

Quality Thresholds

Set alerts when quality drops:

thresholds = QualityThresholds(
    min_overall_score=95.0,  # Alert if score < 95
    max_violation_rate=0.05,  # Alert if violations > 5%
)

Violation Tracking

Every validation error is tracked:

# Query violations
violations = store.query_violations(
    schema_id="user_schema", status="open", severity="high"
)

API Tiers

PyCharter's API is organized into three tiers:

Best performance, full features:

from pycharter import Validator, Pipeline, QualityCheck

validator = Validator.from_file("contract.yaml")

Tier 2: Convenience Functions

Quick start, one-off operations:

from pycharter.pydantic_generator import from_dict
from pycharter.runtime_validator import validate

Model = from_dict(schema, "User")
result = validate(Model, data)

Note

validate / validate_batch are low-level utilities that take a prebuilt Pydantic model. Importing them from the package root (from pycharter import validate) is deprecated — prefer the Validator class (Tier 1), or import them from pycharter.runtime_validator when you already have a model.

Tier 3: Low-Level Utilities

When you need fine-grained control:

from pycharter.json_schema_converter import model_to_schema
from pycharter.runtime_validator import validate_batch

Error Handling

PyCharter uses a structured exception hierarchy:

from pycharter.shared.errors import (
    PyCharterError,  # Base exception
    ConfigError,  # Config loading/parsing
    ConfigValidationError,  # Schema validation
    ExpressionError,  # Expression evaluation
)

try:
    pipeline = Pipeline.from_dir("invalid/")
except ConfigError as e:
    print(f"Config error: {e}")
except PyCharterError as e:
    print(f"PyCharter error: {e}")

Error Modes

Control how pipeline errors are handled:

from pycharter.shared.errors import ErrorMode, ErrorContext

# Strict: raise on first error (default)
result = await pipeline.arun(error_context=ErrorContext(mode=ErrorMode.STRICT))

# Lenient: log and continue
result = await pipeline.arun(error_context=ErrorContext(mode=ErrorMode.LENIENT))

# Collect: gather all errors
result = await pipeline.arun(error_context=ErrorContext(mode=ErrorMode.COLLECT))
print(result.errors)  # List of all errors

Web UI Overview

The PyCharter UI (served by pycharter ui serve or pycharter ui dev) provides:

  • Contracts — Registry, diagram/form editor, validation workspace
  • Pipelines — Overview, pipeline runs, ETL generator (YAML), and Pipeline diagram (visual extract/transform/load editor)
  • Quality — Metrics, thresholds, violations
  • Ontology — Concepts, concept schemes, diagrammatic workspace, proposals, semantic health
  • Documentation — API playground

See REST API & Web UI for details.

Next Steps