API Reference¶

Complete reference documentation for PyCharter's Python API. PyCharter is organized around three pillars — Pipeline, Contract, and Vocabulary — each with a dedicated storage layer. The data contract is the single source of truth for validation, pipeline load checks, and quality monitoring; contracts are stored in the Contract Store by contract name and version.

Three pillars at a glance¶

Pillar	Store	Module	Keyed by
Pipeline	Pipeline Store	`pipeline_store`	`pipeline_name`, `pipeline_version`
Contract	Contract Store	`contract_store`	`contract_name`, `contract_version`
Vocabulary	Semantic Store	`semantic_store`	Schemes: `scheme_key`; concepts: `name`; LinkML configs: name + version

Module Overview¶

pycharter
├── Pipeline              # Pipeline orchestration (DAG-based, config-driven)
├── Validator             # Data validation (contract, store, or file)
├── QualityCheck          # Quality monitoring and violation tracking
├── pipeline_generator/  # Pipeline framework
│   ├── pipeline         # Pipeline, PipelineResult, PipelineContext
│   ├── extractors/      # HTTP, File, Database, CloudStorage; Kafka, RabbitMQ, SQS (optional)
│   ├── transformers/    # Rename, Filter, AddField, Drop, Select, Convert, Map, etc.
│   ├── loaders/         # PostgresLoader, FileLoader, CloudStorageLoader
│   ├── validation       # ETLValidator, resolve_contract, create_etl_validator
│   ├── quality          # PostLoadChecker, pipeline quality checks
│   ├── state            # Incremental extraction (FileStateStore, SqliteStateStore)
│   ├── testing          # MockExtractor, MockLoader, PipelineTestHarness
│   └── factory / orchestrator  # Config-driven pipeline creation
├── contract_parser/     # parse_contract, parse_contract_file, ContractMetadata
├── contract_builder/    # build_contract, build_contract_from_store, DataContract, ContractArtifacts
├── contract_store/      # ContractStoreClient — schema registry (Postgres, SQLite, MongoDB, Redis, InMemory)
├── pipeline_store/            # PipelineStoreClient — pipeline config registry (Postgres, SQLite, MongoDB, Redis, InMemory)
├── semantic_store/       # SemanticStoreClient — vocabulary registry (Postgres, SQLite, InMemory)
├── runtime_validator/   # Validator, get_model_from_contract, get_model_from_store, ValidationResult
├── quality/             # QualityCheck, check_quality, ViolationTracker, QualityReport
├── schema_evolution/    # Schema versioning and compatibility
├── pydantic_generator/  # from_dict, from_file, generate_model
├── json_schema_converter/ # to_dict, model_to_schema
├── docs_generator/      # Generate Markdown/HTML from contracts
├── domain/              # Lifecycle binding (FSM / state alignment)
├── semantic/             # Vocabulary models, parsers, RDF export (used by semantic_store)
└── shared/              # Errors, coercions, validations, protocols

Quick Links¶

Core Classes¶

Class	Description
`Pipeline`	Pipeline orchestration
`Validator`	Data validation
`QualityCheck`	Quality monitoring

Pipeline Components¶

Module	Description
`Extractors`	HTTP, File, Database, Cloud
`Transformers`	Rename, Filter, AddField, etc.
`Loaders`	Postgres, File, Cloud

Contract Management¶

Module	Description
`Contract Parser`	Parse contract files → ContractMetadata
`Contract Builder`	Build DataContract from artifacts or store
Contract-First Guide	Define → parse/build → store → validate

Storage¶

Module	Description
`Contract Store`	Schema registry — contracts (schema, rules, metadata, field bindings) keyed by contract name and version
`Pipeline Store`	Pipeline registry — pipeline configurations keyed by pipeline name and version
`Semantic Store`	Vocabulary registry — concept schemes, concepts, relationships, LinkML schema configs
`Schema Evolution`	Versioning & compatibility for schemas

Utilities & Extensions¶

Module	Description
`Pydantic Generator`	Generate Pydantic models from schemas
`JSON Schema Converter`	Convert Pydantic ↔ JSON Schema
`Docs Generator`	Generate Markdown docs from contracts
`Domain`	Lifecycle binding for FSM engines
`Concepts`	Semantic annotations, knowledge graph, governance
`Testing Framework`	Mock components and pipeline test harness
`Errors`	Exception hierarchy

Import Patterns¶

Recommended Imports¶

# Core classes
from pycharter import Pipeline, Validator, QualityCheck

# Data contract (single source of truth)
from pycharter import (
    DataContract,
    ContractArtifacts,
    parse_contract_file,
    parse_contract,
    build_contract,
    build_contract_from_store,
)

# ETL components
from pycharter import (
    HTTPExtractor,
    FileExtractor,
    DatabaseExtractor,
    CloudStorageExtractor,
    Rename,
    Filter,
    AddField,
    Drop,
    Select,
    Convert,
    CustomFunction,
    PostgresLoader,
    FileLoader,
    CloudStorageLoader,
    PipelineResult,
)

# Contract store (schema registry; keyed by contract name and version)
from pycharter import (
    InMemoryContractStore,
    SQLiteContractStore,
    PostgresContractStore,
    MongoDBContractStore,
    RedisContractStore,
)

# Pipeline store (pipeline registry) and Semantic store (vocabulary) — also available via API dependencies
from pycharter.pipeline_store import (
    PipelineStoreClient,
    PostgresPipelineStore,
    SQLitePipelineStore,
    InMemoryPipelineStore,
)
from pycharter.semantic_store import (
    SemanticStoreClient,
    PostgresSemanticStore,
    SQLiteSemanticStore,
    InMemorySemanticStore,
)

# Validation and quality
from pycharter import (
    Validator,
    validate,
    validate_batch,
    get_model_from_contract,
    get_model_from_store,
    check_quality,
    QualityCheckOptions,
    QualityReport,
)

# Schema → Pydantic and back
from pycharter.pydantic_generator import from_dict, from_file
from pycharter.json_schema_converter import to_dict, model_to_schema
from pycharter import from_json, to_file, to_json

# Errors
from pycharter.shared.errors import (
    PyCharterError,
    ConfigError,
    ConfigValidationError,
    ExpressionError,
)

Type Annotations¶

PyCharter is fully typed with py.typed marker:

from pycharter import Validator, ValidationResult


def process_data(validator: Validator, data: dict) -> ValidationResult:
    return validator.validate(data)

Async Support¶

All pipeline operations are async:

import asyncio
from pycharter import Pipeline

# From script
result = asyncio.run(pipeline.run())


# From async function
async def main():
    result = await pipeline.arun()
    return result

See the Async Execution Model guide for detailed guidance on running pipelines from scripts, FastAPI, notebooks, and Celery.