API Reference¶

Complete reference documentation for PyCharter's Python API. PyCharter is organized around three domains — Pipelines, Contracts, and Ontology — each with a dedicated storage layer. The data contract is the single source of truth for validation, pipeline load checks, and quality monitoring; contracts are stored in the Contract Store by contract name and version.

Three domains at a glance¶

Domain	Store	Module	Keyed by
Pipelines	Pipeline Store	`pipeline_store`	`pipeline_name`, `pipeline_version`
Contracts	Contract Store	`contract_store`	`contract_name`, `contract_version`
Ontology	Semantic Store	`semantic_store`	Schemes: `scheme_key`; concepts: `name`; configs: `ontology_name`, `ontology_version`

Module Overview¶

pycharter
├── Pipeline              # Pipeline orchestration (DAG-based, config-driven)
├── Validator             # Data validation (contract, store, or file)
├── QualityCheck          # Quality monitoring and violation tracking
├── pipeline_generator/  # Pipeline framework
│   ├── pipeline         # Pipeline, PipelineResult, PipelineContext
│   ├── extractors/      # HTTP, File, Database, CloudStorage; Kafka, RabbitMQ, SQS (optional)
│   ├── transformers/    # Rename, Filter, AddField, Drop, Select, Convert, Map, etc.
│   ├── loaders/         # PostgresLoader, FileLoader, CloudStorageLoader
│   ├── validation       # ETLValidator, resolve_contract, create_etl_validator
│   ├── quality          # PostLoadChecker, pipeline quality checks
│   ├── state            # Incremental extraction (FileStateStore, SqliteStateStore)
│   ├── testing          # MockExtractor, MockLoader, PipelineTestHarness
│   └── factory / orchestrator  # Config-driven pipeline creation
├── contract_parser/     # parse_contract, parse_contract_file, ContractMetadata
├── contract_builder/    # build_contract, build_contract_from_store, DataContract, ContractArtifacts
├── contract_store/      # ContractStoreClient — schema registry (Postgres, SQLite, MongoDB, Redis, InMemory)
├── pipeline_store/            # PipelineStoreClient — pipeline config registry (Postgres, SQLite, MongoDB, Redis, InMemory)
├── semantic_store/       # SemanticStoreClient — ontology registry (Postgres, SQLite, InMemory)
├── runtime_validator/   # Validator, validate_with_store, validate_with_contract, ValidationResult
├── quality/             # QualityCheck, check_quality, ViolationTracker, QualityReport
├── schema_evolution/    # Schema versioning and compatibility
├── pydantic_generator/  # from_dict, from_file, generate_model
├── json_schema_converter/ # to_dict, model_to_schema
├── docs_generator/      # Generate Markdown/HTML from contracts
├── domain/              # Lifecycle binding (FSM / state alignment)
├── semantic/             # Ontology models, parsers, RDF/vocabulary (used by semantic_store)
└── shared/              # Errors, coercions, validations, protocols

Quick Links¶

Core Classes¶

Class	Description
`Pipeline`	Pipeline orchestration
`Validator`	Data validation
`QualityCheck`	Quality monitoring

Pipeline Components¶

Module	Description
`Extractors`	HTTP, File, Database, Cloud
`Transformers`	Rename, Filter, AddField, etc.
`Loaders`	Postgres, File, Cloud

Contract Management¶

Module	Description
`Contract Parser`	Parse contract files → ContractMetadata
`Contract Builder`	Build DataContract from artifacts or store
Contract-First Guide	Define → parse/build → store → validate

Storage¶

Module	Description
`Contract Store`	Schema registry — contracts (schema, rules, metadata, field mapping) keyed by contract name and version
`Pipeline Store`	Pipeline registry — pipeline configurations keyed by pipeline name and version
`Semantic Store`	Ontology registry — concept schemes, concepts, relationships, and ontology configs
`Schema Evolution`	Versioning & compatibility for schemas

Utilities & Extensions¶

Module	Description
`Pydantic Generator`	Generate Pydantic models from schemas
`JSON Schema Converter`	Convert Pydantic ↔ JSON Schema
`Docs Generator`	Generate Markdown docs from contracts
`Domain`	Lifecycle binding for FSM engines
`Ontology`	Semantic annotations, knowledge graph, governance
`Testing Framework`	Mock components and pipeline test harness
`Errors`	Exception hierarchy

Import Patterns¶

Recommended Imports¶

# Core classes
from pycharter import Pipeline, Validator, QualityCheck

# Data contract (single source of truth)
from pycharter import (
    DataContract,
    ContractArtifacts,
    parse_contract_file,
    parse_contract,
    build_contract,
    build_contract_from_store,
)

# ETL components
from pycharter import (
    HTTPExtractor, FileExtractor, DatabaseExtractor, CloudStorageExtractor,
    Rename, Filter, AddField, Drop, Select, Convert, CustomFunction,
    PostgresLoader, FileLoader, CloudStorageLoader,
    PipelineResult,
)

# Contract store (schema registry; keyed by contract name and version)
from pycharter import (
    InMemoryContractStore,
    SQLiteContractStore,
    PostgresContractStore,
    MongoDBContractStore,
    RedisContractStore,
)

# Pipeline store (pipeline registry) and Semantic store (ontology) — also available via API dependencies
from pycharter.pipeline_store import PipelineStoreClient, PostgresPipelineStore, SQLitePipelineStore, InMemoryPipelineStore
from pycharter.semantic_store import SemanticStoreClient, PostgresSemanticStore, SQLiteSemanticStore, InMemorySemanticStore

# Validation and quality
from pycharter import (
    validate, validate_batch,
    validate_with_contract, validate_with_store,
    get_model_from_contract, get_model_from_store,
    check_quality, check_quality_with_store,
    QualityCheckOptions, QualityReport,
)

# Schema → Pydantic and back
from pycharter import from_dict, from_file, from_json, to_dict, to_file, to_json, model_to_schema

# Errors
from pycharter.shared.errors import (
    PyCharterError,
    ConfigError,
    ConfigValidationError,
    ExpressionError,
)

Type Annotations¶

PyCharter is fully typed with py.typed marker:

from pycharter import Validator, ValidationResult

def process_data(validator: Validator, data: dict) -> ValidationResult:
    return validator.validate(data)

Async Support¶

All pipeline operations are async:

import asyncio
from pycharter import Pipeline

# From script
result = asyncio.run(pipeline.run())

# From async function
async def main():
    result = await pipeline.run()
    return result

See the Async Execution Model guide for detailed guidance on running pipelines from scripts, FastAPI, notebooks, and Celery.