Skip to content

Templates overview

PyCharter ships starter YAML so you do not author contracts and pipeline configs from a blank page. Templates live in the installed package under pycharter/data/templates/ and are also exposed through the REST API and Web UI when the server is running.

Family What you get Typical use
Data contracts Schema, coercion, validation, metadata, field bindings, DQC, SLA Registry entries, step contract: blocks, API validation
Batch pipelines Extract / transform / load YAML (+ settings, variables) pycharter pipeline run, pipeline store
Streaming workers Worker config shape (no separate template pack yet) pycharter stream run — see example
Concept schemes Domain vocabulary seeds pycharter db seed, ontology workspace 
flowchart LR
  subgraph sources [How you obtain templates]
    CLI["CLI scaffold\n(pipeline init)"]
    API["GET /api/v1/ops/templates/..."]
    PKG["Package files\n(data/templates/)"]
  end
  subgraph artifacts [What you produce]
    C[Contract YAML]
    P[Pipeline YAML]
    S[stream-worker.yaml]
  end
  PKG --> C
  PKG --> P
  API --> C
  API --> P
  CLI --> P
  PKG -.->|examples| S

Data contract templates

Contract templates describe the canonical artifact shape PyCharter parses, stores, and enforces. Identity (title, status, …) belongs under metadata; structural field references are bare field names (for example order_id) — the $field form appears only inside expression rule bodies.

Template files

File Purpose
template_contract.yaml Full contract (schema + rules + metadata + bindings + data_quality + SLA)
template_json_schema.yaml JSON Schema artifact only
template_coercion_rules.yaml Pre-validation coercion (all built-in coercers)
template_validation_rules.yaml Post-validation business rules
template_metadata.yaml Ownership, lifecycle, governance
template_field_bindings.yaml Semantic bindings to vocabulary concepts
template_data_quality.yaml Top-level data_quality (targets + ODCS-shaped rules)
template_sla.yaml sla_default_element + sla_properties

On disk (development checkout):

src/pycharter/data/templates/contract/

The in-package README documents coercion/validation catalogs and a full YAML skeleton — same content is summarized in Contract-first workflow and Data contracts tutorial.

Download contract templates

# Single artifact
curl -O http://localhost:8000/api/v1/ops/templates/json_schema
curl -O http://localhost:8000/api/v1/ops/templates/field-bindings

# All contract artifacts as ZIP
curl -O http://localhost:8000/api/v1/ops/templates/contract-artifacts

Endpoints (under /api/v1/ops/templates/):

  • json_schema, metadata, coercion-rules, validation-rules
  • field-bindings, contract-artifacts (ZIP)

Open Contracts and use Download contract artifact templates to fetch the ZIP (same archive as contract-artifacts).

cp src/pycharter/data/templates/contract/template_contract.yaml ./my_contract.yaml
# edit, then parse or store

Bootstrap without templates

For exploratory work, infer a starter contract from data:

pycharter contract sketch --from-csv orders.csv --out orders_contract.yaml

Then refine against template_contract.yaml before registering in the contract store.

Split vs monolithic contracts

Templates support both styles:

  • Monolithic — one template_contract.yaml (or sketch output) with all sections.
  • Split artifacts — separate files per section (schema, metadata, coercion, …) in a directory; the streaming example uses this layout under examples/streaming/orders/contracts/.

Batch pipeline templates

Pipeline templates use the canonical pipeline.yaml envelope (format_version, id, metadata, settings, variables, steps).

Package path: src/pycharter/data/templates/pipeline/

Location Purpose
template_pipeline.yaml Full HTTP → Postgres starter
template_pipeline_minimal.yaml File → file starter
snippets/*.yaml Step-body fragments to paste into steps:

Scaffold with CLI

mkdir -p pipelines/users
pycharter pipeline init pipelines/users --template http-to-postgres

Presets (--template):

Preset Description
http-to-postgres (default) HTTP extract → transform → PostgreSQL load
database-incremental Incremental DB extract → Postgres load
file-to-file CSV file extract → JSONL file load 
# Validate config before run
pycharter pipeline validate pipelines/users

# Run (provide variables or env for ${...} placeholders)
pycharter pipeline run pipelines/users -v DATABASE_URL=postgresql://...

Snippet catalog (by stage)

Paths are relative to the pipeline template dir; download as snippets/{filename} from the API.

Snippet Source type
snippets/extract_http_simple.yaml HTTP (single request)
snippets/extract_http_paginated.yaml HTTP + pagination
snippets/extract_http_path_params.yaml HTTP + path params
snippets/extract_http_incremental.yaml HTTP incremental
snippets/extract_file_csv.yaml / _json / _parquet / _glob Local files
snippets/extract_database.yaml SQL database
snippets/extract_database_ssh.yaml SQL via SSH tunnel
snippets/extract_database_incremental.yaml Incremental SQL
snippets/extract_cloud_s3.yaml / _gcs / _azure Object storage
snippets/extract_with_validation.yaml Extract + inline contract: block
Snippet Notes
snippets/transform_simple.yaml rename, convert, add, select, drop
snippets/transform_jsonata.yaml JSONata expressions
snippets/transform_custom_function.yaml Python callable
snippets/transform_combined.yaml Mixed operators
Snippet Target
snippets/load_postgresql.yaml PostgreSQL
snippets/load_sqlite.yaml SQLite
snippets/load_file.yaml JSON / CSV / Parquet file
snippets/load_upsert.yaml / load_insert.yaml / load_truncate_and_load.yaml Postgres write modes
snippets/load_with_dlq.yaml Load + dead-letter handling
snippets/load_with_validation.yaml Load + contract gate
snippets/load_with_ssh_tunnel.yaml Postgres via SSH
snippets/load_cloud_s3.yaml / _gcs / _azure Cloud object stores
Snippet Purpose
snippets/settings.yaml contract_store, dlq, lineage
snippets/variables.yaml ${VAR} substitution examples

Contract blocks on pipeline steps

Any extract, transform, join, union, or load step can reference a contract:

contract:
  ref:
    type: file
    path: ./contracts/orders.yaml
  validate: true
  coerce: true
  on_error: fail

Store-backed refs (type: store, name, version) work when settings.contract_store is configured. See Contract vs pipeline quality and Building ETL pipelines.

Download pipeline templates

# List names (starters + snippets/*)
curl http://localhost:8000/api/v1/ops/templates/pipeline

# Download canonical starter or a step snippet
curl -O "http://localhost:8000/api/v1/ops/templates/pipeline/template_pipeline.yaml"
curl -O "http://localhost:8000/api/v1/ops/templates/pipeline/snippets/extract_http_simple.yaml"

Or copy from src/pycharter/data/templates/pipeline/ in a source checkout.

Real-time extractors (batch pipeline runtime)

Streaming extractors (SSE, WebSocket, Kafka, …) use the same source block shape as batch extract steps inside pipeline.yaml, but they run as continuous batch yields — not the streaming worker engine.

Streaming and messaging extractors

There is no extract_websocket.yaml snippet yet; configure from the guide or adapt snippets/extract_http_simple.yaml field names to your extractor type.

Streaming worker configs

The streaming validation worker (pycharter stream run) uses a different top-level document: a streaming: block with one or more streams[], each with source, contract, optional transform, load, dlq, and mode (detect / enforce / shadow).

No bundled stream init scaffold yet

Unlike batch pipelines, PyCharter does not yet ship a pycharter stream init or a data/templates/streaming/ directory. Use the reference schema and the repository example as your starting point.

Reference and example

Resource Location
Field-by-field schema Streaming worker config reference
Runnable example examples/streaming/orders/stream-worker.yaml
Operating modes, DLQ, headers Streaming validation
Kafka cookbook Consume a Kafka topic

Minimal shape:

streaming:
  streams:
    - name: orders-validator
      source:
        type: kafka
        topic: orders
        bootstrap_servers: kafka:9092
        consumer_group: orders-validator
      contract:
        ref:
          type: file
          path: /app/contracts/orders
        validate: true
        coerce: true
      mode: enforce
      delivery: at_least_once
      load:
        target: { type: kafka, topic: orders-validated, ... }
      dlq:
        target: { type: kafka, topic: orders-dlq, ... }
  observability:
    health_port: 8080

The contract block is intentionally aligned with batch pipeline steps so the same contract files work in ETL and streaming.

pycharter stream run examples/streaming/orders/stream-worker.yaml

Install extras as needed: pycharter[streaming], pycharter[streaming], etc. See Installation.

Concept scheme (vocabulary) templates

Domain concept scheme templates seed the semantic store on pycharter db seed:

src/pycharter/data/templates/concept_schemes/

File Pattern
ecommerce.yaml Domain (customers, products, orders)
healthcare.yaml Domain
saas.yaml Domain
party.yaml Reusable pattern
product_catalog.yaml Reusable pattern
event_participation.yaml Reusable pattern

Each file uses a template: envelope (id, name, concepts, relationships). Built-in domain templates for NL parsing also live in this directory; list them from Python with list_builtin_templates() (see Vocabulary).

LinkML schema samples for contracts are separate — see seed examples under data/seed/examples/, not the concept-scheme template folder.

How templates relate to stores

Store What you persist Templates help you…
Contract store Named contract versions Author artifacts before store.save
Pipeline store Named pipeline versions Author extract/transform/load before register
Semantic store Schemes, concepts, LinkML configs Seed vocabulary; bind fields via field_bindings

After authoring from templates, follow Contract-first workflow and Building ETL pipelines to register and run.

Next steps