PyCharter — Python API Consolidation¶

Status: implemented (P0, PAC-1…PAC-8; see the table below) · Created: 2026-06-08 · Audience: every contributor

PyCharter's public Python API grew organically across three capability families — validation, batch pipelines, and (recently) streaming — and across two delivery modes for each: a config file/dict or the same artifact persisted in a versioned database store. The capabilities are strong; the surface is inconsistent. The same concept is spelled four ways, construction idioms differ per class, and ~320 symbols are exported at the package root. New users hit shape mismatches on first touch; contributors cannot tell which form is canonical.

This is the API-surface analogue of the contract-foundation unification (docs/proposals/contract-foundation-unification.md): same principle — one name per concept, one shape per artifact, no back-compat code in the runtime — applied to the public Python surface instead of contract data shapes. PyCharter is pre-1.0 (v0.0.x); per the project stability policy this is the moment to collapse the surface with a hard cut (no aliases, no deprecation shims) before 1.0 freezes it.

The system owns every in-repo consumer (CLI, FastAPI routes, UI backend, docs, examples, tests), so each rename migrates atomically with its call sites. The Kafka/DLQ wire format and REST response shapes are external contracts and are explicitly preserved (see Danger flags).

The four inconsistencies¶

No unified config-or-store addressing. Validator takes three kwargs (store, contract_name, contract_version); pipeline YAML uses a nested {ref: {type: store, name, version}}; streaming threads a ContractRef(contract_id, version) dataclass; quality has a separate check_quality_with_store(...). Top-level run()/preflight() cannot address a stored artifact at all.
Store-layer drift. Four versioned stores (contract, pipeline, streaming, semantic LinkML) expose near-identical-but-divergent CRUD. ContractStoreClient has no list_versions; no store resolves "latest"; "latest" is reimplemented twice, once lexicographically (so 1.10.0 sorts before 1.9.0 — a bug). variables typing differs (dict[str, Any] vs dict[str, str]). Writing a contract is 3–4 separate committing calls (no atomic write). Four separate singleton factories; no user-facing open_store(...).
Construction idiom drift. Validator = 7-kwarg overloaded constructor plus ValidatorBuilder plus from_* classmethods plus six module-level *_with_contract/*_with_store wrappers; Pipeline = four from_* classmethods; streaming ContractStreamProcessor has no config factory at all — the config→engine path is locked inside _cli_streaming.py.
Naming drift + oversized root. data vs data_list; path vs directory vs raw; str vs str | Path; ValidationResult vs ValidationOutcome; run vs run_sync vs consume. Generic names leak to the package root (from pycharter import from_dict).

Target shape¶

A. One address: `StoreRef`¶

@dataclass(frozen=True, slots=True)
class StoreRef:
    name: str
    version: str | None = None  # None == latest

    @classmethod
    def parse(cls, ref: str | StoreRef) -> StoreRef: ...  # "name" / "name@version"
    def __str__(self) -> str: ...

Lives in pycharter/shared/refs.py; exported as pycharter.StoreRef. The single programmatic address accepted (as StoreRef | str) by every store-driven entry point: run/preflight, Validator.from_store, Pipeline.from_store, StreamWorker.from_store, quality.check_quality, build_contract_from_store, kafka.consume(static_contract=...). The "name@version" string mirrors the existing CLI convention.

version=None means "latest", resolved by the store layer's version_sort_key (B). The YAML discriminated-union ref models (ContractFileRef/ContractStoreRef/ContractReference, semantic/pipelines/models.py) and the streaming ContractRef dataclass collapse into this one concept (PAC-2 / PAC-5).

B. One versioned store interface + `open_store`¶

A common VersionedArtifactStore ABC (pycharter/stores/base.py), keyed (name, version), adopted by all four stores:

class VersionedArtifactStore(ABC):
    kind: ClassVar[str]                  # "contract"|"pipeline"|"stream"|"linkml_schema"
    def save(self, name, version, document, *, created_by=None) -> str   # raises ArtifactExistsError on dup
    def get(self, name, version=None) -> dict | None                     # version=None -> latest
    def list(self, name=None) -> list[dict]                              # envelope summaries
    def list_versions(self, name) -> list[str]                           # sorted via version_sort_key
    def latest_version(self, name) -> str | None
    def set_status(self, name, version, status, *, updated_by=None) -> None
    def update(self, name, version, document, *, updated_by=None) -> None  # sectional patch
    def delete(self, name, version) -> bool

Envelope: {id, name, version, status, created_at, updated_at, created_by, updated_by, document}; document is the engine-ready canonical artifact (today's get_pipeline_dict/get_streaming_dict output, the assembled contract dict, or {linkml_yaml, ...}).
Mutability (decided): uniform save-once + update. save() raises ArtifactExistsError on a duplicate (name, version) — no accidental overwrite; update() patches a version in place (the draft-editing flow the UI inspector and pipeline/streaming editors already use). Convention: status="active" artifacts get version-bumped; drafts may be patched. This matches reality — the contract store is already mutated in place via upsert_field_bindings + a revision counter, so declaring per-version immutability would be a fiction enforced only where inconvenient.
Ordering: version_sort_key(version) (in pycharter/utils/version.py, built on the existing parse_version) gives numeric-aware, semver-tolerant total order — fixing the 1.10.0 < 1.9.0 lexicographic bug. Centralized in list_versions so backends cannot drift.
Atomic contract write: ContractStoreClient.save() decomposes a DataContract/dict into schema + rules + metadata + field bindings in a single transaction; the per-artifact store_* writers become internal _save_* primitives. Read-side per-artifact getters stay public.
Factory: pycharter.open_store(kind, url=None, *, connect=True, **backend_kwargs) selects the backend by URL scheme (sqlite:///postgresql:///mongodb:///redis:///None→in-memory), absorbing api/dependencies/_store_factory.resolve_connection. The four api/dependencies singletons reduce to open_store(kind) + HTTP-error wrap; _store_factory.py is deleted.

C. One construction idiom¶

Classmethod factories from_file / from_dir / from_dict / from_store construct from a source; __init__ takes parsed canonical objects only. One factory per source shape — no polymorphic constructors, no builder objects.

Applied to:

Validator — __init__(contract: ContractMetadata | dict, *, strict, include_quality, metrics, state_rules) + the four from_* classmethods. Delete the 7-kwarg constructor, ValidatorBuilder, create_validator, from_files, and the six *_with_contract/*_with_store wrappers. get_model() → model property; get_json_schema() → json_schema().
Pipeline — keep from_file/from_dict; rename from_config_dir → from_dir; delete niche from_config_files; add from_store(ref, *, store=None). run()/run_sync()/preflight() unchanged.
StreamWorker (new, streaming/worker.py) — the config-level API for streaming, the peer of Pipeline. from_file/from_dict/from_store build a StreamingConfig; run()/run_sync()/stop() drive the per-stream processors. ContractStreamProcessor stays as the low-level embedding API (constructor takes wired protocol objects — already correct under this idiom). The config→engine builders move from _cli_streaming.py into streaming/_worker_builder.py; the CLI shrinks to arg-parse + StreamWorker.from_file/from_store + run.

D. Naming conventions (normative)¶

Concept	Canonical	Replaces
Filesystem path	`path: str \\| Path`	`contract_file`, `contract_dir`, `directory`, `config_path`
Parsed config mapping	`config: dict[str, Any]`	`raw`
Record batch	`records: list[dict]`	`data_list`
Single record	`data: dict`	—
Variable overrides	`variables: dict[str, str]`	`extra_variables`
Store address	`ref: StoreRef \\| str`	`contract_name`+`contract_version`, `contract_id`, `--name`/`--version`
Store client param	`store`	`contract_store`
Identity fields	`name`, `version`	`contract_id`, `pipeline_name`, `stream_name`, `schema_name`
Async execution	`async def run()` + `run_sync()`; iterators `consume()`+`consume_sync()`	sync-only / async-only
Result type	`*-Result`; `ValidationOutcome` → `StreamValidationResult`	`ValidationOutcome`
Construction	`from_file`/`from_dir`/`from_dict`/`from_store`	overloaded ctors, builders

E. Smaller root¶

pycharter/__init__.py exports entry points, the pipeline operator classes, Validator/ValidationResult, contract/store clients, errors — and demotes generic leaks to their subpackages: from_dict/from_file → pycharter.pydantic_generator; to_dict/model_to_schema → pycharter.json_schema_converter; merge_rules_into_schema, the validate decorators, the step-contract helpers → their modules. Deletes check_quality_with_store (folded into polymorphic check_quality).

Scope — phased delivery¶

Each phase lands as its own atomic change set with tests green. P0 (the additive foundation) ships first and breaks nothing; the PAC items perform the hard-cut deletions on top of it.

P0 — Additive foundation (this change set)¶

Adds the new canonical surface without removing any old name, so the tree stays green and consumers migrate incrementally:

StoreRef (shared/refs.py); version_sort_key (utils/version.py, fixing the ordering bug); pycharter.open_store(kind, url) (stores/factory.py) returning the existing concrete store clients; ArtifactExistsError/ArtifactNotFoundError (stores/errors.py).
Pipeline.from_store(ref, *, store=None); run/preflight accept a StoreRef.
StreamWorker facade (streaming/worker.py) with from_file/from_dict/from_store/run/run_sync/stop, reusing the existing streaming engine builders.
Top-level exports: StoreRef, open_store, StreamWorker, StreamingConfig.

PAC backlog — status¶

Status legend: ✅ merged · 🔄 in progress · ⛔ blocked · ⬜ not started. Updated 2026-06-08.

ID	Title	Status	PR
P0	Additive foundation (`StoreRef`, `open_store`, `version_sort_key`, `Pipeline.from_store`, `StreamWorker`)	✅	#270
PAC-1a	`VersionedArtifactStore` ABC + pipeline store adoption	✅	#271
PAC-1b	Streaming store adoption	✅	#272
PAC-1c	LinkML/semantic store adoption	✅	#273
PAC-2a	Contract store: `VersionedArtifactStore` + atomic `save` + `get`/`list`/`list_versions`/`delete`/`set_status`/`update`; protocol cleanup; `build_contract_from_store` → `store.get`	✅	#274
PAC-2b	~~Internalize the contract per-artifact writers~~ — resolved: keep writers (by design). The per-artifact writers are the composition API: a contract is an immutable tuple of pointers to independently-versioned, immutable artifacts, and different contract versions reuse unchanged artifact versions. `save(document)` cannot express that reuse, so the writers stay. The real gap was one in-place-mutation hole (`upsert_field_bindings`) that broke artifact immutability; the fix — every editor edit creates a new contract version — is its own initiative: `contract-immutable-versions.md`. (The earlier de-sharing idea, `contract-artifact-ownership.md`, is withdrawn — it would destroy the intended reuse feature.)	✅ (won't-do) → 📋 immutable-versions	—
PAC-3	`Validator` collapse — one `__init__(contract)` + four `from_*`; delete builder/wrappers/`create_validator`/`from_files`; `model`/`json_schema()`; two live FastAPI endpoints (byte-identical), ~130 test sites, ~11 docs	✅	#302
PAC-4	`ContractRef` → `StoreRef` in streaming; `contract_id` → `name` across ~17 files; pin Kafka header + DLQ envelope keys (`contract_id`) + `format_version`; `ValidationOutcome` → `StreamValidationResult`	✅	#299
PAC-5	YAML ref-model unification — collapse `ContractFileRef`/`ContractStoreRef`/`ContractReference` onto `StoreRef` without YAML byte changes	✅	#298
PAC-6	Root surface pruning + naming sweep — demote generic leaks to subpackages; delete `check_quality_with_store`; polymorphic `check_quality(contract)`; `extra_variables`→`variables`; `from_config_dir`→`from_dir`; drop `from_config_files`. Shipped in two parts: root prune + `check_quality` + `extra_variables` in #300; decorator/step-helper demotion (`validate_input`/`validate_output`/`validate_with_contract_decorator`, `apply_step_contract_to_records`/`build_step_contract_context`) + `from_config_dir`→`from_dir` + drop `from_config_files` in the remainder PR	✅	#300 (+ remainder)
PAC-7	`api/dependencies` onto `open_store` — collapse 4 singletons; delete `_store_factory.py`	✅	#297
PAC-8	Route-shadow fixes — `PATCH …/{version}/status` (pipeline) and `GET …/{name}/versions` (LinkML) are swallowed by greedy catch-all routes (404); register the specific routes first, as `/versions` already is for pipelines. Pre-existing; found during PAC-1a/1c	✅	#296

PAC-1 complete (all versioned config stores share one interface). PAC-2a delivered the contract-store atomic write + interface. PAC-2b resolved — the contract per-artifact writers are load-bearing for artifact sharing (above), so the contract store keeps a two-tier API (whole-contract save/update + artifact-level writers) by design. The one immutability hole (upsert_field_bindings) was closed by the immutable-versions cutover (editor edit = new contract version): see contract-immutable-versions.md, shipped across PRs #279–#284, #291, #293 (and the binding-flow fixes #285–#290,

292, #294). The cutover's optional tail — dropping the now-unused `revision`¶

column and removing the per-section /metadata POST routes — is tracked there (CIV-FOLLOWUP-4 / "5c") and deliberately deferred. PAC-3 (Validator) is the next-largest user-facing item; PAC-7/PAC-8 are small, unblocked, and independent.

Danger flags (carry into every phase)¶

REST response field names + URLs are a UI contract — the Next.js UI consumes stream_name, pipeline_version, etc. and fixed URL paths. Do not rename them; map the uniform envelope (name/version) to the existing response models at the route boundary.
DLQ/Kafka wire format — the Kafka CONTRACT_HEADER and the DLQ envelope JSON key stay literally contract_id; format_version on stored documents is unchanged. Only Python attribute names change. A round-trip test pins the envelope keys.
Mongo atomic save is transactional only on replica sets — document the tier caveat (docs/reference/store-backend-tiers.md), do not fake it.
Latest-ordering change (lexicographic → numeric-aware) changes which version pycharter stream run --name X resolves when multi-digit versions exist — prominent changelog entry. Seed data (5.15.0, 3.13.9) shows this is a bug fix, not a regression.
No Alembic migration required — "latest" is computed; status, audit columns, and UniqueConstraint(name, version) already exist on every table. Mongo/Redis key patterns and persisted document shapes are untouched, so the hard cut is safe for users' databases.

Out of scope¶

Semantic store concept/scheme/relationship mixins (single-name keyed, not (name, version)); only the LinkML-config mixin adopts the interface.
diff_contract_in_store(store, name, v1, v2) — addresses two versions of one name; StoreRef does not fit.
The DLQ store (pycharter/dlq/) — a different protocol, not a versioned artifact store.