PyCharter Migration Guide¶

Numeric quality assertion corrections¶

mustNotBe now evaluates inequality in aggregate, expression, and per-column quality rules. In particular, row_count with mustNotBe: 0 fails an empty batch and passes a nonempty batch. Review contracts that relied on the prior ignored assertion or observed-equals-zero fallback.

Native step contract references now read the registry's stored data_quality section. Previously these rules could disappear unless copied into legacy metadata. A stored gate can therefore fail a run that previously bypassed it. An explicit stored section, including an empty one, takes precedence over the legacy metadata copy. Review the exact stored contract version before running.

Numeric operands now reject booleans, nonnumeric strings, NaN, infinity, malformed ranges, and conflicting camelCase/snake case aliases at contract load time. Direct evaluator dictionaries raise a configuration error when the invalid rule is reached. Replace coerced boolean operands with explicit 0 or 1 as intended. Give mustBeBetween exactly two finite endpoints in ascending order; equal endpoints remain valid. Keep one spelling for an assertion if its aliases disagree. Finite numeric strings, negative thresholds, null/unset defaults, and custom callable verdicts remain supported.

Multiple distinct non-null native numeric assertion fields now use logical AND: every requirement must pass. This is an intentional semantic compatibility break from the former first-match behavior. For example, row_count with mustBeGreaterThan: 0 and mustBeLessThan: 1000 now fails a 1,500-row batch; the earlier executor could pass it by checking only the lower bound.

Review every rule declaring multiple assertions before upgrading. Keep all intended requirements, remove accidental assertions, and correct contradictory requirements. Split checks into separate named rules when they need separate severities or outcomes. Each combined rule still measures once and produces one outcome; failure guidance lists all failed requirements in deterministic operator order. Equivalent aliases count once, null is unset, and malformed later operands still fail validation. Single assertions, no-assertion defaults, and custom callable verdict ownership are unchanged. See numeric quality assertions.

Operational SLA comparison direction¶

Quality reports no longer treat every numeric sla_properties commitment as an upper bound. Add comparison: at_most for a maximum (for example, maximum latency or freshness age) or comparison: at_least for a minimum (for example, availability). Existing entries remain valid, but a supplied observation is reported with breached: null until the direction is declared. Invalid comparison values fail contract parsing. See Contract SLA comparison.

Pipeline format 3: one load-target write vocabulary¶

The load target's write vocabulary is unified (breaking, pre-1.0, no aliases):

Legacy key	Current key	Applies to
`write_method:`	`mode:`	Database targets (`insert`, `upsert`, `replace`, `update`, `delete`, `truncate_and_load`)
`write_mode:`	`mode:`	File / cloud-storage targets (`overwrite`, `append`)
`primary_key:`	`match_on:`	Upsert/update/delete row-match key (not necessarily the table PK)

Persisted store artifacts upgrade on read (format 2 → 3). Files rewrite with pycharter migrate <file> --write — including already-current manifests that still carry the old keys. A raw config passed to Pipeline.from_dict with an old key fails fast with an error naming the rename.

Optional dependency extras consolidated (29 → 15)¶

Extras are grouped by capability. This was a breaking change to extra names (clean break, no aliases — pycharter is pre-1.0):

Legacy extras	Current extra
`[api]`, `[ui]`, `[worker]`, `[postgres]`, `[observability]`, `[fastapi]`	`[server]`
`[pipeline]`, `[lineage]` (and the never-defined `[etl]`)	`[pipeline]`
`[streaming]`, `[kafka]`, `[rabbitmq]`, `[redis-streams]`, `[messaging]`, `[collab]`	`[streaming]` (all broker transports bundled)
`[linkml]`, `[semantic-export]`, `[neo4j]`	`[semantic]`
`[ai]`, `[mcp]`	`[ai]`

[extraction], [otel], [cli], [docs], [airflow], [dagster], [prefect], [all], [dev], and [ci] are unchanged. Orchestrator adapters stay separate (heavy and mutually exclusive).

Naming change: ETL → Pipeline¶

As of this release, all "etl" naming has been renamed to "pipeline" across the codebase (no backward compatibility):

Packages: etl_generator → pipeline_generator, etl_store → pipeline_store
API paths: /etl/run → /pipeline/run, /etl/configs → /pipeline/configs
CLI: Subcommand etl → pipeline (e.g. pycharter pipeline run ...)
UI: Routes /pipelines/etl → /pipelines/editor; components and API client use "pipeline"
DB table: etl_pipeline_configs → pipeline_configs (run migrations to rename)
Docs: "ETL Store" → "Pipeline Store", etl-store.md → pipeline-store.md

Update imports from pycharter.etl_generator to pycharter.pipeline_generator and pycharter.etl_store to pycharter.pipeline_store. Use the new API paths and CLI subcommand in scripts and clients.

Canonical `pipeline.yaml` (multi-file removal)¶

Section-based and multi-file pipeline authoring is no longer supported.

Legacy	Canonical
`extract.yaml`, `transform.yaml`, `load.yaml` in one directory	Single `pipeline.yaml` with `steps:`
Top-level `extract:` / `transform:` / `load:` sections	Steps with `type: extract` / `transform` / `load`
`Pipeline.from_config_files(extract=..., load=...)`	`Pipeline.from_dir(...)` or `Pipeline.from_file(...)`
`POST /api/v1/pipeline/run` with `extract_yaml` / `load_yaml`	Same endpoint with `pipeline_yaml` (full canonical document)

Migration checklist:

Run pycharter pipeline init ./my_pipeline --template http-to-postgres and compare with your old files, or compose from data/templates/pipeline/snippets/.
Move identity to id (slash-qualified) and browse folder to metadata.namespace.
Wrap extract config under steps[].source, transform ops under steps[].operations, load config under steps[].target.
Add format_version: "2", version, and metadata block (no metadata.version — the artifact version lives at the top level).
Update API clients and UI to send pipeline_yaml instead of section YAML strings.

Dashboard run metadata: persisted pipeline runs now derive extractor_type / loader_type from the first canonical steps[] entry when the snapshot has no legacy top-level extract / load sections.

Editor folder tables removed (CLEAN-1)¶

The UI file browser no longer uses DB-backed folder trees. Browse paths come from metadata.namespace on contracts, pipelines, and related artifacts.

Legacy	Current
`editor_folders` / `editor_folder_items` tables	Not created (removed from schema)
Folder REST API	Removed — virtual tree only

Ops checklist: reset or recreate the DB (pycharter db truncate then pycharter db upgrade, or delete the SQLite file and run db init). Folder data was presentation-only; no artifact content lived in those tables.

Questions?¶

If you have questions about migration, please open an issue on GitHub.