Skip to content

Offline pipeline plans and version identities

pycharter.pipeline_plan inspects a supplied canonical pipeline without reading sources, loading referenced contracts, resolving environment variables, creating connectors, or importing configured Python. It produces an incomplete structural plan for review. Extraction previews are different: they can read a source.

Which version means what?

Value Meaning
api_version: pycharter.io/v2 The document format understood by the reader. It is independent of a pipeline release.
Pipeline metadata.version The authored version of the complete pipeline definition, including inline steps and its declared references.
Step contract.ref.version The independently selected version of that data contract. Publishing a newer contract does not change an explicit reference.
plan_summary_digest A hash of the exported inspection summary, with the explicit scope offline_structure_v1.
Resolved bundle digest Future identity of captured effective content and dependencies. This slice always leaves bundle_digest null.
Deployment identity and binding revision Optional public context identifying an environment/deployment. This slice records labels; it does not resolve or enforce bindings.
Invocation/run identity Optional public context identifying one run and its input names. This slice records no parameter values and enforces no parameter policy.

Inline steps belong to their pipeline's version. Independently versioned reusable step artifacts require interface and parameter rules and are tracked separately. Legacy spec.versions has no execution consumer: this API reports its presence without activating, reinterpreting, exporting, or deleting its contents.

This is a full authored pipeline example. Its pipeline version 1.2.0 and contract version 1.1.0 refer to different artifacts. Loading it for inspection does not require the fixture, database, referenced contract, or environment variables to exist.

api_version: pycharter.io/v2
kind: Pipeline
metadata:
  namespace: market
  name: daily-prices
  version: "1.2.0"
spec:
  variables:
    DESTINATION_URL: "${PRICES_DATABASE_URL}"
  steps:
    - id: read-prices
      type: extract
      source:
        type: file
        path: fixtures/prices.json
    - id: shape-prices
      type: transform
      input: read-prices
      operations:
        - select: [symbol, date, close]
    - id: save-prices
      type: load
      input: shape-prices
      target:
        type: postgres
        connection_string: "${DESTINATION_URL}"
        table: daily_prices
      contract:
        ref:
          type: store
          name: market/daily-prices
          version: "1.1.0"

Inspect supplied configuration

from pycharter.pipeline_artifact import load_pipeline_from_dict
from pycharter.pipeline_plan import (
    compile_pipeline_plan,
    dump_pipeline_plan_to_dict,
    load_pipeline_plan_from_dict,
)

# document is the already supplied YAML/JSON mapping above.
definition = load_pipeline_from_dict(document)
plan = compile_pipeline_plan(
    definition,
    deployment={"id": "staging", "binding_revision": "revision-2"},
    invocation={"run_id": "run-42", "input_names": ["WINDOW_START", "SYMBOL"]},
)
exported = dump_pipeline_plan_to_dict(plan)
restored = load_pipeline_plan_from_dict(exported)
assert restored == plan
assert plan.spec.bundle_digest is None
assert not plan.spec.frozen and not plan.spec.verified and not plan.spec.executable

Use the canonical artifact loader for supplied manifests. The runtime parser can resolve environment-dependent configuration; it is not an offline inspection boundary. Reading a declared configuration file is an explicit outer operation; the compiler and plan codec themselves perform no file, store, or network I/O.

The plan retains authored step order, input edges, operation names, direct structural dependency declarations, and safe issue codes. An exact declared contract pin still has resolution: unresolved: the compiler has not captured or verified its content. An omitted contract version is floating. Files, configured Python/expressions, engine versions, and runtime bindings remain unresolved. Discovery is direct and structural, not a complete transitive dependency inventory. Native settings.state_store and settings.plugins remain inactive and are reported when present.

Privacy and digest limits

Only public identifiers and allowlisted structural facts are exported. Treat pipeline/step/dependency identifiers, deployment/run labels, and input names as public; do not put secrets in them. Connector options, URLs, paths, variable defaults, expression bodies, callable names/kwargs, policies, cataloguing metadata, and legacy ledger contents are withheld and are not hashed. Use dump_pipeline_plan_to_dict as the export boundary: it revalidates nested values before serialization, including models changed through unchecked Pydantic copying/construction. Direct serialization of arbitrary unchecked Python objects is outside this guarantee. Public labels accept letters, digits, underscores, dots and hyphens in slash-delimited segments; artifact identity uses the existing stricter slug and semantic-version rules.

The digest covers the exported structural summary, excluding the digest itself and deployment/invocation context. It preserves ordered graph and operation names. Changing a destination, secret, rename mapping, policy, or other withheld option can leave the digest unchanged. Equal summary digests do not establish equal execution behavior. It is neither an execution lock nor a signature or proof of origin. The codec rejects unsupported claims and inconsistent summaries; an author can still construct a different valid summary with its own digest.

Retained pipeline components adds a separate supplied-input API that saves admitted native operations and contract rules with scoped content checksums. It leaves complete bundle and execution claims false. Frozen execution, immutable publication, parameter enforcement, deployment validation, and replay evidence are later slices. Configuration reproducibility alone cannot guarantee identical outputs from a changing source, clock, randomness, or custom code. This API does not execute configured Python; connector substitution and offline planning do not sandbox arbitrary Python objects or application code.