Retained pipeline components

pycharter.pipeline_bundle saves supported native operations and contract rules from supplied canonical artifacts. Saved components remain available after the inputs change or disappear. Compilation and the dict codec perform no source, file, store, environment, or configured Python lookup.

The retained artifact has status: retained_incomplete: bundle_digest is null and frozen, verified, and executable are false. Source, destination, quarantine, and engine bindings remain unresolved in the artifact. An explicit runtime entrypoint can run its supported native projection with fixture connectors. Running the original pipeline still uses its normal configuration and dependency resolution.

Save supplied rules

Here is a complete pipeline definition. The pipeline release is 1.2.0; its attached contract is independently pinned to market/daily_prices at 1.1.0. The source and destination need not exist for offline compilation.

api_version: pycharter.io/v2
kind: Pipeline
metadata:
  namespace: market
  name: daily_prices
  version: "1.2.0"
spec:
  steps:
    - id: read-prices
      type: extract
      source:
        type: file
        path: fixtures/prices.json
    - id: shape-prices
      type: transform
      input: read-prices
      operations:
        - rename: {ticker: symbol}
        - convert: {close: number}
        - select: [symbol, date, close]
    - id: save-prices
      type: load
      input: shape-prices
      target:
        type: postgres
        connection_string: "${PRICES_DATABASE_URL}"
        table: daily_prices
      contract:
        ref:
          type: store
          name: market/daily_prices
          version: "1.1.0"

Supply the referenced contract's content explicitly. A store reference in this API is a lookup among the supplied contracts; no registry connection is made.

api_version: pycharter.io/v2
kind: DataContract
metadata:
  namespace: market
  name: daily_prices
  version: "1.1.0"
spec:
  json_schema:
    type: object
    properties:
      symbol: {type: string, minLength: 1}
      date: {type: string, pattern: '^\d{4}-\d{2}-\d{2}$'}
      close: {type: number, exclusiveMinimum: 0}
    required: [symbol, date, close]
    additionalProperties: false

from pycharter.contract_parser import parse_contract
from pycharter.pipeline_artifact import load_pipeline_from_dict
from pycharter.pipeline_bundle import (
    compile_pipeline_bundle,
    dump_pipeline_bundle_to_dict,
    load_pipeline_bundle_from_dict,
)

# pipeline_document and contract_document are the supplied mappings above.
definition = load_pipeline_from_dict(pipeline_document)
contract = parse_contract(contract_document)
bundle = compile_pipeline_bundle(definition, contracts=(contract,))
saved = dump_pipeline_bundle_to_dict(bundle)
restored = load_pipeline_bundle_from_dict(saved)
assert restored == bundle
assert bundle.spec.bundle_digest is None
assert not bundle.spec.frozen and not bundle.spec.verified and not bundle.spec.executable

Run the retained native projection

The separate pycharter.pipeline_bundle.runtime module runs supported retained content using the currently installed DAG engine and explicitly supplied PipelineIO. It bypasses Pipeline.arun: authored environment substitution, original file/store reference loading, configured connector factories, and operational callbacks are not used. The pure parent package exports stay the same. For the restored artifact above:

from pycharter.pipeline_bundle.runtime import arun_retained_pipeline_bundle
from pycharter.pipeline_generator.io import PipelineIO
from pycharter.pipeline_generator.testing import MockExtractor, MockLoader

destination = MockLoader()
bindings = PipelineIO(
    extractors={
        "read-prices": MockExtractor(
            [{"ticker": "FIXTURE", "date": "2026-10-12", "close": "2.5"}]
        )
    },
    loaders={"save-prices": destination},
)
result = await arun_retained_pipeline_bundle(restored, io=bindings)
assert result.success
assert destination.loaded_records == [
    {"symbol": "FIXTURE", "date": "2026-10-12", "close": 2.5}
]
assert not restored.spec.executable

Before the first source call, this API revalidates the complete retained document and component checksums, constructs runtime steps and validators, and checks every extractor, loader, and declared quarantine binding. A missing binding, unsupported operation/contract, unresolved variable declaration, runtime-incompatible step ID, or outgoing edge from a load step fails admission. Contract flags cannot bypass unsupported capture or validator compilation. Selected file references can be captured with the separate adapter below; this runtime entrypoint never reloads them. Explicitly selected pinned store contracts can also be captured below. Package and transitive resolution remain later work.

Binding maps are snapshotted for each invocation; connector resource objects are supplied and owned by the caller. Use separate finite fixture connectors for concurrent runs. Quarantine sinks belong in PipelineIO.dlq_sinks, keyed by the contract-bearing step ID. Contract application closes a used sink, including when processing fails; bindings for steps never reached remain caller-owned. The maintained contract path supplies validation, annotations, routing, and record tallies. A strict run-local context map represents unattached steps with explicit None; missing entries, including an empty map, cannot trigger configured dependency resolution.

Quarantine methods must complete synchronously. Obvious async methods and async callable objects fail binding admission. A synchronous wrapper that returns an awaitable or async generator fails when invoked and is not counted as an acknowledged write. That dynamic failure occurs after extraction; checking a callable's shape does not certify arbitrary connector behavior.

This executes only the admitted native projection. It does not retain or certify the original pipeline's complete operational settings, engine/package versions, source snapshot, deployment, or allowed invocation parameters. Inactive legacy/state/plugin facts stay inactive. The artifact's full digest and completeness flags remain unchanged after a run. Supplied connectors are trusted Python, and the DAG materializes their batches: document-size limits do not bound extracted records, memory, or duration. Use finite fixtures here; production resource policies remain separate work. Connector substitution is not a sandbox for arbitrary Python.

Unselected file references, missing store versions, absent supplied contracts, and ambiguous names stay explicit unresolved slots. A namespace-qualified exact reference must match that identity. Equivalent supplied duplicates deduplicate; conflicting admitted content fails. An unsupported duplicate blocks capture of that identity instead of allowing an easier candidate to win. Unused supplied content is not serialized or hashed.

Capture declared contract files

The separate pycharter.pipeline_bundle.files module reads explicitly selected contract files. Supply an absolute, caller-owned root and an expected exact identity for each selected step. The expectation comes from the caller's reviewed inputs; it is never inferred from the file being read.

This example changes the earlier pipeline's contract declaration to a file, captures its supported rules, then deletes that file. It uses a temporary root and performs no source or destination I/O.

import copy
import json
from pathlib import Path
from tempfile import TemporaryDirectory

from pycharter.pipeline_bundle.files import capture_pipeline_bundle_from_files

file_document = copy.deepcopy(pipeline_document)
file_document["spec"]["steps"][-1]["contract"]["ref"] = {
    "type": "file",
    "path": "contracts/prices.json",
}
file_definition = load_pipeline_from_dict(file_document)
with TemporaryDirectory() as temporary:
    root = Path(temporary).resolve()
    contract_file = root / "contracts" / "prices.json"
    contract_file.parent.mkdir()
    contract_file.write_text(json.dumps(contract_document), encoding="utf-8")
    captured = capture_pipeline_bundle_from_files(
        file_definition,
        root=root,
        expected_contracts={
            "save-prices": {
                "namespace": "market",
                "name": "daily_prices",
                "version": "1.1.0",
            },
        },
    )
    contract_file.unlink()
restored = load_pipeline_bundle_from_dict(dump_pipeline_bundle_to_dict(captured))

The retained runtime example above can now run this restored artifact with fresh finite fixture connectors. It uses saved rules even after file mutation or deletion. Normal execution of file_definition still follows its authored file declaration. Unselected file attachments remain unresolved and are not opened. Explicit contracts= can also supply exact store attachments; the adapter does not open a configured store.

The bundle records resolved_file_components_v1 scope and a distinct attachment with capture: supplied_file_contract_v1. Its authored facts remain ref_kind: file and pin: unknown, with null reference/version. Expected and resolved identities are separate facts; neither becomes an authored version pin. Paths, absolute roots, raw file bytes and raw-file hashes are withheld. Only admitted native behavior receives a component checksum. Original B1 documents keep their exact representation and scope. Older readers reject the new scope; there is no pipeline, contract, constituent or envelope version bump.

The complete selection is validated before any artifact is opened. Supported paths are relative JSON/YAML/YML declarations beneath the trusted root, with no parent traversal, URI/environment forms or descendant symlinks (including symlinks that point inside the root). Descriptor-relative, no-follow regular-file reads require POSIX capabilities; unavailable platforms fail closed. The limits are accepted byte budgets of 1 MiB per file and 4 MiB total, and 256 combined selected files and supplied contracts. Over-limit detection may read one extra byte. Parsing additionally limits nodes/events to 20,000, nesting to 24, and scalar text to 65,536 characters. Duplicate decoded keys, YAML anchors, aliases, merges, custom tags, non-JSON scalars and multiple documents are rejected. Only native contract documents are supported here; ODCS conversion stays a separate operation. ${ is rejected anywhere in parsed data, including literal patterns and ignored catalog text; no interpolation occurs.

File and store capture check normalized section/footer version declarations before parser coalescing and preserve the checked facts. Contradictory or malformed known versions are rejected; a contract release does not rewrite an independently versioned schema or other constituent.

For callers that already hold canonical contracts, the pure pycharter.pipeline_bundle.resolution module exposes FileContractResolution and compile_resolved_pipeline_bundle. Each resolution supplies a step ID, expected identity and canonical contract. This is caller-supplied evidence, not attestation of a filesystem read. The pure API performs no filesystem lookup. Both paths reject conflicting admitted content under the same exact identity; unsupported duplicate candidates cannot be bypassed. A namespace of None means exactly no namespace, rather than a wildcard.

This adapter captures declared contract behavior, not a complete frozen bundle or source snapshot. It does not establish an operating-system sandbox, validate arbitrary Python, certify trusted fixture connectors, resolve transitive references, or pin code/engine/container dependencies. All existing unsupported contract and policy declarations still block retained execution. Broader capture and publication remain separate work.

Capture selected pinned store contracts

The separate pycharter.pipeline_bundle.stores module accepts an already connected, caller-owned synchronous reader. Select existing store attachments by step ID with exact namespace, name and version expectations. The declaration must already pin that same version; floating/latest references are rejected before any read. The adapter never constructs, configures, connects, disconnects or lists a store, and never reads pipeline sources or destinations.

This example uses an isolated temporary SQLite registry with the earlier definition and parsed contract. Initialization is only for this disposable example; initialize production registries through the database workflow.

from pathlib import Path
from tempfile import TemporaryDirectory

from pycharter import SQLiteContractStore
from pycharter.pipeline_bundle.stores import capture_pipeline_bundle_from_store

with TemporaryDirectory() as temporary:
    store = SQLiteContractStore(f"sqlite:///{Path(temporary) / 'registry.db'}")
    store.connect(auto_initialize=True)
    try:
        store.save("daily_prices", "1.1.0", contract.to_dict(), namespace="market")
        captured = capture_pipeline_bundle_from_store(
            definition,
            store=store,
            expected_contracts={
                "save-prices": {
                    "namespace": "market",
                    "name": "daily_prices",
                    "version": "1.1.0",
                },
            },
        )
        store.delete("daily_prices", "1.1.0", namespace="market")
    finally:
        store.disconnect()
restored = load_pipeline_bundle_from_dict(dump_pipeline_bundle_to_dict(captured))

The retained runtime can run restored with fresh finite fixture connectors after the registry changes or disappears. Each distinct selected identity is read once. Acquired content resolves only selected slots, even if another step declares the same identity. Existing explicitly supplied contracts= behavior is preserved. Pure callers can instead supply StoreContractResolution records through compile_resolved_pipeline_bundle(..., stores=(...,)), together with optional file records; these records are caller evidence, not proof of origin.

Envelope identity is checked before native document parsing. Providers return stored native section dictionaries (json_schema, metadata, and optional sections), not authored api_version/kind/spec envelopes. Missing document identity fields are filled only from that verified envelope, and contradictions in identities or known constituent versions fail. Constituent versions remain independent of the pipeline and contract release versions. Legacy assemblies reusing a metadata constituent at another version exceed the current retained representation: their facts are preserved during parsing and the slot remains unsupported, rather than rewriting that version. An expected namespace: None means global identity: current stores may perform a wildcard lookup for None, but foreign results and ambiguous global/named collisions are refused. Use a qualified namespace when that is the intended identity.

The adapter snapshots bounded plain native JSON data: 1 MiB compact UTF-8 JSON per distinct returned document, 4 MiB total, 20,000 nodes, depth 24, scalar text 65,536, and 256 combined selected slots and supplied contracts. These checks include discarded catalog values inside the document. Unrelated envelope catalog fields are neither inspected nor retained. Unknown top-level document fields, ODCS, custom Python values, non-string keys, nonfinite numbers, future formats and ${ syntax are rejected. Unsupported behavioral policies, including empty/disabled mappings, still block retained execution.

Store-only capture uses the existing supplied_native_components_v1 artifact; mixed supplied file/store capture keeps the file scope. No wire or digest version changes, complete registry snapshot, attestation or executable/frozen claim follows. The provider's own I/O, latency and initial allocations are not bounded, and it is trusted Python. Capture and fixture connector substitution do not establish a sandbox for arbitrary Python.

Contract names copy verbatim during authoring-envelope conversion, including the literal name contract. Identity-less partial dictionaries now stay identity-less instead of acquiring a synthetic name; they still fail canonical validation. Old partial wrappers already containing name: contract are indistinguishable from authored identities and need correction from their reviewed source. Ordinary DAG execution now resolves qualified references as namespace plus bare name. Legacy literal slash-bearing global store keys need migration to that representation; there is no fallback to an unrelated global key.

Supported behavior and unresolved slots

Native operations retain their authored order and repetitions. Supported operations are rename, select, drop, convert, default flatten, and default unnest. Ordered mapping entries preserve sequential rename behavior: on {a: 1, b: 2}, {a: b, b: c} yields {c: 1}, while reversing the mapping yields {c: 2, b: 1}. Empty config selections are neutral. flatten: {} is active; flatten: null is inactive. Nondefault flatten/unnest options, literal defaults, additions, filters, JSONata, and custom Python remain unsupported slots. Join/union edges are retained but execution options remain unsupported. One unsupported slot keeps the transform incomplete.

Contract admission covers flat object schemas with string, integer, number, and boolean fields, required fields, boolean additionalProperties, and strict_types. Supported numeric rules are minimum, maximum, exclusiveMinimum, and exclusiveMaximum; string rules are minLength, maxLength, and pattern. Zero bounds, empty patterns, combined assertions, and exact pattern text are preserved. Matching uses the maintained Validator's Python regex search behavior. The compiler does not match source data.

Unhandled behavioral sections block the entire contract. Nested schemas, references, enum/const/default/example payloads, formats, coercion, custom validation, quality rules, and policies are not silently discarded. Any supplied unhandled policy mapping, including {} or an explicitly disabled policy, blocks admission. Empty mappings remain authored declarations even when their current defaults happen to match omission. Known display annotations may be omitted.

Parsed-contract exports and promotion preserve supplied extraction_policy, ai_usage_policy, and kb_capture mappings exactly, including {}; they do not expand defaults. Absent and null policies remain equivalent to omission. This holds through canonical export/reload and parse/store/build on all five contract backends. Earlier parsed-contract exports dropped empty mappings. A record already missing one cannot reveal whether it was omitted or lost: re-export the original reviewed configuration, following the store's existing version/update rules. Retention does not reconstruct historical declarations or implement these policy behaviors.

Stored contracts can include synthesized empty metadata. Empty ownership and lineage mappings, and runtime declarations with only empty storage_locations, serve, and environments, are admitted as neutral. A nonempty value or unknown key in those blocks makes the contract unsupported; this exception does not extend to policy mappings.

If json_schema.version is absent, the effective schema version follows the existing Validator fallback to contract metadata.version. The bundle records that provenance separately from actual constituent-version facts. It does not invent a constituent revision or reinterpret legacy pipeline spec.versions. Native state-store/plugin settings remain inactive. Supplied constituent-version declarations must agree with known section and attachment versions. They are not independently resolved or proof of origin; missing facts remain missing.

Credentials and checksums

Supported rule values and public identifiers are approved to save. This does not make the saved artifact publicly accessible. Authors must keep secrets out of those fields, including regex declarations and field names. Credentials belong in external bindings; connector URLs, paths, auth, environment values, arbitrary kwargs, and opaque payloads are withheld from retained content and hashes. This API does not automatically discover a secret deliberately placed in an approved field.

Use the public dump function as the export boundary: it revalidates nested models before serialization or hashing, including unchecked Pydantic copies. Malformed inputs produce static guidance without echoing private values. Admitted structures and retained documents have bounded sizes; opaque input payloads are not traversed or subject to a complete caller-object size limit. Closed shapes, graph edges, operation slots, attachment identities, and component references are validated independently of checksum equality.

Each component checksum has scope retained_native_component_v1. It identifies admitted component content, including effective defaults and ordered mappings. It is stronger than a structural plan summary, but neither checksum identifies the complete executable pipeline. Checksums are not signatures or proof of origin. Omitted display metadata and unresolved bindings do not affect them.

Package/optional-store/transitive adapters, broader policies, engine/package identity, typed invocation parameters, immutable publication, and full bundle identity remain later work. See Offline pipeline plans for the separate format, definition, contract, deployment, and run identities. Source-reading previews remain distinct from offline compilation. Connector substitution and these pure APIs do not sandbox arbitrary Python objects or application code.