Skip to content

PyCharter — Cross-package decoupling (pyintake handshake)

Status: approved (architectural decision) · Created: 2026-05-28 · Audience: pycharter maintainers and integrators

Purpose & how to use this doc

This document records the architectural decision that governs pycharter's relationship with its sibling package pyintake (the KB-synthesis layer at /Users/IMTAUC/akc/statfyi/pyintake/). It supersedes the prior assumption, captured in docs/proposals/archived/kb-etl-alignment-notes.md, that pycharter would shed code to pyintake under a phased "Option B" migration plan. That migration is withdrawn.

This is a self-contained brief: a fresh session should be able to read it and understand the boundary without prior context. Companion proposal: linkml-as-schema-substrate.md (the schema-shape side of the same picture).

Decision

Pycharter and pyintake are fully decoupled packages. Specifically:

  1. Each package owns its own concept scheme. Pycharter's stays narrow and contract-binding focused. Pyintake's grows into the synthesis / RAG layer. Neither is the system of record for the other.
  2. No runtime coupling in either direction. Pycharter must be installable, runnable, and testable with pyintake absent — and vice versa. Neither package may import or special-case the other.
  3. Integration is opt-in, DTO-mediated, transport-agnostic. When both are deployed, they exchange data through duplicated-shape DTOs over HTTP/file/queue. The DTOs are defined in a shared LinkML schema (docs/contracts/integration-dtos.linkml.yaml) — checked into each repo, generated to Pydantic locally — not imported across the boundary.
  4. Duplication is preferred over interdependence. If two packages would otherwise need to share a model/store/protocol, each gets its own copy with the same DTO shape on the wire. The cost of duplication is accepted as the price of independence.

This decision resolves the open question previously captured in BACKLOG.md under "Concept-scheme registry ownership" — the registry stays in pycharter (and pyintake has its own). No further deferral is needed.

What this withdraws

The prior alignment notes (kb-etl-alignment-notes.md, 2026-05-27, now archived) assumed an Option B migration where modules such as semantic/kb.py, semantic/extraction/*, persistence/collaboration/suggestions.py, semantic/extraction_policy, KB ingest API routes (kb-upload, _ingest_*), and semantic/knowledge_base/* would migrate out of pycharter and into pyintake across Phases 1–4. None of that migration happens. Pycharter retains all of these modules and continues to evolve them for its own needs (narrow, contract-binding-driven, deliberately not a synthesis platform).

The §11 naming-consolidation discussion in the archived doc (field_mapping / field_bindings / ontology artifact naming) is orthogonal to this decoupling decision and remains a separate discussion. The §3 KB-as-ETL-recipe sketch in the archived doc is withdrawn along with the Option B premise.

Integration handshake — three flows, one DTO catalog

When both packages are deployed together, they communicate over three HTTP flows. Each is opt-in via configuration; absent the config flag, the corresponding code paths simply do not fire.

Flow Direction Endpoint (pycharter side) DTO
A — Inbound suggestions pyintake → pycharter POST /api/v1/inbound/suggestions SuggestionBatch → SuggestionBatchReceipt
B — Contract snapshot publish pyintake → pycharter (pull) GET /api/v1/contracts/{id}/snapshot ContractSnapshot
C — ResourceBinding payload pyintake → pycharter POST /api/v1/semantic/resource-bindings (already shipped, DG-1) ResourceBindingPayload

Mapping policy at the receiver. For Flow A, pycharter does the inbound-vocabulary → own-vocabulary mapping. Default rule: match concept_key by name 1:1; unknown keys auto-stage as FREE-tier suggestions for human review. Pycharter never reaches into pyintake's scheme to resolve references.

Idempotency. Inbound flows accept an Idempotency-Key header and reuse the existing maybe_existing_run helper (api/routes/v1/_ingest_helpers.py). Retries are safe.

Auth. All three flows require Role.REVIEWER via the existing require_role dependency. No new auth surface.

Transport-agnostic in principle. HTTP/JSON is the default. The same DTOs work for file-drop, S3-object-watch, or a message bus; transport is a deployment choice, not a code one.

Annotated record substrate

Pyintake's primary input is pycharter's annotated record stream — records carrying a _concept_annotations sidecar key produced by annotate_record (src/pycharter/semantic/contracts/runtime.py) when a pipeline step sets annotate: true. The substrate is:

  1. Pycharter's pipeline loads annotated records to a destination (Parquet, Postgres, Kafka, whatever the target: clause names).
  2. Pyintake reads from the destination, not from pycharter. The destination is the shared substrate; neither package depends on the other.
  3. Pyintake's deeper extractors (LLM, vector retrieval, semantic chunking) produce candidate concepts that flow back as SuggestionBatch via Flow A.

This is the cleanest possible decoupling: data crosses the boundary through a destination they both can read, not through a direct pipe.

Annotation as a record-level contract

The _concept_annotations sidecar shape (src/pycharter/semantic/contracts/runtime.py:312-332) is treated as a stable record-level contract, on equal footing with the integration DTOs above. Any downstream consumer (pyintake, an analytics tool, a search index) may rely on its shape. Breaking changes require a format_version bump on the contract artifact (DG-2 machinery).

What pycharter does as a result

Workstreams driven by this decision are tracked in the implementation plan at /Users/IMTAUC/.claude/plans/system-reminder-the-git-worktree-piped-beacon.md. Summary:

  • A — Codify the decoupled philosophy (this proposal + linkml-as-schema-substrate.md + BACKLOG sync + archive of the superseded notes).
  • B — Close the LinkML loop so domain_schema references resolve and validate field bindings (companion proposal).
  • C — Wire the structural extractor (AutoExtractionSink) into the pipeline runtime, so pycharter's own KB grows from its own pipelines. This proves the standalone story before any cross-package coupling exists.
  • D — Build the integration seam: DTO catalog at docs/contracts/integration-dtos.linkml.yaml + the three endpoints above + the auto-stage mapping policy + the integrations.* config flags.
  • E — Opportunistic internal items (P5-007 MCP server, DG-3 bitemporal spike, optional webhooks) that are independent of this split.

What this decision does NOT do

  • Does not modify pycharter's concept registry storage, models, or governance. The existing ontology layer is adequate for the contract-binding workload.
  • Does not remove any code in anticipation of pyintake taking it over. No migration map. No deprecation shims for KB-adjacent modules.
  • Does not add pyintake as a dependency of any kind (runtime, optional extra, or [dev]). The integration is purely opt-in via config + HTTP DTOs.
  • Does not collapse the LinkML layer into the concept registry. See linkml-as-schema-substrate.md for why shape and meaning stay distinct.
  • Does not introduce webhooks in the initial integration seam. Outbound POST on contract.published is deferred until a real consumer asks for it; the existing SSE ChangeBus covers in-process listeners.

Generalizability note

This decision is framed around pyintake because pyintake is the active consumer driving it. Every clause generalizes: the integration seam is source-neutral. A second sibling package, a third-party consumer, or a future internal product can plug into the same DTOs and endpoints with no pycharter changes. The "duplicate, don't depend" rule applies symmetrically.

References

  • Companion proposal: docs/proposals/linkml-as-schema-substrate.md
  • Withdrawn premise (archived): docs/proposals/archived/kb-etl-alignment-notes.md
  • DG-1 (generic ResourceBinding, shipped 2026-05-25): docs/guides/resource-bindings.md
  • DG-2 (artifact format versioning, shipped): docs/reference/artifact-format-versioning.md
  • Annotation runtime: src/pycharter/semantic/contracts/runtime.py, src/pycharter/semantic/pipelines/runtime.py
  • Implementation plan: /Users/IMTAUC/.claude/plans/system-reminder-the-git-worktree-piped-beacon.md