Skip to content

PyCharter — LinkML as the shape substrate (concepts stay the meaning substrate)

Status: approved (architectural decision) · Created: 2026-05-28 · Audience: pycharter maintainers and integrators

Purpose

This document records the architectural decision to make LinkML the canonical shape substrate in pycharter — for data-contract schemas, DTO catalogs, and codegen — while keeping pycharter's concept registry as the canonical meaning substrate. The two layers are deliberately distinct and bridged by the slot.meaning: / concept_scheme annotation on LinkML slots. Companion proposals: cross-package-decoupling.md, remove-shacl.md (vocabulary governance without SHACL).

Decision

  1. LinkML defines shape. What fields exist, what types they hold, what is required, what enumerations are permissible. Pycharter's contracts may reference a LinkML schema via the domain_schema field on the contract; field bindings target LinkML class/slot pairs via LinkMLSlotTarget (src/pycharter/semantic/contracts/field_bindings.py).
  2. Pycharter concepts define meaning. What a field refers to in the real world, what relationships exist between concepts, what tier governs concept changes. The concept registry (ConceptScheme, Concept, Relationship in src/pycharter/semantic/ontology/models.py) remains the system of record.
  3. The bridge is the slot meaning URI. LinkML slots may carry a meaning: annotation (and pycharter's LinkMLSlotTarget carries an accompanying concept_scheme) that points into the concept registry. LinkML says "this slot is a string of max length 64, required"; the concept registry says "this string identifies a material.glass.borosilicate instance."
  4. LinkML is the source of truth for cross-package DTO catalogs. The integration DTOs (Flow A/B/C shapes from cross-package-decoupling.md) are defined in docs/contracts/integration-dtos.linkml.yaml and compiled to Pydantic via src/pycharter/semantic/generators/_pydantic.py in each package independently. Both packages depend on linkml/linkml-runtime (already an optional extra) but not on each other.

Why now

Most of the plumbing is already in place; the missing piece is wiring it into the contract load path.

Already shipped:

  • src/pycharter/semantic/linkml/ — loader, bridge to ConceptScheme, filesystem registry, editor-JSON serializer. Pure YAML parsing; the optional [linkml] extra is only needed when codegen is invoked.
  • src/pycharter/semantic/generators/ — deterministic codegen to Pydantic, JSON Schema, SQL DDL, SHACL, and OWL from a LinkML schema. All tested.
  • LinkMLSlotTarget and the auto-detection logic in src/pycharter/semantic/ontology/parser.py:_parse_new_format distinguishing new-format (LinkML-aware) from legacy concept-centric bindings.
  • 30+ test files exercising LinkML paths.

The gap. When a contract declares domain_schema: trading:1.0.0, that reference is parsed as a string but never resolved. Field bindings that target class: Order, slot: id are accepted whether or not those class/slot pairs actually exist in the referenced schema. The validation loop is open.

Where LinkML is used (and where it is not)

Used as the shape substrate:

  • Data contract domain_schema blocks
  • FieldBinding.primary.linkml_slot targets
  • The cross-package DTO catalog (docs/contracts/integration-dtos.linkml.yaml)
  • Codegen for Pydantic, JSON Schema (OpenAPI), SHACL (semantic health), SQL DDL, and OWL via semantic/generators/
  • Contract publish (the LinkML serialization is what consumers pull via Flow B of the integration seam)

Not used (intentionally):

  • The concept registry models themselves — Concept, Relationship, ConceptScheme stay as native pycharter dataclasses. They reference LinkML schemas via slot meanings, but they are not a LinkML schema.
  • The ResourceBinding state machine and event store — state machines are not a shape problem.
  • Coercion rules, validation rules, FSM lifecycles — runtime logic, not schema.
  • Extraction adapters and the materializer — they produce candidate concepts, not LinkML structures.

The decoupling-driven payoff

Under the decoupling decision in cross-package-decoupling.md, pyintake and pycharter must exchange data through duplicated-shape DTOs without importing each other. LinkML solves this by construction:

  1. One LinkML file at docs/contracts/integration-dtos.linkml.yaml defines SuggestionBatch, ContractSnapshot, ResourceBindingPayload, and the rest.
  2. Each repo checks the file in (identical bytes) and runs make generate-integration-dtos locally to produce its own Pydantic models under src/<pkg>/api/models/integration.py.
  3. A pre-commit hook fails on drift between the LinkML source and the generated Pydantic.
  4. Neither package imports the other; both depend on the LinkML toolchain (already an optional extra in pycharter).

This is the cleanest expression of "duplicate, don't depend": the wire shape is one file; the storage and Python types are duplicated; nothing crosses the package boundary at runtime.

Migration posture (what stays, what closes)

  • Legacy concept-centric field_bindings (no domain_schema) stays fully supported. Auto-detection in _parse_new_format continues to dispatch between the two. No deprecation, no migration churn.
  • The LinkML loop closes additively. When domain_schema is present on a contract, pycharter resolves the reference via LinkMLSchemaRegistry and validates that each LinkMLSlotTarget resolves to an actual class+slot in the schema. Contracts without a domain_schema are unaffected.
  • pydantic_generator/ and json_schema_converter/ stay. They handle non-LinkML contracts today and continue to. Migration of those modules to LinkML-only codegen is opportunistic, not a project.

Open conventions to settle (during Workstream B)

These are smaller decisions that fall out of closing the loop:

  • meaning: URI namespace policy — do LinkML slot meanings point to pycharter concepts (pycharter:concept/material.glass.borosilicate), to upstream standards (obo:CHEBI_46784, qudt:Pascal), or both? Recommend: both, with namespace conventions documented in the contract-authoring guide.
  • linkml-runtime version pin — pin to a tested minor range. The linkml-runtime subset has a more stable API than the full linkml tool; production code uses linkml-runtime, dev/CI uses the full linkml package for codegen.
  • Generator output location — src/pycharter/api/models/integration.py for the cross-package DTOs; generated contract Pydantic stays in the existing pydantic_generator/ output paths until that subsystem is migrated.

Acceptance criteria

Pycharter is "production-grade on LinkML" when:

  1. A contract with domain_schema: <name>:<version> and field bindings to non-existent class/slot pairs fails to load with a clear, actionable error naming the missing slot.
  2. The same contract with correct bindings loads cleanly and exposes the resolved LinkMLSchema to downstream consumers.
  3. docs/guides/contract-authoring.md (or equivalent) includes a worked LinkML-first example covering domain_schema, LinkML class/slot bindings, and concept bindings for meaning.
  4. The cross-package DTO catalog (docs/contracts/integration-dtos.linkml.yaml) round-trips: LinkML → Pydantic → JSON → LinkML validator passes.

These map directly to Workstream B and Workstream D of the implementation plan.

What this decision does NOT do

  • Does not deprecate the legacy concept-centric field_bindings format. Both formats are first-class. Auto-detection dispatches at parse time.
  • Does not require all contracts to declare a domain_schema. Optional; absent it, validation falls back to the existing concept reference checks.
  • Does not collapse the concept registry into LinkML. Concepts, relationships, tiers, and schemes remain pycharter dataclasses backed by their existing storage.
  • Does not make linkml-runtime a core (non-extra) dependency. The current [linkml] optional extra is sufficient; the loader uses pure YAML parsing so the core path stays light.
  • Does not retire pydantic_generator/ or json_schema_converter/. They cover non-LinkML contracts and continue to. Migration is opportunistic if and when natural pressure to consolidate arises.

References

  • Companion proposal: docs/proposals/cross-package-decoupling.md
  • LinkML loader and bridge: src/pycharter/semantic/linkml/
  • LinkML-driven codegen: src/pycharter/semantic/generators/
  • Field bindings: src/pycharter/semantic/contracts/field_bindings.py
  • Parser auto-detection: src/pycharter/semantic/ontology/parser.py
  • Implementation plan: /Users/IMTAUC/.claude/plans/system-reminder-the-git-worktree-piped-beacon.md