PyCharter — Vocabulary Consolidation and the Contribution Flywheel¶

Status: implemented (2026-07-12; VC-1..16 shipped) · Created: 2026-07-10 · Audience: maintainers · Extends: terminology.md (T-series) · Supersedes: kb-ontology-workflow-proposal.md, vocabulary-governance-rfc.md, vocabulary-implementation-plan.md

The Vocabulary pillar's primitives are unusually complete: a governed suggestion queue with provenance and confidence, tier governance with write-protected core concepts, extraction from records / code / docs / contracts, LinkML projection, RDF / Neo4j / Obsidian export, and agent context packs. The primitives are done; the product isn't. A user still has to distinguish semantic_store, concept graph, LinkML schema, field binding, scheme, suggestion, and (two different things called) "knowledge base" before accomplishing a basic task, across three UI surfaces, ~37 API routers, and three overlapping concept models.

This proposal finishes the consolidation the T-series started and then closes the contribution flywheel — the product thesis that using pipelines and contracts grows your vocabulary by default, governed:

Contract registered → concept suggestions appear. Pipeline or stream run → field-profile suggestions + _concept_annotations. Steward accepts → registry grows → calibrated suggestions improve → coverage and health visibly rise. Nothing auto-lands: the suggestion queue stays the sole write path into the concept registry.

Sixteen VC items across six phases, one-PR-one-story, hard cuts in the T-series style.

Why¶

Symptom → root cause → fix:

#	Symptom	Root cause	VC
1	"KB" means two unrelated things	Vocabulary (governed concepts) vs RAG document-KB (`semantic/knowledge_base/`: chunks, embeddings, `KBVersion.document_count`) coexist in-tree; the latter is pyintake territory per CR-2	VC-4
2	CLI violates its own dictionary	`pycharter kb` manages the Vocabulary; "KB" is a forbidden word for that meaning	VC-2
3	Three concept shapes drift	SKOS dataclass `Concept`, graph Pydantic `ConceptNode`, SQLAlchemy `ConceptModel` — mismatched fields (`label`/`name`, `broader`/`parent_concept_key`), asymmetric coverage	VC-5
4	`semantic/pipeline/` vs `semantic/pipelines/`	Not duplicates: `pipelines/` is the canonical pipeline artifact format — Pillar-1 code misfiled under Pillar 3	VC-3
5	Concepts are second-class in their own store	`SemanticStoreClient`'s versioned-artifact surface (`save/get/list`) operates on `kind="linkml_schema"`; concepts hang off side mixins	VC-7
6	~37 semantic/editor routers, overlapping prefixes	Accretion; three modules mount under `.../semantic/concepts`; `editor_*` BFF routers duplicate resource routers	VC-8
7	Three UI surfaces	`app/vocabulary/` (newer), `app/editor/` (intended future), `app/knowledge-base/` (deprecated, still shipped)	VC-1, VC-9
8	Tiers/types have two sources of truth	Hardcoded `ConceptTier` / `ConceptType` enums duplicate `data/seed/vocabulary_metaschema.yaml`	VC-6
9	Search is unranked ILIKE/token match	In-memory index; the `SearchService` seam already promises a Postgres FTS swap	VC-10
10	The flywheel has gaps	Pipeline-side auto-extraction exists and defaults on; contract publish doesn't extract; accept/reject outcomes never feed back into confidence calibration	VC-11..13
11	Vocabulary is a silo to external catalogs	SKOS export exists but isn't a stated guarantee; ODCS export hides bindings in `customProperties`; OpenLineage runs carry no concept facets	VC-14..16

Corrections to earlier internal analyses, so this proposal starts from verified ground:

The four "dead dirs" (semantic/{ontology,ontology_editor,concept_graph,knowledge}/) are already deleted from git (T6/T7); what remains is untracked __pycache__ litter. No PR needed.
Document-KB versioning is not in-memory-only anymore (semantic/knowledge_base/factory.py has a sqlalchemy backend) — but what it versions is the document KB, which is leaving anyway (VC-4), so the "durable Phase-1 versioning" question dissolves.
ExtractionPolicy.trigger already defaults to "automatic" (semantic/extraction_policy.py); the pipeline side of the flywheel is wired via semantic/extraction/auto_trigger.py. VC-11..13 close the remaining gaps rather than build the loop from scratch.

Scope¶

ID	Title
VC-1	Delete the deprecated standalone `app/knowledge-base/` UI
VC-2	CLI `pycharter kb` → `pycharter vocabulary` (+ tombstone)
VC-3	Relocate `semantic/pipelines/` → `pipeline_artifact/` (Pillar 1)
VC-4	Cut the document-KB module (pyintake territory)
VC-5	One canonical `Concept` / `Relationship` model
VC-6	Metaschema-driven tiers and types
VC-7	Concepts first-class in `semantic_store` + `pycharter.Vocabulary` facade
VC-8	Router consolidation: ~37 → ~10 under `/api/v1/vocabulary/*`
VC-9	One UI surface: `editor/*` wins + Vocabulary Home panel
VC-10	Ranked search with evidence (Postgres FTS)
VC-11	Contract-publish extraction hook
VC-12	Calibration feedback loop from suggestion decisions
VC-13	Streaming extraction parity verification
VC-14	SKOS round-trip as a stated guarantee
VC-15	ODCS export carries `authoritativeDefinitions` concept links
VC-16	OpenLineage vocabulary facet

Decisions¶

D1 — Mental model and naming (finish the T-series)¶

VC-2 — CLI rename. pycharter kb → pycharter vocabulary. Files: _cli_commands_kb.py → _cli_commands_vocabulary.py, _cli_parsers_kb.py → _cli_parsers_vocabulary.py. While touching it, the contract sketch / diff parsers that currently live in _cli_parsers_kb.py split out into _cli_parsers_contract.py — they were never vocabulary code. semantic/kb.py (module named after a forbidden word) is renamed to semantic/ingest.py.

Deprecation story: hard break plus a one-release tombstone — a kb stub that prints "renamed to pycharter vocabulary" and exits 2. terminology.md forbids working aliases; a non-functional pointer for a human at a terminal is not an alias. The stub is deleted the following release.

VC-3 — semantic/pipelines/ → src/pycharter/pipeline_artifact/. It holds CanonicalPipelineConfig and friends — the pipeline artifact format for the editor, i.e. Pillar-1 material misfiled under the Pillar-3 directory. Relocation, not merge; semantic/pipeline/ (the semantic golden path: validate → populate → infer → query → export) keeps its name and becomes unambiguous once the plural sibling is gone.

VC-4 — Cut the document-KB module; don't extract it here. Delete semantic/knowledge_base/, api/routes/v1/editor_knowledge_base.py, db/models/semantic/{kb_version,kb_link,kb_lineage}.py (+ an Alembic migration dropping kb_versions / kb_links / kb_lineage), UI app/knowledge-base/ (VC-1), app/editor/kb/, and ui/src/lib/api/knowledgeBase.ts.

Rationale: KBVersion carries document_count / chunk_count / embedding_model — it versions a document knowledge base, which is pyintake territory per terminology.md and CR-2. PyCharter's flywheel never touches it, and keeping it in-tree preserves the single worst ambiguity ("KB" meaning two things). The pyintake extraction itself remains CR-2's job — the deleted code is referenced from the CR-2 note at its last in-tree revision, not kept alive here. Vocabulary versioning is unaffected: it continues via semantic/versioning/ and the collab branch/proposal machinery, which stay.

⚠ Data loss decision: the Alembic migration drops three tables. Accepted on the grounds that the data is pyintake-bound and pycharter has no reader for it after this change; the migration docstring names the export escape hatch (dump the tables before upgrading).

VC-5 — One canonical concept model. Survivor: the graph Pydantic model (semantic/concepts/graph/models.py::ConceptNode) — the only shape carrying tier, slots, origin, and provenance, which governance and the flywheel require. Migration order:

Promote it to semantic/concepts/models.py as Concept (and ConceptEdge → Relationship), absorbing the SKOS dataclass's useful extras (alt_labels, examples, exact_match) as optional fields.
Rewrite the SKOS importer/parser to emit the canonical model; the old SKOS dataclass is demoted to an rdf/-internal codec DTO (SkosConcept) or deleted outright if the codec can consume the canonical model directly.
SQLAlchemy ConceptModel (db/models/semantic/concept.py) stays the only persistence shape; row mappers align field names (name, parent_concept_key) with the canonical model.

No aliases, no re-exports of the old shapes.

VC-6 — Metaschema-driven tiers and types. Replace the hardcoded ConceptTier / ConceptType enums with a loader over data/seed/vocabulary_metaschema.yaml; validators consume the loaded sets. This makes tier/type vocabulary configuration — consistent with "domain artifacts are configuration" — and ends the two-sources-of-truth drift. (Seed-file edits go through the config-change review gate.)

VC-7 — Concepts first-class in their store + one Python entry point. Promote scheme/concept to a peer versioned-artifact surface next to kind="linkml_schema" in semantic_store/client.py, so the primary CRUD verbs (save / get / list / set_status) work on the things users think the store is for. On top of it, expose pycharter.Vocabulary — one facade with schemes, concepts, suggestions, bindings, health, and context services (the kb-ontology-workflow "SemanticWorkbench" idea, renamed and trimmed), aligned with python-api-consolidation.md.

D2 — The contribution flywheel¶

VC-11 — Contract-publish extraction hook (new). Run the existing contract adapter (semantic/extraction/adapters/_contract.py) when a contract is published — contract_store save and POST /contracts — gated by the contract's ExtractionPolicy (which already defaults to automatic). Idempotent per contract version; suggestions carry origin=contract plus provenance pointing at the contract version. Today the adapter only runs when explicitly invoked; publishing a contract should seed the vocabulary without a second command.

VC-12 — Calibration feedback loop (new). Suggestion decisions are already recorded (SuggestionModel status transitions + semantic/review_events/). Add fit_from_decisions() to semantic/ai/calibrator.py: learn per-extractor / per-evidence-type reliability weights from acceptance rates; calibrated priors feed the displayed confidence and the auto-sampling thresholds. This is what makes the flywheel compound — accepted suggestions make future suggestions better, not just more numerous.

VC-13 — Streaming parity (verification). The batch pipeline attaches the extraction sink via semantic/extraction/auto_trigger.py. Verify the same sink attaches in the streaming runtime (_cli_streaming_run.py) and that the _concept_annotations sidecar flows on validated records; fix if not. This is an audit-plus-patch PR, not new machinery.

Explicitly not changed: extraction defaults (already automatic); the suggestion queue as the sole registry write path; vocabulary versioning (stays on semantic/versioning/ + collab branches — no new versioning system).

D3 — Usability: one journey, one surface, ten routers¶

VC-1 / VC-9 — editor/* wins. Delete app/knowledge-base/ immediately (VC-1, deprecated and tiny). Migrate the app/vocabulary/* pages (suggestions, search, discovery, history, lineage, proposals, configuration) into editor panels/tabs, then delete them; rename app/editor/kb/ → app/editor/vocabulary/. Add a Vocabulary Home / next-best-action panel on the editor dashboard: pending suggestions, unbound contract fields, coverage/health tiles, and an "import a contract" CTA — the §A design from the superseded kb-ontology-workflow proposal, reused as-is.

The journey, identical across UI, CLI, and Python:

register a contract → suggestion inbox (split-pane with evidence) → accept/reject → coverage and health tick up on the home panel.

(pycharter vocabulary import|suggestions|health|search and pycharter.Vocabulary expose the same verbs.)

VC-8 — Router consolidation, ~37 → ~10 under /api/v1/vocabulary/*:

Router	Absorbs
`schemes`	scheme CRUD
`concepts`	concept CRUD + graph + definitions
`relationships`	edge CRUD
`bindings`	resource + field bindings + binding actions
`suggestions`	suggestion queue + inbound suggestions
`extraction`	ingest runs + extraction endpoints
`search`	search + discovery
`health`	health + dashboard + activity + completeness
`governance`	proposals + reviews
`insights`	rdf + export + query + reasoning + context + provenance + lineage

editor_* BFF endpoints fold into their resource routers.

⚠ This reverses T10's recorded decision that HTTP paths stay at /api/v1/semantic/* ("HTTP-level back-compat is expensive to break"). Why reverse it now: T10 predates this consolidation — moving 37 routers one-to-one would indeed have been pointless churn, but collapsing them to 10 is a shape change, and doing it without also fixing the path prefix bakes a forbidden-ish word into the API for another cycle. All known consumers are in-repo (UI, src/pycharter/mcp/ — audit it for hardcoded routes) and land atomically with each router move; the gate is an OpenAPI snapshot diff reviewed per PR; the release notes carry a minor-version bump. If the maintainers veto this, VC-8 proceeds with consolidation under the old prefix and only this sub-decision changes.

VC-10 — Ranked search with evidence. Implement the Postgres FTS backend (tsvector + pg_trgm) behind the existing SearchService seam — semantic/search/indexer.py's docstring already promises exactly this swap. Responses carry evidence: matched fields, binding count, provenance origin, scheme. SQLite/in-memory keep the token index.

D4 — Standards alignment¶

VC-14 — SKOS round-trip as a guarantee. Promote the existing semantic/rdf/ machinery to a stated conformance: a mapping table (Concept ↔ skos:Concept, scheme ↔ skos:ConceptScheme, is_a ↔ skos:broader, same_as ↔ skos:exactMatch, alt_labels ↔ skos:altLabel, …) in docs/reference/, backed by a round-trip test suite (export → import → semantic equality) that becomes the gate for any model change touching these fields.

VC-15 — ODCS authoritativeDefinitions. The exporter (contract_io/odcs.py) currently stuffs field bindings into customProperties only. Add per-schema-property authoritativeDefinitions entries (type businessDefinition, URL = the concept URI from semantic/namespaces.py); the importer (contract_io/from_odcs.py) maps them back to field bindings. customProperties stays for lossless round-trip of pycharter-specific detail. Result: a pycharter contract exported to any ODCS-compliant catalog carries its concept links in the standard slot.

VC-16 — OpenLineage vocabulary facet. Extend build_run_facets in pipeline_generator/lineage.py with a pycharterVocabulary dataset facet: field → concept_key + scheme version for every bound field, aligned with the existing emit_facet flag on data-quality rules (db/schemas/_quality_models.py). Lineage consumers (Marquez etc.) then see meaning, not just dataset names.

Execution¶

Six phases, one-PR-one-story, each independently green.

Phase	Items	Content	Verification
0	VC-1	Land this proposal + BACKLOG section; delete `app/knowledge-base/`	UI build; grep gate: no route/link to `/knowledge-base`
1 — naming	VC-2, VC-3, VC-4	CLI rename + tombstone; `pipeline_artifact/` relocation; document-KB cut + Alembic drop	pytest; CLI smoke (`vocabulary --help`; `kb` exits 2); `alembic upgrade`/`downgrade` both directions; terminology grep checklist
2 — model	VC-5, VC-6, VC-7	Canonical `Concept`/`Relationship`; metaschema-driven tiers/types; concepts first-class in store + `pycharter.Vocabulary`	Round-trip unit tests (yaml → model → db → model); config-change gate on the metaschema; public-API snapshot
3 — API/UI	VC-8, VC-9, VC-10	Router consolidation; vocabulary→editor page migration + Vocabulary Home; ranked FTS search	OpenAPI snapshot diff reviewed per PR; Playwright journey (import → accept → health); search relevance fixtures
4 — flywheel	VC-11, VC-12, VC-13	Contract-publish hook; calibrator feedback; streaming parity	e2e: register contract → suggestions appear with evidence; accept N / reject M → calibrated score shifts; streaming run produces suggestions
5 — standards	VC-14, VC-15, VC-16	SKOS conformance suite; ODCS `authoritativeDefinitions`; OpenLineage facet	Round-trip equality tests; ODCS schema validation; OpenLineage payload snapshot

Ordering rationale: naming/deletion first shrinks the surface everything else touches; the model collapse lands before router consolidation so the routers serialize the canonical model exactly once; the flywheel builds on the stabilized store/model; Phase 5 depends only on Phase 2 and can run in parallel with 3–4.

Blast radius¶

CLI users: pycharter kb * hard-breaks (tombstone for one release, then deleted).
API consumers: /api/v1/semantic/* → /api/v1/vocabulary/* is a hard path break (reverses T10 — see VC-8; minor-version bump + CHANGELOG; in-repo UI and src/pycharter/mcp/ updated atomically).
UI: app/knowledge-base/ deleted; app/vocabulary/* migrated into the editor then deleted; app/editor/kb/ renamed.
Stored data: kb_versions / kb_links / kb_lineage dropped (VC-4, data loss accepted; export escape hatch documented in the migration). Concept tables unchanged in Phase 2 — the model collapse is code-side.
Python API: pycharter.semantic.kb and the SKOS Concept dataclass import paths break; pycharter.Vocabulary is the new front door.

Out of scope / deferred¶

The pyintake extraction of the document-KB itself (CR-2 owns it; this proposal only removes the code from pycharter).
OWL reasoning expansion beyond the existing OWL-RL backend.
Embedding/vector search — VC-10 is FTS only.
A "workspace" model beyond concept schemes.
Contract YAML format changes — that is config-template-harmonization.md territory; the two proposals are independent and touch disjoint files except the contract-publish hook (VC-11), which is envelope-agnostic.
DG-1 ResourceBinding state-machine changes.
CF-5 (single contract-loader entry point) — still deferred as recorded in CF.

Docs superseded¶

The following get a "Superseded by docs/proposals/vocabulary-consolidation.md" banner; their content remains for historical context:

docs/guides/kb-ontology-workflow-proposal.md (+ its kanban, kb-ontology-workflow-kanban.md)
docs/guides/vocabulary-governance-rfc.md
docs/guides/vocabulary-implementation-plan.md

The kb-phase-* review docs stay historical as-is. terminology.md remains canonical; this proposal extends it into the two areas it deliberately stopped at (the CLI subcommand name and the HTTP route prefix).