Skip to content

Ontology Implementation Plan

Objective

Implement ontology governance as a practical layer in PyCharter that:

  1. normalizes and validates ontology bindings in contracts,
  2. enforces proposal/review governance with merge gates, and
  3. operationalizes ontology for validation, routing, and impact analysis.

This plan is derived from:

  • docs/guides/ontology-governance-rfc.md

Scope Summary

In scope:

  1. Canonical contract ontology schema (field_bindings) with backward compatibility.
  2. Semantic health APIs and merge gate validation APIs.
  3. Policy-aware approvals by tier and change type.
  4. UI support in ontology proposal flow for ontology change governance.
  5. Initial ontology-driven contract checks (coverage, references, derived_from).

Out of scope for this implementation cycle:

  1. Advanced graph reasoning and automated ontology inference.
  2. Full ontology recommender/copilot.
  3. Hard global enforcement on all legacy contracts from day one.

Workstreams

WS1: Contract Ontology Canonicalization

Goal: make ontology contract payload strict enough for reliable processing.

Tasks:

  1. Define canonical schema:
  2. ontology.version
  3. ontology.field_bindings.{field_path}.concept_id
  4. optional confidence, relationships, lineage, tags
  5. Implement strict Pydantic models in db/schemas/data_contract.py.
  6. Add parser normalization:
  7. legacy ontology.fields -> canonical ontology.field_bindings
  8. Add structured validation errors and migration warnings.
  9. Add tests:
  10. parse legacy and canonical
  11. invalid relationship type rejection
  12. missing concept_id rejection

Primary files:

  • src/pycharter/db/schemas/data_contract.py
  • src/pycharter/contract_parser/parser.py
  • tests/test_contract_parser.py
  • tests/wiki/test_knowledge_schema.py (ontology enums: ConceptType, RelationshipType)
  • tests/wiki/test_schema_loader.py (loads ontology_schema.yaml)

Acceptance criteria:

  1. Existing contracts still parse via normalization path.
  2. New canonical schema is validated deterministically.
  3. Validation error messages identify field path and fix hint.

WS2: Semantic Health Service

Goal: expose measurable ontology quality for each contract.

Tasks:

  1. Add semantic health service:
  2. mapping coverage (% mapped fields)
  3. invalid concept references
  4. deprecated concept usage
  5. invalid relationship types
  6. Add API endpoint:
  7. GET /api/v1/semantic/contracts/{contract_id}/semantic-health
  8. Add tests for service + route.

Primary files:

  • src/pycharter/wiki/knowledge/* (new helper/service module)
  • src/pycharter/wiki/api/routes/v1/*
  • src/pycharter/wiki/api/models/*

Acceptance criteria:

  1. Endpoint returns deterministic health payload.
  2. Health score works for both legacy and canonical ontology payloads.

WS3: Proposal Governance + Merge Gates

Goal: make ontology changes policy-driven and safely mergeable.

Tasks:

  1. Extend proposal model:
  2. proposal_type
  3. required_approvals
  4. gate_status JSON
  5. Add policy evaluator:
  6. derive required approvals by tier + change type
  7. Add proposal validation endpoint:
  8. POST /api/v1/semantic/proposals/{id}/validate
  9. Implement gates:
  10. concept existence
  11. relationship-type validity
  12. semantic coverage delta
  13. optional SHACL gate (feature flag)
  14. Block merge unless gates pass (for governed tiers).

Primary files:

  • src/pycharter/db/models/semantic/ (proposal-related models)
  • src/pycharter/wiki/collaboration/proposals.py
  • src/pycharter/wiki/collaboration/review.py
  • src/pycharter/wiki/collaboration/permissions.py
  • src/pycharter/wiki/api/routes/v1/proposals.py

Acceptance criteria:

  1. core changes require configured approvers and passing gates.
  2. Gate outputs are persisted and queryable.
  3. Failed gates provide actionable diagnostics.

WS4: Ontology UI Enhancements for Ontology Governance

Goal: make governance decisions visible and auditable in the UI.

Tasks:

  1. Proposal form adds proposal_type and impacted contract summary.
  2. Proposal detail page shows:
  3. required approvers
  4. gate status (pass/fail/pending)
  5. impact summary (contracts/fields/concepts touched)
  6. Review panel shows policy progress to merge readiness.

Primary files:

  • src/pycharter/ui/src/app/ontology/proposals/page.tsx
  • src/pycharter/ui/src/app/ontology/proposals/detail/page.tsx
  • src/pycharter/ui/src/components/ontology/CreateProposalModal.tsx
  • src/pycharter/ui/src/components/ontology/ReviewList.tsx

Acceptance criteria:

  1. User can see exactly what is blocking merge.
  2. Policy and gate outcomes are aligned with backend source of truth.

WS5: Ontology-Driven Contract Checks

Goal: turn ontology into useful validation behavior.

Tasks:

  1. Add optional semantic coverage threshold checks.
  2. Implement references integrity checks (quality checks, not DB FKs).
  3. Implement derived_from lineage consistency checks.
  4. Route incidents to concept + contract owners.

Primary files:

  • src/pycharter/validator/*
  • src/pycharter/api/services/contract_service.py
  • metadata ownership/governance integration points

Acceptance criteria:

  1. Contract validation report includes ontology-derived checks.
  2. Failures identify impacted owners/stakeholders.

Sequencing and Milestones

Milestone 1 (2 weeks)

  1. WS1 complete.
  2. WS2 partial (service + API v1 without full scoring weights).

Deliverable:

  • canonical schema + health endpoint + parser compatibility.

Milestone 2 (2 weeks)

  1. WS3 backend complete.
  2. WS4 partial UI integration for gate and approvals display.

Deliverable:

  • governed proposal workflow with merge block behavior.

Milestone 3 (2 weeks)

  1. WS4 complete.
  2. WS5 baseline checks complete.

Deliverable:

  • ontology-driven validation and end-to-end governance UX.

Testing Strategy

  1. Unit:
  2. schema normalizer
  3. policy evaluator
  4. gate engine
  5. Integration:
  6. proposal creation -> review -> validate -> merge
  7. contract parse -> semantic health -> validation report
  8. Regression:
  9. legacy contract parsing
  10. existing ontology flows without ontology changes

Migration and Backward Compatibility

  1. Accept both legacy ontology.fields and canonical field_bindings.
  2. Normalize at parse time; keep write-path to canonical for new saves.
  3. Add warnings in API responses for legacy payload usage.
  4. Provide migration command (future task):
  5. dry-run summary + optional rewrite output.

Delivery Backlog (Ticket-Ready)

Backend

  1. BE-ONT-001: Add canonical ontology binding models to DataContractSchema.
  2. BE-ONT-002: Implement parser normalizer fields -> field_bindings.
  3. BE-ONT-003: Build semantic health calculator service.
  4. BE-ONT-004: Add semantic health API route + response model.
  5. BE-ONT-005: Extend proposal DB model for governance metadata.
  6. BE-ONT-006: Implement policy evaluator by tier/change type.
  7. BE-ONT-007: Implement proposal gate runner endpoint.
  8. BE-ONT-008: Merge guard enforcement with gate + approval checks.
  9. BE-ONT-009: Add ontology-derived validation checks (references, derived_from).

Frontend

  1. FE-ONT-001: Add proposal type selector in create proposal modal.
  2. FE-ONT-002: Add gate status panel in proposal detail.
  3. FE-ONT-003: Add approval requirement/progress display.
  4. FE-ONT-004: Add impact summary block for changed concepts/fields/contracts.

QA/DevEx

  1. QA-ONT-001: Contract fixture suite for legacy + canonical ontology payloads.
  2. QA-ONT-002: End-to-end test for proposal governance lifecycle.
  3. DX-ONT-001: Author migration guide section + API examples.

Risks and Controls

  1. Risk: concept ID inconsistency across environments.
  2. Control: explicit concept registry validation + seed synchronization check.
  3. Risk: governance friction for domain teams.
  4. Control: tiered policy with configurable thresholds.
  5. Risk: UI/backend policy mismatch.
  6. Control: backend-provided policy status model; UI only renders server state.

Decision Log (To Fill)

  1. Canonical field path format.
  2. Decision: JSONPath in v1.
  3. Rationale: user preference and better support for nested/array structures.
  4. Field-to-concept cardinality.
  5. Decision: many-to-many in v1.
  6. Concept identifier strategy.
  7. Decision: use an immutable concept_key (separate from display name).
  8. Coverage denominator.
  9. Decision: leaf fields only.
  10. Coverage enforcement behavior.
  11. Decision: warn-only in v1 (no hard fail).
  12. Approval policy for core in v1.
  13. Decision: any authenticated user can approve for now.
  14. Merge blocking scope by tier.
  15. Decision: block merges for core only in v1; domain/free are warn-only with audit.
  16. Legacy payload handling (ontology.fields).
  17. Decision: always normalize in memory; persist canonical field_bindings on create/update saves.
  18. Note: no forced bulk rewrite migration for untouched historical contracts.
  19. SHACL gate default (off/on) for first production rollout.
  20. Decision: feature-flagged OFF by default in v1, report-only when enabled.

WS3 API Examples

Create proposal with governance metadata:

POST /api/v1/semantic/proposals/
{
  "contract_id": "9fae5b8a-2a50-4f77-86fc-7f96d130c001",
  "title": "Map customer fields to core identity concepts",
  "author": "alice",
  "proposal_type": "mapping_change",
  "affected_tiers": ["core"]
}

Run proposal validation gates:

POST /api/v1/semantic/proposals/{proposal_id}/validate
{
  "proposed_ontology": {
    "version": "1.1.0",
    "field_bindings": {
      "$.customer_id": [
        {"concept_id": "CustomerIdentifier"}
      ]
    }
  }
}

Semantic health by contract name/version:

GET /api/v1/semantic/contracts/by-name/{contract_name}/{contract_version}/semantic-health

Implementation Start Recommendation

Start with WS1 + WS2 in a single branch to establish canonical data shape and observability first. This minimizes downstream churn before governance and UI work.