Skip to content

Ontology Governance RFC for Data Contracts

Status

  • Draft
  • Author: PyCharter team
  • Target release: phased rollout over 2-3 releases

Context

PyCharter already has the core building blocks:

  • Ontology relationship and concept-type control via ontology_schema.yaml
  • Contract-level ontology section support
  • Ontology collaboration primitives (branches, proposals, reviews)
  • RDF / JSON-LD / SHACL capabilities

What is missing is a clear operating model that connects ontology governance to contract lifecycle and validation behavior.

Problem Statement

Today, ontology is available but not yet fully operationalized as a governed semantic layer for contracts.

Main gaps:

  1. Ontology payload in contracts is structurally permissive (dict[str, Any]) and can drift.
  2. Approval workflow is generic and not policy-aware by concept tier/change type.
  3. Contract validations do not consistently consume ontology relationships as executable checks.
  4. No explicit contract semantic quality gates (coverage, mapping health, impact checks).

Goals

  1. Make ontology first-class and governed, without over-constraining early adoption.
  2. Keep contracts focused on field-to-concept binding, not full concept redefinition.
  3. Convert ontology into practical value:
  4. Better data validation
  5. Better ownership/stakeholder routing
  6. Better impact analysis and change safety
  7. Use existing ontology proposal/review UI and APIs as the governance control point.

Non-goals

  1. Full OWL reasoner or advanced inference engine in this phase.
  2. Enforcing 100% semantic coverage from day one.
  3. Replacing JSON Schema validations with ontology validations.

Industry-Informed Pattern

Adopt a split model seen in ontology-centric data platforms:

  1. Central Concept Registry (global truth):
  2. Concept definitions, tiers, node kinds, lifecycle
  3. Allowed predicate catalog
  4. Contract Semantic Bindings (local truth):
  5. Field -> concept binding
  6. Local relationship hints and lineage references
  7. Governed change workflow:
  8. Proposals + approvals + merge gates
  9. Runtime policy enforcement:
  10. Validation checks and routing derived from ontology

This follows the same core direction as ontology-first enterprise platforms (for example, Palantir-style object/link governance), while remaining lightweight in PyCharter’s current architecture.

Proposed Conceptual Model

1) Ontology Registry

System of record for:

  • concept_id, label, definition, owner, tier, concept_type, status
  • Relationships between concepts (is_a, depends_on, governs, etc.)
  • Allowed relationship types from ontology_schema.yaml

2) Contract Semantic Bindings

Each data contract contains only semantic mappings for its fields:

ontology:
  version: "1.1.0"
  field_bindings:
    customer_id:
      concept_id: CustomerIdentifier
      confidence: high
      relationships:
        - type: references
          target_field: customers.id
    total_amount:
      concept_id: MonetaryAmount
      relationships:
        - type: derived_from
          target_field: line_items.extended_amount
      lineage:
        source_system: billing_service

Notes:

  • concept_id must resolve in the registry.
  • relationships.type must be in allowed predicate set.
  • Contracts should not redefine concept definitions unless in an explicit exception path.

3) Governance Policy

Governance depends on concept tier and change type:

  1. core concepts:
  2. Require central steward + domain owner approvals.
  3. domain concepts:
  4. Require domain owner approval.
  5. free concepts:
  6. Single approval or auto-merge with audit trail (configurable).

Change types:

  • Add concept
  • Update concept definition/tier/node_kind
  • Deprecate concept
  • Add/remove concept relationship
  • Remap contract field to a different concept

Integration With Current PyCharter Components

Contract Parser and Schema

Current:

  • ontology accepted but mostly opaque.

Proposed:

  1. Add strict Pydantic models for ontology.field_bindings.
  2. Enforce:
  3. concept_id required
  4. Known relationship types only
  5. Optional confidence enum (low, medium, high)
  6. Preserve backward compatibility:
  7. Continue reading existing fields shape
  8. Normalize to canonical field_bindings internally

Target files:

  • src/pycharter/db/schemas/data_contract.py
  • src/pycharter/contract_parser/parser.py
  • src/pycharter/wiki/ontology/validator.py

Ontology collaboration layer

Current:

  • Proposals/reviews exist, but policy logic is thin.

Proposed:

  1. Add policy evaluator:
  2. Determine required approvers based on impacted concept tier + change type.
  3. Add merge gates:
  4. Ontology schema validation
  5. Relationship catalog validation
  6. SHACL validation (when RDF artifacts are available)
  7. Impact report generation
  8. Persist governance metadata:
  9. Required approvals
  10. Approval evidence
  11. Gate run results

Target files:

  • src/pycharter/wiki/collaboration/proposals.py
  • src/pycharter/wiki/collaboration/review.py
  • src/pycharter/wiki/collaboration/permissions.py
  • src/pycharter/db/models/semantic/ (proposal-related models; extend as needed)

Ontology UI proposal flow

Use existing /ontology/proposals flow and add ontology-specific UX:

  1. Proposal type selector (concept_change, mapping_change, relationship_change).
  2. Auto-generated impact panel:
  3. affected contracts
  4. affected fields
  5. owners to notify
  6. Merge gate status panel:
  7. pass/fail per gate
  8. Approval requirement panel:
  9. who must approve and why

Target files:

  • src/pycharter/ui/src/app/ontology/proposals/page.tsx
  • src/pycharter/ui/src/components/ontology/CreateProposalModal.tsx
  • src/pycharter/ui/src/components/ontology/ReviewList.tsx

Runtime Use in Data Contracts

Ontology should power concrete checks and operations:

  1. Semantic coverage check
  2. % required fields mapped to concept_id
  3. Fail/warn thresholds by contract tier.
  4. Referential integrity quality checks
  5. references relationships become foreign-key quality checks.
  6. Derivation lineage checks
  7. derived_from relationships require valid upstream mapping evidence.
  8. Policy checks
  9. e.g. pii-tagged concepts require masking/governance metadata.
  10. Stakeholder routing
  11. Validation incidents route to concept owners + contract owners.

API Changes (Draft)

Add or extend endpoints under the semantic API:

  1. POST /api/v1/semantic/proposals:
  2. include proposal_type, change_set, impact_preview
  3. POST /api/v1/semantic/proposals/{id}/validate:
  4. runs merge gates; returns structured results
  5. GET /api/v1/semantic/contracts/{contract_id}/semantic-health:
  6. coverage, invalid mappings, deprecated concept usage
  7. GET /api/v1/semantic/concepts/{concept_id}/impact:
  8. contract and field usage graph

Data Model Changes (Draft)

Proposed additive changes:

  1. proposals table:
  2. proposal_type
  3. required_approvals
  4. gate_status (JSON)
  5. reviews table:
  6. optional review_scope (concept/mapping/global)
  7. optional mapping audit table:
  8. contract_id, field_path, concept_id, status, changed_by, timestamps

Keep existing concepts, concept_relationships, field_concepts models as base.

Rollout Plan

Phase 0 (Foundation)

  1. Formalize canonical ontology-in-contract schema (field_bindings).
  2. Add normalizer for legacy fields shape.
  3. Add semantic health report endpoint (read-only).

Exit criteria:

  • Existing contracts continue to parse.
  • Semantic health can be computed for all active contracts.

Phase 1 (Governed Proposals)

  1. Add proposal types and policy-aware approval requirements.
  2. Add merge validation endpoint and UI gate display.
  3. Require successful gates before merge for core concepts.

Exit criteria:

  • Core ontology changes blocked unless policy + gates pass.

Phase 2 (Contract Enforcement)

  1. Add semantic coverage policy checks to contract validation.
  2. Add references and derived_from quality checks.
  3. Add incident routing to concept + contract owners.

Exit criteria:

  • Ontology adds measurable validation and governance value in production workflows.

Phase 3 (Optimization)

  1. Add impact simulation before concept deprecations.
  2. Add recommendation tooling for unmapped fields (assisted mapping).
  3. Add analytics dashboard (coverage, change lead time, validation defect trends).

Success Metrics

  1. Semantic coverage rate by contract/domain.
  2. % contracts using deprecated concepts.
  3. Proposal lead time (open -> merged).
  4. Rejected/rolled-back ontology changes.
  5. Validation defects detected via ontology-derived rules.
  6. Mean time to route and resolve data incidents.

Risks and Mitigations

  1. Over-governance slows teams.
  2. Mitigation: tiered policy; lighter defaults for free/domain.
  3. Taxonomy sprawl.
  4. Mitigation: concept de-dup checks and required broader/owner metadata.
  5. Mapping quality inconsistency.
  6. Mitigation: confidence levels + review requirements + health score visibility.
  7. Backward compatibility issues.
  8. Mitigation: normalization layer and phased enforcement.

Open Questions

  1. Should field -> concept be one-to-one or many-to-many by default?
  2. Should same_as/equivalent_to be auto-collapsed in lineage/discovery views?
  3. Should domain admins be able to override core concept mappings locally?
  4. What minimum semantic coverage is required per contract maturity tier?

Build the smallest vertical slice first:

  1. Canonical field_bindings schema + parser normalizer.
  2. Semantic health endpoint.
  3. Proposal validation endpoint with:
  4. relationship-type validation
  5. concept existence check
  6. coverage delta report

This delivers immediate value with low migration risk and sets up stricter governance later.