Skip to content

End-to-end user guide

This guide walks a new contributor through the PyCharter knowledge-base (KB) and ontology workflow from a cold start: create a workspace, import a sample domain, review the generated suggestions, bind contract fields to concepts, measure semantic health, and serve governed context to a downstream consumer.

The walkthrough is anchored on the revenue/billing sample shipped by ticket P6-005. Every command below was verified against the develop branch on 2026-05-19 (Phase 0..6 wave 2 merged). When the UI and the CLI both expose a step, the guide names both paths.

Audience:

  • New contributors making their first KB/ontology change.
  • Domain stewards reviewing automatically generated suggestions.
  • Engineers wiring an application or agent to a workspace's context.

Cross references:

Forward references (other Phase 6 docs in flight):

  • Admin / governance guide — covers role assignment, policy tiers, and audit export.
  • Vocabulary migration guide — upgrading an existing deployment to the unified editor and vocabulary routes.

Prerequisites

You need the following before starting:

  • Python 3.11+ — same floor as the package's pyproject.toml (requires-python = ">=3.11").
  • Node.js 24+ — required only if you plan to develop the web UI with hot reload. Skip if you only use the API or the bundled static UI.
  • A database — SQLite (zero setup, default) or PostgreSQL. The sample script defaults to a SQLite file at ./sample.db.

Clone and install:

git clone https://github.com/optophi/pycharter.git
cd pycharter
pip install -e ".[dev]"

Optional: if you only want to run the sample, the semantic-export extra is sufficient instead of the full dev set:

pip install -e ".[dev]"

For wider context, see the Configuration guide (database connection options), the Building the Package guide (UI static-file build), and the Production deployment guide (operator-facing deployment).

Step 1: Create or select a workspace

In PyCharter, a workspace is the user-facing name for a backend concept scheme. One workspace owns one bounded domain — its concepts, mappings, suggestions, source assets, and ingest runs. The program decision (locked in kb-ontology-workflow-kanban.md) is that no new persisted workspace model exists in Phase 1: the UI label "Workspace" is rendered over ConceptSchemeModel. Both names remain greppable in their respective surfaces.

Three ways to create one:

The sample script in Step 2 creates the revenue-billing workspace automatically if it does not already exist. New contributors should let the sample do this for them and revisit workspace creation later.

1b. Via the UI

  1. Open the unified editor at http://localhost:8002/editor.
  2. Use the workspace selector in the editor top bar to either pick an existing workspace or create a new one. The selector reads from GET /api/v1/semantic/schemes.

1c. Via the REST API

curl -X POST http://localhost:8002/api/v1/semantic/schemes \
  -H "Authorization: Bearer ${PYCHARTER_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "scheme_key": "revenue-billing",
    "title": "Revenue / Billing",
    "version": "0.1.0"
  }'

The scheme_key is the stable workspace identifier the rest of the workflow uses. Pick a short, lowercase, dash-separated string.

Step 2: Seed the database

The fastest way to populate a workspace is pycharter db seed. It walks every example under src/pycharter/data/seed/examples/ and loads each one: reference data (owners, domains, systems, ...), contracts, the concept scheme, and the pipeline config.

Two examples ship today:

  • examples/ecommerce/ — full end-to-end example. Four contracts (customer, product, order, order_line_item), two reference docs, four DDL files, one pipeline (ecommerce_orders_ingest) that reads the bundled orders.csv and validates against the Order contract.
  • examples/trading_domain/ — minimal LinkML example that exercises the foreign-schema loader path.

2a. Run the seed

pycharter db seed
# or, equivalently:
make seed

By default this seeds the on-disk SQLite database configured by PYCHARTER_DATABASE_URL (or the package default). Override with the --database-url flag:

pycharter db seed --database-url 'postgresql+psycopg://user:pw@localhost/pycharter'

You can verify the run with the smoke test:

pytest tests/integration/test_seed_ecommerce.py -m integration -v

The seeder, per example:

  1. Loads reference data (owners / domains / systems / environments / data feeds / tags) declared in the manifest.
  2. Persists each contract YAML — DataContractModel plus its JsonSchemaModel, coercion / validation rules, metadata record, and field bindings.
  3. Ingests the example's concept_scheme.yaml into the concepts table.
  4. Loads the example's pipeline_config.yaml.
  5. (PR3) Runs the pipeline and writes a profile artifact so a fresh seed is immediately useful in the UI.
  6. Persists one SemanticAsset row per file (3 contracts + 3 docs + 3 DDL = 9 assets).
  7. Builds an ExtractionResult per asset, runs it through the production extraction_result_to_drafts adapter, and stamps the ingest-run id onto every draft.
  8. Persists the drafts via SuggestionManager.create_batch with require_evidence=True — the same audited path real extractors use.

2b. (Alternative) Import via the UI Import Wizard

The Import Wizard at /editor/import-wizard is the UI surface for the same flow. Phase 1 of the editor UX initiative reshaped how you reach the wizard, what it accepts, and how it reports progress; the six backend steps still exist but most users now jump straight to Review once the wizard's pre-filled defaults validate.

Launching the wizard

Two paths open the wizard. Both pre-fill the target domain workspace from the editor's active canvas and restore the last-used governance tier and extraction options from the previous session.

  1. Toolbar button. Click Import in the editor toolbar. The wizard opens with the current workspace pre-selected. If the pre-filled defaults already validate, the wizard jumps straight to the Review step.
  2. Canvas drop. Drag one or more files from your file manager onto the editor canvas. A full-canvas overlay highlights the drop target while files are over the editor; releasing the files opens the wizard with the files already attached.

Source files

The wizard's first step is a file dropzone, not a paste textarea.

  • Drag and drop. Drop one or more files onto the zone. Multiple files are supported in a single ingest run; the wizard groups them under one suggestion batch.
  • Click to browse. Activates the native file picker. The same multi-file rules apply.
  • Paste instead. A disclosure under the dropzone reveals a textarea for pasting raw text — useful when you only have a snippet and no file.

Source type is auto-detected from the file extension (.md, .pdf, .html, .rst, .csv, .sql, source code via Tree-sitter, and the other adapters the extraction pipeline already ships). The auto-detect result is shown next to each filename and can be overridden manually if the extension is wrong or ambiguous.

Preview before commit

The Review step shows a live preview of what the import will contribute, fetched from the new POST /api/v1/semantic/ingestion/kb-preview route. The preview runs the same extraction adapters as the real import but stops before persistence.

  • Candidate business terms. Each row shows the proposed term, the source file and span, and the governance tier badge (Free, Domain, or Core — see kb-phase-0-vocabulary.md §"Concept (governance) tier" for what each tier means).
  • Candidate relationships. Listed underneath the terms, with the same evidence and tier badges.
  • Toggle and re-run. Extraction options live in a side panel on the Review step itself; toggling them re-runs the preview in place, so you can compare results without backing out to an earlier step.

Submitting and tracking progress

Click Submit to commit. The wizard hands off to the existing POST /api/v1/semantic/kb/ingest-upload route. Small imports finish inline; larger imports flip to the background path (P6-003) and the wizard closes immediately.

For background runs, an Import progress pill lives in the editor toolbar next to the workspace selector. The pill shows the current file name and a progress percentage while a run is active. Click it to open a popover with per-file detail and a Cancel button; the popover dismisses on completion.

Recovery

The wizard persists its state to localStorage on every step change, so accidental navigation, refresh, or browser-tab close will not lose your attached files or chosen options. Reopening the wizard restores the previous session.

  • To start fresh without finishing the saved session, open the wizard with ?reset=1 (for example, /editor/import-wizard?reset=1). This discards the saved state.
  • A successful submit clears the saved state automatically; no manual cleanup is required.

If an extraction adapter fails for a specific file — for example, a corrupt PDF or unsupported encoding — the error surfaces inline on that file's row in the Source step, not as a wizard-blocking modal. The other files in the batch still go through.

What happens after submit

After submit the wizard returns the ingest-run id and a deep link into the suggestion review queue. Extracted business terms and relationships land there as candidates, exactly like CLI-driven imports.

Reviewers triage, accept, or dismiss them via the suggestions inbox (see Step 3 below). Accepted candidates become governed business terms in the workspace; dismissed candidates are archived with the reviewer's rationale.

Importing via the wizard requires the reviewer role or higher (the matrix in role-permissions-matrix.md lists the gated routes). Sign in via POST /api/v1/auth/login first if the API is running with auth enabled.

Step 3: Review the suggestions inbox

After import, the suggestions inbox at /vocabulary/suggestions lists every candidate concept, relationship, and mapping the extractor produced (P3-001). Each row carries the evidence the reviewer needs: source asset, source span, rationale, confidence score, proposed action.

The page supports:

  • Filter by tier, origin, confidence band, source asset, status, ingest-run id. See pagination-and-filtering.md for the query syntax.
  • Row actions — Accept, Dismiss, Request changes. Each materialises a ReviewDecision event on the row and (for Accept) creates a concept or relationship.
  • Batch actions — select multiple rows and accept, dismiss, or request-changes the batch. Batch ops are gated on the reviewer role and share the same accept/dismiss/request-changes semantics as the single-row actions.

For the sample, accept all six suggestions. They are deliberately low-confidence drafts so the inbox exercises the review flow; in a real domain a steward triages on confidence and tier first.

Status labels you see:

Inbox label Backend value Colour Meaning
Needs review pending amber-400 Awaiting a reviewer (default).
Accepted accepted emerald-600 Materialised into a concept or mapping.
Dismissed dismissed gray-300 Reviewer declined; not a failure.
Changes requested request_changes orange-400 Reviewer asked the author to revise.

Source: kb-phase-0-vocabulary.md §"Suggestion status".

The review action requires the reviewer role. See role-permissions-matrix.md for the route-by-route table.

Step 4: Bind concepts in the Mapping Workbench

Open the Mapping Workbench at /editor/contracts/{name}/{version}/mapping-workbench (P3-007). The workbench lists every contract field on the left and the candidate business term on the right. For the revenue/billing sample, open the invoice contract:

/editor/contracts/invoice/1.0.0/mapping-workbench

For each contract field you can:

  • Accept candidate concept — bind the field to the suggested business term. The binding moves from suggested to mapped.
  • Remap — pick a different concept than the suggested one. Useful when the auto-suggestion picked a near-miss.
  • Mark technical-only — flag a field that is plumbing rather than business-relevant (e.g. created_at, updated_at). The binding moves to technical_only and stops counting against coverage.
  • Annotate — attach a free-text note to the binding for the next reviewer. Annotations are stored on the binding-event log.

Binding actions all require the reviewer role. The full route list is in role-permissions-matrix.md (POST /api/v1/contracts/{contract_id}/fields/{field_path}/binding/...).

Binding lifecycle labels (from kb-phase-0-vocabulary.md):

Workbench label Backend value Colour Meaning
Unmapped unmapped gray-300 No binding exists.
Suggested suggested amber-400 Auto-picked binding awaiting human review.
Mapped mapped green-500 Reviewer accepted; awaiting governance approval.
Approved approved emerald-600 Governance-approved.
Technical only technical_only slate-400 Deliberately not modelled (e.g. plumbing columns).

Use the field_bindings blocks in each src/pycharter/data/seed/examples/ecommerce/contracts/<entity>.yaml as a guide. They name the field-to-concept pairs the seed loader materialises (and the smoke test asserts on).

Step 5: Measure semantic health

The Health tab in the editor companion panel (P4-001) lists the actionable findings the system has against your workspace. Open the editor, switch to the workspace, and click the Health tab at the bottom.

The tab surfaces four kinds of check:

  1. Duplicate concept — two concepts with the same name or alias. Action link opens the concept inspector for a merge proposal.
  2. Missing mapping — a required contract field with no binding. Action link opens the Mapping Workbench filtered to the field.
  3. Deprecated concept usage — a mapping references a concept that has been deprecated. Action link opens the concept inspector.
  4. Broken relationship — a relationship endpoint references a concept that no longer exists. Action link opens the relationship inspector.

Severity labels:

UI label Backend severity Colour Meaning
Error error / Violation red-500 Blocks merge or context publication.
Warning warning / Warning amber-400 Should fix; does not block.
Info Info sky-400 Informational only.

For the revenue/billing sample, expect a handful of warnings and zero errors after accepting all six suggestions. The exact counts vary depending on which fields you bound in Step 4.

For the API equivalent, query GET /api/v1/semantic/health/issues. The response is a paginated list with the same shape the UI consumes.

Step 6: Serve governed context

The Context tab in the editor companion panel (P5-004) shows what a downstream agent or application will receive when it requests a context pack for the workspace.

The preview surfaces:

  • Pack id and content hash — stable identifiers an agent can quote back. The id is a UUID per pack; the hash is a SHA-256 of the canonical-form JSON so two semantically identical packs hash the same.
  • Evidence envelope — the per-concept evidence list (source assets, source spans, confidence scores, rationale).
  • Policy filter — which fields and concepts the requesting role can see. The viewer / reviewer / admin / superuser hierarchy decides what is included. See semantic-context-roles.md.

For the API equivalent:

curl http://localhost:8002/api/v1/semantic/context/packs/revenue-billing \
  -H "Authorization: Bearer ${PYCHARTER_TOKEN}"

The response is a JSON context pack with id, content_hash, policy, evidence, and the concept and mapping payload. The response respects the caller's role: a viewer only sees approved bindings; reviewer also sees under_review; admin sees deprecated and rejected; superuser bypasses the workspace filter. The full visibility matrix is in semantic-context-roles.md.

Step 7: Troubleshooting

Common issues and remediations:

sqlalchemy.exc.OperationalError: unable to open database file

The SQLite URL is relative and the working directory is not writable. Pick an absolute path:

--db-url sqlite:////absolute/path/to/sample.db

Note the four slashes after sqlite: for an absolute path.

ModuleNotFoundError: No module named 'networkx'

networkx is a core dependency (context-pack projection). Reinstall the package:

pip install -e '.[ci]'

If you still see this after upgrading, remove stale editable metadata (pip uninstall pycharter then reinstall) or delete .venv-ci-pre-push and push again so the pre-push hook rebuilds the CI venv.

Seed smoke test fails

Either the ecommerce example was edited or the seed loader changed. Re-run against a fresh database file to rule out stale state, then diff src/pycharter/data/seed/examples/ecommerce/ against main.

Import Wizard fails with 403 Forbidden

The caller is below the reviewer role. Either request a higher role from an admin and re-login, or — for local development — bypass auth entirely:

export PYCHARTER_AUTH_DISABLED=true

With that flag set every caller is treated as a superuser. For production, mint a JWT via POST /api/v1/auth/login and send Authorization: Bearer <token>. See role-permissions-matrix.md §"Source of role + workspace".

Mapping Workbench is empty for a contract

The contract has no fields. Confirm the contract YAML defines properties under json_schema: and was imported into the same workspace as the concept candidates (Step 2). The workbench joins on scheme_key.

Context tab says "Policy blocked"

The caller's role does not satisfy the workspace's policy tier. See semantic-context-roles.md for the role-to-policy matrix. Reduce the workspace policy or promote the caller's role.

Step 8: Next steps

You have completed one pass through the workflow: workspace, import, review, bind, health, context. To go further:

  • Admin / governance guide — read it next to learn how to assign roles, configure policy tiers, and export audit logs. Forward reference (Phase 6 wave 3).
  • Migration guide — follow Vocabulary migration when upgrading an existing deployment to the unified editor and /vocabulary/* routes.
  • Phase plans and ADRs — the workflow proposal and the kanban describe the design behind each surface and the residual gaps.
  • Reference docs — the role matrix (role-permissions-matrix.md), the role-to-policy matrix (semantic-context-roles.md), and the query conventions (pagination-and-filtering.md) describe the building blocks the UI and API share.
  • The ecommerce example directory — the contracts, docs, DDL, and manifest under src/pycharter/data/seed/examples/ecommerce/ are the smallest self-contained domain that exercises every Phase 0..5 surface. Copy it as a template for your own domain.