Privacy flags (`pii` and `restricted`)¶

Two per-field flags on a contract's field_bindings say that a value is personal or sensitive. This guide covers what pycharter does about them when records leave its control — batch loaders, streaming sinks, and dead-letter queues — and what it deliberately does not do.

The short version:

You declare	pycharter does
`pii: true` on a field	replaces the value with an HMAC token at an uncleared sink
`restricted: true` on a field	replaces the value with `null` at an uncleared sink
nothing	nothing changes

Enforcement is opt-in per step. The default mode is detect: it never mutates a record, it only counts and reports. Upgrading pycharter therefore changes no pipeline's output.

1. Flag a field¶

Flags live on the contract, next to the field's binding. The contract owner declares them; a pipeline author never does.

# orders.yaml
metadata:
  name: orders
  version: "2.1.0"
json_schema:
  type: object
  properties:
    order_id: {type: string}
    email:    {type: string, format: email}
    ssn:      {type: string}
field_bindings:
  email:
    primary: {concept: {concept: EmailAddress}, concept_scheme: "commerce:1.0.0"}
    pii: true
  ssn:
    primary: {concept: {concept: SSN}, concept_scheme: "commerce:1.0.0"}
    restricted: true

Both flags on the same field means restricted — the stricter of the two wins, and only a restricted clearance passes it through in clear.

Flags match top-level record keys, exactly like the knowledge-base emitter. Nested paths (customer.email) are not matched; see What is not covered.

2. Choose a mode¶

The mode is a per-step privacy block inside the step's contract block:

steps:
  - id: load_orders
    type: load
    input: extract_orders
    contract:
      ref: {type: file, path: orders.yaml}
      privacy:
        mode: enforce        # off | detect | enforce
        clearance: []        # flags this sink may receive in clear
    target:
      type: postgres
      db_schema: public
      table: orders

Mode	Records	Report
`off`	untouched	none — the step is not reported at all
`detect` (default)	untouched	counts every flagged value that reached the sink in clear
`enforce`	uncleared flagged values are tokenized or redacted before the write	counts what was handled

detect is the default when the privacy block is absent, so an existing pipeline that attaches a flagged contract starts reporting immediately and keeps writing exactly what it wrote before.

Enforcement sits immediately before the sink write and after the knowledge-base emitter and extraction-sink tees. Those tees, and the records your transforms see, always get clear values; only what lands in the destination is handled.

`clearance` — the sink owner's declaration¶

clearance lists the flags this particular sink is allowed to receive in clear. It is pipeline-side config, never inferable from the contract, and it shows up in the run report so an auditor can see it:

privacy:
  mode: enforce
  clearance: [pii]     # email lands in clear; ssn is still redacted

clearance is only honoured at a load step's sink. Setting it anywhere else is a preflight error ([clearance_on_non_load]).

3. What the handling actually is¶

restricted → redact. The value becomes null — not a "__redacted__" sentinel — so a typed sink column stays loadable.

pii → tokenize. The value becomes a 64-character hex HMAC-SHA256 token over the KB's normalized form of the value. It is:

deterministic — the same secret and value always produce the same token, so re-running a pipeline produces identical output;
joinable — the token equals the KB match_token for that value, so a tokenized warehouse column joins against knowledge-base records without ever holding the clear value;
not reversible — there is no un-tokenize step. Keep the clear value where it is allowed to live, not in the sink.

A value that is not a scalar (a dict, list, or bytes) cannot be normalized into one token, so it is redacted instead and counted separately as non_scalar_redacted. Failing closed beats raising in the middle of a batch.

The org secret¶

Tokenization needs an HMAC key. It is never read from pipeline YAML. Supply it either way:

export PYCHARTER_KB_ORG_SECRET='…'     # environment (usual for CI / prod)

pipeline.run(privacy_secret="…")        # or await pipeline.arun(privacy_secret="…")

Every pipeline whose records should join by token must use the same secret. If a token is needed and no secret resolves, the run does not start — see [missing_secret] below. It never falls back to writing the clear value.

4. The report¶

Every run that touched a flagged contract carries a report:

PipelineResult.privacy_report (and result.to_dict()["privacy_report"])
StepResult.privacy — the same per-step entry
the pycharterPrivacy OpenLineage run facet, on the COMPLETE / FAIL event
one CLI line per load step from pycharter pipeline run

It carries field names, flags, handling, clearance, and counts — never a value.

{
  "steps": [
    {
      "step_id": "load_orders",
      "mode": "enforce",
      "contract_id": "orders",
      "contract_version": "2.1.0",
      "sink": "postgres:public.orders",
      "clearance": [],
      "records": 1204,
      "fields": [
        {"field": "email", "flag": "pii", "handling": "tokenize",
         "cleared": false, "values": 1204, "handled": 1204,
         "non_scalar_redacted": 0},
        {"field": "ssn", "flag": "restricted", "handling": "redact",
         "cleared": false, "values": 1204, "handled": 1204,
         "non_scalar_redacted": 0}
      ],
      "dlq_records_handled": 3,
      "dlq_records_withheld": 0
    }
  ]
}

Reading it:

values — non-null occurrences of the field at the boundary. In enforce a cleared field is never visited, so it reports values: 0 with cleared: true.
handled — values actually tokenized or redacted. Always 0 in detect.
values > 0 with handled == 0 and cleared == false is the interesting line: those values reached the sink in clear.

The CLI prints the same thing on one line per load step:

privacy [enforce] load_orders → postgres:public.orders: email pii tokenized 1,204; ssn restricted redacted 1,204; dlq handled 3

The field table for the privacy block itself is in Reference → PrivacyBlock.

5. Dead-letter queues¶

A quarantined record is a record that failed validation — its payload is exactly the kind of thing that carries raw personal data, so under enforce the DLQ is handled too:

Batch step DLQ (on_error: quarantine, file or database target) — every flagged field in the payload is redacted, and validation_errors carry the offending type, never the offending value. Redact-all, not the step's handling: a quarantined record never needs to be joinable, redaction needs no secret, and it cannot fail mid-write. clearance does not apply to a DLQ — it is not the cleared sink.
Streaming DLQ — the same, plus: a payload that cannot be classified at all (a decode error, an unresolvable contract) is withheld and replaced by {"_withheld": true, "sha256": …, "bytes": n}. If a record cannot be handled safely (no plan, or no org secret), it is routed to the DLQ withheld with reason privacy_error rather than written in clear.

Under detect and off the DLQ keeps its documented raw-payload behaviour — the change is gated behind opting in to enforce.

Counts appear in the report as dlq_records_handled and dlq_records_withheld.

6. Preflight: what each rule code means¶

pycharter pipeline validate reports privacy problems in the privacy_clearance category, and Pipeline.run() / arun() runs the same pass before the first step — a pipeline never starts in a state preflight would reject. Errors abort the run (nothing executes and the messages come back on PipelineResult.errors); warnings are logged at WARNING and the run proceeds.

Code	Severity	Meaning	What to do
`[missing_secret]`	error	`enforce` must tokenize a `pii` field but no org secret resolves.	Export `PYCHARTER_KB_ORG_SECRET`, pass `privacy_secret=`, or add `clearance: [pii]` if this sink is allowed the clear value.
`[clearance_on_non_load]`	error	`clearance` set on a step that is not a load step.	Remove it, or move it to the load step that writes the records.
`[flagged_cursor]`	error	A flagged field is the incremental `watermark_field`.	Pick an unflagged cursor (an id, an `updated_at`): watermarks persist to the run checkpoint in clear.
`[unenforceable_sink]`	error under `enforce`, warning under `detect`	A load step has no contract block — or attaches a contract that does not flag the same fields — while an upstream step's contract flags fields. Those flags cannot be enforced at this sink.	Attach the upstream contract (or one carrying the same flags) to the load step with its own `privacy` block.
`[unenforceable_field]`	error under `enforce`, warning under `detect`	A flagged binding key is nested (`customer.email`, `tags[0]`); the sink boundary matches top-level record keys only, so the value would pass in clear.	Flatten the field, or flag its top-level parent.
`[in_clear]`	warning	`detect`, and flagged values reach this sink in clear.	Expected while you are surveying. Set `mode: enforce`, or add the flag to `clearance` to say the sink is allowed it.
`[tokenized_non_string]`	warning	A tokenized field's schema type is not a string.	Tokens are 64-char strings — widen the sink column to text, or clear the sink for `pii`.
`[quality_check_on_redacted]`	warning	A post-load `quality_check` targets a field `enforce` redacts.	Drop the check, or clear the sink for `restricted`. A `null_rate` check on a redacted column reads 100%.

7. Streaming¶

Streaming takes the same privacy block, on the stream's contract:

streams:
  - name: orders
    contract:
      ref: {type: store, name: orders}
      store: {type: postgres, connection_string: ${DATABASE_URL}}
      privacy:
        mode: enforce
        clearance: []

Semantics match the batch boundary: applied immediately before every valid-sink write (including each row a window emits), tees untouched, plans cached per contract, identical output on redelivery. Field-level details are in Reference → Streaming Worker Config.

Use ref.type: store. A ref.type: file contract resolves through the deprecated split-directory form (schema.yaml + rules), which carries no field_bindings and therefore no flags. The worker refuses enforce on such a stream at build time and warns that detect — explicit or the default when no privacy block is present — will report nothing; the same applies to kafka.consume(contract_dir=...).

8. What is not covered¶

Known, deliberate gaps — do not read these as promises:

ai_usage_policy.pii_redaction_required is knowledge-base only. It governs extraction capture and embedding, not warehouse sinks. Sink behaviour keys off the per-field pii / restricted flags and nothing else.
Nested paths are not matched at the sink. A binding key matches a top-level record key (same rule as the KB emitter). A flagged nested key is not silently skipped, though: preflight reports [unenforceable_field] (error under enforce), and validation diagnostics do withhold values for a dotted binding key.
ODCS import writes per-field flags. classification: pii | restricted on a foreign ODCS property lands on field_bindings[..].pii / .restricted — the flags the runtime enforces — and no longer on the legacy metadata.governance_rules.pii_fields list. Export unions both, so re-export is unchanged; code that read pii_fields after from_odcs should read the per-field flags.
pycharter stream replay is gated, but has nowhere to quarantine. Pass --contract-file <manifest.yaml> (a single-file kind: DataContract artifact) and every replayed record goes through the same gate a worker uses: --privacy-mode off|detect|enforce (default detect), --clearance pii|restricted (repeatable), org secret from PYCHARTER_KB_ORG_SECRET. The deprecated --contract-dir form carries no field_bindings, so enforce is refused (exit 2) and detect warns that its report will be empty — the same refusal the worker applies to a contract.ref.type: file stream. Replay has no DLQ of its own, so a record enforce cannot handle safely (no plan, or tokenization with no secret) is dropped rather than written in clear; it appears only as the withheld count on the summary line.
kafka.consume() as a library applies detect only when the caller's factory exposes contract metadata (built with annotate=True or the privacy toggle); otherwise it is off.
Post-load quality checks run on handled data. uniqueness on tokens still holds; null_rate on a redacted column reads 100% (preflight warns).
No encryption, no key management, no detectors. pycharter enforces what the contract declares; it does not guess which fields are personal.

Privacy flags (pii and restricted)¶