privacy block (PrivacyBlock)

How one step treats contract-flagged fields (field_bindings entries carrying pii: true or restricted: true) at its sink boundary.

The same block appears in two places:

• a pipeline step's contract block — steps[].contract.privacy (StepContractBlock.privacy);
• a stream's contract block — streams[].contract.privacy (StreamContract.privacy, see Streaming Worker Config).

Model: pycharter.pipeline_artifact.models.PrivacyBlock — frozen, extra: forbid (an unknown key is a config error, not a silent no-op).

For the workflow — flagging fields, reading the report, preflight rule codes — see the guide Privacy flags.

steps:
  - id: load_orders
    type: load
    input: extract_orders
    contract:
      ref: {type: file, path: orders.yaml}
      privacy:
        mode: enforce        # off | detect | enforce
        clearance: [pii]     # flags this sink may receive in clear
    target: {type: postgres, db_schema: public, table: orders}

Fields

Field	Type	Default	Notes
mode	enum off | detect | enforce	detect	off restores pre-feature behaviour: records untouched, step absent from the report. detect never mutates and counts flagged values that reached the sink in clear. enforce tokenizes pii and redacts restricted immediately before the sink write — after the KB-emitter and extraction-sink tees — unless the flag is listed in clearance.
clearance	list of pii | restricted	[]	Flags this step's sink is allowed to receive in clear. Declared by the pipeline author, never by the contract, and echoed in the run report. Only honoured on a load step; preflight rejects it elsewhere ([clearance_on_non_load]). An unknown flag is rejected at parse time.

Absent block means {mode: detect, clearance: []}, so attaching a flagged contract to an existing step changes no output and starts reporting.

Handling derived from the flags

One derivation, shared by the load boundary, the streaming sink, both DLQ wrappers, preflight, and validation diagnostics:

Contract flag	Handling at an uncleared sink	Result in the sink
restricted: true	redact	null (not a sentinel — typed columns stay loadable)
pii: true	tokenize	64-char hex HMAC-SHA256 token, equal to the KB match_token for the same value
both	redact	null; the effective flag is restricted, so only a restricted clearance passes it in clear
a non-scalar value under tokenize	redact	null, counted as non_scalar_redacted

Tokenization requires an org secret: PYCHARTER_KB_ORG_SECRET, or Pipeline.run(privacy_secret=...) / arun(privacy_secret=...). When none resolves, preflight fails the run with [missing_secret] — a flagged value is never written in clear as a fallback.

What the step reports

Each step running under detect or enforce with a flagged contract contributes one entry to PipelineResult.privacy_report (also on StepResult.privacy, the pycharterPrivacy lineage run facet, and the pycharter pipeline run summary line):

Key	Meaning
step_id, mode, sink	Which boundary, under which mode. sink is credential-free (postgres:public.orders).
contract_id, contract_version	The contract whose flags were applied.
clearance	Flags this sink was cleared for.
records	Records that crossed the boundary.
fields[]	Per flagged field: field, flag, handling, cleared, values, handled, non_scalar_redacted.
dlq_records_handled, dlq_records_withheld	Quarantined payloads handled, and (streaming) withheld because they could not be classified.

Counts only — no report, log, or facet ever carries a flagged value.