Worker hardening — operations guide

This document covers operability features added during worker hardening (Phases 4–6).

Metrics (Phase 4)

Enable Prometheus metrics on the worker process:

export PYCHARTER_WORKER_METRICS_ENABLED=true
pycharter worker --mode all

Install the worker extra for real counters (without it, metrics are no-ops):

pip install pycharter[server]

Key metrics:

Metric	Labels	Meaning
pycharter_worker_jobs_claimed_total	queue	Jobs claimed
pycharter_worker_jobs_completed_total	queue	Successful completions
pycharter_worker_jobs_failed_total	queue, reason	Failure attempts
pycharter_worker_jobs_dlq_total	queue, reason	Terminal DLQ writes
pycharter_worker_stale_released_total	queue	Stale claims released
pycharter_worker_lease_lost_total	queue	Heartbeat lease losses
pycharter_worker_job_processing_seconds	queue	Processing latency histogram
pycharter_worker_in_flight	queue	Currently processing
pycharter_worker_retention_deleted_total	table, category	Retention deletions

Health server (Phase 4)

Opt-in HTTP server (default port 9090):

export PYCHARTER_WORKER_HEALTH_SERVER_ENABLED=true
export PYCHARTER_WORKER_HEALTH_SERVER_PORT=9090
pycharter worker --mode validation

Endpoints:

• GET /metrics — Prometheus scrape target
• GET /healthz/live — liveness (always 200 when process is up)
• GET /healthz/ready — readiness (DB reachable + worker started)
• GET /healthz/startup — startup (worker start() completed)

Bind address defaults to 0.0.0.0; override with PYCHARTER_WORKER_HEALTH_SERVER_HOST.

NOTIFY wake-up (Phase 5)

On Postgres, job submission emits pg_notify on channel pycharter_worker_jobs (or {prefix}_pycharter_worker_jobs when PYCHARTER_WORKER_NOTIFY_CHANNEL_PREFIX is set).

Enable the LISTEN consumer in the worker:

export PYCHARTER_WORKER_NOTIFY_LISTENER_ENABLED=true
pycharter worker --mode all

Polling remains the safety net — if NOTIFY is missed, pollers wake on the next poll_interval_ms.

Requires pycharter[server] (psycopg 3).

Retention (Phase 6)

Opt-in DELETE sweeps for terminal queue rows. Never touches pending or claimed.

export PYCHARTER_WORKER_RETENTION_ENABLED=true
export PYCHARTER_WORKER_RETENTION_VALIDATION_COMPLETED_DAYS=7
export PYCHARTER_WORKER_RETENTION_VALIDATION_FAILED_DAYS=90
export PYCHARTER_WORKER_RETENTION_SWEEP_INTERVAL_SECONDS=86400
pycharter worker --mode all

Setting	Default	Meaning
RETENTION_VALIDATION_COMPLETED_DAYS	7	Prune completed validation jobs
RETENTION_VALIDATION_FAILED_DAYS	90	Prune failed validation jobs
RETENTION_PIPELINE_COMPLETED_DAYS	7	Prune completed pipeline jobs
RETENTION_PIPELINE_FAILED_DAYS	90	Prune failed pipeline jobs
RETENTION_SWEEP_INTERVAL_SECONDS	86400	Sweep cadence while worker runs
RETENTION_SWEEP_BATCH_SIZE	10000	Max rows deleted per batch

Set a window to none, forever, or -1 to disable pruning for that category.

Scope boundary

pycharter worker does not cover:

• Streaming validation (pycharter stream run)
• Sync API validation endpoints
• Background ingest helpers (_ingest_background)

Those use separate execution paths.