Skip to content

ADR: PyCharter Worker Roadmap

Status: Accepted — batch worker hardening complete (2026-06) Date: 2026-05-04 (updated 2026-06-04) Deciders: Austin; TBD reviewers

Context

PyCharter ships a database-backed worker under src/pycharter/worker/ that polls validation_jobs and pipeline_jobs, claims work atomically, runs contract-backed quality validation and durable pipeline execution, and records job status/results. The API exposes producer/status routes; the CLI starts the service with pycharter worker --mode {all,validation,pipeline}.

Workers are optional runtime infrastructure — they consume public core/store APIs and shared database models without moving business logic out of the core.

Decision

PyCharter treats workers as long-running execution services with strict layer boundaries:

  • src/pycharter/api/ — HTTP, enqueue/status routes, DLQ replay API
  • src/pycharter/ui/ — observation/control via API only
  • src/pycharter/db/ — migrations and models shared by API and worker
  • src/pycharter/worker/ — poll loops, claim/lease, processors, operability
  • Core packages (quality, pipeline modules) — reusable by API, CLI, and workers

Current batch worker (shipped)

The unified batch worker is production-ready for validation and pipeline queues.

Architecture

  • Single runner: Worker in runner.py with WorkerMode (all, validation, pipeline)
  • Sibling job sources: ValidationJobSource, PipelineJobSource sharing _db_helpers.py
  • Processors: ValidationJobProcessor, PipelineJobProcessor with lease heartbeat
  • Structured DLQ: dead_letter_queue table + /api/v1/dlq/* replay/discard API
  • Operability (opt-in): metrics, health HTTP server, Postgres NOTIFY listener, retention sweeps

CLI

pycharter worker                          # mode=all (default)
pycharter worker --mode validation
pycharter worker --mode pipeline

Removed (no aliases): pycharter worker start, pycharter worker pipeline as separate subcommands.

Terminal failure contract

Artifact Behavior
Queue row failed + error_message + optional dlq_record_id
DLQ record Canonical replay source when PYCHARTER_WORKER_DLQ_ENABLED=true
Pipeline runs Updated to failed on terminal pipeline job failure

Legacy dead_letter queue status is not used for new failures.

Configuration highlights

  • PYCHARTER_WORKER_LEASE_SECONDS + PYCHARTER_WORKER_HEARTBEAT_SECONDS (replaces CLAIM_LEASE_TIMEOUT_S)
  • Per-queue concurrency: VALIDATION_CONCURRENCY, PIPELINE_CONCURRENCY
  • Operability flags documented in docs/operations/worker-hardening.md

Documentation

  • User guide: docs/guides/worker.md
  • Configuration: docs/guides/configuration.md, pycharter.cfg.example
  • Operability runbook: docs/operations/worker-hardening.md
  • Module map: src/pycharter/worker/README.md

Phases (historical)

Phase 1: Validation worker foundation — complete

Database queue, claim/lease, retries, cancellation-safe completion, stats/health API.

Phase 2: Pipeline run persistence — complete

pipeline_runs is the canonical lifecycle record for API, CLI, and worker-driven runs.

Phase 3: Pipeline worker — complete

Dedicated pipeline_jobs queue, PipelineJobSource, unified Worker runner.

Phase 4: Pipelines health UI — complete

Merged health overview under Pipelines/Quality (queue depth, failures, DLQ counts).

Phase 5: Batch worker hardening — complete (2026-06)

Aligned with PyStator worker operability tier:

  • Lease + heartbeat with lease_expires_at
  • Structured DLQ + replay API
  • Metrics, health server, NOTIFY, retention (all opt-in)
  • Unified CLI and package layout refactor

Out of scope (batch worker)

  • Streaming worker observability (pycharter stream run — tracked separately as ST-10)
  • DLQ replay UI (API-first; UI follow-up)
  • Advisory locks, apply-data queues, action outbox (PyStator-specific patterns)
  • Shared optophi-worker extraction package

Consequences

  • Batch worker code is the template for future queue types: sibling source + processor + shared helpers.
  • Postgres is recommended for multi-poller production; SQLite remains supported for local dev.
  • New operability features stay opt-in via WorkerConfig to avoid breaking minimal installs.

Follow-up backlog

The batch worker roadmap is complete, and so is the streaming worker work (ST-*). New worker work goes in BACKLOG.md.