Skip to content

Corporate action PDF ingestion design

Draft, 2026-10-16. Source audit: develop at 55e6ad2bb6c753948284cf639aab7c7753b1fc76. AGPIPE-11 in BACKLOG.md is the only implementation tracker. This proposal defines a second-use-case candidate under AGPIPE-10; it does not widen the HKTVmall pilot or approve a provider, new public API, dependency, numerical operating limit or live deployment.

Outcome and architectural fit

Given a reviewed manifest of official corporate-action PDFs, publish versioned cash-dividend and simple share-subdivision announcement records with retained source evidence, or explain why the required batch is withheld. Consumers may use accepted events alongside market prices; this workflow does not calculate adjusted prices, portfolio entitlements or trading decisions.

PyCharter owns the contracts, pure normalization/evaluation, coverage rules, retained requirements and publication conditions. External adapters acquire, parse/render and optionally interpret documents. The workflow remains useful with direct HTTP and deterministic extraction; browser/model use must demonstrate an operational benefit rather than become an unconditional requirement.

Follow ARCHITECTURE.md: business fields and rules are YAML/JSON validated by Pydantic, core functions operate on supplied data, Store clients translate persistence and wrappers stay thin. Do not add a browser fleet, general agent loop, document task language, scheduler, mandatory sibling package or new top-level domain artifact kind. Operational snapshots/attempts are typed records, not a second artifact catalog. No new cross-package bridge is needed. Existing conventional pipelines and explicit delivery destinations retain their current behavior.

Bounded first reference

Dimension Proposed boundary
Targets Known public PDF URLs with expected issuer/security and document family
Source Reviewed issuer/exchange domains; qualify selected HKEX families first
Events Cash dividends and simple share subdivisions, qualified separately
Parsing Usable text-layer PDFs; retained tables/form fields where relevant
Language Qualified English content; no silent translation or bilingual deduplication
Publication Complete frozen manifest; one unresolved required target withholds release
Deployment Python/CLI terminating batch; existing worker/scheduler optional

Start with cash-dividend semantics and a separately reviewed split corpus before advertising both. Source layouts, event cardinality and permitted field states must be explicit. A document with several relevant events cannot silently yield only one; either account for the supported events or require review.

Defer general announcement search/crawling, OCR, broad language support, scrip/currency-election calculations, complex reorganizations, automatic price/portfolio adjustment and partial publication. A known target returning no usable document is an unresolved acquisition, not evidence of no corporate action. Frozen-manifest completion does not certify all announcements for an issuer, exchange, market or historical universe. Discovery needs a separately reviewed coverage contract and cannot silently shrink the expected set.

Source audit and interface decision

The current source establishes useful foundations and limits:

  • semantic/extraction/adapters/_pdf.py:1-19,99-104 emits document/page vocabulary concepts with bounded descriptions, does not perform OCR and can replace page extraction errors with empty text. It is not a complete financial parser or factual-accuracy check; preserve its existing public behavior.
  • pipeline_generator/steps/_base.py:257-269 declares document_parse, while pipeline_artifact/models.py:524-531 admits only extract/transform/join/union/load. pipeline_generator/_runner.py:396-526 dispatches those five runtime types. Registry discovery is not proof of canonical execution support.
  • pipeline_generator/protocols.py:16-31 describes record-batch extraction, not remote submission/reconciliation. stores/factory.py:24 declares no managed document/dataset client. Use AGPIPE's planned boundaries, not implicit support.

The conceptual flow is:

freeze targets -> acquire original bytes -> retain immutable snapshot
  -> parse pages/tables/forms -> extract candidate facts -> transform
  -> verify facts/evidence/coverage -> stage -> publish records + receipt

A document-source connector under extract, completion of canonical document_parse, and the proposed agent step for interpretation are candidates, not settled API decisions. A small reference experiment must compare these with an application adapter supplying records through public interfaces. Resolve the chosen boundary and remote lifecycle before adding native configuration. An acquisition adapter must not hide outstanding remote operations behind a bare record iterator. Browser-assisted link resolution from approved landing pages is a separately qualified extension of the known-PDF acquisition scope.

Use strict typed configuration, explicit capability discovery and exact references. Unknown fields, missing required evaluators, unsupported step/connector shapes and unavailable mandatory capabilities refuse before external work. Older readers and editors must round-trip admitted fields or refuse clearly. Do not turn unsupported steps into successful no-ops or bump the global manifest merely to introduce the workflow. No runnable YAML is implied by this proposal.

Contracts and retained records

Reuse DataContract for domain records. Define identifiers, optionality, cardinality, null states and transformations before authoring draft examples. Separate expected identity from observed identity: echoing a target symbol does not demonstrate that the document contains that security.

Boundary Required information
Target Stable target ID, expected issuer/security, approved locator/family, exact security-mapping version
Acquired snapshot Source identity, immutable original bytes, digest, media type, byte count, retrieval time and safe source locator
Parsed document Snapshot identity, parser/version/settings, page/table/form elements and explicit completeness diagnostics
Candidate facts Observed identifiers, original value text, qualifiers and field-level evidence references
Normalized announcement Exact event terms and states, security association, source revision and processing identities
Verification Rule/version, pass/fail/indeterminate, reasons and supporting evidence
Publication receipt Frozen targets, accepted revisions, coverage accounting and committed publication identity

Operational manifests, provider status and evidence metadata remain linked run information under AGPIPE-⅔. They are not financial event rows. Document-reference records may carry an opaque snapshot identity; that does not expose or duplicate the acquisition ledger. Review this distinction when selecting the native schema.

Cash dividends

Keep declared amount/currency separate from payment amount/currency. Preserve share basis, dividend type/reporting period, approval conditions and each ex-dividend/record/payment date with its own qualifier. Use exact decimal strings or an equivalently lossless validated representation; do not reuse the grocery pilot's integer-cent rule. A sub-cent dividend must survive normalization and Store/database round trips without rounding or binary-float conversion.

Define not_reported, to_be_announced, not_applicable and ambiguous separately; none implies zero. Preserve proposed/conditional terms. A new announcement is not automatically approval or evidence of payment. An accepted announcement may legitimately report TBA fields; a downstream contract requiring final terms must continue to reject that incompleteness. No implied FX, tax, calendar or investor-specific entitlement calculation belongs in this reference.

Share subdivisions

Represent explicit positive old/new share quantities with a documented direction, not an unexplained scalar ratio. Preserve source wording and distinguish action effective date, expected trading commencement, settlement conversion and certificate exchange dates. Board-lot changes and operational conversion ratios are not separate stock splits. Temporary/parallel counters require a reviewed association to the intended security, not fuzzy ticker matching.

Identities and revisions

Keep document revision, extraction attempt and business event identity distinct. A content digest identifies bytes; (symbol, type, date) cannot reliably identify an action whose dates change or collide with another action. Use official action identifiers when available, otherwise a trusted managed association. Uncertain cross-document associations require review.

Updates and withdrawals append event revisions and preserve prior observations. Partial amendments may merge only against an exact prior accepted revision under explicit reviewed rules; absent fields are not automatic replacements or zeroes. Conflicting updates require a deterministic source policy or attributable review. Repeated bytes may reuse retained parsing without losing the new observation; different language documents are not automatically the same revision.

Keep source publication time when available, observation time, acceptance and dataset publication time separate from event dates. PDF metadata/printed dates do not prove website release time. Historical downloads cannot backdate what the system knew. Define as-of availability from retained times and disclose unknown source release times; correction handling must not rewrite historical knowledge.

Evidence and acceptance

Every material observed field needs a retained snapshot, physical page number, parsed element/cell or form-field reference, and supporting raw text including its label/header context. Qualify page-region evidence where layout is needed; define page numbering, rotation, coordinates and parser-output identity. Do not silently discard tables, qualifiers, footnotes, checked boxes or pages needed to interpret a field. Parser support for form values and rendered/text agreement needs testing against actual published documents, not only templates.

Pure evaluators check supplied evidence availability/integrity, observed identity, field binding, permitted states, exact transformations and target accounting. A citation containing a number does not prove that it describes the claimed field. Deterministic label/value checks may qualify known templates; unsupported layouts or competing interpretations require review. A second model or the extractor's confidence is not ground truth. Review corrections create new candidates and attributable decisions bound to exact evidence/requirements; they cannot rewrite raw documents or bypass mandatory validation. The review interface and authorization require explicit design, not presumed support from existing contract-artifact approval tools.

Persistence and execution

Extend AGPIPE-4's one configured database through public Store clients for the bounded reference: original PDFs, parsed content, required evidence, retained inputs/requirements, attempts, decisions, staged records and receipts. Temporary files are disposable processing material; provider URLs and hashes alone are not recovery sources. Copy required expiring artifacts before accepting the target.

Storage representation and numeric bounds require measured qualification across SQLite, PostgreSQL and MongoDB as required by repository instructions. Implement explicit migrations and conformance tests; verify transaction/payload capability before submissions. No implicit in-memory fallback, task-selected state database or separate file registry is permitted. Large-document object storage requires an explicit design extension with integrity, backup, retention and recovery semantics.

Freeze contracts, mapping/rule implementations, parser settings, instructions, model/adapter identities, non-secret context and effective grants/limits before external work. Retain the required content, not just hashes; unreported provider revisions remain unknown. Replaying retained output, resuming interrupted work and fetching/reinterpreting again are distinct operations with distinct attempts.

Use AGPIPE-6's remote reconciliation, bounded retries and owner/attempt fencing. Record intent before submission and remote identity afterward. Unknown submission outcomes require provider lookup/idempotency or explicit uncertainty, not blind resubmission. A local timeout/cancellation request does not establish remote termination. Stale workers cannot write progress or publish. Resume rechecks current authority and context; no database transaction spans remote waiting.

Parsing runs in a constrained runtime process with explicit page, byte, content, memory and time limits. Model usage, concurrency, retries and total work also need bounds. Limits are measured design decisions, not the provisional quantities from discussion or guarantees inferred from a vendor parameter. Failure or truncation must remain visible; scanned/encrypted/corrupt/oversized or incompletely parsed PDFs cannot become successful empty documents. Empty manifests report no-work under the agreed result projection, not successful collection.

Publication and operations

AGPIPE-7 owns staging and complete publication. Freeze the publication unit before execution; one unresolved required target withholds the batch. Independent batches may be separate runs chosen beforehand, not post-failure denominator changes. Run required checks after the declared transform regardless of optional quality flags. Gate loader calls and reader visibility; post-load checks cannot protect a publication that already happened.

Accepted revisions and their receipt commit atomically using documented backend semantics. Commit-output loss resumes from the receipt without duplicate events. Required evidence/ledger write failures prevent acceptance or publication. Warehouse export is a separate delivery outcome with its own idempotency and receipt; arbitrary loaders or separate databases have no shared atomicity claim.

Extend AGPIPE-8 and existing run/check interfaces with acquired/parsed/verified/ withheld/published targets, event revisions, evidence availability, attempts and actual/estimated/unknown usage. Distinguish execution, document completeness, factual verification, event state and publication. Use PIPE-10/10b/11/13 for inspection, retention, freshness and check output. Source disclosure delay, acquisition freshness and historical backfill have distinct meanings. Required evidence and unresolved reconciliation cannot disappear through routine pruning.

Adapter qualification and source policy

Compare direct HTTP plus deterministic extraction first. Browser use addresses approved navigation/download needs; model use addresses interpretation. Neither implies the other, and provider selection for the price pilot does not qualify PDF acquisition. Capability assessment must distinguish original file retrieval from text conversion, page evidence, session lifecycle and enforceable limits.

PDF/browser/OCR/model engines remain optional adapters behind typed Protocols. No new engine or dependency is selected here. A parser upgrade requires corpus comparison and recorded implementation identity. Qualification includes schema translation limits and independent canonical validation, rather than weakening the contract to fit a provider.

Enforce operator-approved domains, redirect/destination checks, permitted browser operations and resource limits at a trusted runtime boundary. Document content cannot grant access, select secrets/storage/code or change instructions and acceptance policy. Parser execution has no unnecessary network/tools; model interpretation is tool-free unless separately required and enforced. Secret values stay in the approved secret source. Retained locators, evidence and logs must exclude credentials and unnecessary personal data. External processing, retention/access policy, review owner and budget must be settled before live work.

Qualification evidence and delivery boundaries

Use independently annotated supported documents plus negative/failure cases. Hold out issuers/layouts where feasible; reviewers resolve annotation disagreements independently of the extractor. Include declaration/payment confusion, sub-cent values, share bases, TBA states, proposals, amendments/withdrawals, split direction, multiple counters/dates, duplicate or multiple events and unsupported documents. Measure both dividends and splits independently. Scope exclusions and review rates remain visible alongside automated acceptance and critical-field errors.

Establish numerical accuracy/automation, cost, latency, intervention and resource gates from this corpus, a simpler baseline and the operator's objectives before live release. Finite zero-error results do not establish universal accuracy. Tests must prove evidence resolution, complete target accounting, unchanged conventional behavior, the chosen canonical parse/dump/Store/build/execute path, required backend parity, restart/definition mutation, competing/stale workers, commit-output loss, changed/inaccessible evidence and safe rerun. Synthetic fixtures prove only their cases; a separately authorized reviewed live trial must state its actual source/parser/provider/account/destination scope and limitations.

AGPIPE-11 owns document-specific delivery. Reuse AGPIPE-⅔ for schemas/evaluators, AGPIPE-4 for persistence, AGPIPE-⅚ for adapters/recovery, AGPIPE-7 for publication and AGPIPE-8 plus PIPE-4/10/10b/11/13 for retained requirements and operations. Do not copy those implementation cards or add PDF release gates to AGPIPE-1–9. Split eventual code into focused configuration/core, Store, adapter, execution, publication and wrapper changes after qualification settles their interfaces. The first milestone's acceptance is not certification of PDF behavior; the second reference must supply its own end-to-end evidence.

Primary references

Documentation reviewed 2026-10-16; capabilities below are not account/site trials.