Skip to content

Validate a pandas DataFrame

Apply a reviewed contract to every row, then inspect the accepted data and failures separately. This self-contained example works with public pycharter==0.0.67 and the base installation:

import pandas as pd
from pycharter import validate_dataframe

df = pd.DataFrame([
    {"order_id": "A-100", "amount": 19.95},
    {"order_id": "A-101", "amount": -5},
    {"order_id": "A-102", "amount": 0},
])
contract = {
    "json_schema": {
        "version": "1.0.0",
        "type": "object",
        "properties": {
            "order_id": {"type": "string"},
            "amount": {"type": "number", "minimum": 0},
        },
        "required": ["order_id", "amount"],
    }
}

report = validate_dataframe(df, contract, on_error="quarantine")
print(len(report.valid))        # 2
print(len(report.quarantined))  # 1
print(report.summary())

report.valid contains passing rows; report.quarantined contains failures and a _violations column. These are in-memory results. Persist or publish them explicitly when your workflow requires it. A saved contract path can replace the dictionary; see the CSV recipe.

Choose what a failure means

on_error Behavior
"quarantine" (default) Return passing rows and failing rows separately
"fail" Raise DataFrameValidationError; inspect its .report
"warn" Log a warning, collect diagnostics and pass all rows through unchanged; this is not an acceptance gate

For a batch that must be entirely acceptable before proceeding:

from pycharter import DataFrameValidationError

try:
    validate_dataframe(df, contract, on_error="fail")
except DataFrameValidationError as exc:
    print(len(exc.report.quarantined))  # 1
    # Stop or route the failed batch according to your application's policy.

With coercion enabled, accepted output uses the contract-coerced values. Pass coerce=False to keep original values in the returned valid frame; this option is not a substitute for strict-type validation.

Inspect individual records

from pycharter import Validator

validator = Validator.from_dict(contract)
results = validator.validate_batch(df.to_dict(orient="records"))
for row_index, result in enumerate(results):
    if not result.is_valid:
        print(row_index, result.errors)

For large inputs, process bounded chunks appropriate to your memory budget. Keep raw failures out of shared logs unless your access and retention policy allows them. For aggregate quality thresholds and monitoring, continue with quality checks.

See also: CSV validation, local pipeline enforcement, and FastAPI request validation.