{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# PyCharter Getting Started\n",
        "\n",
        "This notebook demonstrates the core features of PyCharter:\n",
        "\n",
        "1. Data validation with contracts\n",
        "2. ETL pipelines with the `|` operator\n",
        "3. Quality monitoring\n",
        "\n",
        "[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/optophi/pycharter/blob/main/docs/notebooks/01_getting_started.ipynb)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Installation\n",
        "\n",
        "First, install PyCharter:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Uncomment to install in Colab or Jupyter\n",
        "# !pip install pycharter"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Part 1: Data Validation\n",
        "\n",
        "Define a schema and validate data against it."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from pycharter import from_dict, validate\n",
        "\n",
        "# Define a JSON Schema\n",
        "schema = {\n",
        "    \"type\": \"object\",\n",
        "    \"version\": \"1.0.0\",\n",
        "    \"properties\": {\n",
        "        \"name\": {\"type\": \"string\", \"minLength\": 1},\n",
        "        \"email\": {\"type\": \"string\", \"format\": \"email\"},\n",
        "        \"age\": {\n",
        "            \"type\": \"integer\",\n",
        "            \"minimum\": 0,\n",
        "            \"coercion\": \"coerce_to_integer\"  # Convert strings to int\n",
        "        }\n",
        "    },\n",
        "    \"required\": [\"name\", \"email\"]\n",
        "}\n",
        "\n",
        "# Generate a Pydantic model\n",
        "User = from_dict(schema, \"User\")\n",
        "print(f\"Generated model: {User}\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Validate valid data\n",
        "result = validate(User, {\n",
        "    \"name\": \"Alice\",\n",
        "    \"email\": \"alice@example.com\",\n",
        "    \"age\": \"30\"  # String will be coerced to int\n",
        "})\n",
        "\n",
        "print(f\"Valid: {result.is_valid}\")\n",
        "print(f\"Data: {result.data}\")\n",
        "print(f\"Age type: {type(result.data.age)}\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Validate invalid data\n",
        "result = validate(User, {\n",
        "    \"name\": \"\",  # Empty string - violates minLength\n",
        "    \"email\": \"invalid-email\",  # Not a valid email\n",
        "    \"age\": -5  # Negative - violates minimum\n",
        "})\n",
        "\n",
        "print(f\"Valid: {result.is_valid}\")\n",
        "print(f\"Errors:\")\n",
        "for error in result.errors:\n",
        "    print(f\"  - {error['loc']}: {error['msg']}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Part 2: Using the Validator Class\n",
        "\n",
        "For production use, the `Validator` class is more efficient."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from pycharter import Validator\n",
        "\n",
        "# Create validator from schema dict\n",
        "validator = Validator.from_dict(schema=schema)\n",
        "\n",
        "# Validate multiple records efficiently (model is cached)\n",
        "users = [\n",
        "    {\"name\": \"Alice\", \"email\": \"alice@example.com\", \"age\": 30},\n",
        "    {\"name\": \"Bob\", \"email\": \"bob@example.com\", \"age\": 25},\n",
        "    {\"name\": \"\", \"email\": \"invalid\", \"age\": -1},  # Invalid\n",
        "    {\"name\": \"Charlie\", \"email\": \"charlie@example.com\", \"age\": 35},\n",
        "]\n",
        "\n",
        "results = validator.validate_batch(users)\n",
        "\n",
        "valid_count = sum(1 for r in results if r.is_valid)\n",
        "print(f\"Valid: {valid_count}/{len(users)}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Part 3: ETL Pipeline\n",
        "\n",
        "Build a simple ETL pipeline using the `|` operator."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import asyncio\n",
        "from pycharter import Pipeline, HTTPExtractor, FileLoader, Rename, Filter, AddField\n",
        "\n",
        "# Build pipeline\n",
        "pipeline = (\n",
        "    Pipeline(HTTPExtractor(url=\"https://jsonplaceholder.typicode.com/users\"))\n",
        "    | Rename({\"username\": \"user_name\"})  # Rename fields\n",
        "    | Filter(lambda r: r.get(\"id\", 0) <= 5)  # Keep first 5\n",
        "    | AddField(\"source\", \"api\")  # Add source field\n",
        "    | FileLoader(path=\"/tmp/users.json\", file_format=\"json\")\n",
        ")\n",
        "\n",
        "print(\"Pipeline defined!\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Run the pipeline\n",
        "result = asyncio.run(pipeline.run())\n",
        "\n",
        "print(f\"Success: {result.success}\")\n",
        "print(f\"Rows extracted: {result.rows_extracted}\")\n",
        "print(f\"Rows loaded: {result.rows_loaded}\")\n",
        "print(f\"Duration: {result.duration_seconds:.2f}s\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# View the output\n",
        "import json\n",
        "\n",
        "with open(\"/tmp/users.json\") as f:\n",
        "    data = json.load(f)\n",
        "\n",
        "print(f\"Loaded {len(data)} users\")\n",
        "print(json.dumps(data[0], indent=2))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Part 4: Quality Monitoring\n",
        "\n",
        "Check data quality and track metrics."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from pycharter import QualityCheck, QualityCheckOptions, QualityThresholds\n",
        "\n",
        "# Use contract (schema) directly - no store required\n",
        "contract = {\"schema\": schema}\n",
        "\n",
        "thresholds = QualityThresholds(\n",
        "    min_overall_score=80.0,\n",
        "    max_violation_rate=0.25,\n",
        ")\n",
        "\n",
        "check = QualityCheck()\n",
        "report = check.run(\n",
        "    contract=contract,\n",
        "    data=users,\n",
        "    options=QualityCheckOptions(\n",
        "        calculate_metrics=True,\n",
        "        check_thresholds=True,\n",
        "        thresholds=thresholds,\n",
        "    ),\n",
        ")\n",
        "\n",
        "print(f\"Quality Score: {report.quality_score.overall_score:.1f}/100\")\n",
        "print(f\"Violation Rate: {report.quality_score.violation_rate:.1%}\")\n",
        "print(f\"Records: {report.record_count} (valid: {report.valid_count}, invalid: {report.invalid_count})\")\n",
        "print(f\"Passed Thresholds: {report.passed}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Summary\n",
        "\n",
        "In this notebook, you learned:\n",
        "\n",
        "1. **Data Validation**: Define schemas with coercion and validation rules\n",
        "2. **Validator Class**: Efficiently validate batches of data\n",
        "3. **ETL Pipelines**: Build pipelines with the `|` operator\n",
        "4. **Quality Monitoring**: Check data quality with thresholds\n",
        "\n",
        "## Next Steps\n",
        "\n",
        "- [Full Documentation](https://optophi.github.io/pycharter/)\n",
        "- [ETL Pipelines Tutorial](https://optophi.github.io/pycharter/tutorials/etl-pipelines/)\n",
        "- [Contracts Tutorial](https://optophi.github.io/pycharter/tutorials/contracts-validation/)"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.0"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 4
}