Problem

University data for tuition, deadlines, English requirements, scholarships, rankings, and program details is scattered across official pages, making manual entry slow, inconsistent, and difficult to audit.

Pipeline

• Scale from a 50-university pilot to a 150-university batch across 22 countries
• Crawl only approved official source pages and store evidence with source URL, content hash, and parser version
• Classify evidence quality before extraction so low-quality pages do not produce product facts
• Extract auditable university facts linked back to `evidence_id` and `source_url`
• Generate product profiles, program rows, matching tags, writer context, QA reports, and import-shaped CSVs
• Normalize and match QS ranking rows for downstream profile enrichment

Data Quality

• 100% of extracted facts linked to evidence and source URL for auditability
• Quality gates prevent facts from being generated from weak or unapproved evidence
• Field gap reports identify missing deadlines, application systems, and tuition ranges
• Retry source maps and source repair workflows isolate batch blockers
• QS ranking normalization and matching improved ranking coverage for 147 of 150 universities
• 38 unit tests cover validation, crawling, extraction, ranking matching, profile generation, repair, and export logic

Problem

Market analysts need a repeatable way to track competitor prices, ratings, product categories, and pipeline quality over time instead of collecting product data manually.

Pipeline

• Run the `ecommerce_market_etl` Airflow DAG daily at 00:00 UTC
• Extract product data with retry, timeout handling, pagination support, and raw JSON staging
• Validate schema drift and transform product rows with Pandas and Pandera
• UPSERT clean daily snapshots into PostgreSQL and store rejected rows with reason codes
• Expose checked-in SQL queries for Metabase product and pipeline dashboards

Data Quality

• Schema drift validation before transformation
• Critical-field rejection for missing product IDs and invalid prices
• Rating clipping and category normalization
• Idempotent UPSERT on product, source, and extraction date
• Pipeline audit rows with row counts, status, and data quality score

Problem

Formula 1 data is fragmented across historical APIs, session endpoints, and telemetry sources, making reliable race performance and strategy analysis difficult to reproduce.

Pipeline

• Use Airflow to ingest historical, session, lap, stint, weather, position, and selected telemetry data
• Load raw API responses into a Bronze layer on MinIO/S3 with source and race/session partitions
• Transform Bronze data into typed, deduplicated Silver Parquet datasets with PySpark
• Build Gold analytical marts with dbt for race overview, driver performance, pit stops, tyre strategy, and weather impact
• Serve Gold datasets through DuckDB and Metabase dashboards

Data Quality

• Raw Bronze preservation for auditability and reprocessing
• Canonical identifier mapping across OpenF1 and Jolpica-F1
• Silver schema checks for race, driver, lap, weather, telemetry, and result datasets
• dbt tests for not-null keys, uniqueness, accepted values, and metric ranges
• Airflow logs for source, parameters, output paths, record counts, and errors

Problem

Risk teams need trusted, low-latency transaction data to detect fraud patterns, audit historical activity, and serve analytics without moving data between systems.

Pipeline

• Generate synthetic users, accounts, and transaction inserts in PostgreSQL
• Capture inserts and updates with Debezium CDC and publish them to Kafka topics
• Evaluate fraud rules in PySpark Structured Streaming and write enriched events to Bronze Iceberg tables
• Use Airflow and dbt to build Silver and Gold models for analyst-ready fraud and account metrics

Data Quality

• CDC completeness checks against source transaction counts
• Great Expectations validation on Bronze and Silver promotion gates
• dbt tests for required keys, accepted values, and model relationships
• SLO checks for 5-second fraud latency and 15-minute Gold freshness