Submit assignment for ticket reservation system

[Poorva28]

Summary

This PR adds a small CDC lakehouse solution for a ticketing reservations system. The goal is to build a simple, reliable CDC pipeline that supports replay safety, schema evolution, historical tracking, current-state reporting and data validation.

What’s Included

• Source schema definitions in sql/source/postgres_schema.sql and source/models.py
• CDC event simulation in pipeline/cdc.py
• Raw CDC event contract in sql/spark/raw_s3_event_contract.sql
• Spark streaming pseudocode for S3-to-Iceberg ingestion in sql/spark/structured_streaming_pseudocode.py
• Iceberg silver table definitions in sql/spark/iceberg_tables.sql
• Local lake writer in pipeline/lake.py
• Warehouse current/history model in pipeline/warehouse.py
• Snowflake table, merge, refresh and quality-check SQL in sql/snowflake/
• Schema compatibility checks in pipeline/schema_contracts.py
• Catalog metadata in catalog/catalog.json
• Local validation guide in LOCAL_VALIDATION.md
• Solution/design explanation in SOLUTION.md

Source Schema Design

The source schema models a ticketing and reservations workflow.

Tables:

• venues
• events
• customers
• seats
• reservations
• reservation_items
• payments
• tickets

Strong entities:

• venues: independent place where events occur
• events: independently scheduled inventory container
• customers: independent buyer identity
• reservations: main transaction root connecting customer, event, status, timing and money

Weak entities:

• seats: dependent on a venue for real-world identity
• reservation_items: dependent on a reservation and seat
• payments: dependent on a reservation lifecycle
• tickets: fulfillment records tied to reservation items

Keys and relationships:

• venues.venue_id is the parent key for events and seats
• events.event_id is referenced by reservations and tickets
• customers.customer_id is referenced by reservations and tickets
• reservations.reservation_id is referenced by reservation_items and payments
• reservation_items.reservation_item_id is referenced by tickets
• reservation_items enforces uniqueness on (reservation_id, seat_id)
• tickets enforces one ticket per reservation item
• payments enforces unique provider references

Indexes:

• event lookup by venue/start time
• seat lookup by venue/status
• reservation lookup by customer/created time
• reservation lookup by event/status
• item lookup by reservation
• payment lookup by reservation/status
• ticket lookup by event/status

Validation rules:

• event end time must be after start time
• held reservations require hold_expires_at
• amounts and fees must be non-negative
• payment capture cannot predate authorization
• ticket use cannot predate issuance
• uniqueness constraints prevent duplicate seats, provider references and ticket codes

Architecture

The design follows this flow:

PostgreSQL
  -> Debezium CDC / Kafka
  -> raw Kafka event files in S3
  -> Spark Structured Streaming every 5 minutes
  -> Iceberg silver tables
  -> Snowflake current and history warehouse tables

CDC Strategy

Changes are captured as Debezium-style CDC events with:

• operation code: create, update, delete, snapshot
• before image
• after image
• source table
• primary key
• source LSN
• transaction ID
• Kafka topic, partition and offset
• schema ID
• event timestamp

Replay/restart behavior:

• the Spark design reads raw CDC files from S3 on a 5-minute trigger
• checkpointing resumes from the last successful micro-batch
• local tests simulate checkpoint replay with offset boundaries
• downstream writes are idempotent

Duplicate handling:

• each CDC event has a deterministic event_id
• lake writes deduplicate by event_id
• replaying the same events does not create duplicate lake rows
• warehouse history also ignores duplicate event_id values

Delete handling:

• delete events carry a before image
• the lake stores delete events as part of the immutable CDC history
• the warehouse current table represents deletes as soft deletes with is_deleted = true
• history retains deleted versions for audit and restore reasoning

Lake and Warehouse Modeling

Lake modeling:

• the lake keeps every unique CDC event in an append-oriented silver table
• inserts, updates, deletes, source LSNs, offsets, timestamps, before images and after images are retained
• this creates a durable historical event log for replay and audit

Warehouse modeling:

• warehouse current tables expose latest known state by business key
• current-state merges only accept newer source_lsn values
• stale or out-of-order events cannot overwrite newer warehouse records
• deletes are represented as soft-deleted current records

Time travel / restore:

• warehouse history stores versioned records with valid_from, valid_to, is_current and is_deleted
• point-in-time restore is supported by querying history for records valid at a requested timestamp
• tests verify that prior reservation state can be reconstructed after later updates

Schema Change Safety

Incompatible changes are detected by comparing incoming/event schemas against the registered source contract.

Detected incompatible changes include:

• dropped required columns
• type changes
• unknown source tables
• key changes
• enum/domain contractions

Ingestion stop behavior:

• incompatible schema changes raise an IncompatibleSchemaChange
• the design stops before downstream lake or warehouse writes
• the solution documents failure surfacing through ops.schema_failures

Failure surfacing:

• local tests verify dropped columns, type changes and unknown tables
• schema validation scripts provide a repeatable local check
• failures are explicit instead of silently allowing drift

Validation Parity

Failure checking:

• tests/test_data_quality.py verifies reservation total mismatches are detected
• scripts/run_data_quality_checks.py provides a local validation script
• schema contract tests verify source compatibility assumptions
• lake/warehouse tests verify replay, delete, latest-state and restore behavior

Catalog Exposure

Dataset metadata is published in catalog/catalog.json.

The catalog includes:

• raw CDC event contract
• silver CDC event table
• silver current/history tables
• Snowflake warehouse current tables
• Snowflake warehouse history table
• schema failure records

Each catalog entry includes metadata such as:

• owner
• layer
• cadence
• description
• keys

Local Setup

cd submission/ticket-reservations
python -m pip install -r requirements.txt

Run Validations

pytest
python scripts/check_schema_contracts.py
python scripts/validate_catalog.py
python scripts/run_data_quality_checks.py

The local test harness uses DuckDB to simulate the critical behavior without requiring live AWS, Spark, Iceberg, Kafka or Snowflake infrastructure.

Assumptions

• Kafka CDC events have already been landed in S3 before Spark reads them
• Debezium-style events contain operation type, before/after images, source LSN, Kafka offset and schema ID
• source_lsn is the primary ordering signal for warehouse current-state merges
• Duplicate delivery is possible and must be ignored by stable event_id
• Breaking schema changes should stop ingestion before downstream writes

Tradeoffs and Limitations

• Spark, Iceberg, S3 and Snowflake code is represented as SQL/pseudocode plus local simulations, not a fully deployed runtime
• Local tests prove behavior using DuckDB rather than actual production services

Responsible AI Usage

AI helped draft the first version and organize the structure. The final design, naming and failure behavior were reviewed and simplified to match the requested pipeline shape.