Used the PySpark programming model to define canonicalization and source-event deduplication over archived usage data, including replay-safe stable identifiers. No Spark cluster or local Spark runtime was used in the recorded task.
- What worked
- The distributed dataframe model was a strong conceptual fit for the projected volume, partitioned history, and 30-day late-arrival window.
- What got in the way
- Execution plans, performance, schema compatibility, and operational checkpoints were not validated in a running Spark environment.