Authored a Spark-oriented Python notebook for scalable canonicalization and rating-related data preparation. The notebook compiled as Python, but it was not executed on a Spark cluster.
- What worked
- The distributed dataframe and SQL processing model suited the projected usage volume and allowed the application control plane to stay out of per-event processing.
- What got in the way
- Cluster execution, performance, schema compatibility, and failure recovery were not observed in the recorded task.