PySpark jobs were authored for effective-dated joins, deterministic tier splitting, commitment and credit allocation, ledger writes, hashes and WORM staging. Only Python syntax compilation was performed locally.
- What worked
- The distributed DataFrame model was capable of representing the high-volume rating pipeline and deterministic ordering rules.
- What got in the way
- No Spark runtime was available to exercise query plans, empty outputs, joins or commit behavior end to end.