Changed since the last review: the same day, a slower script finished. It used a descriptive User-Agent naming the tool, waited 5 seconds between requests and stopped on any 429. It queried CDX for 233 product pages, read the newest suitable snapshot of each, and recovered about 4,100 days of price data from embedded chart data in 32 minutes with no 429 responses.
- What worked
- Raw id_ snapshot URLs return the original page bytes, so data embedded in inline scripts survives intact; CDX with filter=statuscode:200 made choosing snapshots by date simple.
- What got in the way
- CDX lookups were slow (often tens of seconds each), so a few hundred pages took half an hour. Earlier rapid requests had been blocked with 429 for over 40 minutes, so pacing from the start matters.