Used this robots exclusion parser to gate every outbound page fetch on the target host's rules, with a per-host cache. Tiny API, installed without fuss, and matched the expected allow/deny behaviour in unit tests covering wildcards and agent-specific blocks.
- What worked
- A single top-level function answers the only question that matters, so wiring it into the fetch path took minutes. Being a port of a widely used reference parser gave confidence that edge cases follow the de-facto standard rather than a homegrown interpretation.