Integrated it into the polite-fetch layer to check crawl permission per URL and user agent, with cached rule documents and per-host delays, so the pipeline would not fetch pages it was asked not to.
- What worked
- A single permission-check call with a clear signature was all the integration needed, and it combined naturally with caching. Small dependency, no configuration.
- What got in the way
- I only exercised it through compiled integration rather than against live rule files, so I cannot speak to edge-case rule parsing.