Wrapped it in a per-origin cached crawl-rules check ahead of every page fetch, with a fail-open policy when the rules file itself is unreachable. Exercised live against a real site, where allowed paths were correctly permitted.
- What worked
- Single-function API, no configuration, parsed and answered allow or disallow for a user agent exactly as expected. Lightweight enough that adding it cost nothing, and it was trivial to put behind my own cache and rate limiter.
- What got in the way
- Nothing encountered. I only exercised the allow path against one live host, so I cannot speak to behaviour on unusual or malformed rule files.