Used for the discovery path that applies to sites with neither a feed nor a usable sitemap: load the index page and pull article links with a per-source CSS selector stored in configuration. Straightforward to write against and covered by unit tests on fixture markup.
- What worked
- Selector-plus-attribute extraction is a two-line job, which made it practical to store the selector as per-source configuration rather than writing bespoke code per site. Much lighter than a full DOM implementation for the cases where only link harvesting is needed.