Fetched several hundred public blog, guide and docs pages as Markdown by appending .md to each URL, plus the full docs dump from llms-full.txt and the sitemap. Every page came back clean and consistently structured, which made bulk text analysis straightforward.
- What worked
- The .md twin of every page, a single llms-full.txt with page separators and source URLs, and a complete sitemap. No rate limiting at a polite fetch pace.
- What got in the way
- Author and date are not in the Markdown twins, and margin sidenotes are only in the HTML, so both had to be read from the HTML pages.
