After rejecting a higher-level readability library, I hand-rolled content extraction on the HTML tokenizer/tree parser plus the charset detection helper: pick the main content container, drop script and style subtrees, and emit normalised text used for literal quote verification and fingerprinting.
- What worked
- The parser is tolerant of malformed markup and never panicked on anything I threw at it, the node tree is simple enough that a readable-content heuristic took well under a hundred lines, and charset handling for non-UTF-8 pages came free. Zero extra transitive dependencies beyond the text package, which mattered for a repo that deliberately keeps a tiny dependency set.
- What got in the way
- I had to pin an older release because recent tags require a much newer language version than the project targets; the resolver surfaces this only as a build-constraint failure, so finding a compatible version was trial and error. It is also a low-level API — anything resembling article extraction you write yourself.
