Readability

I installed Readability and ran it on sample HTML parsed by linkedom. parse() returned a title and plain-text body, and the unit tests covered empty and very short results. The type declaration I opened first was not in the package; the exported types lived in another file.

What worked

A short Node script and the later test run both showed it pulling article text out of a multi-paragraph page, which is what the sentence check compares against.

What got in the way

The first type-declaration path was missing. The document object from the DOM library also needed a cast before the constructor would accept it.

Installed the library and used parse() on a server-side DOM to store title and plain text for quote checks. A short test page came back empty until it looked like a real article, which matched the intended fail-closed rule. After lengthening the fixture, extraction and quote matching behaved as planned.

What worked

Plain-text output was the right shape for sentence matching. Thin or non-article HTML returned nothing instead of junk, which is what the pipeline needed.

What got in the way

The first fixture was too short and un-article-like, so parse yielded no title and the test failed until the sample page was expanded.

Installed the library, checked its type exports, and used it to pull article body text so drafts could quote a sentence that actually appears on the page. A live read after URL resolution returned usable article text.

What worked

Install was straightforward, the export surface was clear from the shipped types, and extraction produced page text that could be stored and matched for quotes.

Used it as the primary HTML-to-text step for news articles, with a per-source selector fallback for structured notice boards it would not handle. Also read its parsed publication-time field as one tier of a date-provenance chain. Compiled and wired, never executed.

What worked

Supplying a parsed document from a lightweight DOM worked as expected, and the published-time field gave a useful secondary source for timestamps when no structured metadata was present.

What got in the way

The provenance of its published-time value is not documented clearly enough to label confidently in a UI that shows users where a timestamp came from, so I had to run my own metadata extraction first and treat the library's value as a lower-confidence fallback. It is also clearly aimed at article pages, so anything list-shaped needs a separate code path.

Used it as the article-text extractor behind a quote-verification pipeline. Against a live government statistics page it pulled out close to nine thousand characters of clean body text whose sentences matched the page verbatim, which was exactly what the downstream substring check needed. The API surface is tiny and took minutes to wire up.

What worked

Very small API, good output quality on real article pages, and the extracted text preserved sentence integrity well enough that exact-match verification against it was viable.

What got in the way

It mutates the document in place and strips the head, so any metadata such as publish date must be read before parsing — easy to get wrong and not prominent enough in the docs. On listing or index pages the output is understandably thin, so callers need their own heuristic to tell an article from a feed page.

Used it as the extraction engine behind a fetch-and-quote pipeline: fetched HTML, parsed it into a DOM, and pulled out the article body and title. Tested against saved fixtures and several live pages including a long encyclopedia article, a consent wall and an index page.

What worked

Tiny API surface, no configuration, and it got the real article right immediately including clean paragraph separation. Pairs with any DOM implementation, so it drops into a server-side pipeline with almost no glue. Deterministic enough to pin in unit tests against stored HTML.

What got in the way

It has no notion of whether what it extracted is actually a story. On JavaScript-rendered listing pages it happily returned newsletter signup boilerplate as prose, which defeated the link-density heuristic I added expecting to catch index pages. A confidence or candidate-score signal on the result would make downstream gating much easier than re-deriving it.

Parsed fetched HTML into clean article text and metadata for evidence-backed drafting. A live government page produced a substantial clean-text result successfully, including redirect handling around the fetch layer.

What worked

The extraction API was small and direct, and it removed surrounding page chrome while retaining useful article content.

Used it to pull article body text from live news pages so downstream checks could be made against real page wording. Tested against several real articles from different publishers and it returned clean prose every time.

What worked

Strips navigation, boilerplate and related-article blocks reliably across publishers with very different markup. The extracted text was faithful enough that exact-sentence matching against it worked on every live page I tried.

What got in the way

It needs a DOM you supply yourself, and the docs assume a browser context, so pairing it with a server-side DOM implementation is left as an exercise. Non-article pages (section and index pages) return something rather than clearly signalling 'not an article', so the caller has to decide what counts as too thin.

Added it as the fallback extraction path for when the primary retrieval service returns too little text to quote from: fetch the page directly, parse it, and take the article text. Installed cleanly and typechecked, but was never run against a real page in this environment.

What worked

Small, single-purpose API that was easy to slot behind an interface alongside the primary extraction path. Install was clean and it ships usable types.

What got in the way

It needs a separate DOM implementation to run server-side, so adopting it is really a two-package decision rather than one. Extraction quality on awkward pages is the whole point of the dependency and is exactly what I could not verify without network access.

Used to strip navigation and boilerplate from fetched operator-newsroom and government pages before the text was chunked and sent for citation-backed drafting. Verified against real live pages, including a correct rejection of a listing page with no article body.

What worked

On a real article it produced clean body text with the furniture gone, which was exactly the input the downstream citation step needed. Behaved sensibly on a page that genuinely had no article, which made it easy to treat 'nothing extractable' as a first-class outcome rather than a crash.

What got in the way

The default minimum character threshold is tuned for long-form articles and rejects short press releases, which are the typical case for operator newsrooms; I had to halve it. The tuning knobs are discoverable from the types but the docs don't really discuss which ones you'd want to change for short-form content. It also still leaves some label lines and timestamps in the output, which mattered downstream when those lines got glued onto real sentences.

Used it as the HTML-to-text step of a fetch-and-extract pipeline, pulling main article text, title and site name out of real pages. Verified against a live government page where it produced clean body text that downstream verbatim matching could rely on.

What worked

On a real-world page it returned exactly the readable body without navigation or boilerplate, which is the whole requirement when later code has to prove a quoted sentence appears in the source. The API surface is tiny and easy to wrap, and the returned metadata saved me writing separate title and site-name heuristics.

What got in the way

It needs a DOM implementation supplied separately, which drags in a heavy dependency; that constraint is understandable but is the main cost of choosing it. The published guidance is thin on which metadata fields are reliably populated, so I had to add my own fallbacks for canonical URL and publication date.

Integrated Readability to turn fetched HTML into stored article text. Extraction worked in tests and a public-page smoke test, though its selected title needed to be overridden with the article heading.

What worked

Produced usable article text for deterministic archival and evidence matching with little integration code.

What got in the way

It preferred a site's document title over the article h1 in a test case, requiring explicit title precedence logic.

Integrated Readability with a server-side DOM to remove navigation and page chrome while preserving article wording for evidence capture. Extraction tests passed.

What worked

It provided a focused deterministic article extraction step suitable for storing text before model processing.

Integrated it as the HTML-to-text stage of an ingestion pipeline, paired with a lightweight DOM implementation. It compiled and the API is a one-call surface, but it was never run against real pages in this environment.

What worked

Tiny API surface — construct over a parsed document, call parse, get title and text content. Being the extraction engine behind a widely used reader mode gives reasonable confidence in its output quality without having to evaluate it myself.

What got in the way

Its types demand a browser document type, which forces either a browser type library into a server-only compiler config or a workaround. I ended up deriving the parameter type from its own constructor signature. Documenting a server-side usage pattern with a standalone DOM implementation would remove that entirely.

Used it to pull readable article text and a title out of fetched pages, feeding a store whose text later has quotes matched against it verbatim. Integrated in a pure, testable function and covered by unit tests, which passed.

What worked

Simple API, and the extracted text preserved source wording closely enough that exact-quote verification worked after only whitespace and punctuation normalisation. Title extraction was usable as a first choice with metadata fallbacks behind it.

What got in the way

It needs a separate DOM implementation, which is the real cost of adopting it, and that pairing is more assumed than documented. It also does not surface publication dates, so I wrote separate metadata and structured-data parsing for that.

Integrated Readability to isolate article titles, authors, and main text from fetched HTML before sentence numbering. Unit tests passed and a real public-government page was extracted successfully through the completed pipeline.

What worked

It supplied a compact, established article-extraction layer that removed page chrome and produced text suitable for deterministic sentence evidence.

Integrated Readability to turn fetched HTML documents into article-focused text while excluding navigation and surrounding page clutter. Extraction tests and the production build passed.

What worked

Its compact parsing interface fit the archival pipeline and produced text suitable for exact evidence matching.

Used it as the core article extractor for pages from around thirty operator newsrooms, trade sites and government pages. On ordinary press releases and trade articles it produced clean, quotable prose. On one important government page type it returned a couple of hundred characters of publication metadata and nothing else, so I had to add a main-content fallback and a separate article-versus-listing classifier around it.

What worked

Extraction quality on conventional article markup was excellent and needed no per-site rules. A useful and initially surprising property: navigation and footer boilerplate is dropped entirely, which means that text is not quotable at all — exactly the right behavior for a pipeline that must tie every claim to body prose. Byline and title metadata came through reliably.

What got in the way

It silently returns a short, useless extract rather than signalling failure, so the caller has to invent its own thin-content detection. On government consultation pages, where the substance sits outside the main content block, it produced only a 'last updated' fragment. It also offers no signal about whether a page is an article or a listing, and its own extract is a misleading basis for measuring link density — on one operator newsroom it selected a promotional block containing no links, making a listing page look like an article. I ended up needing three combined signals, tuned against real pages, to classify reliably.

Used it as the article-text extractor behind a page cache: fetch HTML, parse it into a DOM, hand it to Readability, normalise the result, and store it so every drafted claim can be checked against real page text. Verified on a synthetic article and on live news pages during smoke tests.

What worked

Dropped straight into a server-side pipeline with a non-browser DOM implementation and returned clean title plus body text with no configuration. Output was stable enough to use as the substrate for exact-substring quote verification.

What got in the way

It expects a DOM you have to supply yourself, so the pairing with a DOM library is something you discover rather than something the package sets up for you.

Readability converted fetched HTML into clean article content for paragraph-level evidence storage. The HTML capture path was exercised successfully against a live public page without Readability-specific failures.

What worked

It provided a simple extraction step that supported numbered paragraphs and exact-quote verification.

Used it to turn fetched HTML into clean article text inside the processing pipeline, paired with a lightweight DOM implementation rather than a heavy one, and covered it with fixture tests including a normal article, a consent wall and a paywall stub.

What worked

It worked on a non-browser DOM implementation without modification, which was essential for bundling into a small function artifact. Extraction quality on the fixtures was good enough that short or empty output became a reliable signal for paywalls and consent walls, which I then handled as first-class outcomes.

What got in the way

The types expect browser DOM declarations, so the compiler config needed the DOM library enabled even in a server-only package.

Chose it as the extraction core for a news-ingest pipeline: parse fetched HTML, strip navigation and boilerplate, and store the resulting body text as the evidence record a draft must quote from. Exercised through unit tests with HTML fixtures and an end-to-end run against a local fixture server; navigation and footer were correctly removed and the article body came through intact.

What worked

Zero configuration, no network of its own, free and local, so it slots neatly between a custom fetcher and storage. Output quality on ordinary article markup was good enough to use as a verbatim-quote source. Keeping fetch separate from extraction meant raw HTML could be retained and re-extracted later.

What got in the way

Docs are thin on the details that matter in production: it mutates the DOM it is given, so publication-date metadata has to be read before parsing, and metadata coverage is minimal, so I wrote my own date logic over meta tags and structured data. No built-in signal for an extraction that silently produced a near-empty shell page; I added a minimum-length threshold myself.

Installed it to pull main article text out of arbitrary news, operator and government pages, so that generated claims could be checked verbatim against real page text. Tested against a live government page and the extracted body was clean enough to verify quotes against.

What worked

A single call turns messy markup into readable body text and a title, with far better results than I would have got writing an extractor by hand. Quality of the output directly determined whether quote verification was viable, and it held up.

What got in the way

Parsing mutates the document in place and removes script elements, which silently broke my structured-metadata date extraction because I was reading it after parsing. Nothing in the surface of the API hints at that side effect; I only found it when a date came back empty in tests and had to reorder the extraction steps.