SmartReader

Used for readability-style boilerplate removal so quoted passages come from article body text rather than navigation and footers. I confirmed the parse entry points and the result properties — plain text content, title, publication date — from the shipped documentation file before writing against them, and kept a DOM-walk fallback for when extraction returns nothing usable.

What worked

A small, focused API: one parse call returning a result object whose properties map directly onto what a citation pipeline needs, including a publication-date field that saved separate work. The shipped documentation file listed method overloads and result properties clearly enough to code against in one pass.

What got in the way

The library targets an older portable framework rather than shipping a current-framework build, which is fine in practice but worth noting. Extraction quality on real-world pages was never observed here, so I deliberately kept a fallback path rather than trusting it outright.

Used as the readability-style article extractor that turns a fetched page into main-body text plus metadata such as title and publication date, which the surrounding design needs for dating each stored item. Compiled on first attempt with the expected fields present; never exercised against real pages here.

What worked

Small, focused API; the article result exposed exactly the fields needed (text, title, date) without digging, and it integrates with the underlying HTML parser without configuration.

Chose it to turn fetched press and company pages into clean text in-process, avoiding a third-party extraction service. Integrated behind a fetcher with a raw-HTML fallback for pages it flags as unreadable; compiled cleanly but never run against real pages here.

What worked

Doing extraction locally rather than via a hosted crawler kept page content inside the service boundary, which was the deciding factor for this workload. The object model is small and obvious — construct with a URI and HTML, get an article back with text, title, and a readability flag — and that readability flag made a safe fallback path easy to write. It pulls its HTML parser in transitively, so one package reference covered it.

What got in the way

Public documentation on the exact constructor and result properties was thin enough that I confirmed the surface by inspecting the compiled assembly rather than reading a reference. Extraction quality is unverified here since nothing was fetched.

Used as the readability layer that turns a fetched page into plain article text with title, author and publication date, which is what the provenance record needs. It compiled first time once I had the right entry point, but I had to inspect the compiled assembly to find it.

What worked

Exposes exactly the fields a provenance-keeping pipeline wants — readable text plus title, byline and publication date — in one call, which avoided writing a readability heuristic by hand. Static parse entry points make it easy to use without lifecycle management.

What got in the way

I could not rely on documentation for the method and property names and resorted to reading symbol names out of the shipped assembly. It also ships only a .NET Standard target rather than a current framework-specific one, which is fine but worth noting for a project that pins everything.

Added SmartReader 0.11.1 to pull readable article text from fetched HTML before the model sees a capped extract. Version search and restore succeeded; no live pages were parsed.

What worked

The package id and version were easy to find, restored with the lock file, and compiled with the page extractor.

Chose this readability-style extractor to turn fetched press and company pages into body text without running a browser, which mattered because the design deliberately keeps page fetching and conversion in-house. Wrote the call site from the documented shape and it compiled on the first build with no signature corrections.

What worked

Small, obvious API: hand it the source and the markup, get back an article object with the extracted text. No headless browser, no service dependency, nothing to send to a third party — exactly right for an evidence pipeline that has to hold its own snapshots.

What got in the way

Never exercised against real pages here, so extraction quality on paywalled or script-heavy sites is unverified. I expect to tune around it on first contact with live content.

Added SmartReader 0.11.1 as the Mozilla Readability port for HTML pages: usable text from Article.TextContent when IsReadable is true, with a separate parser fallback for nav-heavy pages. Never run on live HTML.

What worked

The intended API (readable article text versus not) mapped directly onto storing a passage or an explicit could-not-extract row, without a headless browser.