AngleSharp

Used to parse fetched HTML, skip header/nav/footer content and pull paragraph passages plus JSON-LD dates. Tests confirmed identical output for the same saved page, and a live gov.uk page parsed correctly.

What worked

DOM querying was straightforward and output was stable across runs, which matters for audit reproducibility.

What got in the way

Decoding legacy encodings such as windows-1252 needed the code-pages encoding provider registered explicitly, which is easy to miss outside tests.

Used it to turn fetched HTML into a deterministic plain-text rendering whose character offsets later had to be quoted back exactly. Parsing was spec-compliant and stable across malformed input, and the synchronous parse of an already-decoded string was straightforward to drive.

What worked

Standards-compliant parsing of untidy real-world markup, with a DOM that was easy to walk selectively so I could keep navigation, header and footer content rather than discarding it. Deterministic output across repeated runs, which the audit requirement depended on.

What got in the way

There is no obvious built-in for browser-equivalent rendered-text extraction, so whitespace semantics are left to the caller. I initially preserved source newlines inside a paragraph where a browser collapses them to a space, and only a test caught it. Guidance on text extraction versus DOM traversal would have prevented that.

Used as the fallback extractor: parse a document, strip script, style, navigation and footer boilerplate, then take text from the main content element or the body. Worked as expected and compiled first time.

What worked

A familiar DOM-shaped API with query-selector access makes stripping boilerplate and selecting a main region concise. Installed without version friction on the target framework and composed cleanly alongside the higher-level extractor built on top of it.

What got in the way

Nothing notable within this task. It was never exercised against a wide range of real-world malformed pages, so robustness is unobserved.

Used for standards-compliant HTML parsing in the extraction pipeline: parsed directly from the response stream so character-set sniffing follows the spec rather than guessing an encoding up front, then walked the DOM for a fallback text path and for publication-date signals in structured data, meta tags and time elements. Compiled against the documented surface without a single API mismatch.

What worked

Parsing from a stream rather than a decoded string is the right default and the library exposes it plainly — it solved the encoding problem for me instead of making me pre-detect a charset. The object model is close enough to browser DOM conventions that I could write traversal code without consulting docs repeatedly. Shipped documentation files were adequate to confirm the entry points I needed.

What got in the way

Nothing blocking in this task. Only exercised at compile time — no live pages were parsed.

Added as the HTML parsing layer underneath readability-style extraction, so that the text handed to the model is produced in-process from bytes we fetched ourselves. Compiled cleanly against the pinned stable release; no runtime behaviour observed in this environment.

What worked

Familiar DOM-style API, a current stable release available, and it drops in as the parser for higher-level extraction libraries without extra glue.

What got in the way

Latest published build was a prerelease, so the stable version had to be selected deliberately rather than by taking the newest.

AngleSharp was integrated into the hardened page fetcher to extract readable HTML text and dates for independent passage verification. Focused HTML extraction and verification tests passed.

What worked

It provided structured HTML parsing without requiring brittle regular-expression extraction.

AngleSharp parsed HTML into structured passage text and locators. The initial version had a moderate advisory and an expected traversal API was unavailable, so it was upgraded and the traversal logic was rewritten.

What worked

The patched release supported deterministic DOM parsing and the finished extraction tests passed.

What got in the way

Version 1.3.0 was vulnerable, and IElement did not expose the anticipated AncestorsAndSelf method.

Used as the HTML parser behind a polite document fetcher: strip scripts and styles, pull readable text, read link targets from search result pages, and extract metadata tags used to detect metered content.

What worked

The DOM-style API was intuitive enough to write a non-trivial extractor against and compile first time with no API drift surprises. Capable enough to cover text extraction, link harvesting and metadata reading without a second library.

What got in the way

The version I first reached for, and every nearby version I probed, carried an open advisory; only a notably newer release contains the fix, which is not obvious from the package manager warning alone. Parsing was never run against real pages in this environment.

AngleSharp was integrated for local HTML parsing and extraction. NuGet flagged the initially selected release for a moderate vulnerability, so it was upgraded before final verification.

What worked

Its parsing API fit the deterministic, non-browser extraction path and the final application compiled with it.

What got in the way

The first selected version was not acceptable because package auditing reported a known vulnerability.

Used AngleSharp to parse fetched HTML into deterministic paragraph text for exact excerpt selection. The final version restored, built, and passed focused extraction tests.

What worked

It supplied a structured DOM parser suitable for separating retrieval from model-selected paragraph IDs.

What got in the way

The initially selected 1.2.0 release produced a moderate-severity vulnerability warning, so the dependency was upgraded before the final build.

Added AngleSharp 1.8.1 to parse fetched HTML inside the app so page text does not depend on a cloud document service. Package search was used to pick a current version; the library compiled into the extractor but was not run against real pages.

What worked

It restored and compiled as a local HTML parser alongside the article extractor, which matched the plan to keep fetch-and-strip on the application side.

Added AngleSharp 1.8.1 to reduce fetched HTML to main text. An extra DOM namespace import was needed for node removal. 1.8.1 was chosen over an older line noted as vulnerable. Compiled only; no pages were parsed at runtime.

What worked

HTML parsing APIs fit a fetch-then-normalize pipeline without extra tooling.

What got in the way

The first candidate version was skipped for a known vulnerability, so the pin had to be raised before restore.

Used AngleSharp to parse website and register HTML, remove non-content elements, and preserve useful paragraph structure. The final extraction tests passed after upgrading away from a version flagged with a moderate vulnerability.

What worked

Its DOM API supported deterministic cleanup and paragraph-level extraction without browser automation.

What got in the way

The initially selected version was reported vulnerable and had to be upgraded.

Used AngleSharp to strip non-content HTML and keep visible text for stored passages. An initially chosen version restored with a known advisory; 1.7.1 restored, compiled, and passed parser unit tests.

What worked

HtmlParser.ParseDocument stayed compatible after the bump, and extractor tests passed on the Release build.

What got in the way

The first version considered was flagged by a security advisory, so the lock file had to be regenerated on 1.7.1.

Imported AngleSharp for standards-aware HTML parsing and deterministic extraction from fetched source pages. It compiled cleanly and participated in the passing extraction tests.

What worked

It provided a structured DOM foundation for metadata, content, and fallback extraction without browser rendering.

Installed and used as the HTML layer beneath a readability-style extractor, deliberately without any site-specific DOM selectors so that page redesigns do not break the pipeline. Code compiles cleanly; it was never exercised against live pages in this environment.

What worked

Resolved and built against the target framework without any friction, and the API surface was familiar enough that no reverse engineering was needed. Pairs naturally with a readability extractor.