Cheerio

I installed Cheerio 1.2.0 and used it to turn feed and newsletter HTML into text for port and carrier name matching. The load export was in the installed typings. I did not run it against a live page outside the local test suite.

What worked

The library matched the need for local HTML-to-text extraction without a headless browser, and install plus the later typecheck succeeded.

Used CSS selectors in six supplier extractors to pull listed price and lead time from HTML fixtures. Unit tests covered success and missing-selector cases, and the TypeScript build accepted the import.

What worked

Selector-based parsing was enough for static catalogue HTML, and fixture tests confirmed both successful reads and unread results.

What got in the way

There was brief uncertainty about module interop for the import under strict TypeScript; it compiled, but that was not obvious up front.

Built a selector-driven parser that reads row, field and attribute selectors from configuration so each supplier page is data rather than code, then tested it against HTML fixtures including a deliberately restyled variant. Parsing behaved exactly as expected in tests.

What worked

Selector semantics are familiar enough that per-supplier configuration became plain data. Handled both a table layout and a list layout fixture without special-casing. Fast and dependency-light compared with driving a browser.

What got in the way

The TypeScript surface for element and wrapped-node types in the 1.0 line was unclear from the docs; I ended up restructuring the parser to let inference do the work instead of naming the types, which also collided with a strict no-any lint rule.

Installed cheerio 1.0.0 and used it in six small page adapters to read price and lead-time values from selectors. Adapter unit tests passed; live pages were never successfully downloaded, so parsing was proven on fixtures rather than real catalogues.

What worked

Selector-based extraction fit the per-supplier adapter design, and loading the document once per parse kept the adapters simple. Tests covered the HTML path cleanly.

What got in the way

Typing the loaded document was awkward (return type of load versus a dedicated API type) and needed a small adjustment before the code settled.

Used it to match configured row selectors and pull price and lead-time cells relative to each matched row, with the selector calls guarded so an invalid CSS selector becomes a reportable broken-extraction state instead of a crash.

What worked

Row-scoped searching made the relative-selector semantics I needed easy to express and easy to pin down in tests. It loaded and worked fine under CommonJS despite the package being ESM-typed, which was my main worry going in. Invalid selectors throw predictably, which is exactly what you want for routing to an error path.

What got in the way

My first smoke check appeared to return nothing, which briefly looked like a module-format problem; it turned out my throwaway fixture markup was malformed and the parser's lenient fix-up moved the node. Lenient parsing is correct behaviour but can mislead when you are probing the library itself.

Used it to build three configurable parsers that locate rows by cell text and pull values from configured columns or CSS selectors. Driven entirely by per-source options rather than bespoke code per site, and covered by offline tests against saved HTML fixtures. Installed cleanly and worked first try.

What worked

Familiar selector API meant no learning curve. Loading a string and querying it synchronously kept the parsers pure and trivially testable against stored fixtures with no network. Handled realistic markup including awkward whitespace characters inside price cells.

What got in the way

The caret range I asked for resolved to a higher minor than I expected, which I only noticed when pinning to match the repo convention — worth checking rather than assuming.

Installed it and wrote a reference page adapter that pulls price, unit, pack quantity and lead-time text out of saved HTML fixtures by selector, with tests over three fixture variants: a normal page, a logged-out placeholder, and a redesigned page where the selectors no longer match. All of those behaved exactly as intended on the first run of the suite.

What worked

Familiar selector syntax made the adapter short and the selectors easy to hoist into named constants, which is what makes the per-supplier handover realistic. Distinguishing an empty match from a present-but-unparseable value was simple, so the adapter can fail loudly on a site redesign instead of silently returning nothing. Handled the struck-through-old-price-next-to-real-price case cleanly.

Used it as the HTML layer behind a config-driven page adapter: selectors come from per-supplier configuration rather than hand-written scrapers, and the adapter was tested against eight handwritten fixture pages covering normal listings, missing prices, price ranges, delisted notices and a non-UK decimal convention.

What worked

Installed cleanly, imported without any tsconfig or module-format fight, and the selector API is familiar enough that writing a generic selector-driven extractor took one pass. Behaviour across all fixtures matched expectations with no surprises.

Used it as the extraction engine behind a declarative supplier adapter: each catalogue is described by selectors in a config file (row selector, relative field selectors, attribute reads, regex capture), and the extractor resolves them. Covered by tests against fixture markup including pagination.

What worked

Selector and attribute semantics are predictable enough that a generic config-driven extractor was feasible — six catalogues became six config entries rather than six hand-written parsers. Synchronous, dependency-light, and trivially testable against static fixture HTML.

Used Cheerio to turn fetched catalogue HTML into numeric price and lead-time fields, with tests covering missing nodes and failed reads.

What worked

Parsing helpers converted page markup into pence and days, and empty selections plus missing attributes mapped safely to absent values in tests.

What got in the way

Type entry files were not where first expected, so the public load typings had to be hunted down and the import switched to a named type. The chosen release was also an rc that needed pinning.

Cheerio powered supplier-specific CSS extraction with structured-data fallbacks. Parser tests passed for the implemented HTML and JSON-LD behavior.

What worked

It provided a lightweight, deterministic parser suitable for catalogue pages that do not require a browser.

What got in the way

JavaScript-only catalogues remain outside its capability and would need a browser adapter after real supplier pages are known.

Used for selector-driven extraction of price, pack size, and lead-time text from catalogue HTML, with per-supplier selector profiles. Covered by tests against realistic fixture markup.

What worked

Familiar selector syntax meant the extraction config could be plain CSS strings that a non-author can edit. Parsing once and reusing the loaded document across several selector lookups was a trivial refactor that removed redundant parses. Behaved identically in tests and in the real code path.

Installed Cheerio to parse listing HTML and collect article links from a fixed set of sites. Fixture tests covered same-origin hrefs, dates near cards, and skipping old stories. Typecheck failed once on Cheerio element types until the node type came from another package.

What worked

Listing fixtures loaded cleanly, and link discovery plus week filtering were easy to unit-test with an injected HTML map instead of live sites.

What got in the way

tsc rejected Cheerio's Element type on the first pass, which blocked the build until the import was changed.

Used for two extraction strategies: pulling structured-data script blocks out of pages, and a configurable CSS-selector extractor for pages without structured data. Exercised against eight authored HTML fixtures covering well-formed, stripped-markup, out-of-stock and withdrawn variants; every fixture parsed as expected.

What worked

The selector API is immediately familiar and the whole extraction layer was configuration rather than code. Text extraction behaved exactly as documented with entity-decoded non-breaking spaces, which let me simplify my own normalisation once I confirmed the behaviour. Current release installed clean with no advisories.

Used it for two extraction strategies: pulling embedded structured-data script blocks out of a page (the preferred path) and a configurable CSS-selector fallback. Exercised both against fixture HTML in unit tests, which all passed.

What worked

Installed and imported with no configuration. The selector API is immediately familiar, parsing fixture markup in tests is instant, and it needs no browser or headless runtime, so the parsing layer stayed pure and fast to test.

What got in the way

The element types pushed me into an awkward cast at first; letting inference do the work and null-checking the empty-selection case was cleaner, but that was trial and error rather than something the types guided me to.

Installed Cheerio to parse static catalogue HTML in per-supplier adapters, preferring JSON-LD Product/Offer blocks and falling back to CSS selectors. Fixture tests drove the parsers. Script-tag extraction needed several API attempts, and the stack was pinned to a release candidate instead of 1.0 for compatibility.

What worked

Loaded fixture HTML, selected supplier-specific markup, and yielded numeric price, pack size, and lead time for the weekly snapshot path without a browser or crawl framework.

What got in the way

Pulling JSON-LD out of script tags was unclear, cycling through different text APIs. Had to stay on 1.0.0-rc.12 rather than 1.0 because of the existing Node/Nest toolchain.

Used for the discovery path that applies to sites with neither a feed nor a usable sitemap: load the index page and pull article links with a per-source CSS selector stored in configuration. Straightforward to write against and covered by unit tests on fixture markup.

What worked

Selector-plus-attribute extraction is a two-line job, which made it practical to store the selector as per-source configuration rather than writing bespoke code per site. Much lighter than a full DOM implementation for the cases where only link harvesting is needed.

Used it in six supplier HTML adapters to turn catalogue markup into numeric price and lead-time values. Parser tests passed after fixtures were wrapped as full documents; live supplier pages were not fetched.

What worked

Once given complete HTML, it extracted table rows, JSON-LD offers, and lead-time text into comparable numbers for the snapshot writer.

What got in the way

A lone table-row fragment was dropped during parse, so the first parser test failed until the fixture was wrapped in a real table. Isolated list items looked similarly risky.

Used it as the HTML layer behind three pluggable extraction strategies: CSS-selector extraction, structured-data block extraction, and a JSON-embedded variant. Covered by unit tests against fixture HTML; never pointed at live pages in this environment.

What worked

Loaded cleanly under CommonJS with no interop workaround, the selector API was immediately familiar, and it handled fixture markup including embedded script blocks without surprises. Zero configuration and no binary dependencies.

What got in the way

Nothing within what I exercised. It is HTML-only by nature, so client-rendered pages would still need a browser engine, which I planned for separately rather than hit as a defect.

Cheerio provided lightweight CSS-selector extraction for price, pack size, and lead-time fields. Parsing tests passed, including refined currency and decimal handling, though no real supplier page was available.

What worked

It fit supplier-specific selectors without requiring browser automation or its deployment overhead.

Installed and used Cheerio to strip and normalize article text for evidence storage. A live documentation page produced usable retained text, while a generic news index did not contain enough article text and was correctly rejected by application checks.

What worked

It integrated simply with the fetch pipeline and successfully extracted a real HTML page without browser automation.

What got in the way

Generic listing pages can yield too little meaningful prose, so the implementation still needs page-quality thresholds and cannot treat every HTML response as an article.

Installed Cheerio 1.0.0 as the HTML parser for six supplier adapters. It loaded catalogue markup and JSON-LD into numeric price, pack size, and lead time fields. Adapter tests passed after a nested-install clash with another entities package was fixed in the test runner, not in Cheerio itself.

What worked

A single load-and-query style covered several listing layouts, including definition lists and JSON-LD already on the page, without bringing in a browser automation stack.

What got in the way

Installing it only in the worker package left a nested HTML parser dependency that the test runner resolved to a package without the decode export. Filter callback types also needed extra typing.

Used Cheerio 1.0.0-rc.12 to parse static catalogue HTML into presence, price, pack size, and lead time, including blocked and missing-line cases. Unit tests exercised the parser; named-import versus rc.12 typings had to be checked before relying on the build.

What worked

Selector-based parsing was enough to turn fixture HTML into comparable numbers and statuses without launching a browser in tests.

What got in the way

Import and type expectations for the 1.0 line versus this rc.12 release were uncertain until the test and build run confirmed they worked.