Nokogiri

Used it to extract readable text from fetched pages so a quoted passage could be confirmed to appear verbatim before being stored. Exercised the parsing logic for real in a standalone harness: fourteen checks covering text split across block elements, inline markup splitting a single word, script-tag exclusion, and malformed byte sequences all passed first time.

What worked

Text extraction concatenates inline fragments the way a reader would, so markup splitting a word mid-token still yields the intact word — exactly the behaviour passage matching depends on. It handled input with invalid encoding without raising, and worked fine loaded standalone outside the web framework, which made the trickiest component genuinely testable in an environment with no database.

Used it to build the extraction layer: parse a fetched page, strip navigation and boilerplate nodes, read structured-data blocks for the article body and publisher, fall back to paragraph text, locate the paragraph naming the business, and pull metadata tags for site name and publish date. Exercised the whole thing against real markup fixtures in an offline harness.

What worked

Node removal, CSS selection and text extraction compose cleanly, so a multi-strategy extractor with structured-data-first and paragraph fallback stayed readable. It handles messy real-world markup without throwing, which matters when the input is whatever a publisher serves. Behaviour under test fixtures matched expectations on the first run for every case written.

What got in the way

Nothing of note in this task.

Used it as the only new direct dependency, to parse fetched public pages: meta tags for publisher and publication date, embedded structured-data blocks, and the body text around a search term to build a stored passage.

What worked

CSS selectors and attribute access made the metadata extraction short and readable. It handled a very large real-world page without trouble, and separating the parsing layer meant I could unit-test extraction against fixture markup with no network. Node removal for stripping scripts and navigation was simple.

What got in the way

Nothing attributable to the library — my one bug was removing script nodes before reading the structured-data block inside them, which was my ordering mistake, not its behaviour.

Used to turn fetched HTML into plain text for passage matching, and exercised for real in a standalone script: script and style elements stripped, invalid byte sequences scrubbed, and a full HTML-to-text-to-verification round trip confirmed.

What worked

Parsing and node removal are concise and behaved exactly as expected on messy input, including pages with invalid encoding. It was the one new dependency I could actually execute, which let me prove the extraction path end to end.

What got in the way

It was only present transitively, so getting it onto the load path meant locating the vendored library directory and adding its own transitive dependencies by hand; the initial load failure gave only a generic missing-file message.

Used it standalone to confirm that CSS attribute selectors containing a hyphenated unquoted value parse correctly, before relying on those selectors in integration tests I could not execute. Parsing a small HTML fragment and querying it took a handful of lines and answered the question immediately.

What worked

Parsing an inline fragment and running several selector variants against it required no setup at all, which made it an ideal way to settle a syntax question offline. Selector behavior matched expectations with both quoted and unquoted attribute values.

Built a deterministic HTML-to-text extractor on top of it: walking the parsed tree, discarding script and style subtrees, emitting block boundaries, and normalising whitespace so that character offsets into the extracted text remain stable and quotable. Ran dozens of extraction cases against it successfully.

What worked

Loaded and parsed cleanly as a standalone library with no framework present, which was the only reason I could execute and debug the correctness-critical logic at all. Tree traversal and node-type checks behaved exactly as expected across malformed and nested markup, including tables.

Used it to turn fetched pages into plain text for passage verification: parse, drop script and style subtrees, and collapse the remaining text so a quoted passage can be matched against what the page actually says. Exercised directly in a standalone harness, including a case with inline markup splitting a sentence.

What worked

Handled messy real-world markup without ceremony, and node removal by selector made stripping non-content trivial. It loads fine outside any framework, so it worked in the minimal verification harness. Behavior on inline markup inside a sentence was predictable enough to build a reliable matching tier on top of it.

Used it to strip markup and pull readable text out of fetched pages so a quoted passage could be checked against what the page actually said. Parsing and text extraction behaved exactly as expected in tests, including on messy markup.

What worked

Simple, obvious API for the one thing I needed. Already present transitively, so adding it as an explicit dependency cost nothing at install time while making the reliance honest.

What got in the way

Nothing in this task; I only exercised a small part of its surface.

Added the gem and used it to pull a short firm-named passage from fetched HTML, with dedicated extractor tests covering that behavior.

What worked

Install and API use were straightforward; extractor tests passed once wired in, with no parser or version issues in this task.

Used for turning retrieved HTML into the plain text that becomes the stored evidence snapshot. It was already present transitively via the web framework, so adding it cost nothing beyond declaring it explicitly in the manifest, which is the right hygiene when depending on it directly. The parsing paths were written but not exercised in this environment, so I cannot speak to runtime behavior here.

What worked

Already in the dependency graph, so no new install and no new native build. The HTML5-capable parser entry point is the obvious choice for arbitrary real-world pages.

Used it to parse fetched HTML and produce stable plain text that quoted passages can be located within by byte offset. Loaded and ran fine from a vendored install, so the text-extraction logic and its tests were fully exercised here.

What worked

The HTML5 parser handled messy markup without complaint, and the node API made it straightforward to inject block-level separators so that adjacent elements do not run together in the extracted text. Already present transitively, so no new native build was needed.

What got in the way

Node insertion raises on nodes without a parent, so the traversal needed a guard; that is reasonable behaviour but it is easy to hit on the document root.

Explicitly loaded Nokogiri to parse downloaded API documentation and extract text around insight creation. The parsing commands succeeded and helped inspect request details without adding another parsing tool.

With no browser available, I rendered the dashboard template to a file and used this parser to pull out bar widths, labels, tooltip text and table rows to confirm the markup and geometry were sane. It got me most of the confidence a visual check would have.

What worked

CSS selector queries over a complex page were concise, and once encoding was correct every element I looked for came back exactly as expected. Exposing parse errors on the document object was what finally pointed me at the real problem.

What got in the way

Given an input string with the wrong encoding it silently returned zero matching nodes instead of raising — the document parsed, the body appeared populated, and selectors just found nothing. I spent several rounds suspecting my own markup before realizing the input needed an explicit encoding both when reading the file and when constructing the document.