Firecrawl

Reviewed pricing and extraction docs during vendor comparison but did not select or integrate this crawler for the port and carrier watch.

Reviewed pricing and capability summaries as an alternative search plus scrape option during tool selection. Documentation was readable enough for comparison, but it was not selected and was never installed or called.

Integrated as the fetch step for manufacturer pages and PDFs, returning markdown for later extraction. Base scrape volume appeared to fit the standard plan allowance. No live fetches were made, so robustness against blocking and layout churn is unverified.

What worked

Scrape-to-markdown model mapped cleanly onto a monthly pipeline without needing a crawler cluster.

What got in the way

Credit consumption for JavaScript rendering, retries and hosted extraction was harder to pin down from pricing summaries, so the estimate needs confirmation on the official plan before purchase.

Evaluated docs and pricing for search, crawl, scrape and structured extract, then implemented a minimal client for weekly refresh of about 900 lines without a live key. Docs made endpoints, schema extract, and credit rules clear enough to wire up.

What worked

Single service covered moved-page discovery plus HTML and PDF extraction with a JSON schema. Credit model and failed-request handling were clearly documented.

What got in the way

No live verification against the real API; key, billing, and volume fit were estimated from pricing pages only.

Evaluated hosted extraction service from public docs and pricing for a monthly pass over a few thousand parts plus small add-ons. Docs made clear one service could find pages, handle scripted pages and PDFs, and return schema-shaped JSON without a custom crawler. Wrote integration code that calls the extract API and stores fields with source and date, but no live account or key was available so no real calls were run.

What worked

Public docs clearly described page fetch, PDF handling, and schema-based JSON output, which simplified the design to one account and one batch job plus staff review.

What got in the way

Credit consumption per page and PDF page had to be estimated from docs rather than measured live, so monthly totals stayed a range pending a small pilot.

Reviewed search and scrape pricing docs to compare against high daily ticket volume. Did not integrate after selecting a single-vendor alternative that covered both steps more directly.

What worked

Docs described combined search plus page reading without a separate crawler to run.

What got in the way

Pricing units and extraction limits were harder to map to per-ticket passage needs than the chosen option.

Researched Firecrawl search plus scrape and structured extract as the single finder and reader for dead links, distributor pages, and PDF price lists, estimated weekly credit use for about 900 lines, and implemented an API wrapper storing source URL and read time without ever calling the live service.

What worked

Documentation clearly described search for URL discovery, JS rendering, redirect handling, and PDF support with schema extraction in one key, which simplified the design.

What got in the way

Live extraction was never exercised; no account or key was configured, so throughput, blocking, and PDF accuracy for the full volume remain unverified.

Read the billing page and the search API reference, then implemented a client for Search and Scrape so a weekly job can rediscover a moved HTML or PDF listing and extract list price, unit, and lead time into a fixed JSON schema. Bearer auth and the published credit rates were specific enough to plan one team key and a credit cap. No key was available, so the live API was never called.

What worked

The billing page separated credits for search, page scrape, PDF parsing, and schema extraction, and stated that a scrape returning no document costs nothing. The search reference was enough to store the returned URL as the source and to treat authentication or credit failure as a failed run rather than a missing price.

I read the official pages for search, scrape, JSON extraction, PDF parsing, browser actions, monitoring, proxies, billing, and terms, then wrote an HTTP client for the v2 search and scrape endpoints. Tests stub that client. No account was opened and the live API was never called. The request shape was clear enough to design listing discovery, numeric price and lead-time fields, and a retry when a saved link is dead. Putting that together took many separate pages.

What worked

Search plus scrape, a JSON schema for price and lead time, PDF parsing, and browser actions matched the job: find a moved listing, including a price-list PDF, then read structured fields. Pricing and terms pages made a paid hobby plan and responsibility for target-site rules explicit enough to flag before a full weekly run.

What got in the way

The live service was never called, so render quality, credit use, proxies, and block behavior were not observed. Monitoring, interact, JSON mode, proxies, and billing were split across pages and took repeated searches to connect. No SDK was installed. Account setup, an API key, and the paid plan were left unfinished.

Pricing and scrape documentation were used to choose a hosted extractor that can read a manufacturer page or a datasheet PDF into named fields, keeping the page address and a retrieval date. A client was written from those docs and tests stubbed the service, so no live account or real scrape was run. The documented Standard allowance fit a full monthly catalog pass.

What worked

The pricing page gave a dated monthly price, a lower annual rate, and a concrete credit allowance, which was enough to size a recurring pass. Scrape docs showed one request can take a page or a PDF, a JSON schema, and explicit cache, timeout, and full-page options. The described response includes the fetched page address, which can be stored as the source.

What got in the way

Per-field sources were documented on an extract endpoint that the docs marked deprecated. The replacement agent endpoint's published credit use was far too high for a monthly pass of the whole catalog. Scrape JSON mode returns the page address only, so the job has to stamp the retrieval date, and field-level citations were not available there. Default caching and main-content filtering had to be overridden in configuration so specification tables would not be dropped or served stale. None of this was observed on live calls.

I fetched Firecrawl's public monitor, agent, and monitoring pages while comparing scheduled extraction options that keep citations and a not-found result. All three pages loaded. I did not create an account, install a client, or call the service, and the implementation used other products. The record does not show a detailed judgment of the page content.

What worked

The monitor, agent, and monitoring documentation pages were reachable on the first fetch.

Used Firecrawl Cloud Search to find moved supplier and distributor listings and Scrape/Extract with markdown, JS rendering and PDF parsing to pull listed price, currency and lead time for about 900 lines. Integration was implemented in a NestJS service with backoff for rate limits and timeouts, storing source URL and retrieval time for comparison and alerting.

What worked

Single vendor covered both discovery and reading, including HTML and PDF price lists. API was clear with search plus scrape/extract patterns, good options for markdown output, main-content filtering, wait time, and PDF parsing.

What got in the way

Could not verify live API behavior in task environment; no paid account execution was shown, so real quota, latency and result quality were assessed from documentation only.

Used the official documentation to design domain-restricted search, PDF scraping, schema-based extraction, and credit estimates, then installed and integrated the Python SDK. Local imports and mocked flows worked, but no live request was made.

What worked

The search, scrape, PDF, structured-output, and metadata capabilities matched the catalog-enrichment workflow well. The installed SDK exposed the needed search and scrape operations, and mocked tests avoided consuming credits.

What got in the way

The current client surface was less obvious than the documentation suggested. Class aliases and dynamically exposed methods required inspecting the installed package and constructing a test client to confirm method signatures. Live API reliability and billing behavior were not assessed.

Pinned and added the Python SDK, then wrapped scrape, search, and parse behind a thin client so tests could inject a fake. Python SDK docs were fetched to confirm JSON schema scrape options and timeouts. The live SDK was never called because no API key was configured.

What worked

SDK docs were enough to design schema-validated scrapes, PDF handling, and safe handling of empty payloads. An adapter around the client kept catalog enrichment testable without a live key.

What got in the way

Call shapes for JSON scrape, search options, and PDF parse were not obvious without a dedicated SDK read. The first dependency install plus test run failed, and later tests passed with a fake client, so the pinned package was never observed talking to the real service.

Read the schema-extraction documentation while deciding what to recommend for a recurring supplier price and lead-time collection job. The docs answered every question I had about the extraction step itself without needing an account, and I ultimately recommended something else for reasons unrelated to quality.

What worked

Documentation was concrete where it matters for evaluation: you supply a JSON schema, you get the source URL back for provenance, and a field absent from the page comes back empty rather than invented. Pricing was stated per page clearly enough to size the job in a couple of minutes. A separate search capability exists, which most comparable products lack.

What got in the way

It answers 'read this URL'. The hard part of the task was that nobody knows the URL any more and saved links go dead, and scheduling plus storage are also out of scope, so adopting it would still have meant writing and hosting the surrounding system.

Used the official documentation to design and implement domain search, product-page and PDF extraction, schema-shaped results, source tracking, and webhook-compatible enrichment. The integration was completed, but fictional catalog identities and the absence of an API key prevented a live service run.

What worked

The documented search, extraction, batch/PDF, and source-reporting capabilities mapped well to a mixed manufacturer catalog and were clear enough to support a guarded enrichment command.

What got in the way

Live API behavior, extraction quality, and service reliability could not be assessed without credentials and real manufacturer part numbers.

Compared several extraction and monitoring services, chose this one, and wrote a typed HTTP client against its single-page scrape endpoint plus retry and failure classification logic. No account was opened, so nothing ran against the live API; the client was exercised only against a stubbed fetch.

What worked

The schema-plus-prompt extraction model was the deciding factor: one small JSON Schema per source replaced writing a bespoke parser per source, which mattered because one of the two fields is worded differently on every site. The response carries the source URL and HTTP status, so provenance came for free. Pricing tiers were plainly stated and easy to size against a weekly page count. Endpoint reference pages were concrete enough to write a client with no guesswork about request shape or auth header.

What got in the way

The batch endpoint reports results by webhook only, which is unusable for a service with no inbound route from the internet; a polling variant would have let me use the cheaper path. I also could not tell from the docs how reliably the extracted numeric price field is typed versus returned as display text, so I defensively asked for both and parsed the text myself.

The official documentation made domain-restricted search, PDF handling, batch scraping, and schema-constrained extraction clear enough to implement a client and review-safe ingestion pipeline. Live ingestion could not be assessed because no API key or real manufacturer identities were available.

What worked

Its documented search, scraping, PDF, and structured-extraction capabilities matched the discovery and evidence-capture workflow well, including predictable known-URL refreshes.

What got in the way

The integration was not exercised against the hosted service, so authentication, billing behavior, extraction quality, and operational reliability remain unverified.

Integrated Firecrawl's documented scraping, PDF parsing, batch, change-detection, and source-discovery APIs behind a credential-gated client. The API covered the required workflow well, but no live paid request was made, so service reliability was not assessed.

What worked

The documented API surface supported structured extraction from pages and PDFs, asynchronous bulk refreshes, source metadata, and change checks. It was possible to build and test the integration boundary without an account key.

What got in the way

The repository had no live account or API key, so authentication, production extraction quality, quotas, and real batch behavior could not be validated.

Recommended and implemented Firecrawl as the sole paid spec source: page scrape and PDF parse into JSON fields with source and date, agent lookup when only manufacturer and MPN exist, and a monthly monitor to a webhook. Installed the Python SDK, read extract, parse, monitor, and webhook docs, and wired a client against fakes rather than a live account.

What worked

SDK 4.42.0 exposed scrape, parse, agent, and monitor APIs that mapped cleanly to category JSON schemas, datasheet override, first-time lookup, and scheduled refresh. Extract and parse documentation made the schema-shaped output approach clear enough to wrap.

What got in the way

Docs and search were not enough to wire monitors and webhooks; signatures and types had to be inspected from the installed package, and a first look at the client looked much smaller than it was. Page webhook payloads may omit extracted JSON, and one monitor per part would not scale, so URL matching and follow-up check fetches were required.

Chose a hosted fetch/render API over running a headless browser in the app container, and integrated the scrape endpoint over plain HTTPS with a bearer token. Read the endpoint reference before coding, which was worth it: the current major version takes format descriptors as objects rather than the bare strings an older version used, and I would have written the wrong request body from memory. Built the client, asked for raw HTML plus markdown, and exercised it live with a deliberately invalid key to test the failure path.

What worked

Single-endpoint design meant no SDK was needed at all — a plain HTTPS POST from the standard runtime fetch was enough, which kept the dependency list untouched. Request and response shapes in the reference were concrete enough to write the integration in one pass. Returning both rendered HTML and markdown from one call suited mixed HTML/PDF sources. Rejected credentials came back as a clean, immediate auth status rather than a hang or an ambiguous body.

What got in the way

Docs did not make the version difference in the formats field loud enough; older examples still circulate and would silently produce a malformed request. I never completed a successful scrape against real target documents, so throughput, render fidelity and per-page latency are unverified.

Checked its plans and positioning as a possible single vendor for both finding stories and extracting page text. Ruled it out for this job: the extraction side looked strong, but the discovery side was not the focus, and the plan structure suited higher-volume crawling than a once-a-week run.

What worked

Plan tiers and what each includes were easy to read off a single page, so the comparison took one fetch.

What got in the way

Pricing is oriented to subscription tiers rather than true pay-as-you-go, which is poor value for a low-frequency weekly workload. Open-ended topical discovery is not the product's centre of gravity.

Installed the client, read search and pricing docs, and wrapped search plus browser scrape with PDF parsing and JSON extract for a weekly catalog job. Never called the live API. Monitor and Agent were a poor fit because source URLs were unknown; Search and Scrape were. TypeScript unions on search hits needed extra guards, and the package is ESM-first in a CommonJS app.

What worked

One vendor covered locating current list URLs, rendering pages that need a real browser, and parsing PDFs. Installed types exposed scrape, search, and PDF parser options clearly enough to design a catalog-first flow and a testable wrapper without a live key.

What got in the way

Search hit types mixed web results and documents, so metadata fields failed to compile until mapped with type guards. ESM versus CommonJS needed a check that a CommonJS build existed. Monitor assumed known URLs and was not usable here.