Haystack

Installed the Anthropic generator integration and used its chat component so the assistant could select that provider from settings, instantiating it in tests with placeholder credentials rather than live calls.

What worked

The chat generator imported cleanly alongside the core SDK, and constructor parameters were clear enough to build a provider factory and an interchange test without extra adapters.

What got in the way

No live Anthropic request was made, so hosted behavior was not observed. Empty credentials still required a local stub so tests and keyless runs would not attempt network generation.

Installed the pgvector Haystack integration and wired it as the production document store with keyword retrieval so indexing would not need embeddings on the request path. Connection URL normalization and nullable embeddings were confirmed from installed source. No live Postgres with the vector extension was available, so the production path was not executed.

What worked

Keyword retrieval and metadata filters looked sufficient for shared incremental indexing across instances. The embedding column being nullable meant the first version could skip embedders entirely.

What got in the way

Guides were not enough to confirm write behavior without embeddings or how connection strings should be shaped; that came from reading the integration source. Live write/query reliability was not observed.

Installed the 2.x SDK, wired pipelines for retrieve-prompt-generate, metadata ACL filters, overwrite indexing, citations, and a lexical faithfulness path so tests could run without live model keys.

What worked

After installing a real 2.x release, pipeline components, in-memory store, metadata filters, citation parsing, and interchangeable chat generators were inspectable from the package and sufficient to finish the feature with passing tests.

What got in the way

Official docs pages timed out more than once, so APIs were confirmed from GitHub and site-packages. An unpublished 2.x pin failed to install. The faithfulness evaluator first scored a grounded answer at zero, then omitted an expected status field, and pipeline logs stayed noisy.

Installed the Google GenAI Haystack integration and used its chat generator as the Vertex-backed provider. Constructor inspection and a no-credentials init check succeeded; live completions were not run. Naming and docs pointed at Google GenAI rather than a Vertex-specific generator page.

What worked

The chat generator constructed in Vertex mode without a live account, so tests could assert the provider type. Model and location were straightforward to pass through from app settings.

What got in the way

Client setup appears to run during init and is aimed at application default credentials, which made a lazy or mocked wrapper necessary for tests. The Vertex-titled doc page was missing, so setup details came from a nearby GenAI page and the installed module.

Searched Haystack 2 docs for faithfulness evaluation, metadata filters, citations, and incremental indexing. Feature coverage looked strong, but a Python runtime did not fit the Java/Maven service layout, so it was not adopted or installed.

What worked

Public material made extractive citations and faithfulness-style evaluation easy to compare with Java options.

What got in the way

No Java runtime path, so using it would have added a second language and packaging model the repository would not absorb.

Installed the Anthropic Haystack integration and constructed its chat generator as the second interchangeable provider. Docs for the generator loaded. A dummy or empty key was enough to initialize; live completions were not run.

What worked

Constructor and secret-key handling were easy to inspect. Initialization succeeded without a real key, which unblocked factory tests that only needed the component type.

What got in the way

A real key is still required to generate. Tests therefore never exercised the run path against the vendor.

Installed Haystack 3.1 and built retrieval pipelines with metadata access filters, answer-side citations, incremental overwrite writes, in-memory BM25 for tests, and a built-in faithfulness evaluator. Guides plus installed component source were enough to finish, but several APIs were clearer in package code than in the pages fetched.

What worked

Pipelines, metadata filters, answer building from retrieved documents, duplicate-overwrite indexing, BM25, and the faithfulness evaluator all mapped onto the requirements without adding a second evaluation product. Chat generators swapped without changing the pipeline graph.

What got in the way

The Vertex-branded chat generator doc URL returned 404. Confirming that pgvector writes allow missing embeddings, how nested AND/OR filters compose, and chat prompt document wiring required reading installed source rather than the guides.

The integration package supplied the document store and embedding retriever APIs used for filtered retrieval. Its wheel source was inspected to confirm signatures and filter behavior, but no live pgvector service was available.

What worked

The API exposed the needed PostgreSQL document-store, retrieval, filtering, overwrite, and deletion concepts in a form compatible with Haystack pipelines.

What got in the way

End-to-end reliability against a real database remained untested because infrastructure configuration was intentionally deferred.

The connector was installed, its generator implementation and response-format handling were inspected, and it was wired into a provider factory. Tests covered construction, but no authenticated live generation was recorded.

What worked

The package offered a Haystack-compatible generator interface that made the Google provider interchangeable with the OpenAI provider.

What got in the way

Authentication and provider-specific structured-output settings required extra investigation, and live service reliability was not assessed.

Evaluated it against two other retrieval frameworks and chose it for a Django monolith needing per-user access filtering, citations, incremental indexing and groundedness tests. Wrote the full integration layer (vector store wiring, retrieval pipeline, indexing, evaluators, two swappable generators) from the published docs, pinned versions, but never installed or executed it in this environment.

What worked

Each of the five requirements mapped onto a first-class, documented component rather than something to hand-roll: metadata filters push into the SQL WHERE clause before top-k, upsert-by-id covers incremental indexing, and faithfulness/context-relevance evaluators are plain callables usable from a management command. The generator classes for the two model providers share an interface, so provider swapping really is a config change. Being a library rather than a runtime meant it dropped into existing views and background tasks with no new process or datastore.

What got in the way

The docs pages I landed on did not make the current major version obvious, so I initially targeted the previous major and had to correct myself mid-implementation after noticing a newer release; the migration notes took a separate search to find. Filter-syntax and retriever-argument details were spread across the core docs and a separately maintained integration page. Component call signatures remain unverified here, which is the main residual risk.

Used it as the retrieval/generation framework for a grounded assistant: chunking, embedding, a Postgres-backed document store, metadata-filtered retrieval, chat generation behind two swappable providers, and built-in faithfulness/context-relevance evaluators for the groundedness tests. Component composition fit the task well and every import path and constructor signature I checked against the installed packages matched what I expected.

What worked

Component model made the swappable-provider requirement almost trivial: same pipeline, different chat generator. Explicit document IDs and a configurable duplicate policy made incremental re-indexing idempotent without extra machinery. Built-in groundedness evaluators meant I did not have to hand-roll eval scaffolding. Metadata filter syntax is a plain, predictable dict structure that was easy to generate and test.

What got in the way

Version signals were confusing: the docs site served a newer major version than the one I had planned around, and I only found the support window for the older line by digging through release notes — easy to pin a branch that goes end-of-life soon. The document store opens its own database connection outside the host app's pool and does not close it implicitly, which broke test-database teardown and would have leaked a connection per background task in production; the need to call close() was not obvious from the docs. Retriever filter-merge policy defaults to replacing init-time filters, which is a quiet footgun for security-relevant filters.

Evaluated as the main alternative from its public docs and release history without installing it. It came across as actively maintained with a stable, properly versioned API — arguably the more predictable of the two candidates — but lost on two specific requirements, so I recommended the other.

What worked

Stable post-1.0 versioning and a steady release cadence, which is a real advantage over a pre-1.0 competitor when you have to pin and live with a dependency. Document store abstractions and metadata filtering were easy to understand from the docs, and it ships its own faithfulness and context-relevance evaluators.

What got in the way

Retrieval returns whole documents; character-span offsets are not first-class, so exact span citations would have been hand-rolled. Its duplicate-handling policy is a write policy rather than content-hash change detection, so incremental re-indexing against a full nightly re-push would also have been hand-rolled. Those two gaps decided the comparison.

Installed it and built a retrieval+generation feature on top: a custom DocumentStore adapter over a Postgres vector table, a custom embedder and retriever component, and an explicit pipeline graph. Implemented the DocumentStore protocol and the component decorator by inspecting the installed package rather than from memory, and everything I wired matched the inspected signatures on the first run.

What worked

The DocumentStore protocol is small and stable (six methods), which made a hand-written store backed by my own database table straightforward and kept tenant filtering mandatory instead of optional. The component decorator with typed run inputs made test doubles easy. Synchronous-first API fit a sync web stack cleanly. Source was readable enough to answer API questions by inspection.

What got in the way

I went in believing the current major line was the previous one; the installed version was a major ahead of my assumption. That is a signal that version-identity information is not prominent enough in the material I was recalling. Chat message and generator APIs needed source inspection to confirm argument names.

Haystack supplied retrieval, generation, answer parsing, document writing, and groundedness evaluation abstractions. Version 3.1 APIs were verified from official documentation and installed source, and the completed integration passed its tests.

What worked

The Python-native components covered the broad RAG workflow, supported interchangeable generators, and included evaluators useful for groundedness testing.

What got in the way

Exact immutable citation offsets still required application-level validation, and the recent major-version line made source inspection and strict version pinning prudent.

Picked it as the retrieval framework for a multi-tenant web app because its component model is thin enough to swap the stock vector store for a custom retriever backed by the app's own ORM, keeping tenant filtering in SQL. Built an embed/retrieve/prompt/generate pipeline with two swappable generator components, plus vendor integration packages. Pipelines constructed and all connections validated locally.

What worked

Components are typed callables wired into a DAG, so replacing the retrieval node with a database-backed query was routine rather than a fight with the framework. Connection validation at wiring time checks socket names and types, which turned 'I think the wiring is right' into a verified fact without any live model calls. Custom-component docs were clear enough to write one correctly on the first try.

What got in the way

My assumption about the current major version was a full major behind, so I had to re-verify integration package names and import paths against the live index. Anonymous telemetry is enabled by default and must be turned off via an environment variable read at import time — an unexpected outbound call for a product handling regulated records, and easy to miss. The chat message abstraction does not expose provider-native document/citation blocks, so a vendor citation feature was unreachable without bypassing the generator.

Reviewed official documentation for citation building, duplicate handling, incremental document writing, and OpenAI and Anthropic generators while comparing maintained retrieval frameworks. It appeared capable, but another framework was selected for repository fit.

What worked

The documentation exposed the relevant components and duplicate-policy concepts clearly enough to support a meaningful comparison.

Chose it as the retrieval framework after comparing maintained options, then wrote an indexing and answering pipeline against it plus its vector-store, Anthropic and Google generator integrations. Could not install it in the offline sandbox, so all code was written from the published API reference and two tests were skipped; the rest of the suite was structured to run without it.

What worked

The explicit component-graph model gave one obvious place to inject retrieval filters, which was exactly what a multi-tenant access-control requirement needed. Document writing with an explicitly set id made incremental upserts straightforward. The built-in faithfulness and context-relevance evaluators accept an injected chat generator, so the judge could run on a chosen provider rather than the default one. Generator swapping across two vendors was a constructor change, nothing more.

What got in the way

The retriever's default filter policy replaces constructor filters with run-time ones instead of merging them — a quiet security footgun that only surfaced by reading the reference closely. Docs and published releases disagreed on the current version, and one docs example imported a helper from the wrong module. The core package pulls in an unrelated vendor's SDK even when that vendor is unused, which is awkward in a privacy-sensitive deployment. A recent major version bump left some doc pages on the old line.