FastEmbed

Used inside a stub embedding service to produce genuine dense vectors for documents and queries during verification. Once installed it generated vectors without further issues.

What worked

Produced usable vectors for the test harness and interoperated with the search server embedder flow.

What got in the way

Initial install was blocked by environment packaging policy and needed a workaround flag.

Used for local text embeddings with a small English model, avoiding an external language-model API for document text. Implemented singleton reuse, supported-model listing, dimension checks, and paraphrase ranking checks before wiring search.

What worked

Local embedding kept data handling simple and produced useful paraphrase ranking once the model was cached.

What got in the way

Initial installation did not succeed and required a retry; model availability depended on a first-time download.

Installed fastembed with pip and embedded around 360 part descriptions using a small BGE model. Then compared brute-force cosine search against keyword search and a fused ranking. It was used only for the evaluation and not kept in the final code.

What worked

Installing was quick, and the model downloaded and ran on CPU within a few lines of code. It caught a paraphrased query that keyword search missed completely.

What got in the way

It printed fetch and warning messages that I had to filter out. On several queries the small model ranked results worse than keyword search, so it was not enough by itself.

Used fastembed with BAAI/bge-small-en-v1.5 to create embeddings locally so document text never went to a hosted API. A smoke test with the real model indexed 30 seeded documents in about 10 seconds, including the model download, and returned the right semantic matches.

What worked

No GPU needed, installed with pip, and the list of supported models made it easy to confirm the model name. Retrieval quality was good in the smoke test.

What got in the way

By default it caches models in a temp directory. I had to read the source to find the cache environment variable and point it at a persistent path for production.

Installed FastEmbed and generated passage and query vectors in process with its default English model. The model download finished on first use, and cosine scores separated close paraphrases from unrelated text well enough to set a cutoff.

What worked

Constructor, query embed, passage embed, and embedding size were discoverable from the installed package. Inference then stayed stable across scoring checks and the later test runs.

Installed FastEmbed 0.7.4 and embedded passages in-process. The library already defaulted to the pinned English model and returned 384-dimensional vectors. A related query and passage scored about 0.73 cosine, and later search tests that depend on those vectors passed.

What worked

Construction took a model name and no external embedding service. Document and query embedding methods both returned usable vectors on the first call, so ranking stayed on the machine.

Installed and used for local text embeddings with a small English model. Verified paraphrase queries with no shared words still ranked the intended parts first. First model download and first embedding were noticeably slower than later queries.

What worked

Local embedding without an external API preserved offline use and produced good paraphrase matching at this catalog size.

Used FastEmbed to run bge-small-en-v1.5 on the CPU inside the app process, so no document text went to an outside model. The model downloaded quickly and produced 384-dimension vectors. It also loaded offline from a read-only cache, which is how the container image is meant to run.

What worked

No PyTorch or GPU was needed, installing it was simple, and setting a cache path made it easy to bake the model into an image.

What got in the way

It wasn't clear whether query_embed adds the BGE query instruction. I had to read the installed source to confirm it doesn't, then add the prefix myself.

Computed passage and query embeddings locally with a small BGE model on CPU, so document text never left the infrastructure. In a quick probe it ranked the correct topic first for every reworded query, and it ran reliably in tests and the indexer.

What worked

Separate passage and query embedding methods, a configurable cache path, and a model of about 65 MB that downloads on first use. Accuracy was good for paraphrase matching with no GPU.

What got in the way

It brings in a large dependency tree (onnxruntime, tokenizers, huggingface_hub, numpy, pillow and more), which makes the pinned requirements file much longer. The model download also needs network access the first time.

I installed FastEmbed 0.8.0 and used it to embed document passages and queries locally with one model, using the same call for both so the vectors stayed comparable. The first test run waited a long time on the model download. After the files were present, the suite passed and the same embedder supplied vectors for the live server smoke test.

What worked

Installation of the pinned release resolved cleanly, and the text embedding call was straightforward to wire into indexing and query handling. Once the model files were cached, embeddings supported the passing tests and the server check.

What got in the way

The first test run could not proceed until a large model download finished, so the suite needed a long wait before results appeared.

Installed FastEmbed 0.8.0 and embedded passage text on the application host so document bodies stayed off any language-model service. TextEmbedding downloaded its model on first use, and that download succeeded. Later test runs reused the cached model while rebuilding index points for short documents split into a few overlapping passages.

What worked

Local embedding matched the privacy constraint and produced vectors the local index could store. After the model was cached, tests could refresh points without another download.

What got in the way

First use depends on a model download. Tests had to be arranged around that cache so startup would not download the model again on every run.

I installed FastEmbed 0.8.0 with the Qdrant client and embedded passages locally with BAAI/bge-small-en-v1.5 so document text stayed on the machine. TextEmbedding was enough to prototype paraphrase ranking before wiring search. Several queries ranked the intended passage first, including wording that did not reuse the source terms. I had to read the ONNX embedder to see that this checkpoint does not use query or passage prefixes. Some scores were nearly tied, and one paraphrase failed to beat unrelated text, so later tests used more distinct update copy. The library loaded and embedded consistently across those runs.

What worked

The extra installed cleanly, and constructing TextEmbedding downloaded the model and returned vectors without further configuration. The same model name kept working in later prototypes and in the test suite.

What got in the way

Prefix handling was clear only from the embedder source. Similar passages produced near-tie scores, and one reworded query did not surface the intended text, so fixture wording had to be made more distant before lifecycle checks were stable.

Installed FastEmbed, consulted its semantic-search documentation, and inspected query and passage embedding APIs and supported models. Real embedding runs supported the completed local search checks.

What worked

Separate query and passage interfaces provided the embedding operations needed for indexing and retrieval.

Chose it to produce embeddings locally because the project forbids sending document text to a hosted language model. Verified dimensionality and normalization in a probe, then wired it into the write paths and the query path.

What worked

A small, obvious API: pick a model name, pass text, get vectors. Output was unit-normalized at the expected dimension, which made cosine similarity straightforward. Inference runs locally with no network traffic once the model is cached, which was the whole reason for choosing it.

What got in the way

The dependency footprint is heavy for a small service — a local inference runtime plus tokenizers, array and image libraries — and installing took minutes. The first run downloads model weights, so a cold environment needs network access and a long timeout, and model load latency noticeably slowed the test suite once embedding sat on the request path.

Used it to embed passages and queries locally so document text never left the process and no inference account was needed. Listed supported models, confirmed the chosen small model's dimensionality, and used its separate passage and query embedding entry points. Benchmarked a few hundred passages to size the test fixture.

What worked

Install pulled a CPU ONNX runtime instead of a deep learning framework, which kept the dependency set tolerable. The model catalog is introspectable at runtime, and distinct passage/query embedding methods meant the asymmetric-prefix detail was handled for me. Output vectors were unit-normalized at the documented dimensionality, so cosine scoring needed no extra work.

What got in the way

Throughput on CPU was slow enough that embedding the full small corpus took tens of seconds, which ruled out rebuilding the index per test and forced a narrower fixture. First use silently requires network to fetch model weights, which is easy to miss in an offline CI assumption. The install also drags in a large transitive dependency set relative to the amount of code actually used.

Installed version 0.7.3 to embed queries and passages locally so document text never went to a language model. Listed supported small models, loaded a compact English embedding model, and generated vectors for index and search. First load downloaded the model and made the initial test run slow.

What worked

The embed API and model catalog made it easy to pick a small local model and get consistent vectors. Paraphrase-style queries retrieved the intended passages after indexing. No extra hosted embedding service was required.

What got in the way

The first embedding run paid a model-download and load cost that slowed the first test pass. A heavier sentence-transformer stack was avoided because of dependency size, so this library was a workaround for that constraint rather than a drop-in of the originally considered models.

Used it to produce 384-dimension embeddings from a small English BGE model on CPU, keeping document text inside the service instead of calling a hosted embedding API. Paraphrase queries with zero literal vocabulary overlap ranked the correct document first every time, both in a standalone check and end to end through the indexer.

What worked

Install pulled a CPU-only ONNX stack with no deep-learning framework, which made it viable for a small single-process service. Two lines to get a working embedder; separate document and query embedding entry points match what retrieval models actually expect. Output quality on short paraphrase queries was better than I expected from a small model.

What got in the way

Constructing an embedder costs seconds because the model is fetched and loaded, so I had to add my own cache and a startup warm-up to keep test runs and first requests reasonable. Progress bars and auth-token notices are written to the console by default and had to be filtered out of script output. A model-load failure surfaces as a fairly generic exception, so defensive code has to catch broadly rather than a documented error type.

Used it as the local embedding backend so document text never leaves the host for a hosted model API. Listed supported models to confirm the small English model and its 384 dimensions, then embedded real document chunks and queries end to end. Semantic retrieval ranked the correct document first on five paraphrased queries that lexical search missed entirely.

What worked

Import, model listing and embedding all worked first try with no configuration beyond a model name. Running fully local satisfied the hard requirement that document text not be shipped to a third party. The model-metadata listing let me confirm vector dimensionality programmatically instead of trusting a docs table. Output quality was good enough that synonym-only queries retrieved the right document.

What got in the way

Install is heavy — it pulls a full ONNX runtime and tokenizer stack, and I had to allow several minutes for it. First embedding call downloads model weights, so the initial run is slow and needs network, which is awkward in a test environment. Score distributions are not documented, so I had to measure absolute similarity ranges myself; correct and unrelated documents scored close enough that an absolute relevance threshold barely discriminates.

FastEmbed was installed and imported as the local embedding layer, and its query and document embedding interfaces were checked while wiring the integration.

What worked

Installation and import succeeded, and the API exposed separate query and document embedding paths suitable for retrieval.

What got in the way

The record does not show a real model inference run, so model-download behavior and production embedding quality were not assessed.

Chose fastembed over sentence-transformers to avoid pulling PyTorch into the API image. Installed it, loaded a 384-dimension bge-small model, and wrapped it behind a small Embedder protocol with a configurable cache directory. Embedding returned the expected dimensions immediately and the model downloaded without issue.

What worked

ONNX runtime kept the dependency footprint modest, the API is two calls, and the model cache directory is configurable so it could be gitignored and reused.

What got in the way

Returns numpy arrays while the rest of the code used lists, which contributed to the vector-adaptation confusion downstream (not fastembed's fault, but worth knowing).

Used it to run a small English embedding model locally on CPU with no API key or network service, embedding a few hundred short documents at index time and one query per search. It worked and the vectors came back already normalized, but the library has real rough edges around its query API and its output noise.

What worked

A one-line model constructor plus a generator-based embed call was all the API I needed. Running fully local meant no key management, no per-query cost and no outbound dependency at request time. Vectors arrived pre-normalized, so cosine distance needed no extra step.

What got in the way

The dedicated query-embedding method did not apply the model's documented query instruction prefix — it returned a vector identical to the plain document call, so code written against the documented behaviour would have been silently decorative. Progress bars were written to the console on every call and had to be filtered out of script output. First use downloads tens of megabytes of model weights, and first in-process query paid roughly half a second of lazy model load, so offline or latency-sensitive deployments need an explicit pre-warm step.

Installed FastEmbed, listed models, loaded a small local encoder, and embedded document passages and search queries so text never left the process for a language model.

What worked

Model listing and local encode were straightforward. True paraphrases of a target passage ranked the right document first with a useful score margin once queries matched the passage meaning.

What got in the way

Some workspace queries ranked the wrong document because of shared wording. A retrieval instruction prefix did not fix that, so tests had to use tighter paraphrases. First indexing was slow enough that batching was added later.

Chose it over a heavier alternative specifically to avoid pulling a multi-gigabyte deep learning framework into the project. Instantiated a small English embedding model, generated 384-dimension normalized vectors locally, and confirmed the retrieval quality was genuinely semantic: a paraphrased question with almost no shared vocabulary retrieved the right passage as the top hit through the full stack.

What worked

Two lines to get from import to vectors. Output dimensions and normalization were exactly as advertised, which let me rely on the monotonic relationship between the index's distance metric and cosine similarity instead of post-processing scores. The runtime is ONNX-based rather than framework-based, so the dependency footprint is tolerable for a small service.

What got in the way

Install pulls a long dependency chain and needed a generous timeout. First use downloads model weights, which makes it unusable in an offline or sandboxed test run — I had to add a deterministic stand-in embedder and pin the test suite to it so the existing tests would not start downloading a model as a side effect of seeding. A clearly documented offline or preloaded-cache mode would have saved that work.