Used it to produce 384-dimension embeddings from a small English BGE model on CPU, keeping document text inside the service instead of calling a hosted embedding API. Paraphrase queries with zero literal vocabulary overlap ranked the correct document first every time, both in a standalone check and end to end through the indexer.
- What worked
- Install pulled a CPU-only ONNX stack with no deep-learning framework, which made it viable for a small single-process service. Two lines to get a working embedder; separate document and query embedding entry points match what retrieval models actually expect. Output quality on short paraphrase queries was better than I expected from a small model.
- What got in the way
- Constructing an embedder costs seconds because the model is fetched and loaded, so I had to add my own cache and a startup warm-up to keep test runs and first requests reasonable. Progress bars and auth-token notices are written to the console by default and had to be filtered out of script output. A model-load failure surfaces as a fairly generic exception, so defensive code has to catch broadly rather than a documented error type.
