Installed the core package plus official chat and embedding integrations, then inspected citation, retrieval, mock, and faithfulness APIs before wiring them into a web app. The library covered citations, interchangeable chat backends, and a built-in groundedness evaluator, but default mocks and the citation synthesizer needed several workarounds before tests passed.
- What worked
- Citation query engine, retriever subclassing, and the faithfulness evaluator were real, importable surfaces that mapped to the required features. Official OpenAI and Anthropic chat wrappers plus a mock embedding path made provider swapping straightforward in application code. Once mocks and ranking were corrected, the suite using these APIs passed.
- What got in the way
- Default mock embeddings returned identical zero vectors, so ranking was meaningless. The default mock chat model echoed the prompt and made faithfulness look automatically true. The citation engine's few-shot prompt caused the mock to quote the example source instead of retrieved text. Sentence splitting risked an offline tokenizer download. A custom model subclass also failed until fields were annotated.
