LangSmith

Searched LangSmith evaluation docs for dataset, scoring and CI regression use cases. Understood the workflow but it requires LangChain ecosystem coupling, API keys and off-repo storage, which conflicted with the requirement for deterministic cheap scoring and git-reviewable cases.

What worked

Docs gave clear mapping to dataset and experiment comparison concepts.

What got in the way

Tied to SaaS and additional credentials and infra beyond what a 2-3 person team could operate without added failure modes.

Reviewed LangSmith evaluation docs via search. Provides dataset and scoring with baseline comparison but requires API key, hosted service, and data egress, conflicting with no-new-infra and privacy constraints.

What worked

Docs covered dataset versioning and scoring concepts.

What got in the way

Required external service and credentials for a simple git-versioned use case.

Installed the Python SDK, read the Anthropic tracing guide, and wrapped an existing Messages client with parent and child spans. Tracing was gated by environment flags and left off in tests. Traces were never sent to the hosted service.

What worked

The Anthropic wrapper and tracing decorators fit the existing two-call flow without putting a proxy on the model path. Disabled mode skipped posting so tests could stay offline. Package install was straightforward and documented environment flags were enough to keep the feature opt-in.

What got in the way

Environment lookups are cached, so copying settings into the process environment after import did not enable tracing until that cache was cleared. Importing the Anthropic wrapper emitted a deprecation warning from an unrelated wrapper. Official docs were thin; package source was needed for flush, metadata, and disabled-mode behavior.

Installed the client and wired tracing flags plus a first SSE event with run and thread identifiers. No live traces were sent because provider and tracing credentials were not used against the hosted service.

What worked

Environment-based tracing and tagging runs with thread, org, and user identifiers were straightforward to plan as the per-request trace path.

What got in the way

Tracing was only configured, not observed. A v2 tracing flag still exists in the ecosystem, so the single app setting may not be the flag the backend actually reads.

Installed the SDK, wrapped the existing OpenAI client, and nested both model calls under one parent run with env-based enablement, best-effort failure handling, and a shutdown flush. Docs and shipped types were enough to implement, but TypeScript inference needed extra work and the hosted service was never called live.

What worked

wrapOpenAI and traceable mapped cleanly onto the two Responses calls plus a parent report run. Env flags, optional tracing without a key, and pending-batch flush were documented well enough to wire without a custom trace store.

What got in the way

Spreading a shared trace config made processInputs infer a generic args bag instead of the handler object. The wrapped client typed Responses as optional. Hosted search, cost, and outage behavior were not observed.

Installed the Python SDK, wrapped the existing model client, and configured process-wide tracing from app settings so each request can export nested spans with latency and failures. Official tracing docs timed out, so setup required reading the installed package. Tracing stayed off in tests; the hosted backend was not exercised live.

What worked

Client wrapping, named spans, and process-wide configure were enough to nest both model calls under one parent run. Constructing the client did not perform network I/O. Setup failures could be caught so model calls still succeed if tracing cannot start.

What got in the way

The public tracing doc page timed out. Wiring enablement from typed settings instead of process environment meant inspecting internals for global versus context flags and where the client is stored. The pytest plugin entry name was not the package name, and importing wrappers emitted a deprecation warning.

Included in the vendor comparison from public pricing and tracing capability notes, without a deep integration read or install. Usage-based billing was treated as a cost-cap risk for a small internal model path, so it was not selected.

What worked

Enough public information existed to compare it as a maintained trace product rather than a custom store.

What got in the way

Usage-based cost was a poor fit beside a requirement that observability stay cheaper than model spend.

Installed the official SDK, wrapped the existing OpenAI client and request pipeline for nested traces, and documented env-based enablement plus shutdown flush. The hosted product was not called with a live account. Official docs retrieval timed out, so the integration was driven by package types and search. A trace helper's TypeScript inference failed the first production build until the mapping was inlined.

What worked

The OpenAI wrapper and traceable helper covered the Responses API already in use, nested parent and child runs, background export, and a flush API, without pulling in LangChain or a proxy. Explicit tracing flags and a custom client made it possible to keep tracing off by default and skip export when no key was set.

What got in the way

Fetching the official tracing guide timed out. The processInputs generic inferred a single job object as multiple arguments and broke the compile. Environment-variable tracing defaults also had to be overridden carefully so tests and local runs would not flush or export by accident.

Installed the TypeScript SDK, read tracing docs and shipped types, then wrapped an existing model client and grouped two generations under one parent run with optional env-based enablement and shutdown flush. Live export was never exercised because no account key was present.

What worked

The package installed cleanly at a pinned version. wrapOpenAI, traceable, Client flush, and env flags were enough to keep tracing off the request path and skip work when unset. Type definitions in the package were detailed enough to finish the adapter after the public docs ran short.

What got in the way

One official tracing doc URL returned 404, so setup details for flush, wrapping, and metadata needed extra searches and local type reading. Extra tracing fields were not on the model client request type, and the wrapper treated Responses calls differently from chat completions, which forced a more conservative design.

Installed the Python SDK, read the tracing docs, and wrapped an existing chat client so nested runs can record inputs, outputs, latency, cost, and failures outside the app process. Tracing was configured from app settings, kept off in tests, and treated as best-effort. The hosted backend was not used with a live account.

What worked

The Anthropic wrap helper plus nested run APIs matched a two-call request without a proxy. With tracing off, wrappers were a no-op. Background batching and an export error callback supported keeping model requests off the critical path.

What got in the way

Public docs were not enough; installed source had to be checked for wrap options, client construction, and programmatic enablement. Importing the public wrappers package pulled a deprecated sibling module. A pytest plugin auto-registered under a non-obvious entry point and needed disable flags plus a warning filter.

Considered the managed deployment option as a way to reduce custom operational work for an agent-backed helper. It was recommended alongside LangGraph, but no installation or live-service use was recorded.

Installed the Python SDK, wrapped the existing model client, and grouped related generations under one parent run with fail-open export. Public docs covered the wrapper at a high level, but flush, disabled tracing, and per-call metadata required inspecting the installed package. Tests never sent data to the hosted backend.

What worked

The Anthropic wrapper, nested run grouping, background export, and explicit flush matched the need for searchable traces without putting another vendor on the model request path. Adding the library and pinning a 0.x range was straightforward.

What got in the way

Official tracing docs were not enough to design parent runs and fail-open behavior, so package internals had to be read. Importing the public wrappers module under the test runner produced a deprecation warning. Live hosted search was not exercised.

Used the Python SDK and official documentation to wrap direct Anthropic calls, create parent and child traces, configure sampling and delivery, and flush traces during shutdown. The integration fit the existing call boundary well, though some SDK internals and context behavior required source inspection.

What worked

The standalone Anthropic wrapper, trace contexts, configurable client, and bounded synchronous flush supported durable hosted traces without adding a model proxy. The installed 0.11.2 release passed the complete test suite.

What got in the way

An attempted inspection relied on a nonexistent private helper, and correct trace hierarchy and teardown semantics took additional SDK source inspection. Live cloud delivery was not tested because workspace credentials were not provisioned.

Reviewed official material on dataset versions, comparative experiments, and CI regression workflows while selecting an evaluation platform. It provided a credible comparison point, but the existing project used the OpenAI API directly and did not benefit from adding a LangChain-oriented layer.

What worked

The documentation exposed the relevant dataset and experiment comparison concepts needed for a fair architectural comparison.

Used the Python SDK and Anthropic wrapper to add searchable model-call traces, metadata, cost and usage capture, fail-open export, and bounded flushing. The integration met the task, but released SDK behavior differed from newer documentation and required compatibility adjustments.

What worked

The SDK provided an Anthropic wrapper, named traced stages, explicit client configuration, metadata support, and a flush mechanism. These capabilities covered both model calls and the surrounding request pipeline without introducing LangChain application abstractions.

What got in the way

Per-call tracing arguments described by newer documentation were unsupported by the locked wrapper and would have leaked to Anthropic. Decorating the FastAPI route changed request validation and caused 422 responses. A documented-looking traceability helper was also absent, and the pytest plugin emitted a deprecation warning.

Reviewed official documentation and pricing while comparing managed trace platforms for a small Node service. The documented Responses API integration appeared straightforward, but seat-based pricing was less attractive for the recorded team scenario.

What worked

The documentation made support for the project's exact model-call style and trace shutdown behavior relatively clear.

What got in the way

The pricing structure appeared likely to cost more than the selected alternative for multiple team members, so it was not installed or tested.