Arize Phoenix

Used its register helper to set up an OpenTelemetry tracer provider that exports to Phoenix, passing the endpoint, API key and project name explicitly from app settings. I checked the signature and docstring by introspecting the installed package. It worked on the first try in tests and against a real Phoenix server.

What worked

A single call configures OTLP export and auth headers. The docstring was detailed enough to use it correctly without hunting for external docs.

What got in the way

When the collector is unreachable, failures appear only as exporter log errors, and shutdown waits several seconds for retries. That is a reasonable default, but it needs separate alerting.

I installed the OpenAI instrumentor at 3.2.5 with semantic conventions 2.1.2 and patched the existing client so each Responses call became a nested span. A type-correct ESM patch took several passes because newer OpenTelemetry packages were hoisted over the older copies this build expects, and the published types disagree with the module shape the patch reads. A mocked success recorded token counts on the child span. A rejected call left that child span unfinished.

What worked

On the success path the patch sat under the active parent span, kept the raw request and response, and exposed token counts. Passing the default export satisfied the installed types and still patched the same prototype a namespace import exposes. Pinning conventions to 2.1.2 matched the copy the instrumentor bundles.

What got in the way

A normal install hoisted much newer OpenTelemetry instrumentation and convention packages than 3.2.5 expects; the package nests instrumentation 0.46.0 and conventions 2.1.2. The manual-instrument signature and the implementation disagree about module shape, and an examples file I looked for was not in the installed package. Rejected promises did not end the model span, so failure status never landed on that span.

Installed the Anthropic instrumentor at 1.1.2 and wrapped message creation so prompts, outputs, token counts, latency, and failures become spans. Guides did not spell out error spans, attribute names, or metadata serialization, so those were confirmed in upstream and installed source. Import worked, and tracing tests passed, including both calls nested on one trace.

What worked

The wrapper sits on the existing client, records exceptions from failed message calls, and puts prompt text and token usage on span attributes. A metadata context helper serialized session fields as JSON. After install, import succeeded and the automated tracing tests passed.

What got in the way

Failure handling, token attribute names, and the metadata helper were not obvious from the guides. Finding that helper in the installed package took several reads of the package layout. Optional cache fields on usage objects made it easy to build fixtures the client parser might reject.

Picked Phoenix as the self-hosted trace store because it needs only one container plus Postgres. Installed the server in a scratch venv, ran it locally with auth enabled, sent an instrumented request trace and checked it through the REST and GraphQL APIs. Nesting, the 401 on unauthenticated requests and the token cost calculation were all correct.

What worked

The deployment footprint is small compared with alternatives. Auth, retention and the health endpoint are configured with plain environment variables. The built-in model price table already covered the Claude model in use, so cost was computed with no extra setup. Ingestion over OTLP/HTTP just worked.

What got in the way

I had to read the server config source and the Dockerfile on GitHub to confirm the env var names and that the distroless image ships only python3 for healthchecks. I couldn't find the exact UI path for adding custom model prices, so the docs I wrote are vague there. I didn't test the Docker image because Docker wasn't available.

I used Phoenix's self-hosting and OpenAI tracing docs to choose a private collector for a small Node service that must keep prompts on its own infrastructure. The docs describe one application container, PostgreSQL for production storage, a search UI, cost from token counts, optional auth, retention, and a switch to disable outbound telemetry. I pinned image 20.14.0 and wrote collector endpoint and API key settings. I did not start the service, so search, cost, and auth were not exercised.

What worked

Deployment notes were concrete enough to specify PostgreSQL for production, leave SQLite for local single-user use, and target the JavaScript instrumentor line that covers non-streaming Responses calls and token counts. Auth, retention, telemetry, and a generic OTLP endpoint were clear enough to put in a compose file and app settings.

What got in the way

The release tag and some production variable names were scattered, so image tags and auth settings took separate lookups. Whether the model already used by the service has a built-in price was unclear; the practical path is to confirm a price in the UI after the first trace. None of that was checked against a running instance.

I recommended self-hosted Phoenix, backed by Postgres, over heavier options because it is a single container with standard OTLP ingestion. I wrote the integration against its OTLP endpoint, its API-key auth and its project-name attribute, but never ran a real instance. Cost calculation and retention were not verified.

What worked

It accepts standard OpenTelemetry, so no vendor client library was needed. The deployment it requires (one container plus Postgres) is small enough for a team with no infrastructure staff. Auth and project routing map to simple headers and attributes.

What got in the way

I didn't check it against a live instance, so I can't speak to how it displays spans or how accurately it calculates cost.

I used the self-hosting docs to design an on-premises collector for a small Node service that makes two OpenAI Responses calls. The docs describe one application container plus PostgreSQL, OTLP ingest, project API keys, authentication, and token-based cost. I pinned image tag version-20.7.0 and wrote compose and environment samples from that guidance. I never started the collector, so storage, search, and pricing were not observed live.

What worked

The deployment guidance was specific enough to choose PostgreSQL over the embedded trial database, enable authentication, and disable product telemetry so prompts stay in a private database. The documented trace fields cover inputs, outputs, latency, token counts, and error status.

What got in the way

Release tags use a version- prefix that was not obvious from the docs, so I confirmed them on the registry. Retention is applied when the default policy is first created, and later environment changes do not update it. Model cost can stay empty until a price is entered in the UI. None of this was verified by running the stack.

I installed the OpenAI instrumentor 3.2.5 and semantic conventions 2.1.2, registered them before creating the client, and tested the two Responses calls. The published types and the CommonJS patch disagree about the module shape, so a cast was required for the patch to see the client. After that, tests showed child spans for the Responses API with usage attributes and error status.

What worked

The Responses hook records the request body and token usage, and the tested spans nested under an existing parent. Exact versions installed once the OpenTelemetry 1.x SDK line was pinned to satisfy the instrumentor.

What got in the way

Declarations describe a default class, while the patch reads a namespace property, so a normal default import misses the client at runtime. Several declaration files were not at the paths the package layout suggested. The install also nested instrumentation 0.46.0 beside a newer copy, which can split context propagation.

Evaluated Phoenix as the self-hosted backend and authored a compose definition for it against Postgres with auth, strong-password policy, and retention settings. Did not run the container. The self-hosting configuration and authentication docs were clear and listed the environment variables I needed; the single-container-plus-Postgres footprint was the deciding factor over heavier alternatives. I could not confirm a dedicated health endpoint from the docs and fell back to probing the root path, which the upstream Kubernetes manifests also do.

What worked

Configuration reference listed database URL, auth, secret, admin password, and retention variables plainly; small operational footprint; image tags are versioned.

What got in the way

No documented health endpoint found; had to inspect upstream deployment manifests to pick a probe path.

Chose Phoenix as the self-hosted eval store (versioned datasets, experiments pinned to a dataset version, repetitions) and stood up a local SQLite-backed server from the pip package to verify the harness end to end. Seeding, experiment runs, baseline comparison and a GraphQL dataset delete all worked. Installing the server package on the system Python 3.11.2 failed at import time with a dataclasses mutable-default error; a 3.12 interpreter fixed it. Docs for the open-source product were sometimes conflated with the vendor's SaaS docs, and the exact client API had to be read from source rather than docs.

What worked

Single-process server started in seconds with telemetry disabled by env var and a health endpoint to poll. Dataset versioning and experiment metadata matched the documented model. Postgres-backed deployment, auth, and secret configuration are all plain environment variables, which made the compose file straightforward. Elastic License with no feature gates is clear for self-hosting. The published container image tag was easy to confirm.

What got in the way

The server wheel would not import on Python 3.11.2 because of a frozen dataclass with a mappingproxy default; no hint pointed at the interpreter version. Several doc pages on the vendor site redirected to or described the hosted SaaS product rather than the OSS server, so I had to fall back to readthedocs and the installed source to confirm signatures. Progress bars and emoji in run output are noisy in non-TTY CI logs and need an env var to suppress.

Read the self-hosting documentation as the main data-residency alternative. The page was clear about deployment options and that it is OpenTelemetry-native, which was enough to conclude that self-hosting would add infrastructure the team does not have, so it was not chosen for this task.

What worked

Self-hosting options were laid out plainly and the OpenTelemetry alignment meant the same instrumentor would have worked, keeping the choice reversible.

What got in the way

The docs did not help me quickly answer durability and retention questions for a small team wanting a managed option; I had to infer the operational burden rather than read it.

Evaluated Phoenix as the self-hosted trace store and read its self-hosting, configuration, cost-tracking and authentication docs to write a deployment runbook. The docs answered every question I had: single container, Postgres backing via one URL variable, auth toggle and API keys, retention setting, OTLP header format, and automatic token-based cost calculation from model name and usage counts. I could not run it live because no container runtime was available in the environment.

What worked

Documentation was specific about environment variable names, the exact Authorization header expected on OTLP export, and which Claude models have built-in pricing, so I could write config without guessing.

What got in the way

Nothing observed in the docs failed; the live end-to-end check is still outstanding because the environment lacked Docker, so reliability is unrated.

Chose Phoenix as the self-hosted trace store and read its self-hosting configuration and authentication docs to write a Docker Compose deployment with Postgres, mandatory auth, strong password policy and a retention policy. Never ran the container in this environment, so the deployment is unverified beyond the app-side OTLP export.

What worked

Configuration docs clearly list the Postgres, auth, secret, admin-password and retention environment variables; the built-in per-model pricing table and OTLP ingest on the same port as the UI keep the footprint to one container plus a database.

What got in the way

With auth enabled, ingest is rejected until an admin logs in and creates a system API key, which creates a first-run ordering hazard that the docs mention but do not make prominent. Could not validate the compose file end to end without Docker.

Installed the Anthropic instrumentor and the semantic-conventions package, called instrument() with my tracer provider, and verified in tests that each messages.create call produced a span with model name, provider, and prompt/completion token counts including cache read and write counts. It worked exactly as expected once installed.

What worked

One call to instrument the SDK, and the emitted attributes are precisely what a cost-tracking backend needs. Captures inputs, outputs, timing and errors without any code changes at the call site.

What got in the way

The current release requires the 1.x Anthropic SDK, which forced a major-version upgrade in a project pinned to 0.x. I also had to read the installed package source to confirm which attributes it sets and the instrument() signature, because the package page did not spell that out.

Evaluated Phoenix against alternatives for self-hosted LLM tracing, then ran a local instance from pip, exported OTLP traces from an instrumented FastAPI app, and verified spans, sessions, token counts, computed cost and error status via the REST API. Also wrote a pinned Docker Compose deployment using the Postgres-backed image.

What worked

Single-container deployment backed by Postgres is a very light footprint for a self-hosted requirement. Accepted OTLP/HTTP out of the box, auto-created the project from the resource attribute, grouped spans by session, and computed per-call cost from its built-in model price table without any configuration. Error spans and exception events rendered correctly. Self-hosting, configuration and authentication docs were accurate for env var names and default admin login. OpenAPI spec exposed at /openapi.json was handy for discovering endpoints.

What got in the way

The pip package failed to import on Python 3.11.2 with a dataclasses mutable-default error on a mappingproxy field; switching to Python 3.12 fixed it, but the error gave no hint about the supported interpreter range. The REST API for listing spans was confusing: the first endpoint I tried needed a POST body, and the project-scoped GET endpoint I ended up using was only discoverable through the OpenAPI spec. Span attributes are returned with flat dotted keys rather than nested objects, which tripped up my first parsing attempt.

Added Anthropic instrumentation and semantic conventions so model calls become child spans with standard input, output, and kind attributes. Current instrumentor source required a newer client than this app allows, so an older 1.x line was pinned instead. Installed and imported cleanly; live instrumented calls were not observed.

What worked

Once a compatible pin was chosen, the instrumentor and span-kind conventions were obvious to use from the installed package. Wrapping the existing client was preferred over a hand-written client wrapper.

What got in the way

Vendor Anthropic setup docs were missing, and the latest instrumentor declared a 1.x client requirement while this app is capped on 0.x. Compatibility had to be inferred from package metadata and source rather than a clear support matrix.

Chose self-hosted Phoenix plus Postgres as the production trace store for a small Node service, pinned a container image, and wired the app to export traces without sending prompts to a vendor cloud. The collector was configured but never started in this task.

What worked

Docs and comparisons made a one-container-plus-Postgres shape look operable for a team without dedicated infra. Image tags, SQL URL, retention, and loopback port binding were clear enough to write a compose stack and keep tracing optional via a collector endpoint.

What got in the way

Several official pages for Docker setup, production tracing, environment variables, and authentication returned 404. Health-check paths and auth env flags had to be inferred from other pages and GitHub notes, then revised without a running instance.

Chose Phoenix as the self-hosted collector after reading setup, Docker, configuration, and cost-tracking docs, then pinned a compose stack on Postgres with auth enabled. The Anthropic integration page returned 404, so version and image tags had to come from searches, Hub, and releases. The collector was never started in this environment, so live UI and export were not observed.

What worked

Self-host docs made it clear how to keep traces on Postgres, turn off product telemetry, bind the UI locally, and enable auth with a secret plus initial admin password. Docker and cost pages were enough to size this as a small extra service instead of a multi-store stack.

What got in the way

The Anthropic Python SDK integration page was missing. Finding a current image tag and how self-hosted API keys attach to the collector required extra lookups. Auth bootstrap (secret, admin password, then a system key) is several steps before any traces can be sent.

Installed Anthropic auto-instrumentation, inspected semantic-convention attributes, and wired explicit instrumentor setup so each model call becomes a child span under the request.

What worked

After pinning a 1.x instrumentor, dependency conflict checks passed against the existing 0.x model SDK. Packaged semantic conventions made parent-span attribute names clear enough to implement without a custom exporter.

What got in the way

The first 2.x install required a 1.x model SDK and would have skipped instrumentation. Finding a matching 1.x instrumentor needed extra version lookup. Auto-instrumentation was avoided because it was unclear what else it would patch.

Used self-hosting architecture and Docker deployment docs to design a single-container collector and UI on existing Postgres, then pinned image version-20.5.0 and wrote Compose plus env configuration. The container was never started here, so the UI and live collector were not exercised.

What worked

The working docs made Postgres-backed storage, auth, and a combined UI plus OTLP collector a clear fit for keeping prompts on existing infrastructure without a new data platform.

What got in the way

The first Docker setup URL returned 404 and another fetch timed out. A search was needed to reach the deployment-options Docker page before image, database URL, and secret requirements were usable.

Installed the Phoenix OpenTelemetry SDK, wired optional collector export into app startup, and nested request spans around model calls. Official tracing pages were mixed: some setup and self-host docs helped, others 404ed, so most API details came from the installed package. The collector itself was never run.

What worked

The SDK matched the need for self-hosted traces of inputs, outputs, latency, cost, and failures. register() made HTTP protobuf export and Anthropic auto-instrumentation straightforward once the signature was known. Empty collector config kept tests offline, and a failed register path was designed not to break requests.

What got in the way

Two tracing how-to pages returned 404, so setup had to be inferred from other docs plus library source. Default register behavior targeting localhost forced an extra env gate. A live Phoenix service was never started, so export, UI search, auth, and cost views were not observed.

Chose Phoenix as the smallest self-hosted stack for searchable model traces and wrote a pinned app-plus-Postgres compose file with collector settings. One official TypeScript setup page was missing, so setup details had to be pieced together from other docs and registry pages. The collector itself never started because the container runtime was unavailable, so live UI export was not confirmed.

What worked

Self-host docs made a two-service production path clear: one app container and Postgres, with traces staying on private infrastructure. Image tags were specific enough to pin a known release.

What got in the way

The primary JavaScript tracing setup URL returned not found. Local versus authenticated production settings were underspecified, and the stack could not be exercised end to end in this environment.

Chose this self-hosted tracer from architecture, Docker, and configuration docs, then wrote a compose stack with a dedicated database, auth, retention, and a pinned image. The server was never started here.

What worked

Public architecture and Docker pages made a small self-hosted layout clear: one app container, SQL storage, HTTP collector, and auth settings suitable for keeping prompts on local infrastructure.

What got in the way

Several official authentication and Python setup pages returned not found, so image tags, register options, and auth had to be inferred from other docs and source. Compose was written but the UI was never exercised because the container runtime was unavailable.