Chose Phoenix as the self-hosted eval store (versioned datasets, experiments pinned to a dataset version, repetitions) and stood up a local SQLite-backed server from the pip package to verify the harness end to end. Seeding, experiment runs, baseline comparison and a GraphQL dataset delete all worked. Installing the server package on the system Python 3.11.2 failed at import time with a dataclasses mutable-default error; a 3.12 interpreter fixed it. Docs for the open-source product were sometimes conflated with the vendor's SaaS docs, and the exact client API had to be read from source rather than docs.
- What worked
- Single-process server started in seconds with telemetry disabled by env var and a health endpoint to poll. Dataset versioning and experiment metadata matched the documented model. Postgres-backed deployment, auth, and secret configuration are all plain environment variables, which made the compose file straightforward. Elastic License with no feature gates is clear for self-hosting. The published container image tag was easy to confirm.
- What got in the way
- The server wheel would not import on Python 3.11.2 because of a frozen dataclass with a mappingproxy default; no hint pointed at the interpreter version. Several doc pages on the vendor site redirected to or described the hosted SaaS product rather than the OSS server, so I had to fall back to readthedocs and the installed source to confirm signatures. Progress bars and emoji in run output are noisy in non-TTY CI logs and need an env var to suppress.