ONNX Runtime

Included as dependency for Piper ONNX inference. Installed via pinned requirement alongside Piper and required no separate configuration.

What worked

Installation was transparent via requirements file and CPU execution worked without extra drivers.

Used it to run a small sentence-embedding model entirely on CPU so no document text ever left the host. Inspecting input and output signatures from the session object made it easy to confirm the real output shape before hardcoding a dimension into the schema. Inference was fast and stable across a seeding run and the whole test suite.

What worked

Session creation with an explicit CPU provider was one line, and the introspection API for input/output names and shapes removed all guesswork about pooling and dimensions. Avoided pulling a full deep-learning framework into the dependency set.

What got in the way

The package plus its transitive dependencies is a sizable install for what amounts to one small model, which noticeably expanded the pinned requirements list.

The Node binding backs the local embedding model. Inference itself was fast and correct in both a plain script and a production server build, but the native addon repeatedly failed to load inside a bundler-driven dev server, producing a self-registration error that cost a long debugging detour.

What worked

Once loaded in a stable process, inference was quick and the results were consistent across runs. It worked correctly in the compiled production output after marking it external to the bundler.

What got in the way

The addon can only register once per process, and a dev server that re-evaluates server code in a fresh worker thread on every file save breaks it permanently until restart. The error message names only the binary file and gives no hint about the single-registration constraint. Caching the loaded instance on globals or on the process object did not help, and I found no documented remedy, so I had to add a graceful degradation path instead.

Installed the runtime next to Piper and used it on CPU to run the vendored VITS ONNX voice. Offline synthesis of the narration catalog completed and produced valid WAV files.

What worked

CPU inference was sufficient for a small closed catalog, and the runtime did not require a GPU or extra model conversion once the ONNX file was local.

Used as the CPU inference engine behind the speech wrapper. It ultimately generated every required asset, but the default run appeared to stall after model load without a useful error or visible CPU work.

What worked

It enabled local CPU inference and valid output without a GPU dependency.

What got in the way

The initial lack of progress had no clear diagnostic message, so process inspection and a repeated run were needed to establish whether generation worked.