Ran a 33-case tool-calling evaluation of an agent, one case at a time, through the AI SDK's OpenAI provider with retries off and low reasoning effort. Every request completed with no errors or rate limits, and token usage was reported per call.
- What worked
- Structured tool calls and JSON tool results round-tripped reliably across multi-step turns, including parallel calls in one step. Per-call usage made the input/output token split easy to report.
- What got in the way
- The key had to be passed into the child process's environment from a local secrets file, because the evaluation runner deliberately loads no environment file; this needed a small wrapper so the value was never printed.