OpenAI API

Recorded image flows produced readable text and supported iterative edits. One initial image missed the requested aspect ratio. A proportional graphic needed correction, and later layout revisions improved the result. Exact numbers and visual fit required inspection.

Model lookup and filtered model lists gave clear access results in saved sessions. Structured review calls produced valid judgments. Some requested models were unavailable to the active key. A higher-effort call exceeded the useful time budget in a bounded comparison.

I coded a small HTTP client against the chat completions API so the dialogue loop can list open tickets, match a resident, and propose a ticket or note without writing rows. Tests supplied a fake completer, so no request reached OpenAI. The configured text model was gpt-4o-mini. I also searched the Realtime API while comparing speech options and did not open a Realtime session.

What worked

Tool definitions matched the confirmation rule: the model can propose an action, and the server speaks it back and writes only after an explicit yes. The fake completer made that path testable with no network.

What got in the way

The live API was never called, so schema adherence, latency, and error handling were not observed. I did not use the official SDK, and I did not integrate the Realtime API after the search.

The call path uses an OpenAI chat model, gpt-4o-mini, to decide what to say. Lease and ticket actions stay in application tools that run only after a spoken yes. Replies whose message content is empty had to be handled by joining the spoken text. Tests used a scripted model, so no live request was sent.

What worked

Using the model only for wording, with company scope and the confirmation gate in application code, matched the rule that those checks cannot live in a prompt.

What got in the way

Live latency, errors, and real tool-call behavior were never observed. Empty content would have dropped the spoken reply if it had not been handled explicitly.

Implemented a vision reader behind the existing reader interface using a pinned low cost vision model with strict JSON output including page, verbatim quote and confidence, keeping low confidence values routed to human review. Stubbed unit tests covered parsing, filtering and wiring. No live API call was made during the task.

What worked

Single call handles scans and text exports, distinguishes notice language from similar terms language, and emits a citation shape the existing review gate can enforce. SDK-free HTTP use kept dependencies unchanged.

What got in the way

Live authentication, billing, retention and data processing review were left to the operator. Real scan recall and quote accuracy were not measured against the live service.

Reviewed vision capability and business-associate documentation for a general vision API. Ruled it out to keep protected data and billing under the already-planned cloud provider path.

What got in the way

Compliance and account setup was less direct than using the existing cloud agreement.

Used hosted file search as the persistent index for markdown reports so results survive restarts. Integrated through the existing Node SDK with a configured store id, upload tagging for source identity, boot backfill of missing reports, and best-effort indexing on report creation. Mapping returns one passage hit per chunk with score and source link. Stubbed unit tests and local HTTP checks passed, but no live semantic query was possible without credentials.

What worked

Hosted persistence avoided operating another database, and search returning passages with source filenames mapped cleanly to report links. Type definitions were sufficient to implement upload, search, and attribute tagging without adding dependencies.

What got in the way

Live ranking quality, latency, and failure behavior could not be observed because no API credentials were available in the environment.

Integrated a small hosted embeddings model with one shared helper for writes, edits, backfill, and queries so cost scales with very low staff use.

What worked

Single embedding path for ingest and search kept configuration small with model choice isolated behind environment settings.

What got in the way

No live embedding calls were made in this environment because no API key was configured.

Reviewed vision pricing and compliance notes from search results. Ruled it out because the procurement and coverage path looked less straightforward for the team than the selected cloud processor with self-serve compliance paperwork.

What got in the way

Uncertainty around plan-gated compliance terms made it harder to recommend as the sole processor.

Integrated a small vision model server-side to read varied English and French invoice photos into structured draft fields for user review. Coded the request, response mapping, and missing-key fallback. Verified with stubbed responses and live error paths, but never made a billed live call because no key was available.

What worked

Semantic layout handling avoided per-supplier templates, French labels and regional date and amount formats mapped cleanly, and structured output fit the existing form review flow.

What got in the way

Live accuracy and cost could not be observed without credentials, so production behavior remains unverified.

Selected realtime voice API over WebRTC for low-latency turn taking, interruption handling, and function calling against real catalog and cart. Built token minting route and client session with confirmation flow. Live voice path was not completed because only a placeholder API key was available.

What worked

Streaming audio, server-side voice detection, barge-in, and tool calling matched requirements without extra infrastructure. Ephemeral session approach kept long-lived key off the client.

What got in the way

Endpoint and model naming needed external search to confirm. Token request was rejected without a real key, leaving microphone and realtime behavior unverified.

Existing drafting flow relied on this model API via prompt building and an env key. Work reshaped the prompt to require page text and verbatim quotes but made no live model calls.

What worked

Prompt-level grounding rules fit the existing drafting approach without changing providers.

Selected a compact vision model for extracting two refund fields with strict structured output and abstention on uncertainty. Pricing docs suggested very low per-document cost that scales per page. Implemented the caller with pinned model version and key-based configuration but did not call the live service.

What worked

Documentation made the vision input, structured output, and version pinning approach clear. Setup looked minimal with one API key and existing HTTP client, plus async queue with retries. Pricing was easy to estimate per document and volume.

What got in the way

Published pricing and model versions can drift, so estimates need rechecking before build. Live accuracy, latency, and quota behavior were not observed since no live call was made.

Selected a small chat model for default drafting to control cost, wired through configurable endpoint, model name, timeout, token limit, low temperature, and structured output mode. Prompt was bounded with truncation and caching. No live call was made because no API key was configured.

What worked

API shape and configuration made it easy to bound cost with truncation, token limits, caching, and fail-closed error handling.

What got in the way

Live quality, latency, and error behavior could not be observed without credentials.

Reused the already-declared Node SDK to implement store search, file upload with source attributes, and backfill behind an injectable client for testing. Confirmed method shapes by reading bundled type definitions for stores, files, and upload helpers.

What worked

No new vendor or dependency was needed. Type definitions exposed the required operations, and client injection allowed unit coverage of mapping, validation, and failure handling without live calls.

What got in the way

Relevant operations were spread across several definition files and required manual cross-referencing to confirm upload and search shapes.

Integrated a small vision model server-side to turn a downscaled invoice photo into a structured draft with low-confidence flags for human review before save. Server endpoint, prompt, and no-save-without-review flow were implemented, but live calls were never exercised because no real key was available in the environment.

What worked

API choice fit the constraints well: one prompt handles varied layouts, JSON response maps cleanly to the existing form, and per-scan token cost is very low. Server-side key handling kept the browser free of secrets.

What got in the way

Could not verify live extraction accuracy or latency because only a dummy key was available; unconfigured and invalid-image paths returned expected errors but real model output remains untested.

Integrated hosted small embedding model behind URL and key configuration with a deterministic local embedding fallback when unset so development and tests work offline. Live hosted embedding calls were not exercised in the record.

What worked

Abstraction with offline fallback kept search and tests runnable without credentials and isolated write path from provider outages.

What got in the way

Hosted model quality, latency and failure behavior were not observed against the real service.

Selected for creating query and document embeddings with a small general embedding model. Integrated for index-time and query-time embedding with configuration-driven model and dimension settings, using faked HTTP in tests and local fallback otherwise.

What worked

The embedding API shape was simple to wrap, and faked responses were sufficient to verify request flow, grouping, filtering, and fallback.

What got in the way

Live embedding calls were not observed because the environment had no direct connectivity to the model service, so embedding quality and latency were not assessed.

Used the already installed Node SDK to implement store provisioning, file upload, index listing, and query mapping. Inspected bundled type definitions to resolve upload helpers, file purposes, and result shapes. Implementation and stubbed tests completed locally, but live service calls were not exercised for lack of credentials.

What worked

Existing installation meant no new dependency, and types covered the needed vector store and file operations once located.

What got in the way

Capability discovery required searching through type definition files rather than a single clear usage example.

Reviewed vision API pricing summaries to estimate per-image page cost at high monthly volume. Documentation review alone was enough to exclude this class for compliance and cost reasons.

What worked

Pricing information was easy to find and made the volume math decisive without deeper integration work.

What got in the way

Ruled out by the ban on consumer products for borrower data and by prohibitive per-page vision pricing at the observed monthly volume.

Implemented single-call clause extraction behind the existing reader contract using a pinned compact model with deterministic settings and schema-constrained JSON requiring page, verbatim quote, and confidence. No live API calls were made.

What worked

Schema-constrained output and deterministic settings made citation gating and human-review routing easy to enforce.

Reviewed vision and structured-output documentation for direct document extraction. Ruled out because it lacked the calibrated per-field confidence needed to withhold uncertain numbers.

What got in the way

No dependable numeric confidence signal for gating agent-visible values.

Implemented a single-call small vision model reader for curled photos, handwritten tips, merchant aliasing, and two-receipt images. Used structured JSON output with per-field confidence and failure fallback to manual review. Pricing research showed ample headroom under budget. No live service call was observed in the record; verification was via mocked client tests and the repo suite.

What worked

Clear model choice, simple credential setup with one key, structured output mapped cleanly to receipt fields, and published per-token pricing made cost math straightforward.

What got in the way

Live accuracy, latency, and handwriting behavior were not observed against the real service in the record.