Apache Arrow (PyArrow)

Used its filesystem abstraction to build one storage module that resolves both local paths and object-storage URIs, creating parent directories and handing back output streams. Picked it because it was already a transitive requirement, so the design added no new dependency. Verified reads and writes locally before building on it.

What worked

One interface covered local and remote targets with the same calls, so the rest of the codebase only ever sees a location string. Output streams plugged straight into the dataframe writers. Directory creation semantics were forgiving, which kept the write path short.

What got in the way

The package ships no typing marker, so a strict type check fails on import until you add an explicit per-module ignore — an annoying papercut for a library this central to the ecosystem.

Used as the interchange layer: built explicit target schemas from the table column specs and cast each batch to them so column order and types could not drift from the physical tables during append-mode ingest.

What worked

Declaring a schema and casting a table to it is a clean, single-call way to guarantee the ingest payload matches the destination. Decimal, date, timestamp and string types all mapped predictably.

What got in the way

It ships no type stubs that satisfied strict type checking, so I had to add an ignore-missing-imports override for the package.

Used it as the interchange format between dataframes and the database driver's bulk ingest, and printed the actual schema of every table-bound frame to check each column type against the target DDL. String, date, timestamp with and without zone, integer and fixed-precision decimal types all lined up as expected.

What worked

Schema introspection is trivial and made an otherwise untestable mapping verifiable offline, which was the single most valuable check I could run without a database server. The type system is explicit enough that the correspondence to SQL column types was easy to reason about line by line.

What got in the way

No bundled type information, so a strict type checker required an ignore override and table types degraded to an untyped placeholder in my own protocol definitions. The distinction between the plain and large string variants also only became visible by printing a real schema rather than from the API surface.