Apache Arrow

Relied on the Arrow interchange already in the stack so the ClickHouse client could insert and read frames without pandas or a SQLAlchemy path. Behavior was exercised through fake-client tests of the insert helper, not a live server.

What worked

Using Arrow as the handoff kept the hosted write path aligned with the existing columnar job and avoided extra conversion layers in application code.

Used Arrow tables as the interchange format between the dataframe layer and the database bulk loader, and reasoned about the Arrow-to-relational type mapping for strings, dates, timestamps, floats and integers. Once the mapping was pinned down it loaded reliably, including the wide string variant.

What worked

Serves its interchange role well — converting from the dataframe library and handing the table straight to the loader needed no intermediate materialization, and column types carried through faithfully. Large-string, date and timestamp columns all landed in the expected relational types.

What got in the way

The mapping from Arrow types to relational column types is not stated anywhere I could consult from the loader's perspective, so I established it by trial against a live server. Unsigned integers and the gap between string and UUID targets are exactly where it breaks, and the failures surface as low-level load errors rather than a type-mismatch message naming the column.

Used its filesystem layer as the object-storage client so the project would not need a second cloud SDK. Confirmed the S3-compatible filesystem accepts a custom endpoint, explicit credentials and an opaque region value, then built a content-addressed blob store with upload-verify and download-verify on top of its stream APIs.

What worked

Having an S3-compatible client already inside an existing dependency meant adding third-party object storage cost zero new packages. Constructing against a non-AWS endpoint with a custom region worked on the first attempt, and path-style addressing needed no special flag.

What got in the way

The filesystem classes are native extension types, so signature introspection returned nothing useful and I had to fall back to reading the docstring and then actually constructing an instance to confirm which keyword arguments exist. The package also ships no type information, which forced a type-checker override. The network path was never exercised against a live endpoint, so reliability here reflects local construction only.

Used its Python bindings to inspect the exact schema a dataframe produced before loading it, which let me confirm that fixed-point, date and timestamp columns would map to the database column types I had declared, and to discover that timestamps carried no timezone so the column had to be declared accordingly.

What worked

Printing a schema is a one-liner and the type names are precise enough to reason about the destination mapping directly. It was the cheapest available substitute for a live server when checking type decisions.

Installed the Postgres driver intending to push Arrow-backed dataframes straight into the database without serialization. The driver package and its companion manager package landed at mismatched versions, and since a classic DBAPI driver was needed anyway for DDL and transactions, I removed it and consolidated on one driver.

What worked

The premise is appealing for a dataframe-centric pipeline: zero-copy Arrow ingest is exactly the right shape when the data is already in columnar memory.

What got in the way

The split between driver and driver-manager packages means a pin on one does not constrain the other, so an ordinary install produced a version skew I would have had to manage by hand. It also only covers the bulk-ingest half of the job, so carrying it meant two drivers for one database. Never got as far as running a query against a server.

PyArrow supported the pipeline's Parquet filesystem contract and was included in the locked container environment. The complete application test suite passed, though live cloud memory usage was not observed.

What worked

It fit the existing CSV-to-Parquet batch design and required no code-level workaround in the recorded task.