Configured a delivery stream to buffer newline-delimited JSON, convert it to a columnar format using a catalog schema, and land it under date-based prefixes with a separate prefix for failed records. Wrote the producer side to batch within the per-call record cap and retry only the rejected subset.
- What worked
- Format conversion against a catalog table removed the need for any separate transform job, and the split between success and error prefixes makes failures recoverable rather than invisible.
- What got in the way
- Destination configuration has many interdependent fields (role references, buffering hints, serializer settings, two prefix schemes) and getting them consistent was fiddly. The biggest trap is that batch writes report per-record rejections inside an otherwise successful response, so a naive producer drops data silently.
