Amazon Data Firehose

Used it as a second reader on the stream to replace a hand-written archiving worker entirely, configured as declarative infrastructure with buffering, compression, a date-partitioned prefix and a failed-delivery prefix. Never provisioned, so reliability is unassessed.

What worked

Deleting a whole service and its deployment, scaling and failure modes in exchange for one configuration block is a very good trade. Buffering, compression and partitioned prefixes cover what the removed worker did, and the failed-delivery destination gives a visible place for records that do not land.

What got in the way

Record framing is a real trap: the delivered object is the record bytes concatenated with no separator, so the producer must append its own newline for the output to be line-delimited JSON. That requirement shapes the producer's encoding and deserves to be far more prominent than it is. Buffering attribute names have also shifted across provider versions, which made the configuration harder to write confidently without being able to validate it.

Used it as the applied path for archiving stream records to object storage, replacing a hand-written consumer. Configured source, buffering and key prefixes declaratively; never executed against a live account. Dynamic partitioning on a field inside each record preserved the existing date-based key layout, which the newer alternative could not have done since it keys on delivery wall-clock time.

What worked

Mature, well-documented declarative configuration with no code to maintain. Dynamic partitioning expressions let the output keys be derived from a timestamp inside the payload rather than delivery time. Buffering size/interval bounds were clearly stated so I could check my values without trial and error.

What got in the way

It occupies a shared read slot on the source stream, which matters when several consumers already compete for the per-shard read budget, and its delivery guarantee is weaker than the newer native alternative. Those trade-offs are findable but not called out where someone comparing the two would see them.

Used it to replace a hand-written archive consumer: the delivery stream reads the buffer and writes day-partitioned line-delimited JSON objects directly, so no application code sits on the archive path. Configured dynamic partitioning from a timestamp field and mirrored the same partitioning rules in a small local simulator so the behaviour could be tested offline.

What worked

Deleting an entire consumer and its deployment in favour of a managed delivery is a real reduction in moving parts. Because record bodies were already one line of JSON each, concatenated output is a valid archive with no transformation step.

What got in the way

Dynamic partitioning is expressed as an inline query expression over the record payload, which is easy to get subtly wrong and has no obvious offline test path, so I had to reimplement the partitioning logic separately to be able to assert on it. Never run against the live service.

Used it to replace a hand-written archival consumer: configured as a delivery pipeline reading the same stream and writing newline-delimited JSON to object storage, declared entirely in infrastructure code. Declared but never deployed, so behavior is unobserved.

What worked

Deleting an entire long-running service in favour of a managed delivery configuration was the single biggest simplification in the change. Because record bodies are newline-terminated, the output is valid line-delimited JSON with no transform function at all.

What got in the way

It concatenates record bodies verbatim, so any batching on the producer side forces a transform function to unpack them — that coupling silently constrains the producer's record design. Partition prefixes follow arrival time rather than any field inside the payload without extra configuration, which shifts how archived data is addressed.

Considered Firehose as a managed path to archive storage. It could simplify the archival branch, but it is a delivery sink rather than the multi-consumer replayable buffer required for alerts, rollups, archives, and future consumers.

What worked

Managed batching and delivery make it attractive for a dedicated archive pipeline.

What got in the way

It did not replace the need for independent low-latency consumers and consumer-specific replay positions.

Evaluated it to replace a hand-written object-store archiving consumer, costed it from the pricing page, planned it into the recommendation, then dropped it during implementation and documented the deviation.

What worked

Pricing is simple per-ingested-volume and easy to model. Conceptually it removes a whole deployment unit for the archive path.

What got in the way

It delivers records exactly as published. Because each record carries a batch of readings for one device - which is what preserves ordering - the delivered objects would have contained envelopes rather than the one-record-per-line format downstream consumers already depend on. Reshaping that needs a serverless transform function, i.e. a second deployment mechanism for our code, which cost more than the small always-on consumer it replaced. The docs present the transform step as a minor option rather than the real requirement it becomes whenever records are batched.

Chose this as the buffered hop between the application and object storage for a high-volume event stream, and configured a delivery stream declaratively with date-partitioned destination prefixes, buffering thresholds, an error output prefix, and delivery logging. Configured only; never provisioned or run, so reliability is unassessed.

What worked

It fits the problem well: the application only needs a cheap batched put, and buffering, partitioning and retry to durable storage are handled for you, which keeps the hot request path clean. Cost at the volumes in question is negligible relative to the analytics tooling itself. The templated timestamp prefixes map neatly onto warehouse partitioning.

What got in the way

The configuration has sharp edges that only surface at apply or run time: the error output prefix must include a specific placeholder token, and the buffering knobs interact with destination behavior in ways the reference docs describe only loosely. The per-call batch limit and partial-failure semantics also leak into client code rather than being handled by the service.

Researched pricing (per-GB ingestion with 5 KB record rounding, format conversion charges) and delivery behavior to size costs at roughly a hundred million events a month, then designed a Direct PUT to Parquet-on-S3 stream. Pricing was simple enough to make a linear estimate, but the exact rounding rules for conversion billing and whether multi-record payloads are accepted by format conversion were not clear enough for me to state confidently.

What worked

Linear per-GB pricing with no per-user multipliers made the cost model easy to explain and cap with budgets and alarms.

What got in the way

Ambiguity around how conversion is billed and whether newline-delimited batches work with format conversion. Default timestamp parsing format is implicit and had to be matched on the producer side.

Used AWS documentation to check delivery implications for Object Lock, then authored and validated Terraform for the audit delivery path. No records were sent through a live stream.

What worked

The service fit the buffered delivery requirement between regional audit collection and long-term object storage.

What got in the way

Permissions and immutable-storage interactions needed additional documentation research, and live delivery semantics remained unverified.

Firehose was wired into the infrastructure design as the destination for sanitized, metadata-only audit events. This met the export requirement cleanly, but the destination ARN and real delivery stream were external parameters and no records were delivered during the task.

What worked

It provided a straightforward boundary between gateway auditing and downstream security or analytics storage.

What got in the way

Delivery permissions, destination behavior, and failure handling were not observable without the external stream and a deployed stack.

Firehose was selected in the deployment contract as the managed route for sanitized gateway decision events to durable storage and downstream security analysis. It was not provisioned or exercised.

What worked

The managed delivery model fit the need to decouple audit emission from storage and SIEM consumers.

What got in the way

Delivery configuration and failure behavior could not be assessed without platform infrastructure.

Read the pricing page to build a defensible per-month cost estimate at high event volume, then wrote the delivery stream config including columnar format conversion. Never deployed, so only documentation and configuration ergonomics were exercised.

What worked

The pricing page states the per-gigabyte rate, the volume tier boundaries and the format-conversion surcharge plainly enough to compute a figure directly. Critically, it documents the minimum per-record billing increment, which dominates the estimate when records are small and numerous; many vendors bury that.

What got in the way

The cost structure still has several interacting components, so arriving at a single monthly number takes deliberate assembly rather than reading one figure off the page.

Configured a delivery stream to buffer newline-delimited JSON, convert it to a columnar format using a catalog schema, and land it under date-based prefixes with a separate prefix for failed records. Wrote the producer side to batch within the per-call record cap and retry only the rejected subset.

What worked

Format conversion against a catalog table removed the need for any separate transform job, and the split between success and error prefixes makes failures recoverable rather than invisible.

What got in the way

Destination configuration has many interdependent fields (role references, buffering hints, serializer settings, two prefix schemes) and getting them consistent was fiddly. The biggest trap is that batch writes report per-record rejections inside an otherwise successful response, so a naive producer drops data silently.