Amazon Kinesis Data Streams

Selected as the buffer for bursty ordered readings with multi-day retention and independent consumers. Implemented a stream abstraction with an in-memory backend for local use and tests plus provisioning config for the managed stream. Live stream was never provisioned or called during the task.

What worked

Capability fit was clear for per-key ordering, multiple independent readers, adding a new reader without disturbing others, and multi-day replay.

What got in the way

Live behavior, throughput sizing, retention cost, and provisioning were not verified in the record; provisioning was explicitly left unapplied.

Chose the stream as the durable buffer so ingest can acknowledge quickly while each consumer tracks its own position with replay. Implemented per-unit ordering, multi-day retention sizing, and size-aware batching with retry of throttled records, verified only with a fake client.

What worked

Ordering, retention, independent checkpoints, and replay mapped cleanly to burst absorption and adding a fourth consumer without disturbing others.

What got in the way

Never ran against the live service in the record, so throughput, scaling, permissions, and error behavior remain unobserved.

Selected as the ingest buffer between the gateway and four independent consumers. Designed batched publishing with per-trailer ordering and multi-day replay to address sustained and burst throughput, deploy-time loss, and late alerts while staying on the existing cloud bill. Authored the stream abstraction and infrastructure definition and verified behavior with a local in-memory stand-in.

What worked

Documented concepts for partition keys, batch publishing, independent consumer offsets, retention, and iterator-age monitoring mapped cleanly to the decoupling, ordering, replay, and late-join requirements. Cost and throughput guidance made shard sizing tractable within the stated budget.

What got in the way

No live stream was exercised in the record. Throughput, throttling, retention, and consumer-lag behavior were validated only through the local stand-in, so production performance remains unobserved.

Recommended and built an ingest-to-consumers buffer on Kinesis: PutRecords keyed by unit, per-consumer shard reading with checkpoints stored in Postgres, provisioned shards and alarms in Terraform. Never ran against the real service; exercised only via an emulator and the botocore model.

What worked

Shard limits, PutRecords batch size, retention and per-shard read limits were clear enough to size the stream and decide enhanced fan-out was unnecessary for four consumers. Partial-failure responses on PutRecords make retry logic straightforward.

What got in the way

Without the KCL, resharding (parent/child shards), iterator expiry and checkpointing all have to be handled by hand, which was the hardest part of the consumer loop.

Recommended Kinesis and built against it: partition key per trailer, 7-day retention, and enhanced fan-out so each consumer reads on its own. Wrote the writer, polling and fan-out readers, and Terraform. I never ran it against the real service, only against a local emulator and fake clients, so I can't speak to its reliability.

What worked

The model matched the requirements well. Partition-key ordering, extended retention for replay, and up to 20 fan-out consumers per stream make adding a new consumer cheap. It is also cheaper and simpler for a small team than running brokers.

What got in the way

There is no first-class Python consumer library (KCL), so I had to write checkpointing, shard handling and resubscription myself. A partial PutRecords failure that gets retried can break ordering, so I grouped all of a unit's readings into one record.

Selected as the buffer between request validation and four independent consumers, using partition by unit for ordering, multi-day retention, on-demand capacity for bursts, and per-consumer enhanced fan-out with checkpoints. Implemented publish and shard-read wrappers plus an in-memory substitute for tests. Live stream was not exercised and infra validation was still pending.

What worked

Throughput, retention, ordering, replay, and independent fan-out mapped cleanly to sustained load, burst peaks, deploy-loss, lag, and fourth-consumer requirements.

What got in the way

Live behavior, scaling, permissions, and alarm routing could not be confirmed without the real service and infra validation.

Picked Kinesis as the buffer between an HTTP ingest endpoint and several consumers. It fit the load (about 1,600 records/s at peak), per-key ordering, replay, and adding consumers independently through enhanced fan-out. I wrote the producer with batched puts and partial-failure retries, plus Terraform for a 4-shard encrypted stream. I had no AWS credentials, so I never ran it against the real service.

What worked

The capacity model (per-shard write limits, partition key ordering, retention for replay) was easy to reason about when sizing shards. Enhanced fan-out cleanly met the requirement to add a fourth consumer without touching the others. Because PutRecords reports per-record failures, retrying only the refused records was simple.

What got in the way

Kinesis gives no backlog count per consumer, only iterator age, so the lag endpoint had to report time-based lag and a null row count. Pricing could only be estimated, not confirmed.

Selected as the buffer to decouple bursty ingest from multiple independent consumers, preserve per-unit order, retain data across deploys, and allow a new consumer without changing existing ones. Implemented behind a local memory backend and validated with a fake client; live cloud verification was out of scope.

What worked

Partitioning, batched writes, multi-day retention, and independent checkpoints mapped cleanly to burst, deploy-loss, lag, and fan-out requirements.

What got in the way

Live throughput, retention, and replay were not observed because cloud access was unavailable, so shard and retention choices remain unproven in production.

Chose Kinesis Data Streams as the buffer for a few hundred to ~1,600 small readings per second, keyed by unit for per-unit ordering, with independent consumer positions and multi-day retention so a new consumer can replay. Wrote the producer (PutRecords in 500-record chunks with partial-failure retries) and Terraform for a 4-shard stream. Never ran against the real service; exercised only through a local mock, so reliability is unrated.

What worked

The model fit the requirements cleanly: per-key ordering, per-consumer checkpoints, replay from oldest retained data, and low cost at this volume compared with a managed Kafka cluster. PutRecords partial-failure semantics are clear enough to build a bounded retry loop.

What got in the way

Needed care to keep producer latency under a hard client deadline: per-call limits and SDK-level retries had to be accounted for explicitly. Shared-throughput read limits per shard mean more than a few consumers require enhanced fan-out, which adds configuration.

I read the pricing page and ordering notes, then coded a producer that puts one record per partition key with an ordering token, and readers that each use enhanced fan-out against seven-day retention. Tests ran against an in-memory stand-in, not the live service. On-demand, extended-retention, and fan-out prices were not together on the first pricing view, so I searched again. Sequence numbers are variable-length strings and must be compared as integers.

What worked

Partition order, a separate fan-out consumer per reader, on-demand capacity, and multi-day retention matched the buffer, the extra consumer, and the replay window.

What got in the way

Pricing took two page fetches and extra searches before the on-demand, retention, and fan-out rates were clear. I never saw a live put, subscribe slot, or throttle. A string compare of sequence numbers would stall or rewind a cursor when lengths differ, and I only fixed that in code.

Chose an on-demand stream as the durable buffer between accept and consumers, partitioned so one source stays ordered, with each consumer on its own checkpoint and records kept for seven days. Publish and read were implemented against the data-plane operations, and on-demand pricing was looked up once. The live service was never called; tests used a stand-in client.

What worked

Independent checkpoints let another consumer subscribe without a schema change or a shared group. Partition ordering, retention, and acknowledging ingest only after a successful put match the loss and fan-out constraints. The pricing search returned figures that were usable for the budget check.

What got in the way

No call reached the real service, so throughput, throttling, and iterator behavior were not observed. Shard listing must not send the stream name together with a continuation token, and some read actions are not authorized on the stream ARN alone, so the access policy needed a broader resource to be safe.

Public pricing pages were specific enough to choose on-demand standard capacity, enhanced fan-out, and seven-day retention, and to reject a higher on-demand tier whose ingest minimum exceeded the spend cap. Ingest publishes each batch once and returns, and each consumer keeps its own cursor so another reader can join without moving the others. The live service was never called; a fake client and unit tests stood in for it.

What worked

On-demand mode matched a small but spiky workload without shard planning. Separate enhanced fan-out subscriptions give each consumer an independent position, which is what adding another reader required. Seven-day retention covers a consumer that is many hours behind, which a short-lived queue would not.

What got in the way

Enhanced fan-out pricing on the on-demand tier was ambiguous, in particular whether those reads replace or add to the standard retrieval rate. Subscription details (child shards only on the final event, reconnect when an iterator ends, and lag as iterator age rather than an unread count) were easy to mis-handle and were never checked against a live stream.

Selected a provisioned stream as the buffer between ingest and independent consumers, then coded batch puts, a stable partition key, enhanced fan-out, and multi-day retention. Public pricing pages were checked first and supported a monthly estimate inside the approved cloud spend. Wrapper tests passed with local stand-ins. The live service was never called, and the infrastructure templates were not applied.

What worked

Published shard and record-rate limits were concrete enough to size a small shard count for the steady rate and the short reconnect burst. Independent fan-out consumers and per-shard checkpoints match a design where another consumer joins without changing the others. Retention covers the deploy window that was dropping data. Pricing pages were sufficient to compare provisioned capacity with on-demand fan-out and to keep the design on the existing cloud bill.

What got in the way

No live put, subscribe, or infrastructure plan was run, so service behavior under load was not observed. The reader API is easy to misuse: shard listing rejects a stream name combined with a pagination token, fan-out registration is not ready immediately, resubscribe can fail as resource-in-use, and a restart from an old checkpoint can pull the retention window into a small task unless the client closes the subscription and bounds its buffer. An early on-demand reading was later replaced by a provisioned estimate.

Kinesis Data Streams was chosen as the buffer after checking on-demand pricing, retention, and fan-out. The implementation uses one on-demand stream retained for seven days, a per-entity partition key, and enhanced fan-out so another consumer can replay without taking shared read capacity. Pricing pages were specific enough to size the stream. No live stream was opened.

What worked

Published per-GB, retention, and shard limits were concrete enough to reject the high-floor on-demand mode and to match ordering plus a seven-day replay. Independent consumers and enhanced fan-out fit a reader joining later.

What got in the way

The service was never called, so iterator expiry, consumer registration, and real throughput were not observed. Shard-iterator settings are easy to assemble incorrectly when the iterator type is passed both as its own argument and inside an unpacked parameter map.

Chose a provisioned stream as the buffer between batch ingest and separate consumers, after checking shard-hour and extended-retention pricing. The integration batches puts, retries rejected records, partitions so one source stays ordered, and gives each consumer its own cursor. A new consumer can start at the oldest retained record without moving the others. Live calls were not made; tests used stand-ins.

What worked

Independent cursors and multi-day retention matched the need for another reader to join without a shared lock or a schema change. Pricing pages were specific enough to size a small shard count and a week of retention inside the existing cloud bill. Put retries and a full-batch refusal path were straightforward to fake in tests.

What got in the way

Lag estimated by reading one page per shard becomes a lower bound once a consumer is further behind than that page. Sequence numbers had to be zero-padded before they could order archive keys. The ingest role also needed read access, not only the consumers, because the lag check runs there.

Chose an on-demand stream as the table change destination so the API write stayed a single update, with organization-level ordering, separate consumers, and timestamp replay inside a seven-day window. Regional list prices were estimated from the public offer file, and the stream was defined in the stack. No stream was created in a live account.

What worked

On-demand capacity, partition-key ordering, enhanced fan-out, and timestamp replay lined up with the burst, ordering, and replay requirements. After the offer file was parsed, hourly and per-gigabyte rates were specific enough to price a month of overnight updates.

What got in the way

The public pricing page did not include usable rate tables in the fetched text, so prices had to come from the raw offer file. Ingest, fan-out, and replay were never observed on a real stream.

Chose it as the buffer between a high-rate HTTP ingest path and several independent consumers, then wrote producer and reader code plus infrastructure config against it. Never ran it against a live account, so behavior is unobserved. Sizing, ordering-per-partition-key and multi-reader semantics all fit the workload well, and the per-consumer replay model is exactly what let a fourth consumer be added without touching the existing ones.

What worked

Partition-key ordering and independent per-consumer cursors mapped cleanly onto the problem. On-demand capacity removed shard-count guesswork for a bursty workload. Sizing and service-limits docs were concrete enough to check peak record rate, per-key hot-partition limits and the reader-count ceiling before committing.

What got in the way

Pricing is spread over several pages and modes, and the write-side minimum-size rounding is easy to miss while estimating; one retrieval rate I never found authoritatively. A recently added capacity mode has a large minimum commitment that is a trap for small workloads. A newer managed object-store delivery feature is documented for the API but has no infrastructure-as-code path yet, so I had to fall back to a different service for that leg.

Implemented a provisioned, encrypted stream with retention, batched publishing, partial-failure retries, deterministic event identities, independent consumers, and enhanced fan-out. The design fit ordering, replay, burst handling, and consumer isolation well, but it was not exercised against a live AWS account.

What worked

The partition-key, replay, retention, and independent-consumer model mapped directly to the workload. PutRecords supported efficient batch ingest while exposing individual failed records for targeted retry.

What got in the way

Live service behavior and production capacity were not observed because no AWS resources were applied.

Installed and integrated amazon-kclpy 3.1.3 with a Java-backed worker, DynamoDB checkpoints, initial-position controls, and independent consumer applications. Sample-helper packaging and fan-out configuration required source and documentation investigation, and an eager helper import initially broke test collection.

What worked

The library supplied the required lease coordination, checkpointing, replay start position, and multi-worker model. Its packaged samples helped establish the worker configuration.

What got in the way

An existing environment lacked the package's samples module, causing test collection to fail until the import was deferred. The enhanced fan-out property was not obvious from the immediately available Python documentation, and no live worker was run.

Chose an on-demand stream as the ingest buffer so producers acknowledge quickly and each consumer keeps its own iterator. Implemented put-records publishing, per-consumer checkpoints, and infrastructure for retention and partition keys, with a local log when the stream name is unset. Did not exercise a live stream.

What worked

The service model matched the need: fan-out without coupling consumers, ordering by partition key, replay after a crash, and staying on the existing cloud bill so a new vendor review was not required.

What got in the way

Live put, get, and iterator behavior were never observed; production apply and cutover were left as follow-up, so reliability in the hosted service itself is unassessed.

Selected it as the buffer for a few-hundred-per-second telemetry feed with multi-thousand reconnect bursts, then wrote the producer path, a shard poller with external checkpoints, and the infrastructure definition. The per-key ordering guarantee and multi-day replay were exactly the two properties the workload needed, and independent readers let a fourth consumer join without touching the existing three. Never executed against the real service in this environment.

What worked

Ordering per partition key plus retention-based replay solved both the ordering state machine and the new-consumer-from-the-beginning requirement that a plain queue could not. Shard sizing rules are simple enough to reason about capacity and cost up front. Sequence numbers are durable, which made an external checkpoint store straightforward.

What got in the way

The built-in lag metric is per stream and per shard, not per reader, so with several shared-throughput consumers it cannot distinguish which one is behind; I had to emit a custom per-consumer lag metric to get usable alerting. Shard iterators expiring on a short timer forces extra bookkeeping on top of checkpoints. Batch writes can fail partially, so the producer needs its own per-record retry logic rather than a simple success or failure.

Chose it as the buffer between a high-rate HTTP ingest path and several independent consumers, then implemented publishing, shard reading with checkpointing and leases, plus on-demand capacity and retention in infrastructure code. Semantics were modelled faithfully in an in-process fake; nothing ran against the live service.

What worked

Independent reader positions per consumer are exactly the fan-out property the task needed — a new consumer joins without any change to the existing ones. On-demand capacity removed shard-count planning from the design entirely, and retention gives replay for free, which also solved the deploy-time data loss.

What got in the way

Pricing semantics are subtle enough to change the architecture: per-record rounding up to a minimum payload size made one-record-per-item noticeably more expensive than packing, and the packing alternative conflicts with per-item sequence numbers and with downstream delivery formats. Reader-side correctness (expired iterators, closed shards, lease ownership) is all caller responsibility unless you adopt the heavier client library, which is a lot of behavior to reimplement.

Integrated an on-demand stream as the ingest buffer: validate, publish batched records with a per-unit partition key, return an accepted response, and let each consumer checkpoint on its own. Covered the production adapter with a fake client in tests rather than a live account.

What worked

Independent iterators and checkpoints matched the need to add another consumer without schema or poller changes. Partitioning kept per-unit order for sequential alert logic. Fast publish replaced row-by-row inserts on the request path.

What got in the way

A publishing failure after a dedupe claim would have dropped a gateway retry unless the claim was released. Lag checks that walk large shard histories looked costly, so they stayed conservative.