Amazon Web Services

I listed roughly ten thousand objects to check which local files already had a remote copy, which is what made deletion safe rather than a guess. The listing was fast and consistent. Credential resolution was the only real friction, and it produced errors that pointed at the wrong problem.

What worked

Recursive listing returned about ten thousand keys quickly and in a stable format, so I could compare it against local filenames directly. Repeated runs during the session agreed with each other, which is what let me treat the result as evidence rather than a hint.

What got in the way

A stale profile name in the environment took precedence over explicit credentials, and the resulting error named the missing profile rather than the precedence rule, so it read like a configuration problem rather than an ordering one. Separately, a key scoped to one service returned a long authorization error for another service, which is correct but buries the one useful line.

Used the AWS CLI to confirm a VM was hung (EC2 status checks green but SSM agent lost), read CPU metrics, reboot it, wait for SSM to return, and pull logs from the previous boot with SSM run-command. Everything needed was available from one tool.

What worked

SSM ping status and CloudWatch CPU history quickly told the real story when EC2 status checks still said healthy. Reboot plus SSM run-command gave a clean recovery and log forensics with no inbound network access.

What got in the way

SSM send-command is asynchronous, so the first get-command-invocation calls returned empty output while still in progress, and a shell polling loop was needed. Nesting shell quoting inside the JSON commands parameter is awkward. EC2 status checks staying green during a guest livelock is misleading.

Read API keys for a test harness from Secrets Manager. The SSO session had expired, so the person had to sign in again before the harness could start.

What worked

Once signed in, secret reads were quick and scriptable.

What got in the way

An expired SSO session only shows up when a command fails, and an agent cannot renew it.

SSO device sign-in worked, but the first device code expired before approval and the login had to be restarted.

About 170 aws calls across 21 sessions: Secrets Manager reads, sts get-caller-identity, logs filter-log-events, Lambda configuration and S3 copy. With --query and --output text the answers were compact. The main friction was auth: several profiles, and an SSO session that expired mid-task and needed a person to sign in again in a browser.

What worked

sts get-caller-identity is a cheap first check of which account and role are active. logs filter-log-events with a time window and pattern found the failing invocations without a console. lambda get-function-configuration showed the live concurrency settings. --query cut output to the one value needed.

What got in the way

An SSO token expired mid-session, and every call then failed with 'Token has expired and refresh failed'. An agent cannot finish the browser sign-in, so the task had to wait for a person. With several named profiles, the agent sometimes had to copy the config file to point at the right profile. The error does not say which profile was in use.

The CLI supported precise resource, queue, scheduling, and runtime checks across saved sessions. Account, region, and profile selection were recurring requirements. Expired SSO sessions and narrow roles blocked some wider inventories. Metadata checks helped verify the active context.

Function code hashes and update status made deployment checks precise. Object storage supported shared immutable data and private evidence archives. Explicit profile selection and an account check were needed because the default profile targeted a different environment.

Relied on log, alarm, identity policy, and notification concepts to design a narrowly scoped read-only path for the SRE agent. Drafted infrastructure and policy placeholders with no secret values committed and did not apply or call the cloud APIs.

What worked

Read-only log and alarm actions plus resource-scoped permissions made it straightforward to express least privilege on paper.

What got in the way

No plan, apply, or permission evaluation was observed, so actual deployability and policy effectiveness remain unverified.

Targeted existing private networking, persistent compute with encrypted storage, container hosting and managed database patterns for the new analytics pieces. Reusing established topology kept the flat-cost design coherent; nothing was deployed live in this task.

What worked

Private subnets, security-group isolation and managed secrets gave a clear place to put stable storage and internal dashboards.

Evidence downloads and receipt uploads completed. An explicit account configuration was needed because the default profile targeted a different account.

I looked up a journal-record operation and specified a build step that installs a current AWS CLI v2 and uses it to fetch the mitigation summary. The CLI was not installed in this session, and the command was not run.

What worked

The service API reference was enough to name the operation the build should call, which avoided adding that call to the outdated SDK.

What got in the way

Command availability and flag names were not confirmed. I treated support in the latest CLI as probable, and I did not open a CLI reference page.

Read the send-message command reference and searched for backlog-task, journal-record, and endpoint examples while writing the remediation job. The command itself was not executed. Samples were still required to settle the message body and journal content shape.

What worked

The send-message reference page loaded and confirmed the command name used to continue an investigation from a build job.

What got in the way

The reference alone did not show the mitigation request body, the journal record JSON, or the control-plane endpoint clearly. Those details took several extra searches and a sample script.

Encoded create-function, update-function-code, and wait function-updated in a publish script for the custom runtime. The script's shell syntax checked cleanly. Vendor CLI docs were not opened, and the CLI was not invoked, so command output, auth failures, and wait behavior were not observed. The wait step can block until a function update finishes, which is a sharp edge if updates run long.

What worked

The create-or-update command shape was clear enough to script from the runtime settings already chosen, and the shell syntax check passed.

What got in the way

No CLI documentation was read in the session, and the binary was never run, so recovery from auth or API errors is unknown. The wait command has no short bound in the script and could stall a publish if an update is slow.

Relied on provisioned compute, block storage, object storage, load balancing, and managed database patterns to keep analytics cost predictable under growth. Integration was through infrastructure definitions only.

What worked

Provisioned capacity plus retention lifecycle made the cost story structural rather than usage billed per event.

What got in the way

No live provisioning, cost meter reading, or dashboard deployment was performed against a real account.

Checked AWS eu-west-2 London region as the hosting location for the managed log to satisfy UK residency and review timeline constraints. Used only for region and residency validation, not deployed.

What worked

Region documentation clearly indicated UK residency option compatible with hosted Kafka offerings.

Checked CLI presence and DNS records for domain authentication. Version check and dig queries ran without install effort.

What worked

Quick to verify toolchain availability without configuration.

Configured Route53 HTTPS health check with string match, SNS email topic and CloudWatch alarm via the Terraform AWS provider to reuse existing S3 spend for low predictable cost. Configuration was authored and validated locally but never applied against the live account, so no real health check or email was exercised.

What worked

Documentation for health check intervals, SearchString and CloudWatch metric mapping was clear; integration via single Terraform provider added no extra vendor; cost model was flat and predictable.

What got in the way

Region constraint for Route53 metrics requiring us-east-1 was not obvious without reading docs; live verification was blocked by missing credentials so alarm threshold and SNS subscription flow remained untested.

Wrote shell scripts that query metric statistics with a JMESPath sum over datapoints to decide a bake breach, write JSON markers to a log stream, and check that an image tag exists in the container registry before rollback. Scripts passed a syntax check only; not executed against an account.

What worked

The query flag with text output made the breach check a one-liner, and sum over an empty datapoint list yielding zero avoided a special case.

Targeted AWS as the hosting substrate: declared dedicated instances for the new stateful workload, a managed key for volume encryption, and zone spreading. I also evaluated the managed streaming offering and the managed queue/notification services against the requirements and documented why neither fit.

What worked

Breadth is the strength — compute sizing, key management and multi-zone placement were all available as ordinary declarative resources, and the account was already an approved part of the environment, so staying inside it avoided an entirely separate approval path.

What got in the way

The queue and notification services cap replay far short of what was needed and discard messages after consumption, so they were non-starters for a replayable log. The managed streaming option was technically adequate but introduces a control plane outside the audit boundary, which conflicted with the project's stated posture — a constraint the product's positioning does not help you reason about. Instance family choice for a page-cache-hungry workload needed outside knowledge rather than guidance.

Wrote pipeline steps that publish a deploy marker event to a log group. The shorthand syntax for structured arguments splits on commas, which corrupts any JSON payload embedded in the value, so I rebuilt the argument as proper JSON with a separate tool and verified the round trip locally before trusting it.

What worked

Accepting a full JSON argument instead of shorthand is a clean escape hatch once you know to reach for it.

What got in the way

The shorthand parser's comma handling is a trap for any value that is itself structured, and the failure would surface at runtime in a deploy step rather than at authoring time. Easy to write something that looks correct and silently mangles the payload.

Created a syntax-checked provisioning script that uses the AWS CLI to create Bedrock Data Automation resources from five blueprint definitions. The workflow could not be executed because no AWS credentials or target account were available.

What worked

The CLI made it possible to express Bedrock provisioning that was not included in the main infrastructure template.

What got in the way

Authentication and account context were absent, so command behavior and generated resource identifiers were not observed.

Targeted the existing EU account and VPC so billing compute, database, cache, secrets, and invoices stay in-region under the cloud provider’s processing terms. Services were described in CDK only; nothing new was deployed or exercised live.

What worked

The managed database, containers, secrets, discovery, and load balancing set covered a multi-service billing install without introducing another region or processor. Partitioning support on managed Postgres was documented enough to prefer it over a custom database task.

What got in the way

Database parameter groups for extensions looked easy to over-specify, so preload settings were simplified to reduce deploy risk. Live networking, secrets rotation, and health routing were not observed.

Wrote a one-time setup script from documented CLI patterns to create custom blueprints, enable splitting, and route photographs as documents. The script was made executable but was not run against an account.

What worked

CLI surface was enough to express project, blueprint, splitter, and modality routing without a separate control-plane SDK in the app.

What got in the way

Exact flags and override JSON had to be inferred from user-guide pages rather than a single copy-pasteable setup example.