Splunk

Added log inputs for the new indexer to match existing service monitors. An existing worker monitor was dropped during the edit and had to be restored. Collection was not verified against a live instance.

What worked

The existing input layout made it obvious how to attach another service’s logs.

What got in the way

A wholesale replace of the inputs file removed an unrelated monitor, which needed a follow-up restore. No ingest was observed.

Inspected existing forwarder inputs while deciding the search store, then added a log path for the new indexer to match API and worker collection. Rejected this product as the identifier-search engine. No forwarder was run.

What worked

Existing input and output files made it obvious how to attach another service log stream without inventing a new pattern.

What got in the way

It is not a fit for tens of millions of exact-id operational lookups beside a live billing database, so it stayed logging-only.

Reviewed local collector and parsing configuration to see how application logs reach operations search. That was enough to reject log search as the system of record for order lookup, while still shaping lookup log lines so they can be found later. The live product was not queried.

What worked

The checked-in input, parsing, and output settings made the log pipeline and JSON event handling clear enough to decide where operations lookup should and should not live.

What got in the way

Log search could not answer the lookup need: identifiers were missing from the relevant events, retention on disk was short, and current order state lived in the database rather than in indexed logs.

Reviewed existing forwarder and sourcetype settings and treated indexed JSON fields as the operations search path. Did not run queries on a live instance; recommended searches on the newly emitted identifier fields.

What worked

Existing JSON extraction and index routing made it clear that flattening identifiers onto log events would enable order and subscriber lookup without a new API.

What got in the way

Subscriber identity was not on searchable fields yet, so the current logging contract was not enough for operations until those keys were added to relevant events.

Ruled Splunk out as the order catalog because it is a log path with limited retention and incomplete identifiers. Still added a log input for the new service so ops search logs follow the same sidecar pattern. No Splunk instance was queried.

What worked

Existing input and layout files made adding another JSON log path straightforward.

What got in the way

Log search cannot hold tens of millions of order records or reliably find by subscriber identifier, so it could not be the primary lookup system.

Considered the existing log pipeline as a cheap way to let staff search by a partial identifier, and read the forwarder input and sourcetype configuration to check feasibility. Ruled it out: the identifier in question is never emitted in logs, and retention was far too short to serve as an order lookup. No queries were run.

What worked

The forwarding and parsing configuration files are terse and readable, so I could establish what is ingested and how long it is kept without needing access to the service itself. That was enough to make a confident negative decision quickly.

What got in the way

Nothing was wrong with the product; it was simply the wrong store for this need. Worth noting that the configuration does not surface effective retention in an obvious place, so judging whether a log index can back a user-facing lookup takes more inference than it should.

Inspected the existing forwarder sidecar and inputs as a possible search backend, rejected it for production order and subscriber lookup, then only added a log input so the new indexer ships logs the same way as the other services.

What worked

The existing forwarder config was easy to extend for another service’s log file, and the log-shipping role was already obvious from the in-pod setup.

What got in the way

Splunk was the wrong product for CRM-style identifier search: it is aimed at operational logs, not isolated low-latency business-record lookup, and mixing those workloads would have put search on the observability path.

Read the forwarder input, parsing and output configuration to work out which fields reach the log index, in order to judge whether an existing log search could serve the new identifier lookup instead of a database change. It could not, so I ruled the option out.

What worked

The configuration files spell out the monitored paths, the index and the field extraction rules clearly enough that I could determine coverage from the repo alone, with no access to a search head. That made it a cheap option to evaluate and reject.

What got in the way

The approach is only as good as what the application chose to log, and the identifier in question is deliberately never written to the logs, so no amount of search configuration could have covered it. That is a sensible privacy decision upstream, but it does mean log search was a dead end here.

Inspected the forwarder input and source-type configuration to judge whether existing log search could already serve as a stopgap for the lookup staff needed. It could not: the relevant identifier is never written to the log lines being collected, so nothing indexed contains the field to search on. Useful as a quick elimination, but it did not contribute to the delivered solution.

What got in the way

The limitation is upstream, not in the product: a field that is not logged cannot be searched. Still, the config files alone do not tell you which fields actually reach the index, so confirming this meant cross-reading application logging code rather than the collection config.

Inspected existing forwarder config, ruled Splunk out as the application index, and added log inputs for the new indexer service using the same pattern as the other processes. Nothing was sent to a Splunk instance.

What worked

The existing input definitions were easy to clone for another service, which kept observability wiring consistent.

What got in the way

Splunk was not suitable as the dedicated application search store. Forwarding itself was not executed or verified.

Added log collection for the new indexer using the same sidecar input pattern as the other services. No collector or indexer was run here.

What worked

Copying the existing input stanza made it obvious how the new process should ship logs without inventing a separate observability path.

Inspected existing forwarder inputs and added log tailing for the new indexer. Treated this stack as observability only, not as the application search engine for order identifiers.

What worked

Forwarder input files in the repo made it obvious how to attach the new service’s logs without changing how application documents are indexed.

What got in the way

Not a substitute for dedicated application search: it was already scoped to app and GC logs and would not isolate identifier lookup from observability.

Extended the existing input configuration for the new service's structured logs while retaining OpenSearch as the operational record index. No Splunk instance or forwarder was run, so ingestion behavior was not observed.

What worked

The input configuration was simple to extend for the additional log source.

Added a log input for the new search service using the same Universal Forwarder input style already present for the API and worker.

What worked

Existing input stanzas were obvious to copy, so the new service could be monitored the same way as the others.

What got in the way

No forwarder or index was run to confirm ingestion.

Extended the existing file-monitor configuration and deployment sidecar wiring so the new indexer's structured logs follow the established collection path. No Splunk instance or ingestion result was available for validation.

What worked

The existing repository conventions made the new log source straightforward to add.

What got in the way

Timestamp extraction, source typing, and end-to-end ingestion could only be reasoned about from configuration.

Added input and parsing stanzas so the new service's JSON logs flow into the same index as the existing modules via the sidecar forwarder. Config only; nothing was indexed or queried.

What worked

Stanza-based config is easy to extend by analogy. Adding a new source with the right index and sourcetype was a small, low-risk edit that fit the existing pattern exactly.

What got in the way

The config format is positional and convention-heavy, so correctness depends entirely on matching existing stanzas; there is no local way to validate a change before it reaches a forwarder.

Read the existing forwarder input and parsing configuration to work out whether a new workload sharing the same image and log path would be distinguishable in the log platform, and documented the answer rather than changing the setup.

What worked

The input and parsing configuration files are plain key-value stanzas that were quick to read and reason about, and the forwarder adds a host field automatically, which turned out to be the one reliable way to separate the two workloads.

What got in the way

Because routing is keyed on file path and the two workloads share an image, log path and config map, there is no way to give them distinct source or sourcetype values without duplicating forwarder configuration. The structured log fields that would naturally distinguish them are not usable for routing at ingest, so separation has to happen at search time instead.

Already present as the log tier, so I evaluated it as the obvious reuse candidate for entity search and ruled it out, then made a small forwarder input change for the new service's logs. The config file format itself was easy to read and extend.

What worked

The forwarder configuration files are plain, well-structured, and easy to extend for a new application's logs; adding the new service's log source was a few lines that mirrored existing stanzas. As a log and operational-telemetry tier it clearly does its job here.

What got in the way

Wrong shape for the actual requirement. Events are append-only, so an entity that moves through several states would need deduplication at query time rather than storing one current document per entity. Licensing is tied to ingest volume, which makes indexing tens of millions of business records a cost decision rather than an engineering one, and the query model is not built for low-latency exact-identifier lookup with filters and aggregations. Reusing it would have traded a correct answer for the appearance of not adding infrastructure.

Added input stanzas so the two new services' structured logs land in the same place as the existing ones. The configuration was written by pattern-matching the existing file; it was never deployed or verified against a running forwarder.

What worked

The flat stanza format is easy to extend by copying an existing block, and source/index routing is explicit enough that the diff is self-explaining to a reviewer.

What got in the way

There is no local way to validate a stanza — correctness is only observable after deployment, which is a poor fit for a change that otherwise ships as reviewable text alongside code.

Added forwarder input and property stanzas so the new service's structured JSON logs, including a search audit trail, land in the existing logging estate under their own source type. Config was authored to match the existing stanzas; no Splunk instance was available to confirm ingestion.

What worked

The configuration model is just additive text stanzas, so extending an existing setup for a new source is a small, reviewable diff with no tooling required. Declaring a JSON source type lines up neatly with a structured log layout on the application side.

What got in the way

Nothing is verifiable without an instance — field extraction, timestamp parsing and index routing all stay unproven until it runs somewhere real, and a stanza typo would be silent until then.

Reviewed the existing Splunk input and parsing configuration and extended log collection coverage for the new indexer. No Splunk instance ingested the logs during the task.

What worked

The existing file-based convention was straightforward to extend to another service.

What got in the way

End-to-end parsing, routing, and ingestion could not be assessed without the live logging environment.

Added forwarder input stanzas for the new service's structured application and GC logs so they land in the same index as the existing modules, and assessed whether the existing deployment could double as the order search tier.

What worked

The input configuration format is simple and repetitive in a good way — adding a new source was a copy of an existing stanza with the path and source type changed, with no coordination needed beyond the index already in use.

What got in the way

It is the wrong product for low-latency record search over structured business data, and ingest-volume licensing makes reusing an existing deployment for that a costly mistake rather than a free one; I recommended against it. Operationally, the forwarder sidecar never exits, so a pod that is supposed to run to completion cannot carry one — the scheduled job ships without log forwarding as a documented gap.

Extended forwarder input configuration so the new service's structured logs are collected the same way as existing services. I deliberately stopped short of adding a source type for the search server's own logs, because nothing would emit it yet and the collection path and licensing headroom were open questions.

What worked

The stanza-based input configuration is simple to extend by analogy, and the existing per-service convention made the right shape obvious.

What got in the way

Early in the task it was tempting to treat the existing log platform as the search backend for structured business records; it is a poor fit there given ingest-volume licensing and the lack of low-latency exact-match lookup semantics, and that distinction is not obvious from the fact that it is already deployed. Adding collection for the new server's logs is blocked on a licensing-headroom question that the configuration format gives no visibility into.