Wrote a Data Prepper-style pipeline definition with a Kafka source, date processor, conditional add_entries for status, and an OpenSearch sink using date-based index names and document versioning. Only checked YAML syntax; semantics stay unverified until a staging dry run.
- What worked
- Declarative pipeline meant no custom indexer service had to be written in the repo.
- What got in the way
- Processor details (date format into index name, conditional add_when expressions, how version conflicts are reported to the DLQ) needed care, and there was no local way to validate the pipeline beyond YAML parsing.
