Retain event.original value upon pipeline errors

[andrewkroh]

When an ingest pipeline error occurs, our log pipelines should retain the event.original value to ensure that no data loss occurs and to facilitate correcting the failure. If processing is interrupted due to an error, some data may not have been extracted (i.e. incomplete processing) so it's important to retain the event.original. And secondly, in order for the package maintainers to be able to take action on reports of pipeline failures they nearly always need the event.original value to reproduce and understand the issue.

To implement this we should complete the work related to #10072. This not strictly required, but it helps ensure the event.original is consistently handled. We want the Fleet final_pipeline to be responsible for deleting event.original when tags does not contain preserve_original_event.

Next, in the primary pipeline of each log data stream we update the global on_failure handler to inject preserve_original_event into tags. This will accompany event.kind: pipeline_error. With this mechanism, users can still override this behavior through the various levels of @custom pipelines by deleting the tag value. I expect the work to be accomplished "mechanically", and this mechanism can be applied separately on the integrations owned by each SIT team.

Tasks

Preview Give feedback

1. Remove event.original removal processors from ingest pipelines #10072
   Integration:All enhancement +2
2. Tag events with preserve_original_event in primary on_failure handlers

[elasticmachine]

Pinging @elastic/sec-deployment-and-devices (Team:Security-Deployment and Devices)

[elasticmachine]

Pinging @elastic/sec-linux-platform (Team:Security-Linux Platform)

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[elasticmachine]

Pinging @elastic/security-scalability (Team:Security-Scalability)

[elasticmachine]

Pinging @elastic/sec-windows-platform (Team:Security-Windows Platform)