Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

security-service-integrations: retain event.original value upon pipeline errors #12067

Closed
efd6 opened this issue Dec 10, 2024 · 2 comments
Closed

security-service-integrations: retain event.original value upon pipeline errors #12067

efd6 opened this issue Dec 10, 2024 · 2 comments
Assignees
@efd6
Labels
enhancement New feature or request Team:Security-Service Integrations Security Service Integrations Team [elastic/security-service-integrations]

Comments

@efd6
Copy link
Contributor

efd6 commented Dec 10, 2024 •
edited
Loading

This is the security-service-integrations sub issue for #12045.

  • add preserve_original_event in primary on_failure handlers ssi_all: add "preserve_original_event" tag to documents with event.kind set to "pipeline_error" #12046 google_workspace,jamf_protect,ti_mandiant: add "preserve_original_event" tag to documents with event.kind set to "pipeline_error" #12108

  • add preserve_original_event in cases of manual setting event.kind to "pipeline_error" ssi_all: add "preserve_original_event" tag to documents with event.kind manually set to "pipeline_error" #12109

  • ensure no remove event.original processors remain ssi_all: do not remove event.original in main ingest pipeline #12076 okta: do not remove event.original in main ingest #12127

     for f in $(
 	(
 		for p in $(
 			yq 'select(.owner.github == "elastic/security-service-integrations")|.name' packages/**/manifest.yml \
 			| grep -v -- ---
 		); do
 			find packages/$p -name default.yml
 		done
 	)|sort|uniq
 ); do
 	yq -o=json $f|quatch -l $f -p '{"processors":{"remove":{"field":["event.original"]}}}'
 done

    quatch

    • packages/bitdefender/data_stream/push_notifications/elasticsearch/ingest_pipeline/default.yml
    • packages/bitwarden/data_stream/collection/elasticsearch/ingest_pipeline/default.yml
    • packages/bitwarden/data_stream/event/elasticsearch/ingest_pipeline/default.yml
    • packages/bitwarden/data_stream/group/elasticsearch/ingest_pipeline/default.yml
    • packages/bitwarden/data_stream/member/elasticsearch/ingest_pipeline/default.yml
    • packages/bitwarden/data_stream/policy/elasticsearch/ingest_pipeline/default.yml
    • packages/cisco_meraki/data_stream/events/elasticsearch/ingest_pipeline/default.yml
    • packages/darktrace/data_stream/ai_analyst_alert/elasticsearch/ingest_pipeline/default.yml
    • packages/darktrace/data_stream/model_breach_alert/elasticsearch/ingest_pipeline/default.yml
    • packages/darktrace/data_stream/system_status_alert/elasticsearch/ingest_pipeline/default.yml
    • packages/eset_protect/data_stream/detection/elasticsearch/ingest_pipeline/default.yml
    • packages/eset_protect/data_stream/device_task/elasticsearch/ingest_pipeline/default.yml
    • packages/eset_protect/data_stream/event/elasticsearch/ingest_pipeline/default.yml
    • packages/f5/data_stream/bigipafm/elasticsearch/ingest_pipeline/default.yml
    • packages/f5/data_stream/bigipapm/elasticsearch/ingest_pipeline/default.yml
    • packages/forcepoint_web/data_stream/logs/elasticsearch/ingest_pipeline/default.yml
    • packages/forgerock/data_stream/am_access/elasticsearch/ingest_pipeline/default.yml
    • packages/forgerock/data_stream/am_activity/elasticsearch/ingest_pipeline/default.yml
    • packages/forgerock/data_stream/am_authentication/elasticsearch/ingest_pipeline/default.yml
    • packages/forgerock/data_stream/am_config/elasticsearch/ingest_pipeline/default.yml
    • packages/forgerock/data_stream/am_core/elasticsearch/ingest_pipeline/default.yml
    • packages/forgerock/data_stream/idm_access/elasticsearch/ingest_pipeline/default.yml
    • packages/forgerock/data_stream/idm_activity/elasticsearch/ingest_pipeline/default.yml
    • packages/forgerock/data_stream/idm_authentication/elasticsearch/ingest_pipeline/default.yml
    • packages/forgerock/data_stream/idm_config/elasticsearch/ingest_pipeline/default.yml
    • packages/forgerock/data_stream/idm_core/elasticsearch/ingest_pipeline/default.yml
    • packages/forgerock/data_stream/idm_sync/elasticsearch/ingest_pipeline/default.yml
    • packages/github/data_stream/audit/elasticsearch/ingest_pipeline/default.yml
    • packages/github/data_stream/code_scanning/elasticsearch/ingest_pipeline/default.yml
    • packages/github/data_stream/dependabot/elasticsearch/ingest_pipeline/default.yml
    • packages/github/data_stream/issues/elasticsearch/ingest_pipeline/default.yml
    • packages/github/data_stream/secret_scanning/elasticsearch/ingest_pipeline/default.yml
    • packages/gitlab/data_stream/api/elasticsearch/ingest_pipeline/default.yml
    • packages/gitlab/data_stream/pages/elasticsearch/ingest_pipeline/default.yml
    • packages/gitlab/data_stream/production/elasticsearch/ingest_pipeline/default.yml
    • packages/gitlab/data_stream/sidekiq/elasticsearch/ingest_pipeline/default.yml
    • packages/google_scc/data_stream/asset/elasticsearch/ingest_pipeline/default.yml
    • packages/google_scc/data_stream/audit/elasticsearch/ingest_pipeline/default.yml
    • packages/google_scc/data_stream/finding/elasticsearch/ingest_pipeline/default.yml
    • packages/google_scc/data_stream/source/elasticsearch/ingest_pipeline/default.yml
    • packages/google_workspace/data_stream/access_transparency/elasticsearch/ingest_pipeline/default.yml
    • packages/google_workspace/data_stream/admin/elasticsearch/ingest_pipeline/default.yml
    • packages/google_workspace/data_stream/alert/elasticsearch/ingest_pipeline/default.yml
    • packages/google_workspace/data_stream/context_aware_access/elasticsearch/ingest_pipeline/default.yml
    • packages/google_workspace/data_stream/device/elasticsearch/ingest_pipeline/default.yml
    • packages/google_workspace/data_stream/drive/elasticsearch/ingest_pipeline/default.yml
    • packages/google_workspace/data_stream/gcp/elasticsearch/ingest_pipeline/default.yml
    • packages/google_workspace/data_stream/group_enterprise/elasticsearch/ingest_pipeline/default.yml
    • packages/google_workspace/data_stream/groups/elasticsearch/ingest_pipeline/default.yml
    • packages/google_workspace/data_stream/login/elasticsearch/ingest_pipeline/default.yml
    • packages/google_workspace/data_stream/rules/elasticsearch/ingest_pipeline/default.yml
    • packages/google_workspace/data_stream/saml/elasticsearch/ingest_pipeline/default.yml
    • packages/google_workspace/data_stream/token/elasticsearch/ingest_pipeline/default.yml
    • packages/google_workspace/data_stream/user_accounts/elasticsearch/ingest_pipeline/default.yml
    • packages/imperva_cloud_waf/data_stream/event/elasticsearch/ingest_pipeline/default.yml
    • packages/infoblox_bloxone_ddi/data_stream/dhcp_lease/elasticsearch/ingest_pipeline/default.yml
    • packages/infoblox_bloxone_ddi/data_stream/dns_config/elasticsearch/ingest_pipeline/default.yml
    • packages/infoblox_bloxone_ddi/data_stream/dns_data/elasticsearch/ingest_pipeline/default.yml
    • packages/infoblox_nios/data_stream/log/elasticsearch/ingest_pipeline/default.yml
    • packages/jamf_compliance_reporter/data_stream/log/elasticsearch/ingest_pipeline/default.yml
    • packages/jamf_protect/data_stream/alerts/elasticsearch/ingest_pipeline/default.yml
    • packages/jamf_protect/data_stream/telemetry/elasticsearch/ingest_pipeline/default.yml
    • packages/jamf_protect/data_stream/telemetry_legacy/elasticsearch/ingest_pipeline/default.yml
    • packages/jamf_protect/data_stream/web_threat_events/elasticsearch/ingest_pipeline/default.yml
    • packages/jamf_protect/data_stream/web_traffic_events/elasticsearch/ingest_pipeline/default.yml
    • packages/jumpcloud/data_stream/events/elasticsearch/ingest_pipeline/default.yml
    • packages/keycloak/data_stream/log/elasticsearch/ingest_pipeline/default.yml
    • packages/lastpass/data_stream/detailed_shared_folder/elasticsearch/ingest_pipeline/default.yml
    • packages/lastpass/data_stream/event_report/elasticsearch/ingest_pipeline/default.yml
    • packages/lastpass/data_stream/user/elasticsearch/ingest_pipeline/default.yml
    • packages/lyve_cloud/data_stream/audit/elasticsearch/ingest_pipeline/default.yml
    • packages/m365_defender/data_stream/alert/elasticsearch/ingest_pipeline/default.yml
    • packages/m365_defender/data_stream/event/elasticsearch/ingest_pipeline/default.yml
    • packages/m365_defender/data_stream/incident/elasticsearch/ingest_pipeline/default.yml
    • packages/m365_defender/data_stream/log/elasticsearch/ingest_pipeline/default.yml
    • packages/mattermost/data_stream/audit/elasticsearch/ingest_pipeline/default.yml
    • packages/menlo/data_stream/dlp/elasticsearch/ingest_pipeline/default.yml
    • packages/menlo/data_stream/web/elasticsearch/ingest_pipeline/default.yml
    • packages/microsoft_defender_cloud/data_stream/event/elasticsearch/ingest_pipeline/default.yml
    • packages/microsoft_defender_endpoint/data_stream/log/elasticsearch/ingest_pipeline/default.yml
    • packages/microsoft_exchange_online_message_trace/data_stream/log/elasticsearch/ingest_pipeline/default.yml
    • packages/mimecast/data_stream/archive_search_logs/elasticsearch/ingest_pipeline/default.yml
    • packages/mimecast/data_stream/audit_events/elasticsearch/ingest_pipeline/default.yml
    • packages/mimecast/data_stream/dlp_logs/elasticsearch/ingest_pipeline/default.yml
    • packages/mimecast/data_stream/message_release_logs/elasticsearch/ingest_pipeline/default.yml
    • packages/mimecast/data_stream/threat_intel_malware_customer/elasticsearch/ingest_pipeline/default.yml
    • packages/mimecast/data_stream/threat_intel_malware_grid/elasticsearch/ingest_pipeline/default.yml
    • packages/mimecast/data_stream/ttp_ap_logs/elasticsearch/ingest_pipeline/default.yml
    • packages/mimecast/data_stream/ttp_ip_logs/elasticsearch/ingest_pipeline/default.yml
    • packages/mimecast/data_stream/ttp_url_logs/elasticsearch/ingest_pipeline/default.yml
    • packages/netskope/data_stream/alerts/elasticsearch/ingest_pipeline/default.yml
    • packages/netskope/data_stream/events/elasticsearch/ingest_pipeline/default.yml
    • packages/o365/data_stream/audit/elasticsearch/ingest_pipeline/default.yml
    • packages/okta/data_stream/system/elasticsearch/ingest_pipeline/default.yml
    • packages/opencanary/data_stream/events/elasticsearch/ingest_pipeline/default.yml
    • packages/panw_cortex_xdr/data_stream/alerts/elasticsearch/ingest_pipeline/default.yml
    • packages/panw_cortex_xdr/data_stream/incidents/elasticsearch/ingest_pipeline/default.yml
    • packages/ping_one/data_stream/audit/elasticsearch/ingest_pipeline/default.yml
    • packages/pps/data_stream/log/elasticsearch/ingest_pipeline/default.yml
    • packages/prisma_access/data_stream/event/elasticsearch/ingest_pipeline/default.yml
    • packages/prisma_cloud/data_stream/alert/elasticsearch/ingest_pipeline/default.yml
    • packages/prisma_cloud/data_stream/audit/elasticsearch/ingest_pipeline/default.yml
    • packages/prisma_cloud/data_stream/host/elasticsearch/ingest_pipeline/default.yml
    • packages/prisma_cloud/data_stream/host_profile/elasticsearch/ingest_pipeline/default.yml
    • packages/prisma_cloud/data_stream/incident_audit/elasticsearch/ingest_pipeline/default.yml
    • packages/proofpoint_tap/data_stream/clicks_blocked/elasticsearch/ingest_pipeline/default.yml
    • packages/proofpoint_tap/data_stream/clicks_permitted/elasticsearch/ingest_pipeline/default.yml
    • packages/proofpoint_tap/data_stream/message_blocked/elasticsearch/ingest_pipeline/default.yml
    • packages/proofpoint_tap/data_stream/message_delivered/elasticsearch/ingest_pipeline/default.yml
    • packages/pulse_connect_secure/data_stream/log/elasticsearch/ingest_pipeline/default.yml
    • packages/qualys_vmdr/data_stream/user_activity/elasticsearch/ingest_pipeline/default.yml
    • packages/rapid7_insightvm/data_stream/asset/elasticsearch/ingest_pipeline/default.yml
    • packages/rapid7_insightvm/data_stream/vulnerability/elasticsearch/ingest_pipeline/default.yml
    • packages/santa/data_stream/log/elasticsearch/ingest_pipeline/default.yml
    • packages/sentinel_one_cloud_funnel/data_stream/event/elasticsearch/ingest_pipeline/default.yml
    • packages/sentinel_one/data_stream/activity/elasticsearch/ingest_pipeline/default.yml
    • packages/sentinel_one/data_stream/agent/elasticsearch/ingest_pipeline/default.yml
    • packages/sentinel_one/data_stream/alert/elasticsearch/ingest_pipeline/default.yml
    • packages/sentinel_one/data_stream/group/elasticsearch/ingest_pipeline/default.yml
    • packages/sentinel_one/data_stream/threat/elasticsearch/ingest_pipeline/default.yml
    • packages/slack/data_stream/audit/elasticsearch/ingest_pipeline/default.yml
    • packages/snyk/data_stream/audit/elasticsearch/ingest_pipeline/default.yml
    • packages/snyk/data_stream/audit_logs/elasticsearch/ingest_pipeline/default.yml
    • packages/snyk/data_stream/issues/elasticsearch/ingest_pipeline/default.yml
    • packages/snyk/data_stream/vulnerabilities/elasticsearch/ingest_pipeline/default.yml
    • packages/sophos_central/data_stream/alert/elasticsearch/ingest_pipeline/default.yml
    • packages/sophos_central/data_stream/event/elasticsearch/ingest_pipeline/default.yml
    • packages/symantec_edr_cloud/data_stream/incident/elasticsearch/ingest_pipeline/default.yml
    • packages/symantec_endpoint/data_stream/log/elasticsearch/ingest_pipeline/default.yml
    • packages/symantec_endpoint_security/data_stream/event/elasticsearch/ingest_pipeline/default.yml
    • packages/symantec_endpoint_security/data_stream/incident/elasticsearch/ingest_pipeline/default.yml
    • packages/tanium/data_stream/action_history/elasticsearch/ingest_pipeline/default.yml
    • packages/tanium/data_stream/client_status/elasticsearch/ingest_pipeline/default.yml
    • packages/tanium/data_stream/discover/elasticsearch/ingest_pipeline/default.yml
    • packages/tanium/data_stream/endpoint_config/elasticsearch/ingest_pipeline/default.yml
    • packages/tanium/data_stream/reporting/elasticsearch/ingest_pipeline/default.yml
    • packages/tanium/data_stream/threat_response/elasticsearch/ingest_pipeline/default.yml
    • packages/teleport/data_stream/audit/elasticsearch/ingest_pipeline/default.yml
    • packages/tenable_io/data_stream/asset/elasticsearch/ingest_pipeline/default.yml
    • packages/tenable_io/data_stream/plugin/elasticsearch/ingest_pipeline/default.yml
    • packages/tenable_io/data_stream/scan/elasticsearch/ingest_pipeline/default.yml
    • packages/tenable_io/data_stream/vulnerability/elasticsearch/ingest_pipeline/default.yml
    • packages/tenable_sc/data_stream/asset/elasticsearch/ingest_pipeline/default.yml
    • packages/tenable_sc/data_stream/plugin/elasticsearch/ingest_pipeline/default.yml
    • packages/tenable_sc/data_stream/vulnerability/elasticsearch/ingest_pipeline/default.yml
    • packages/thycotic_ss/data_stream/logs/elasticsearch/ingest_pipeline/default.yml
    • packages/ti_abusech/data_stream/malwarebazaar/elasticsearch/ingest_pipeline/default.yml
    • packages/ti_abusech/data_stream/malware/elasticsearch/ingest_pipeline/default.yml
    • packages/ti_abusech/data_stream/threatfox/elasticsearch/ingest_pipeline/default.yml
    • packages/ti_abusech/data_stream/url/elasticsearch/ingest_pipeline/default.yml
    • packages/ti_anomali/data_stream/threatstream/elasticsearch/ingest_pipeline/default.yml
    • packages/ti_cif3/data_stream/feed/elasticsearch/ingest_pipeline/default.yml
    • packages/ti_crowdstrike/data_stream/intel/elasticsearch/ingest_pipeline/default.yml
    • packages/ti_crowdstrike/data_stream/ioc/elasticsearch/ingest_pipeline/default.yml
    • packages/ti_cybersixgill/data_stream/threat/elasticsearch/ingest_pipeline/default.yml
    • packages/ti_eset/data_stream/apt/elasticsearch/ingest_pipeline/default.yml
    • packages/ti_eset/data_stream/botnet/elasticsearch/ingest_pipeline/default.yml
    • packages/ti_eset/data_stream/cc/elasticsearch/ingest_pipeline/default.yml
    • packages/ti_eset/data_stream/domains/elasticsearch/ingest_pipeline/default.yml
    • packages/ti_eset/data_stream/files/elasticsearch/ingest_pipeline/default.yml
    • packages/ti_eset/data_stream/ip/elasticsearch/ingest_pipeline/default.yml
    • packages/ti_eset/data_stream/url/elasticsearch/ingest_pipeline/default.yml
    • packages/ti_maltiverse/data_stream/indicator/elasticsearch/ingest_pipeline/default.yml
    • packages/ti_misp/data_stream/threat_attributes/elasticsearch/ingest_pipeline/default.yml
    • packages/ti_misp/data_stream/threat/elasticsearch/ingest_pipeline/default.yml
    • packages/tines/data_stream/audit_logs/elasticsearch/ingest_pipeline/default.yml
    • packages/tines/data_stream/time_saved/elasticsearch/ingest_pipeline/default.yml
    • packages/ti_otx/data_stream/pulses_subscribed/elasticsearch/ingest_pipeline/default.yml
    • packages/ti_otx/data_stream/threat/elasticsearch/ingest_pipeline/default.yml
    • packages/ti_rapid7_threat_command/data_stream/alert/elasticsearch/ingest_pipeline/default.yml
    • packages/ti_rapid7_threat_command/data_stream/ioc/elasticsearch/ingest_pipeline/default.yml
    • packages/ti_rapid7_threat_command/data_stream/vulnerability/elasticsearch/ingest_pipeline/default.yml
    • packages/ti_recordedfuture/data_stream/threat/elasticsearch/ingest_pipeline/default.yml
    • packages/ti_threatconnect/data_stream/indicator/elasticsearch/ingest_pipeline/default.yml
    • packages/ti_threatq/data_stream/threat/elasticsearch/ingest_pipeline/default.yml
    • packages/trellix_edr_cloud/data_stream/event/elasticsearch/ingest_pipeline/default.yml
    • packages/trellix_epo_cloud/data_stream/device/elasticsearch/ingest_pipeline/default.yml
    • packages/trellix_epo_cloud/data_stream/event/elasticsearch/ingest_pipeline/default.yml
    • packages/trellix_epo_cloud/data_stream/group/elasticsearch/ingest_pipeline/default.yml
    • packages/trendmicro/data_stream/deep_security/elasticsearch/ingest_pipeline/default.yml
    • packages/trend_micro_vision_one/data_stream/alert/elasticsearch/ingest_pipeline/default.yml
    • packages/trend_micro_vision_one/data_stream/audit/elasticsearch/ingest_pipeline/default.yml
    • packages/trend_micro_vision_one/data_stream/detection/elasticsearch/ingest_pipeline/default.yml
    • packages/vectra_detect/data_stream/log/elasticsearch/ingest_pipeline/default.yml
    • packages/wiz/data_stream/audit/elasticsearch/ingest_pipeline/default.yml
    • packages/wiz/data_stream/issue/elasticsearch/ingest_pipeline/default.yml
    • packages/zerofox/data_stream/alerts/elasticsearch/ingest_pipeline/default.yml
    • packages/zeronetworks/data_stream/audit/elasticsearch/ingest_pipeline/default.yml
    • packages/zscaler_zpa/data_stream/app_connector_status/elasticsearch/ingest_pipeline/default.yml
    • packages/zscaler_zpa/data_stream/audit/elasticsearch/ingest_pipeline/default.yml
    • packages/zscaler_zpa/data_stream/browser_access/elasticsearch/ingest_pipeline/default.yml
    • packages/zscaler_zpa/data_stream/user_activity/elasticsearch/ingest_pipeline/default.yml
    • packages/zscaler_zpa/data_stream/user_status/elasticsearch/ingest_pipeline/default.yml
@efd6 efd6 added enhancement New feature or request Team:Security-Service Integrations Security Service Integrations Team [elastic/security-service-integrations] labels Dec 10, 2024
@elasticmachine
Copy link

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

@efd6 efd6 mentioned this issue Dec 10, 2024
Merged
5 tasks
@efd6 efd6 self-assigned this Dec 10, 2024
@andrewkroh andrewkroh mentioned this issue Dec 10, 2024
Open
This was referenced Dec 11, 2024
Merged
Merged
Merged
Merged
@efd6
Copy link
Contributor Author

efd6 commented Dec 17, 2024

Confirmed no removal of event.original in any pipeline.

$ (for d in $(
        (
                for p in $(
                        yq 'select(.owner.github == "elastic/security-service-integrations")|.name' packages/**/manifest.yml \
                        | grep -v -- ---                                                                                              
                ); do                           
                        find packages/$p -name ingest_pipeline        
                done                                              
        )|sort|uniq
); do
        for f in $d/*.yml; do
                yq -o=json $f|quatch -l $f -p '{"processors":{"remove":{"field":["event.original"]}}}'
        done
done)|wc -l
0

@efd6 efd6 closed this as completed Dec 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@efd6 efd6

Labels
enhancement New feature or request Team:Security-Service Integrations Security Service Integrations Team [elastic/security-service-integrations]
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@elasticmachine @efd6