Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ssi_all: add "preserve_original_event" tag to documents with event.kind set to "pipeline_error" #12046

Merged
merged 1 commit into from
Dec 11, 2024
Merged

ssi_all: add "preserve_original_event" tag to documents with event.kind set to "pipeline_error" #12046

merged 1 commit into from
Dec 11, 2024

Conversation

efd6
Copy link
Contributor

@efd6 efd6 commented Dec 10, 2024 •
edited
Loading

Proposed commit message

ssi_all: add "preserve_original_event" tag to documents with event.kind set to "pipeline_error"

Omit problematic packages: google_workspace, jamf_protect and ti_mandiant_advantage.

[git-generate]
for f in $(
    (
        for p in $(
            yq 'select(.owner.github == "elastic/security-service-integrations")|.name' packages/**/manifest.yml \
            | grep -v -- ---
        ); do
            rg -l -g 'default.yml' "value: pipeline_error" packages/$p
        done
    )|sort|uniq|egrep -v 'google_workspace|jamf_protect|ti_mandiant_advantage'
); do
    (grep 'value: preserve_original_event' $f >/dev/null 2>&1) && continue
    perl -i -pe 'BEGIN{undef $/;} s/([a-z:"])
( *)(- set:.*value: pipeline_error)/$1
$2$3
$2- append:
$2    field: tags
$2    value: preserve_original_event
$2    allow_duplicates: false/smg' $f
done
for p in $(git diff --name-only HEAD~1|cut -d/ -f1,2|sort|uniq); do
    (
        cd $p
        elastic-package changelog add \
            --description 'Add "preserve_original_event" tag to documents with `event.kind` set to "pipeline_error".' \
            --type enhancement \
            --next minor \
            --link https://github.com/elastic/integrations/pull/12046
    )>/dev/null 2>&1
done

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

  • [ ]

How to test this PR locally

Related issues

Screenshots

@efd6 efd6 added enhancement New feature or request Integration:All Applies to all integrations [Integration not found in source] Team:Security-Service Integrations Security Service Integrations Team [elastic/security-service-integrations] labels Dec 10, 2024
@efd6 efd6 self-assigned this Dec 10, 2024
efd6 added a commit to efd6/integrations that referenced this pull request Dec 10, 2024
@efd6
ssi_all: add "preserve_original_event" tag to documents with event.ki…
6296699 
…nd set to "pipeline_error"

[git-generate]
for f in $(
        (
                for p in $(
                        yq 'select(.owner.github == "elastic/security-service-integrations")|.name' packages/**/manifest.yml \
                        | grep -v -- ---
                ); do
                        rg -l -g 'default.yml' "value: pipeline_error" packages/$p
                done
        )|sort|uniq
); do
        (grep 'value: preserve_original_event' $f >/dev/null 2>&1) && continue
        perl -i -pe 'BEGIN{undef $/;} s/( *)(- set:.*value: pipeline_error)/$1$2
$1- append:
$1    field: tags
$1    value: preserve_original_event
$1    allow_duplicates: true/smg' $f
done
for p in $(git diff --name-only HEAD~1|cut -d/ -f1,2|sort|uniq); do
    (
            cd $p
            elastic-package changelog add \
                    --description 'Add "preserve_original_event" tag to documents with `event.kind` set to "pipeline_error".' \
                    --type enhancement \
                    --next minor \
                    --link elastic#12046
    )>/dev/null 2>&1
done
efd6 added a commit to efd6/integrations that referenced this pull request Dec 10, 2024
@efd6
ssi_all: add "preserve_original_event" tag to documents with event.ki…
369c577 
…nd set to "pipeline_error"

[git-generate]
for f in $(
        (
                for p in $(
                        yq 'select(.owner.github == "elastic/security-service-integrations")|.name' packages/**/manifest.yml \
                        | grep -v -- ---
                ); do
                        rg -l -g 'default.yml' "value: pipeline_error" packages/$p
                done
        )|sort|uniq
); do
        (grep 'value: preserve_original_event' $f >/dev/null 2>&1) && continue
        perl -i -pe 'BEGIN{undef $/;} s/([a-z:"])
( *)(- set:.*value: pipeline_error)/$1
$2$3
$2- append:
$2    field: tags
$2    value: preserve_original_event
$2    allow_duplicates: true/smg' $f
done
for p in $(git diff --name-only HEAD~1|cut -d/ -f1,2|sort|uniq); do
    (
            cd $p
            elastic-package changelog add \
                    --description 'Add "preserve_original_event" tag to documents with `event.kind` set to "pipeline_error".' \
                    --type enhancement \
                    --next minor \
                    --link elastic#12046
    )>/dev/null 2>&1
done
@andrewkroh
Copy link
Member

I think we will want allow_duplicates: false on that append processor.

@efd6
Copy link
Contributor Author

efd6 commented Dec 10, 2024 •
edited
Loading

Yeah, it's not just that. There is some weirdness with the perl mutation that is brittle. I'm trying to fix other parts to see if that will fix this (I'm not optimistic).

efd6 added a commit to efd6/integrations that referenced this pull request Dec 10, 2024
@efd6
ssi_all: add "preserve_original_event" tag to documents with event.ki…
282a7a6 
…nd set to "pipeline_error"

[git-generate]
for f in $(
        (
                for p in $(
                        yq 'select(.owner.github == "elastic/security-service-integrations")|.name' packages/**/manifest.yml \
                        | grep -v -- ---
                ); do
                        rg -l -g 'default.yml' "value: pipeline_error" packages/$p
                done
        )|sort|uniq
); do
        (grep 'value: preserve_original_event' $f >/dev/null 2>&1) && continue
        perl -i -pe 'BEGIN{undef $/;} s/([a-z:"])
( *)(- set:.*value: pipeline_error)/$1
$2$3
$2- append:
$2    field: tags
$2    value: preserve_original_event
$2    allow_duplicates: false/smg' $f
done
for p in $(git diff --name-only HEAD~1|cut -d/ -f1,2|sort|uniq); do
    (
            cd $p
            elastic-package changelog add \
                    --description 'Add "preserve_original_event" tag to documents with `event.kind` set to "pipeline_error".' \
                    --type enhancement \
                    --next minor \
                    --link elastic#12046
    )>/dev/null 2>&1
done
efd6 added a commit to efd6/integrations that referenced this pull request Dec 10, 2024
@efd6
ssi_all: add "preserve_original_event" tag to documents with event.ki…
4e2052b 
…nd set to "pipeline_error"

Omit problematic package: eset_protect, jamf_protect and ti_mandiant_advantage.

[git-generate]
for f in $(
        (
                for p in $(
                        yq 'select(.owner.github == "elastic/security-service-integrations")|.name' packages/**/manifest.yml \
                        | grep -v -- ---
                ); do
                        rg -l -g 'default.yml' "value: pipeline_error" packages/$p
                done
        )|sort|uniq|egrep -v 'eset_protect|jamf_protect|ti_mandiant_advantage'
); do
        (grep 'value: preserve_original_event' $f >/dev/null 2>&1) && continue
        perl -i -pe 'BEGIN{undef $/;} s/([a-z:"])
( *)(- set:.*value: pipeline_error)/$1
$2$3
$2- append:
$2    field: tags
$2    value: preserve_original_event
$2    allow_duplicates: false/smg' $f
done
for p in $(git diff --name-only HEAD~1|cut -d/ -f1,2|sort|uniq); do
    (
            cd $p
            elastic-package changelog add \
                    --description 'Add "preserve_original_event" tag to documents with `event.kind` set to "pipeline_error".' \
                    --type enhancement \
                    --next minor \
                    --link elastic#12046
    )>/dev/null 2>&1
done
@elastic-vault-github-plugin-prod
Copy link

elastic-vault-github-plugin-prod bot commented Dec 10, 2024 •
edited
Loading

🚀 Benchmarks report

Package tines 👍(0) 💚(1) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
time_saved 35714.29 25000 -10714.29 (-30%) 💔

To see the full report comment with /test benchmark fullreport

efd6 added a commit to efd6/integrations that referenced this pull request Dec 10, 2024
@efd6
ssi_all: add "preserve_original_event" tag to documents with event.ki…
829bdb6 
…nd set to "pipeline_error"

Omit problematic package: eset_protect, jamf_protect and ti_mandiant_advantage.

[git-generate]
for f in $(
        (
                for p in $(
                        yq 'select(.owner.github == "elastic/security-service-integrations")|.name' packages/**/manifest.yml \
                        | grep -v -- ---
                ); do
                        rg -l -g 'default.yml' "value: pipeline_error" packages/$p
                done
        )|sort|uniq|egrep -v 'eset_protect|google_workspace|jamf_protect|ti_mandiant_advantage'
); do
        (grep 'value: preserve_original_event' $f >/dev/null 2>&1) && continue
        perl -i -pe 'BEGIN{undef $/;} s/([a-z:"])
( *)(- set:.*value: pipeline_error)/$1
$2$3
$2- append:
$2    field: tags
$2    value: preserve_original_event
$2    allow_duplicates: false/smg' $f
done
for p in $(git diff --name-only HEAD~1|cut -d/ -f1,2|sort|uniq); do
    (
            cd $p
            elastic-package changelog add \
                    --description 'Add "preserve_original_event" tag to documents with `event.kind` set to "pipeline_error".' \
                    --type enhancement \
                    --next minor \
                    --link elastic#12046
    )>/dev/null 2>&1
done
@efd6 efd6 marked this pull request as ready for review December 10, 2024 08:12
@efd6 efd6 requested review from a team as code owners December 10, 2024 08:12
@elasticmachine
Copy link

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

@elastic-vault-github-plugin-prod
Copy link

Package ti_cybersixgill - 1.31.0 containing this change is available at https://epr.elastic.co/package/ti_cybersixgill/1.31.0/

@elastic-vault-github-plugin-prod
Copy link

Package ti_eclecticiq - 1.3.0 containing this change is available at https://epr.elastic.co/package/ti_eclecticiq/1.3.0/

@elastic-vault-github-plugin-prod
Copy link

Package ti_eset - 1.3.0 containing this change is available at https://epr.elastic.co/package/ti_eset/1.3.0/

@elastic-vault-github-plugin-prod
Copy link

Package ti_maltiverse - 1.3.0 containing this change is available at https://epr.elastic.co/package/ti_maltiverse/1.3.0/

@elastic-vault-github-plugin-prod
Copy link

Package ti_misp - 1.36.0 containing this change is available at https://epr.elastic.co/package/ti_misp/1.36.0/

@elastic-vault-github-plugin-prod
Copy link

Package ti_opencti - 2.4.0 containing this change is available at https://epr.elastic.co/package/ti_opencti/2.4.0/

@elastic-vault-github-plugin-prod
Copy link

Package ti_otx - 1.26.0 containing this change is available at https://epr.elastic.co/package/ti_otx/1.26.0/

@elastic-vault-github-plugin-prod
Copy link

Package ti_rapid7_threat_command - 2.1.0 containing this change is available at https://epr.elastic.co/package/ti_rapid7_threat_command/2.1.0/

@elastic-vault-github-plugin-prod
Copy link

Package ti_recordedfuture - 1.27.0 containing this change is available at https://epr.elastic.co/package/ti_recordedfuture/1.27.0/

@elastic-vault-github-plugin-prod
Copy link

Package ti_threatconnect - 1.4.0 containing this change is available at https://epr.elastic.co/package/ti_threatconnect/1.4.0/

@elastic-vault-github-plugin-prod
Copy link

Package ti_threatq - 1.29.0 containing this change is available at https://epr.elastic.co/package/ti_threatq/1.29.0/

@elastic-vault-github-plugin-prod
Copy link

Package tines - 1.13.0 containing this change is available at https://epr.elastic.co/package/tines/1.13.0/

@elastic-vault-github-plugin-prod
Copy link

Package trellix_edr_cloud - 1.3.0 containing this change is available at https://epr.elastic.co/package/trellix_edr_cloud/1.3.0/

@elastic-vault-github-plugin-prod
Copy link

Package trellix_epo_cloud - 1.12.0 containing this change is available at https://epr.elastic.co/package/trellix_epo_cloud/1.12.0/

@elastic-vault-github-plugin-prod
Copy link

Package trend_micro_vision_one - 1.22.0 containing this change is available at https://epr.elastic.co/package/trend_micro_vision_one/1.22.0/

@elastic-vault-github-plugin-prod
Copy link

Package trendmicro - 2.4.0 containing this change is available at https://epr.elastic.co/package/trendmicro/2.4.0/

@elastic-vault-github-plugin-prod
Copy link

Package tychon - 0.2.0 containing this change is available at https://epr.elastic.co/package/tychon/0.2.0/

@elastic-vault-github-plugin-prod
Copy link

Package vectra_detect - 1.10.0 containing this change is available at https://epr.elastic.co/package/vectra_detect/1.10.0/

@elastic-vault-github-plugin-prod
Copy link

Package wiz - 2.5.0 containing this change is available at https://epr.elastic.co/package/wiz/2.5.0/

@elastic-vault-github-plugin-prod
Copy link

Package zerofox - 1.26.0 containing this change is available at https://epr.elastic.co/package/zerofox/1.26.0/

@elastic-vault-github-plugin-prod
Copy link

Package zeronetworks - 1.16.0 containing this change is available at https://epr.elastic.co/package/zeronetworks/1.16.0/

@elastic-vault-github-plugin-prod
Copy link

Package zoom - 1.21.0 containing this change is available at https://epr.elastic.co/package/zoom/1.21.0/

@elastic-vault-github-plugin-prod
Copy link

Package zscaler_zia - 3.5.0 containing this change is available at https://epr.elastic.co/package/zscaler_zia/3.5.0/

@elastic-vault-github-plugin-prod
Copy link

Package zscaler_zpa - 1.19.0 containing this change is available at https://epr.elastic.co/package/zscaler_zpa/1.19.0/

@efd6 efd6 mentioned this pull request Dec 11, 2024
Merged
5 tasks
efd6 added a commit to efd6/integrations that referenced this pull request Dec 16, 2024
@efd6
google_workspace,jamf_protect,ti_mandiant: add "preserve_original_eve…
aab6a0a 
…nt" tag to documents with event.kind set to "pipeline_error"

This manually replays the changes in elastic#12046.
@efd6 efd6 mentioned this pull request Dec 16, 2024
Merged
5 tasks
efd6 added a commit to efd6/integrations that referenced this pull request Dec 16, 2024
@efd6
google_workspace,jamf_protect,ti_mandiant: add "preserve_original_eve…
6511315 
…nt" tag to documents with event.kind set to "pipeline_error"

This manually replays the changes in elastic#12046.
@efd6 efd6 mentioned this pull request Dec 16, 2024
Merged
5 tasks
efd6 added a commit that referenced this pull request Dec 17, 2024
@efd6
google_workspace,jamf_protect,ti_mandiant: add "preserve_original_eve…
d51dd29 
…nt" tag to documents with event.kind set to "pipeline_error" (#12108)

This manually replays the changes in #12046.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ishleenk17 ishleenk17 ishleenk17 approved these changes

@shmsr shmsr shmsr approved these changes

@kcreddy kcreddy kcreddy approved these changes

Assignees

@efd6 efd6

Labels
enhancement New feature or request Integration:abnormal_security Abnormal Security Integration:akamai Akamai Integration:amazon_security_lake Amazon Security Lake Integration:atlassian_bitbucket Atlassian Bitbucket Integration:atlassian_confluence Atlassian Confluence Integration:atlassian_jira Atlassian Jira Integration:auth0 Auth0 Integration:authentik authentik Integration:aws_bedrock Amazon Bedrock Integration:azure_frontdoor Azure Frontdoor Integration:azure_network_watcher_nsg Azure Network Watcher NSG Integration:azure_network_watcher_vnet Azure Network Watcher VNet Integration:barracuda_cloudgen_firewall Barracuda CloudGen Firewall Logs Integration:barracuda Barracuda Web Application Firewall Integration:bbot BBOT (Bighuge BLS OSINT Tool) Integration:bitdefender BitDefender Integration:bitwarden Bitwarden Integration:blacklens blacklens.io Integration:box_events Box Events Integration:canva Canva Integration:carbon_black_cloud VMware Carbon Black Cloud Integration:carbonblack_edr VMware Carbon Black EDR Integration:checkpoint_email Check Point Harmony Email & Collaboration Integration:checkpoint_harmony_endpoint Check Point Harmony Endpoint Integration:cisa_kevs CISA Known Exploited Vulnerabilities Integration:cisco_duo Cisco Duo Integration:cisco_meraki Cisco Meraki Integration:cisco_secure_endpoint Cisco Secure Endpoint Integration:cisco_umbrella Cisco Umbrella Integration:claroty_ctd Claroty CTD Integration:cloudflare_logpush Cloudflare Logpush Integration:cloudflare Cloudflare Integration:crowdstrike CrowdStrike Integration:cyberark_pta Cyberark Privileged Threat Analytics Integration:cyberarkpas CyberArk Privileged Access Security Integration:cybereason Cybereason Integration:cylance CylanceProtect Logs Integration:darktrace Darktrace Integration:digital_guardian Digital Guardian Integration:entityanalytics_ad Active Directory Entity Analytics Integration:entityanalytics_entra_id Microsoft Entra ID Entity Analytics Integration:entityanalytics_okta Okta Entity Analytics Integration:eset_protect ESET PROTECT Integration:f5_bigip F5 BIG-IP Integration:f5 F5 Logs (Deprecated) Integration:1password 1Password Team:obs-ds-hosted-services Label for the Observability Hosted Services team [elastic/obs-ds-hosted-services] Team:Obs-InfraObs Label for the Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations] Team:Security-Service Integrations Security Service Integrations Team [elastic/security-service-integrations]
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@efd6 @andrewkroh @elasticmachine @kcreddy @shmsr @ishleenk17