Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

all: migrate ssi packages to ecs@mappings #10135

Merged
merged 121 commits into from
Jun 21, 2024
Merged

all: migrate ssi packages to ecs@mappings #10135

merged 121 commits into from
Jun 21, 2024

Conversation

efd6
Copy link
Contributor

@efd6 efd6 commented Jun 11, 2024 •
edited
Loading

Proposed commit message

Migration performed using ecs-update.

  go run github.com/andrewkroh/go-examples/ecs-update@014b35dfe4c9832b51e7c909a39a48257d6a005d \
    -ecs-version=8.11.0 \
    -ecs-git-ref=v8.11.0 \
    -fields-yml-drop-ecs \
    -kibana-version=^8.13.0 \
    -drop-import-mappings \
    -pr=10135 \
    -owner=elastic/security-service-integrations \
    packages/*

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

  • [ ]

How to test this PR locally

Related issues

Screenshots

@efd6 efd6 added enhancement New feature or request Team:Security-Service Integrations Security Service Integrations Team [elastic/security-service-integrations] labels Jun 11, 2024
@efd6 efd6 self-assigned this Jun 11, 2024
andrewkroh
andrewkroh reviewed Jun 12, 2024
View reviewed changes
packages/1password/changelog.yml Outdated
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "1.29.0"
changes:
- description: Update the kibana constraint to ^8.13.0. Modified the field definitions to remove ECS fields where possible.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From a user perspective, do you think this second sentence is confusing? Maybe we should mention the ecs@mappings component template in some way. Like

Removed ECS field definitions that have been made redundant by the ecs@mappings component template.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That seems reasonable.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

andrewkroh/go-examples#36

packages/cel/changelog.yml Outdated Show resolved Hide resolved
@efd6 efd6 mentioned this pull request Jun 12, 2024
Merged
4 tasks
@agithomas agithomas mentioned this pull request Jun 12, 2024
Closed
@elasticmachine
Copy link

elasticmachine commented Jun 12, 2024 •
edited
Loading

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

@efd6 efd6 force-pushed the ecsmapping branch 4 times, most recently from 09c8746 to 3e06b13 Compare June 18, 2024 20:56
@efd6 efd6 mentioned this pull request Jun 18, 2024
Merged
4 tasks
@efd6 efd6 marked this pull request as ready for review June 19, 2024 02:52
@efd6 efd6 requested a review from a team as a code owner June 19, 2024 02:52
@elasticmachine
Copy link

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

@chrisberkhout
Copy link
Contributor

chrisberkhout commented Jun 19, 2024 •
edited
Loading

An issue to be aware of for integrations with transforms: elastic/elastic-package#1641

Correction: actually not an issue since we're not importing them now. They should just be there, presumably for transform destination indexes as well as data streams.

@chrisberkhout
Copy link
Contributor

An issue to be aware of for integrations with transforms: elastic/elastic-package#1641

Correction: actually not an issue since we're not importing them now. They should just be there, presumably for transform destination indexes as well as data streams.

Update: Transform destination indexes don't get the ECS dynamic templates that data streams get. This PR doesn't remove the manual definitions in transforms, so it still works (checked for ti_opencti).

chrisberkhout
chrisberkhout approved these changes Jun 19, 2024
View reviewed changes
Copy link
Contributor

@chrisberkhout chrisberkhout left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I looked at the full diffs for 4 or 5 integrations and checked out the impact on transforms in detail for ti_opencti. All looks good.

packages/ti_opencti/data_stream/indicator/fields/ecs.yml Show resolved Hide resolved
efd6 added 9 commits June 20, 2024 13:22
@efd6
[1password] - Updated fields definitions
9b151d7 
The conditions.kibana.version in the package manifest changed from ^8.12.0 to
^8.13.0. Modified the field definitions to remove ECS fields made redundant by
the ecs@mappings component template.

[git-generate]
go run github.com/andrewkroh/go-examples/[email protected] -ecs-version=8.11.0 [email protected] -drop-import-mappings -kibana-version=^8.13.0 -pr=10135 -fields-yml-drop-ecs packages/1password
@efd6
[akamai] - Updated fields definitions
447cd63 
The conditions.kibana.version in the package manifest changed from ^8.12.0 to
^8.13.0. Modified the field definitions to remove ECS fields made redundant by
the ecs@mappings component template.

[git-generate]
go run github.com/andrewkroh/go-examples/[email protected] -ecs-version=8.11.0 [email protected] -drop-import-mappings -kibana-version=^8.13.0 -pr=10135 -fields-yml-drop-ecs packages/akamai
@efd6
[amazon_security_lake] - removed ecs import_mappings
1c018cf 
Removed import_mappings. The conditions.kibana.version in the package manifest
changed from ^8.12.0 to ^8.13.0. Modified the field definitions to remove ECS
fields made redundant by the ecs@mappings component template.

[git-generate]
go run github.com/andrewkroh/go-examples/[email protected] -ecs-version=8.11.0 [email protected] -drop-import-mappings -kibana-version=^8.13.0 -pr=10135 -fields-yml-drop-ecs packages/amazon_security_lake
@efd6
[atlassian_bitbucket] - Updated fields definitions
731f66e 
The conditions.kibana.version in the package manifest changed from ^8.12.0 to
^8.13.0. Modified the field definitions to remove ECS fields made redundant by
the ecs@mappings component template.

[git-generate]
go run github.com/andrewkroh/go-examples/[email protected] -ecs-version=8.11.0 [email protected] -drop-import-mappings -kibana-version=^8.13.0 -pr=10135 -fields-yml-drop-ecs packages/atlassian_bitbucket
@efd6
[atlassian_confluence] - Updated fields definitions
2fe7ad4 
The conditions.kibana.version in the package manifest changed from ^8.12.0 to
^8.13.0. Modified the field definitions to remove ECS fields made redundant by
the ecs@mappings component template.

[git-generate]
go run github.com/andrewkroh/go-examples/[email protected] -ecs-version=8.11.0 [email protected] -drop-import-mappings -kibana-version=^8.13.0 -pr=10135 -fields-yml-drop-ecs packages/atlassian_confluence
@efd6
[atlassian_jira] - Updated fields definitions
7bc15c5 
The conditions.kibana.version in the package manifest changed from ^8.12.0 to
^8.13.0. Modified the field definitions to remove ECS fields made redundant by
the ecs@mappings component template.

[git-generate]
go run github.com/andrewkroh/go-examples/[email protected] -ecs-version=8.11.0 [email protected] -drop-import-mappings -kibana-version=^8.13.0 -pr=10135 -fields-yml-drop-ecs packages/atlassian_jira
@efd6
[auth0] - Updated fields definitions
794fc08 
The conditions.kibana.version in the package manifest changed from ^8.12.0 to
^8.13.0. Modified the field definitions to remove ECS fields made redundant by
the ecs@mappings component template.

[git-generate]
go run github.com/andrewkroh/go-examples/[email protected] -ecs-version=8.11.0 [email protected] -drop-import-mappings -kibana-version=^8.13.0 -pr=10135 -fields-yml-drop-ecs packages/auth0
@efd6
[aws_bedrock] - Updated fields definitions
a549f10 
The conditions.kibana.version in the package manifest changed from ^8.12.0 to
^8.13.0. Modified the field definitions to remove ECS fields made redundant by
the ecs@mappings component template.

[git-generate]
go run github.com/andrewkroh/go-examples/[email protected] -ecs-version=8.11.0 [email protected] -drop-import-mappings -kibana-version=^8.13.0 -pr=10135 -fields-yml-drop-ecs packages/aws_bedrock
@efd6
[azure_blob_storage] - removed ecs import_mappings
3a1173a 
Removed import_mappings. The conditions.kibana.version in the package manifest
changed from ^8.12.0 to ^8.13.0. Modified the field definitions to remove ECS
fields made redundant by the ecs@mappings component template. The ecs.version in
sample_event.json files was changed to 8.11.0. Previously sample_event.json
files contained 8.0.0.

[git-generate]
go run github.com/andrewkroh/go-examples/[email protected] -ecs-version=8.11.0 [email protected] -drop-import-mappings -kibana-version=^8.13.0 -pr=10135 -fields-yml-drop-ecs packages/azure_blob_storage
@elasticmachine
Copy link

Package okta - 2.11.0 containing this change is available at https://epr.elastic.co/search?package=okta

@elasticmachine
Copy link

Package opencanary - 0.1.0 containing this change is available at https://epr.elastic.co/search?package=opencanary

@elasticmachine
Copy link

Package panw_cortex_xdr - 1.27.0 containing this change is available at https://epr.elastic.co/search?package=panw_cortex_xdr

@elasticmachine
Copy link

Package ping_one - 1.16.0 containing this change is available at https://epr.elastic.co/search?package=ping_one

@elasticmachine
Copy link

Package pps - 0.1.0 containing this change is available at https://epr.elastic.co/search?package=pps

@elasticmachine
Copy link

Package prisma_cloud - 1.3.0 containing this change is available at https://epr.elastic.co/search?package=prisma_cloud

@elasticmachine
Copy link

Package proofpoint_tap - 1.22.0 containing this change is available at https://epr.elastic.co/search?package=proofpoint_tap

@elasticmachine
Copy link

Package pulse_connect_secure - 2.1.0 containing this change is available at https://epr.elastic.co/search?package=pulse_connect_secure

@elasticmachine
Copy link

Package qualys_vmdr - 3.3.0 containing this change is available at https://epr.elastic.co/search?package=qualys_vmdr

@elasticmachine
Copy link

Package rapid7_insightvm - 1.12.0 containing this change is available at https://epr.elastic.co/search?package=rapid7_insightvm

@elasticmachine
Copy link

Package santa - 3.18.0 containing this change is available at https://epr.elastic.co/search?package=santa

Jaos1992
Jaos1992 reviewed Jul 2, 2024
View reviewed changes
packages/akamai/changelog.yml
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

__

@chemamartinez chemamartinez mentioned this pull request Jul 13, 2024
Merged
10 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Jaos1992 Jaos1992 Jaos1992 left review comments

@andrewkroh andrewkroh andrewkroh left review comments

@chrisberkhout chrisberkhout chrisberkhout approved these changes

Assignees

@efd6 efd6

Labels
enhancement New feature or request Integration:akamai Akamai Integration:amazon_security_lake Amazon Security Lake Integration:atlassian_bitbucket Atlassian Bitbucket Integration:atlassian_confluence Atlassian Confluence Integration:atlassian_jira Atlassian Jira Integration:auth0 Auth0 Integration:aws_bedrock Amazon Bedrock Integration:azure_blob_storage Custom Azure Blob Storage Input Integration:azure_frontdoor Azure Frontdoor Integration:azure_network_watcher_nsg Azure Network Watcher NSG Integration:azure_network_watcher_vnet Azure Network Watcher VNet Integration:barracuda_cloudgen_firewall Barracuda CloudGen Firewall Logs Integration:barracuda Barracuda Web Application Firewall Integration:bbot BBOT (Bighuge BLS OSINT Tool) Integration:bitdefender BitDefender Integration:bitwarden Bitwarden Integration:1password 1Password Team:Security-Service Integrations Security Service Integrations Team [elastic/security-service-integrations]
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@efd6 @elasticmachine @chrisberkhout @andrewkroh @Jaos1992