[meta] Upgrade integrations to ECS 8.16

[taylor-swanson]

Guide

It is recommended to split the changes into smaller PRs to limit the number of changed files. Generally speaking, 10 integrations per PR is a good number to target.

Automated method

Use the ecs-update tool which can be found here.

cd packages/
ecs-update -pr 999999 -ecs-git-ref v8.16.0 -ecs-version 8.16.0 -owner elastic/owner-name-here package_1 [package_2 ...]

Once PR is filed, the changelogs will need to be updated with the correct PR number.

Manual method

Update ECS references in integrations to version 8.16.

• Update reference in _dev/build/build.yml to 8.16.0
• Update package changelog/manifest and regenerate README.

dependencies:
  ecs:
-    reference: "[email protected]"
+    reference: "[email protected]"

ECS 8.16 Changes

Schema Changes

Bugfixes

• Fix broken link in docs for vulnerability.id. Fix broken link for vulnerabilty.id (correct commits) ecs#2328

Added

• Added volume.* as beta field set. Stage 2 beta changes for RFC 0040, volume.* ecs#2269
• Advanced process.env_vars to GA. Promote process.env_vars to GA ecs#2315
• Advanced process.io and process.tty fields to GA. Promote process.io and process.tty fields to GA ecs#2317
• Added threat.indicator.id. [RFC] added new field: threat.indicator.id - resolves GH-2252 ecs#2324
• Added process.group to generated schemas. Add process.group fields to generated schemas ecs#2335

Improvements

• Define base encoding of x509.serial_number. Define base encoding in x509.serial_number ecs#2383

Tooling and Artifact Changes

Bugfixes

• Fix broken link for vulnerabilty.id Fix broken link for vulnerabilty.id (correct commits) ecs#2328

Added

• Documentation in README.md providing instruction on contributions to ECS during the OTel donation Special guidance for ECS contributions during OTel donation ecs#2325

Integrations

@elastic/cloud-security-posture:

• cloud_security_posture

@elastic/ecosystem:

• elastic_package_registry

@elastic/elastic-agent:

• elastic_agent

@elastic/elastic-agent-data-plane:

• filestream
• journald
• linux
• log
• windows

@elastic/fleet:

• fleet_server

@elastic/obs-cloudnative-monitoring:

• containerd
• docker
• istio
• kubernetes
• kubernetes_otel
• nginx_ingress_controller

@elastic/obs-ds-hosted-services:

• aws
• aws_logs
• awsfirehose
• azure
• azure_metrics
• gcp
• gcp_metrics

@elastic/obs-ds-intake-services:

• apm
• profiler_collector
• profiler_symbolizer

@elastic/obs-infraobs-integrations:

• activemq
• airflow
• apache
• apache_spark
• apache_tomcat
• awsfargate
• azure_app_service
• azure_application_insights
• azure_billing
• azure_functions
• azure_logs
• azure_openai
• cassandra
• ceph
• cisco_meraki_metrics
• citrix_adc
• cockroachdb
• coredns
• couchbase
• couchdb
• etcd
• gcp_vertexai
• golang
• hadoop
• haproxy
• ibmmq
• iis
• influxdb
• jolokia
• kafka
• kafka_log
• memcached
• microsoft_sqlserver
• mongodb
• mongodb_atlas
• mysql
• nagios_xi
• nats
• nginx
• nginx_ingress_controller_otel
• oracle
• oracle_weblogic
• panw_metrics
• php_fpm
• postgresql
• prometheus
• prometheus_input
• rabbitmq
• redis
• redisenterprise
• salesforce
• spring_boot
• sql
• stan
• statsd_input
• system
• tomcat
• traefik
• vsphere
• websphere_application_server
• zookeeper

@elastic/sec-deployment-and-devices:

• Update Deployment and Devices integrations to ECS 8.16.0 (part 1)  #12119

  • arista_ngfw
  • cef
  • checkpoint
  • cisco_aironet
  • cisco_asa
  • cisco_ftd
  • cisco_ios
  • cisco_ise
  • cisco_nexus
  • cisco_secure_email_gateway
  • citrix_waf

• Update Deployment and Devices integrations to ECS 8.16.0 (part 2) #12120

  • fortinet_forticlient
  • fortinet_fortiedr
  • fortinet_fortigate
  • fortinet_fortimail
  • fortinet_fortimanager
  • fortinet_fortiproxy
  • goflow2
  • hashicorp_vault
  • imperva
  • iptables

• Update Deployment and Devices integrations to ECS 8.16.0 (part 3) #12121

  • juniper_srx
  • modsecurity
  • netflow
  • osquery
  • panw
  • pfsense
  • proxysg

• Update Deployment and Devices integrations to ECS 8.16.0 (part 4) #12122

  • qnap_nas
  • snort
  • sonicwall_firewall
  • sophos
  • squid
  • stormshield
  • suricata
  • tcp
  • udp
  • watchguard_firebox
  • zeek

@elastic/sec-linux-platform:

• auditd
• auditd_manager
• cloud_defend
• fim
• network_traffic
• sysmon_linux
• system_audit

@elastic/sec-windows-platform:

• hid_bravura_monitor
• microsoft_dhcp
• microsoft_dnsserver
• microsoft_exchange_server
• mysql_enterprise
• windows_etw
• winlog

@elastic/security-asset-management:

• osquery_manager

@elastic/security-service-integrations:

• 1password
• abnormal_security
• akamai
• amazon_security_lake
• atlassian_bitbucket
• atlassian_confluence
• atlassian_jira
• auth0
• authentik
• aws_bedrock
• azure_blob_storage
• azure_frontdoor
• azure_network_watcher_nsg
• azure_network_watcher_vnet
• barracuda
• barracuda_cloudgen_firewall
• bbot
• bitdefender
• bitwarden
• blacklens
• box_events
• canva
• carbon_black_cloud
• carbonblack_edr
• cel
• checkpoint_email
• checkpoint_harmony_endpoint
• cisa_kevs
• cisco_duo
• cisco_meraki
• cisco_secure_endpoint
• cisco_umbrella
• claroty_ctd
• cloudflare
• cloudflare_logpush
• corelight
• cribl
• crowdstrike
• cyberark_pta
• cyberarkpas
• cybereason
• cylance
• darktrace
• digital_guardian
• entityanalytics_ad
• entityanalytics_entra_id
• entityanalytics_okta
• eset_protect
• f5
• f5_bigip
• falco
• fireeye
• first_epss
• forcepoint_web
• forgerock
• gcp_pubsub
• gigamon
• github
• gitlab
• google_cloud_storage
• google_scc
• google_workspace
• http_endpoint
• httpjson
• imperva_cloud_waf
• infoblox_bloxone_ddi
• infoblox_nios
• jamf_compliance_reporter
• jamf_pro
• jamf_protect
• jumpcloud
• keycloak
• lastpass
• lumos
• lyve_cloud
• m365_defender
• mattermost
• menlo
• microsoft_defender_cloud
• microsoft_defender_endpoint
• microsoft_exchange_online_message_trace
• microsoft_sentinel
• mimecast
• netskope
• o365
• okta
• opencanary
• panw_cortex_xdr
• ping_one
• pps
• prisma_access
• prisma_cloud
• proofpoint_on_demand
• proofpoint_tap
• pulse_connect_secure
• qualys_vmdr
• rapid7_insightvm
• santa
• sentinel_one
• sentinel_one_cloud_funnel
• servicenow
• slack
• snyk
• sophos_central
• spycloud
• sublime_security
• symantec_edr_cloud
• symantec_endpoint
• symantec_endpoint_security
• sysdig
• tanium
• teleport
• tenable_io
• tenable_sc
• threat_map
• thycotic_ss
• ti_abusech
• ti_anomali
• ti_cif3
• ti_crowdstrike
• ti_custom
• ti_cybersixgill
• ti_eclecticiq
• ti_eset
• ti_maltiverse
• ti_mandiant_advantage
• ti_misp
• ti_opencti
• ti_otx
• ti_rapid7_threat_command
• ti_recordedfuture
• ti_threatconnect
• ti_threatq
• ti_util
• tines
• trellix_edr_cloud
• trellix_epo_cloud
• trend_micro_vision_one
• trendmicro
• tychon
• vectra_detect
• websocket
• wiz
• zerofox
• zeronetworks
• zoom
• zscaler_zia
• zscaler_zpa

@elastic/stack-monitoring:

• beat
• elasticsearch
• enterprisesearch
• kibana
• logstash
• platform_observability

[consulthys]

@elastic/stack-monitoring:

• beat[ ] elasticsearch[ ] enterprisesearch[ ] kibana[ ] logstash[ ] platform_observability

Hey @taylor-swanson thanks for initiating this. What is the deadline for this change?
Also, can you share whether this should be done before or after https://github.com/elastic/ingest-dev/issues/4484 ?

[taylor-swanson]

Hey @consulthys!

While it is nice to have ECS updated, it isn't critical it needs to be done immediately, unless an integration needs to take advantage of a change in the new version. We haven't issued a new ECS release in a while (8.11.0 was the previous version), but at least back then, we'd work on getting the integrations updated within a short period of time (perhaps a week or so?) after the release.

The ecs-update tool (see the issue description above) definitely helps make the process of updating integrations less painful.

Regarding how it relates to the issue you linked, that's actually a good question. As far as I know, I don't think 9.0 will have any issues using a 8.16 (or any 8.x version) ECS. There isn't a strong version enforcement like there is for kibana.version in a package manifest. I'll raise that question with the ECS team, though. If we do need a 9.0 version, we'll for sure get another issue out like this one.