Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CI] Add support for running tests with stack 9.0.0 #11138

Merged
merged 23 commits into from
Sep 19, 2024
Merged

[CI] Add support for running tests with stack 9.0.0 #11138

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
2348f2b
Add support for 9.0 stack
mrodm Sep 16, 2024
90042ff
Test STACK_VERSION 9.0.0-SNAPSHOT
mrodm Sep 16, 2024
40f864d
Allow to test packages with constraint ^8.x for STACK_VERSION 9.0.0
mrodm Sep 16, 2024
ea11883
Fix comments
mrodm Sep 16, 2024
4d3ba1e
Test again just 9.0.0 stack
mrodm Sep 16, 2024
be020b3
Restore allowing constraint 8.0.0 when testing with 9.0.0 stack
mrodm Sep 16, 2024
5538d14
Update pipelines
mrodm Sep 16, 2024
0dcfdcc
Update label for weekly step
mrodm Sep 17, 2024
c4f13fd
Remove STACK_VERSION env var
mrodm Sep 17, 2024
a0cbd5b
Update CI docs
mrodm Sep 17, 2024
72f4e8d
Update cron links to daily and weekly pipelines
mrodm Sep 17, 2024
3455508
Update stack version references
mrodm Sep 17, 2024
e6c4f01
Update docs ci
mrodm Sep 17, 2024
5163e39
Update depends relationships in daily job
mrodm Sep 18, 2024
c36dd31
Update cron for weekly job
mrodm Sep 18, 2024
9954b20
Revert depends_on changes
mrodm Sep 18, 2024
441f7e8
Test skipping reporting in 9.0
mrodm Sep 18, 2024
846b407
Update condition
mrodm Sep 18, 2024
19ea0ef
Update condition
mrodm Sep 18, 2024
db0bfed
Test without refspec
mrodm Sep 18, 2024
c8ed84b
restore regex
mrodm Sep 18, 2024
bbeef79
Test with other pipeline
mrodm Sep 18, 2024
26db499
Remove testing
mrodm Sep 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions .buildkite/pipeline.schedule-daily.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,18 @@ steps:
- step: "check"
allow_failure: false

- label: "Check integrations local stacks - Stack Version v9.0"
trigger: "integrations"
build:
env:
SERVERLESS: "false"
FORCE_CHECK_ALL: "true"
STACK_VERSION: 9.0.0-SNAPSHOT
PUBLISH_COVERAGE_REPORTS: "false"
depends_on:
- step: "check"
allow_failure: false
Comment on lines +62 to +64
Copy link
Contributor Author

@mrodm mrodm Sep 17, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be interesting to run this step after finishing the tests with 8.x ?
It would be updating the step defined in the depends_on field.

To avoid creating another build to test 290 packages at the same time (not sure if that could be an issue). WDYT?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, this is going to be temporary, I guess that some time after 9.0 is released, 7.x will become unmaintained and we will remove its stage.

If it depends on 8.x, will it be executed if 8.x fails? We want both to be executed even if the other fails, what is pretty common 😅

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it should be able to do that by setting depends_on as:

    depends_on:
      - step: "local-8-stack"
        allow_failure: true

Once it is removed 7.x, it can be reverted the depends_on configuration too.

Copy link
Contributor Author

@mrodm mrodm Sep 18, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll add this for now at least for daily

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But is it a problem to run so many stages at the same time? It looks like we are fixing an issue we don't have (yet?).

I don't have a strong preference with any of the options, but it looks simpler to just run everything at the same time if this is not a problem by now.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's true, I'm just moving ahead and probably there would not be any issue.
I'll change it back, and if there are issues we can do this change then.


- label: "Check integrations in serverless - project: Observability"
key: "trigger-integrations-serverless-obs"
trigger: "integrations-serverless"
Expand Down
13 changes: 13 additions & 0 deletions .buildkite/pipeline.schedule-weekly.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,3 +27,16 @@ steps:
depends_on:
- step: "check"
allow_failure: false

- label: "Check integrations local stacks and non-wolfi images for Elastic Agent - Stack Version v9.0"
trigger: "integrations"
build:
env:
SERVERLESS: "false"
FORCE_CHECK_ALL: "true"
STACK_VERSION: 9.0.0-SNAPSHOT
PUBLISH_COVERAGE_REPORTS: "false"
ELASTIC_PACKAGE_DISABLE_ELASTIC_AGENT_WOLFI: "true"
depends_on:
- step: "check"
allow_failure: false
Comment on lines +40 to +42
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as in the daily job? Would it be interesting to trigger this after the other step testing 8.x has finished? (update depends_on field).

7 changes: 5 additions & 2 deletions .buildkite/pipeline.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,5 +102,8 @@ steps:
# not fail build if this step fails
soft_fail: true
# run this step when if it is triggered by the daily job
if: >
build.source == "trigger_job" && build.env('BUILDKITE_TRIGGERED_FROM_BUILD_PIPELINE_SLUG') == "integrations-schedule-daily"
# skipped for now notification for tests related to STACK_VERSION: 9
if: |
build.source == "trigger_job" &&
build.env('BUILDKITE_TRIGGERED_FROM_BUILD_PIPELINE_SLUG') == "integrations-schedule-daily" &&
build.env('STACK_VERSION') !~ /^9/
11 changes: 9 additions & 2 deletions .buildkite/scripts/common.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -464,12 +464,19 @@ is_supported_stack() {
if [ "${kibana_version}" == "null" ]; then
return 0
fi
if [[ ! "${kibana_version}" =~ \^7\. && "${STACK_VERSION}" =~ ^7\. ]]; then
if [[ ( ! "${kibana_version}" =~ \^7\. ) && "${STACK_VERSION}" =~ ^7\. ]]; then
return 1
fi
if [[ ! "${kibana_version}" =~ \^8\. && "${STACK_VERSION}" =~ ^8\. ]]; then
if [[ ( ! "${kibana_version}" =~ \^8\. ) && "${STACK_VERSION}" =~ ^8\. ]]; then
return 1
fi

# TODO: Allowed temporarily to test packages with stack version 9.0 if they have as constraint ^8.0 defined too.
# This workaround should be removed once packages have been updated their constraints for 9.0 stack.
if [[ ( ! ( "${kibana_version}" =~ \^9\. || "${kibana_version}" =~ \^8\. ) ) && "${STACK_VERSION}" =~ ^9\. ]]; then
return 1
fi

return 0
}

Expand Down
4 changes: 2 additions & 2 deletions ...atecli.d/bump-latest-snapshot-version.yml → ...cli.d/bump-latest-8x-snapshot-version.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
name: Bump latest elastic-stack test version
pipelineid: 'bump-latest-elastic-stack-version'
name: Bump latest 8.x-SNAPSHOT elastic-stack test version
pipelineid: 'bump-latest-8x-elastic-stack-version'

actions:
default:
Expand Down
84 changes: 84 additions & 0 deletions .github/workflows/updatecli/updatecli.d/bump-latest-9x-snapshot-version.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
---
name: Bump latest 9.x-SNAPSHOT elastic-stack test version
pipelineid: 'bump-latest-9x-elastic-stack-version'

actions:
default:
title: '[updatecli] Update latest snapshot to {{ source "latestSnapshot" }}'
kind: github/pullrequest
scmid: default
spec:
labels:
- automation
- dependency

scms:
default:
kind: github
spec:
owner: '{{ .scm.owner }}'
repository: '{{ .scm.repository }}'
user: '{{ requiredEnv "GITHUB_ACTOR" }}'
username: '{{ requiredEnv "GITHUB_ACTOR" }}'
token: '{{ requiredEnv "GITHUB_TOKEN" }}'
commitusingapi: true
branch: main

sources:
latestSnapshot:
name: Get latest snapshot
kind: json
spec:
file: https://storage.googleapis.com/artifacts-api/snapshots/main.json
key: .version
latestSnapshotMajorMinor:
name: Get latest snapshort major and minor
kind: json
spec:
file: https://storage.googleapis.com/artifacts-api/snapshots/main.json
key: .version
transformers:
- findsubmatch:
pattern: '(^\d+\.\d+)\.[^\s]+$'
captureindex: 1

targets:
update-snapshot-daily:
name: '[updatecli] [daily] Update latest snapshot to {{ source "latestSnapshot" }}'
kind: file
sourceid: latestSnapshot
scmid: default
spec:
file: '.buildkite/pipeline.schedule-daily.yml'
matchpattern: '(STACK_VERSION:) 9\.[^\s]*\.[^\s]*'
replacepattern: '$1 {{ source "latestSnapshot" }}'

update-snapshot-label-daily:
name: '[updatecli] [daily] Update latest snapshot label step to {{ source "latestSnapshotMajorMinor" }}'
kind: file
sourceid: latestSnapshotMajorMinor
scmid: default
spec:
file: '.buildkite/pipeline.schedule-daily.yml'
matchpattern: '(Stack Version) v9\.\d+"'
replacepattern: 'Stack Version v{{ source "latestSnapshotMajorMinor" }}"'

update-snapshot-weekly:
name: '[updatecli] [weekly] Update latest snapshot to {{ source "latestSnapshot" }}'
kind: file
sourceid: latestSnapshot
scmid: default
spec:
file: '.buildkite/pipeline.schedule-weekly.yml'
matchpattern: '(STACK_VERSION:) 9\.[^\s]*\.[^\s]*'
replacepattern: '$1 {{ source "latestSnapshot" }}'

update-snapshot-label-weekly:
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These steps are now failing in the current check, but they should work once this PR is merged and all the changes are in main branch.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding the updatecli config too 👍

name: '[updatecli] [weekly] Update latest snapshot label step to {{ source "latestSnapshotMajorMinor" }}'
kind: file
sourceid: latestSnapshotMajorMinor
scmid: default
spec:
file: '.buildkite/pipeline.schedule-weekly.yml'
matchpattern: '(Stack Version) v9\.\d+"'
replacepattern: 'Stack Version v{{ source "latestSnapshotMajorMinor" }}"'
2 changes: 1 addition & 1 deletion catalog-info.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -142,7 +142,7 @@ spec:
schedules:
main_weekly:
branch: "main"
cronline: "30 3 * * 1"
cronline: "00 5 * * 1"
message: "Run the weekly jobs"
provider_settings:
trigger_mode: none # don't trigger jobs from github activity
Expand Down
33 changes: 28 additions & 5 deletions docs/ci_pipelines.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ Currently, there are five different pipelines:
- https://buildkite.com/elastic/integrations-serverless: pipeline in charge of testing all packages using a Elastic Serverless project. More info at [section](#serverless-pipeline).
- https://buildkite.com/elastic/integrations-publish: pipeline to publish the new versions of packages. More info at [section](#publish-packages).
- https://buildkite.com/elastic/integrations-schedule-daily/: pipeline running every night to test packages in different scenarios. More info at [section](#daily-job).
- https://buildkite.com/elastic/integrations-schedule-weekly/: pipeline running once per week to test packages in different scenarios. More info at [section](#weekly-job).
- https://buildkite.com/elastic/integrations-backport/: pipeline to create backport branches (just from UI). More info at [section](#backport-branches-pipeline).

## Pull Requests and pushes to specific branches
Expand All @@ -22,14 +23,15 @@ Special comments that can be added in the Pull Request (by Elastic employees):
There are some environment variables that can be added into this pipeline:
- **FORCE_CHECK_ALL**: If `true`, this forces the CI to check all packages even if those packages have no file updated/added/deleted. Default: `false`.
- **STACK_VERSION**: Force the CI steps to spin up a specific Elastic stack version to run the tests. Default: unset.
- **STACK_LOGSDB_ENABLED**: Enable LogsDB setting in Elasticsearch service. Default: false.
- **PUBLISH_COVERAGE_REPORTS**: If `true`, it enables reporting coverage reports.
Currently, it is just set for the build triggered with the current major Elastic stack from the daily job. Default: `false`.

These environment variables can be defined in different locations:
- In the [global `env` section](https://github.com/elastic/integrations/blob/5276ef63712f8f2311818770881688870e8422fe/.buildkite/pipeline.yml#L2).
- In the case of the scheduled daily job here:
- [Running tests with previous major Elastic stack](https://github.com/elastic/integrations/blob/5276ef63712f8f2311818770881688870e8422fe/.buildkite/pipeline.schedule-daily.yml#L21).
- [Running tests with current major Elastic stack](https://github.com/elastic/integrations/blob/d6d99792b90838d18844f6df9343bc5f16130666/.buildkite/pipeline.schedule-daily.yml#L32).
- In the case of the scheduled daily job in each step. Some examples:
- [Running tests with 7.x major Elastic stack](https://github.com/elastic/integrations/blob/5276ef63712f8f2311818770881688870e8422fe/.buildkite/pipeline.schedule-daily.yml#L21).
- [Running tests with 8.x major Elastic stack](https://github.com/elastic/integrations/blob/d6d99792b90838d18844f6df9343bc5f16130666/.buildkite/pipeline.schedule-daily.yml#L32).

More details about this CI pipeline:

Expand Down Expand Up @@ -124,9 +126,13 @@ The schedule of this job can be checked [here](https://github.com/elastic/integr

The scenarios that are tested in this daily job are:

- Test packages with a local Elastic stack running the previous major version of the stack (7.17.X SNAPSHOT).
- Test packages with a local Elastic stack running the latest 7.x version of the stack (7.17.X SNAPSHOT).
- Triggered pipeline: https://buildkite.com/elastic/integrations
- Test packages with a local Elastic stack running the latest major version of the stack (currently 8.X.Y-SNAPSHOT).
- Test packages with a local Elastic stack running the latest 8.x version of the stack (8.X.Y-SNAPSHOT).
- Triggered pipeline: https://buildkite.com/elastic/integrations
- Test packages with a local Elastic stack running the latest 8.x version of the stack with LogsDB setting enabled (8.X.Y-SNAPSHOT).
- Triggered pipeline: https://buildkite.com/elastic/integrations
- Test packages with a local Elastic stack running the latest major version of the stack (currently 9.X.Y-SNAPSHOT).
- Triggered pipeline: https://buildkite.com/elastic/integrations
- Test packages with an Elastic Serverless Observability project.
- Triggered pipeline: https://buildkite.com/elastic/integration-serverless
Expand All @@ -141,6 +147,23 @@ have been published by triggering the pipeline https://buildkite.com/elastic/int
Each step triggering a new pipeline can be customized through environment variables. Environment variables that can
be used in each pipeline are detailed in the corresponding sections of each pipeline.

## Weekly job

**Note**: Just available for Employees at Elastic.

Every week it is configured to run a job that will be in charge of testing all packages with non-Wolfi Elastic Agent docker images: https://buildkite.com/elastic/integrations-schedule-weekly/

The schedule of this job can be checked [here](https://github.com/elastic/integrations/blob/2e72e8524728daca2d47c814d8042031b8f5804f/catalog-info.yaml#L145).

The scenarios that are tested in this weekly job are:

- Test packages with a local Elastic stack running the latest 8.x version of the stack with Elastic Agent images based on Ubuntu images (8.X.Y-SNAPSHOT).
- Triggered pipeline: https://buildkite.com/elastic/integrations
- Test packages with a local Elastic stack running the latest major version of the stack with Elastic Agent images based on non-Wolfi images (currently 9.X.Y-SNAPSHOT).
- Triggered pipeline: https://buildkite.com/elastic/integrations

Each step triggering a new pipeline can be customized through environment variables. Environment variables that can
be used in each pipeline are detailed in the corresponding sections of each pipeline.

## Backport branches pipeline

Expand Down
Loading