Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CI] Add weekly job to test packages with Ubuntu Elastic Agent docker image #10844

Merged
merged 33 commits into from
Aug 29, 2024
+111 −5
Merged

[CI] Add weekly job to test packages with Ubuntu Elastic Agent docker image #10844

Show file tree
Hide file tree
Changes from 25 commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
dc999f5
Add support to override Elastic Agent docker image
mrodm Aug 21, 2024
c5bd547
Add weekly schedule for integration pipeline
mrodm Aug 21, 2024
e10d784
Add automation to update snapshot
mrodm Aug 21, 2024
aec9e31
Show docker images used in elastic-package stack up
mrodm Aug 21, 2024
24661e3
Test with one package
mrodm Aug 21, 2024
127d267
Test environment variables
mrodm Aug 21, 2024
35f1535
Add end of line in regex
mrodm Aug 21, 2024
dc7f622
Test other regex
mrodm Aug 21, 2024
f2042b6
try wide regex
mrodm Aug 21, 2024
17d1b7b
Test other line
mrodm Aug 21, 2024
250ea83
Wide regex
mrodm Aug 21, 2024
2eb841f
Test regex
mrodm Aug 21, 2024
deeb675
Teat regex
mrodm Aug 21, 2024
f6d1452
Test with other branch
mrodm Aug 21, 2024
e9cd8b4
Test without STACK_VERSION
mrodm Aug 22, 2024
99e4ec1
Disable updatecli target
mrodm Aug 22, 2024
5c2220e
Test with another package
mrodm Aug 22, 2024
44ab466
Test without elastic-agent var
mrodm Aug 22, 2024
0835d91
Show docker images
mrodm Aug 22, 2024
d111acb
Test with elastic-package to show agent images
mrodm Aug 22, 2024
971c2cd
Use Elastic Agent override
mrodm Aug 22, 2024
888a2c2
Test defining STACK_VERSION and overriding EA image
mrodm Aug 22, 2024
608e233
Add weekly pipeline
mrodm Aug 22, 2024
27ac181
Update updatecli configuration
mrodm Aug 22, 2024
95969c0
Test with all packages
mrodm Aug 22, 2024
c6aba2f
Remove changes about Elastic Agent docker images
mrodm Aug 26, 2024
980a890
Revert bump elastic-package dep
mrodm Aug 26, 2024
e959100
Merge remote-tracking branch 'upstream/main' into add-weekly-job-elas…
mrodm Aug 26, 2024
d3d5002
Update slack channel
mrodm Aug 26, 2024
cb736d6
Update env. vars for weekly pipeline
mrodm Aug 27, 2024
8dbe594
Test env. var substitution
mrodm Aug 27, 2024
e7c986b
Update labels updatecli
mrodm Aug 28, 2024
68da185
Remove testing just one package
mrodm Aug 28, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions .buildkite/pipeline.schedule-weekly.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/buildkite/pipeline-schema/main/schema.json
name: integrations-schedule-weekly

env:
SETUP_GVM_VERSION: "v0.5.2"
LINUX_AGENT_IMAGE: "golang:${GO_VERSION}"

# The pipeline is triggered by the scheduler every week
steps:
- label: ":white_check_mark: Check go sources"
key: "check"
command: ".buildkite/scripts/check_sources.sh"
agents:
image: "${LINUX_AGENT_IMAGE}"
cpu: "8"
memory: "4G"

- label: "Check integrations local stacks and Elastic Agent Ubuntu docker - Stack Version v8.16"
trigger: "integrations"
build:
env:
SERVERLESS: "false"
FORCE_CHECK_ALL: "true"
STACK_VERSION: 8.16.0-SNAPSHOT
PUBLISH_COVERAGE_REPORTS: "false"
ELASTIC_AGENT_DOCKER_IMAGE: "docker.elastic.co/elastic-agent/elastic-agent-complete"
depends_on:
- step: "check"
allow_failure: false
4 changes: 3 additions & 1 deletion .buildkite/pipeline.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,14 +1,16 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/buildkite/pipeline-schema/main/schema.json
env:
SETUP_GVM_VERSION: "v0.5.2"
LINUX_AGENT_IMAGE: "golang:${GO_VERSION}"
DOCKER_COMPOSE_VERSION: "v2.24.1"
DOCKER_VERSION: "26.1.2"
KIND_VERSION: 'v0.20.0'
K8S_VERSION: 'v1.30.0'
YQ_VERSION: 'v4.35.2'
JQ_VERSION: '1.7'
GH_CLI_VERSION: "2.29.0"

# Agent images used in pipeline steps
LINUX_AGENT_IMAGE: "golang:${GO_VERSION}"
IMAGE_UBUNTU_X86_64: "family/core-ubuntu-2204"

# Elastic package settings
Expand Down
15 changes: 15 additions & 0 deletions .buildkite/scripts/common.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -493,23 +493,38 @@ prepare_stack() {
echo "--- Prepare stack"

local args="-v"
local stack_version=""
if [ -n "${STACK_VERSION}" ]; then
args="${args} --version ${STACK_VERSION}"
stack_version="${STACK_VERSION}"
else
local version
version=$(oldest_supported_version)
if [[ "${version}" != "null" ]]; then
args="${args} --version ${version}"
stack_version="${version}"
fi
fi

if [[ "${ELASTIC_AGENT_DOCKER_IMAGE:-""}" != "" ]]; then
export ELASTIC_AGENT_IMAGE_REF_OVERRIDE="${ELASTIC_AGENT_DOCKER_IMAGE}:${stack_version}"
echo "Using Elastic Agent docker image: ${ELASTIC_AGENT_IMAGE_REF_OVERRIDE}"
fi

echo "Boot up the Elastic stack"
if ! ${ELASTIC_PACKAGE_BIN} stack up -d ${args} ; then
return 1
fi
echo ""
${ELASTIC_PACKAGE_BIN} stack status
echo ""

if [[ "${ELASTIC_AGENT_DOCKER_IMAGE:-""}" != "" ]]; then
echo ""
echo "Images used for Elastic Agent:"
docker ps --format "{{.Names}} {{.Image}}" |grep "elastic-agent"
echo ""
fi
mrodm marked this conversation as resolved.
Show resolved Hide resolved
}

is_serverless() {
Expand Down
1 change: 1 addition & 0 deletions .buildkite/scripts/trigger_integrations_in_parallel.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,7 @@ for package in ${PACKAGE_LIST}; do
FORCE_CHECK_ALL: "${FORCE_CHECK_ALL}"
SERVERLESS: "false"
UPLOAD_SAFE_LOGS: ${UPLOAD_SAFE_LOGS}
ELASTIC_AGENT_DOCKER_IMAGE: "${ELASTIC_AGENT_DOCKER_IMAGE:-""}"
artifact_paths:
- build/test-results/*.xml
- build/test-coverage/*.xml
Expand Down
30 changes: 26 additions & 4 deletions .github/workflows/updatecli/updatecli.d/bump-latest-snapshot-version.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,8 +43,8 @@ sources:
captureindex: 1

targets:
update-snapshot:
name: '[updatecli] Update latest snapshot to {{ source "latestSnapshot" }}'
update-snapshot-daily:
name: '[updatecli] [daily] Update latest snapshot to {{ source "latestSnapshot" }}'
kind: file
sourceid: latestSnapshot
scmid: default
Expand All @@ -53,12 +53,34 @@ targets:
matchpattern: '(STACK_VERSION:) 8\.[^\s]*\.[^\s]*'
replacepattern: '$1 {{ source "latestSnapshot" }}'

update-snapshot-label:
name: '[updatecli] Update latest snapshot label step to {{ source "latestSnapshotMajorMinor" }}'
update-snapshot-label-daily:
name: '[updatecli] [daily] Update latest snapshot label step to {{ source "latestSnapshotMajorMinor" }}'
kind: file
sourceid: latestSnapshotMajorMinor
scmid: default
spec:
file: '.buildkite/pipeline.schedule-daily.yml'
matchpattern: '(Stack Version) v8\.\d+"'
replacepattern: 'Stack Version v{{ source "latestSnapshotMajorMinor" }}"'

## TODO: Looks like these targets are running with the contents from main branch
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be uncommented in a following PR.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Umm, it is configured to trigger also on pull requests and I remember it took the changes 🤔

integrations/.github/workflows/bump-elastic-stack-version.yml

Line 8 in 1fb5746

pull_request:

But maybe this only works for existing targets? Or for branches pushed to upstream repository?

Or it could be related to the conflicting names, now they are duplicated as mentioned in other comments.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or it could be related to the conflicting names, now they are duplicated as mentioned in other comments.

When I was testing this, the names where update-snapshot and update-snapshot-weekly-build
deeb675

For that commit, that this line exists in main branch the github action was working successfully. The other commits where the expected line did not exist yet in main branch, github actions were failing.

Maybe it is related to not create the branch in the upstream repository, I did not test that.

## Uncomment this target once this code is merged
#update-snapshot-daily:
mrodm marked this conversation as resolved.
Show resolved Hide resolved
# name: '[updatecli] [weekly] Update latest snapshot to {{ source "latestSnapshot" }}'
# kind: file
# sourceid: latestSnapshot
# scmid: default
# spec:
# file: '.buildkite/pipeline.schedule-weekly.yml'
# matchpattern: '(STACK_VERSION:) 8\.[^\s]*\.[^\s]*'
# replacepattern: '$1 {{ source "latestSnapshot" }}'

#update-snapshot-label-daily:
mrodm marked this conversation as resolved.
Show resolved Hide resolved
# name: '[updatecli] [weekly] Update latest snapshot label step to {{ source "latestSnapshotMajorMinor" }}'
# kind: file
# sourceid: latestSnapshotMajorMinor
# scmid: default
# spec:
# file: '.buildkite/pipeline.schedule-weekly.yml'
# matchpattern: '(Stack Version) v8\.\d+"'
# replacepattern: 'Stack Version v{{ source "latestSnapshotMajorMinor" }}"'
51 changes: 51 additions & 0 deletions catalog-info.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,57 @@ spec:
# yaml-language-server: $schema=https://gist.githubusercontent.com/elasticmachine/988b80dae436cafea07d9a4a460a011d/raw/e57ee3bed7a6f73077a3f55a38e76e40ec87a7cf/rre.schema.json
apiVersion: backstage.io/v1alpha1
kind: Resource
metadata:
name: pipeline-integrations-schedule-weekly
description: 'Weekly pipeline for the Integrations project'
links:
- title: Pipeline
url: https://buildkite.com/elastic/integrations-schedule-weekly

spec:
type: buildkite-pipeline
owner: group:ingest-fp
system: buildkite
implementation:
apiVersion: buildkite.elastic.dev/v1
kind: Pipeline
metadata:
name: integrations-schedule-weekly
description: 'Weekly pipeline for the Integrations project'
spec:
pipeline_file: ".buildkite/pipeline.schedule-weekly.yml"
schedules:
main_weekly:
branch: "main"
cronline: "30 3 * * 1"
message: "Run the weekly jobs"
Comment on lines +142 to +146
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should these be commented until the wolfi is used ? or should we already running this job weekly ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should already run this weekly to capture any early bug.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This pipeline will run the system tests using the Ubuntu Elastic Agent docker image. So it would be running for now the same tests as the daily builds.

This will change once the upcoming elastic-package release is merged into this repository. From there, packages will be tested with the wolfi images in the daily builds.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I guess we can wait, but it would be good to have it running at least once before wolfi images are available, to have a base to compare with. Though the dailies can be this base too.

provider_settings:
trigger_mode: none # don't trigger jobs from github activity
build_pull_request_forks: false
build_pull_requests: true
build_tags: false
filter_enabled: true
filter_condition: >-
build.pull_request.id == null || (build.creator.name == 'elasticmachine' && build.pull_request.id != null && build.source == 'api')
repository: elastic/integrations
cancel_intermediate_builds: true
cancel_intermediate_builds_branch_filter: '!main'
skip_intermediate_builds: true
skip_intermediate_builds_branch_filter: '!main'
env:
ELASTIC_SLACK_NOTIFICATIONS_ENABLED: 'true'
SLACK_NOTIFICATIONS_CHANNEL: '#beats-build'
mrodm marked this conversation as resolved.
Show resolved Hide resolved
SLACK_NOTIFICATIONS_ALL_BRANCHES: 'true'
SLACK_NOTIFICATIONS_ON_SUCCESS: 'true'
teams:
ingest-fp:
access_level: MANAGE_BUILD_AND_READ
everyone:
access_level: READ_ONLY
---
# yaml-language-server: $schema=https://gist.githubusercontent.com/elasticmachine/988b80dae436cafea07d9a4a460a011d/raw/e57ee3bed7a6f73077a3f55a38e76e40ec87a7cf/rre.schema.json
apiVersion: backstage.io/v1alpha1
kind: Resource
metadata:
name: pipeline-integrations-serverless
description: 'Pipeline for the Integrations project in serverless'
Expand Down
44 changes: 24 additions & 20 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,7 @@ require (

require (
cloud.google.com/go v0.112.1 // indirect
cloud.google.com/go/compute v1.24.0 // indirect
cloud.google.com/go/compute/metadata v0.2.3 // indirect
cloud.google.com/go/compute/metadata v0.3.0 // indirect
cloud.google.com/go/iam v1.1.7 // indirect
cloud.google.com/go/pubsub v1.37.0 // indirect
cloud.google.com/go/storage v1.40.0 // indirect
Expand All @@ -39,6 +38,7 @@ require (
github.com/asaskevich/govalidator v0.0.0-20210307081110-f21760c49a8d // indirect
github.com/aymerick/raymond v2.0.2+incompatible // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/blang/semver/v4 v4.0.0 // indirect
github.com/cbroglie/mustache v1.4.0 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/chai2010/gettext-go v1.0.2 // indirect
Expand All @@ -65,15 +65,16 @@ require (
github.com/fatih/color v1.17.0 // indirect
github.com/felixge/httpsnoop v1.0.4 // indirect
github.com/fsouza/fake-gcs-server v1.48.0 // indirect
github.com/fxamacker/cbor/v2 v2.7.0 // indirect
github.com/go-errors/errors v1.4.2 // indirect
github.com/go-logr/logr v1.4.1 // indirect
github.com/go-logr/logr v1.4.2 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/go-ole/go-ole v1.2.6 // indirect
github.com/go-openapi/errors v0.20.3 // indirect
github.com/go-openapi/jsonpointer v0.19.6 // indirect
github.com/go-openapi/jsonreference v0.20.2 // indirect
github.com/go-openapi/strfmt v0.21.3 // indirect
github.com/go-openapi/swag v0.22.3 // indirect
github.com/go-openapi/swag v0.22.4 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
github.com/golang/protobuf v1.5.4 // indirect
Expand Down Expand Up @@ -121,7 +122,7 @@ require (
github.com/mitchellh/go-wordwrap v1.0.1 // indirect
github.com/mitchellh/mapstructure v1.5.0 // indirect
github.com/mitchellh/reflectwalk v1.0.2 // indirect
github.com/moby/spdystream v0.2.0 // indirect
github.com/moby/spdystream v0.4.0 // indirect
github.com/moby/term v0.5.0 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect
Expand Down Expand Up @@ -153,6 +154,7 @@ require (
github.com/tklauser/go-sysconf v0.3.12 // indirect
github.com/tklauser/numcpus v0.6.1 // indirect
github.com/ulikunitz/xz v0.5.11 // indirect
github.com/x448/float16 v0.8.4 // indirect
github.com/xeipuuv/gojsonpointer v0.0.0-20190905194746-02993c407bfb // indirect
github.com/xeipuuv/gojsonreference v0.0.0-20180127040603-bd5ef7bd5415 // indirect
github.com/xi2/xz v0.0.0-20171230120015-48954b6210f8 // indirect
Expand All @@ -177,36 +179,38 @@ require (
golang.org/x/crypto v0.26.0 // indirect
golang.org/x/mod v0.20.0 // indirect
golang.org/x/net v0.28.0 // indirect
golang.org/x/oauth2 v0.18.0 // indirect
golang.org/x/oauth2 v0.21.0 // indirect
golang.org/x/sync v0.8.0 // indirect
golang.org/x/sys v0.23.0 // indirect
golang.org/x/term v0.23.0 // indirect
golang.org/x/text v0.17.0 // indirect
golang.org/x/time v0.5.0 // indirect
google.golang.org/api v0.171.0 // indirect
google.golang.org/appengine v1.6.8 // indirect
google.golang.org/genproto v0.0.0-20240213162025-012b6fc9bca9 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20240314234333-6e1732d8331c // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20240314234333-6e1732d8331c // indirect
google.golang.org/grpc v1.62.1 // indirect
google.golang.org/protobuf v1.33.0 // indirect
google.golang.org/protobuf v1.34.2 // indirect
gopkg.in/evanphx/json-patch.v4 v4.12.0 // indirect
gopkg.in/inf.v0 v0.9.1 // indirect
gopkg.in/yaml.v2 v2.4.0 // indirect
helm.sh/helm/v3 v3.15.3 // indirect
helm.sh/helm/v3 v3.15.4 // indirect
howett.net/plist v1.0.0 // indirect
k8s.io/api v0.30.3 // indirect
k8s.io/apiextensions-apiserver v0.30.0 // indirect
k8s.io/apimachinery v0.30.3 // indirect
k8s.io/cli-runtime v0.30.3 // indirect
k8s.io/client-go v0.30.3 // indirect
k8s.io/component-base v0.30.0 // indirect
k8s.io/klog/v2 v2.120.1 // indirect
k8s.io/api v0.31.0 // indirect
k8s.io/apiextensions-apiserver v0.30.3 // indirect
k8s.io/apimachinery v0.31.0 // indirect
k8s.io/cli-runtime v0.31.0 // indirect
k8s.io/client-go v0.31.0 // indirect
k8s.io/component-base v0.30.3 // indirect
k8s.io/klog/v2 v2.130.1 // indirect
k8s.io/kube-openapi v0.0.0-20240228011516-70dd3763d340 // indirect
k8s.io/kubectl v0.30.0 // indirect
k8s.io/utils v0.0.0-20230726121419-3b25d923346b // indirect
k8s.io/kubectl v0.30.3 // indirect
k8s.io/utils v0.0.0-20240711033017-18e509b52bc8 // indirect
sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd // indirect
sigs.k8s.io/kustomize/api v0.13.5-0.20230601165947-6ce0bf390ce3 // indirect
sigs.k8s.io/kustomize/kyaml v0.14.3-0.20230601165947-6ce0bf390ce3 // indirect
sigs.k8s.io/kustomize/api v0.17.2 // indirect
sigs.k8s.io/kustomize/kyaml v0.17.1 // indirect
sigs.k8s.io/structured-merge-diff/v4 v4.4.1 // indirect
sigs.k8s.io/yaml v1.4.0 // indirect
)

replace github.com/elastic/elastic-package => github.com/mrodm/elastic-package v0.53.1-0.20240822111528-e141bcf7b550
Loading