Add worker node to run test operator on uni04delta-ipv6 deployments #403

gmarcian · 2024-09-18T18:50:23Z

In order to run disruptive tests on the uni04delta env, We need to have a worker node dedicated for the test operator.
Thus we can ensure that test pods won't be affected by the failures.

Running disruptive tests is part of HA testing of the system. Please see OSPRH-7602

openshift-ci · 2024-09-18T18:50:34Z

Hi @gmarcian. Thanks for your PR.

I'm waiting for a openstack-k8s-operators member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

softwarefactory-project-zuul · 2024-09-18T18:56:08Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/605f01ec091f444fb44e5598bc3cf611

✔️ noop SUCCESS in 0s
✔️ rhoso-architecture-validate-uni04delta SUCCESS in 5m 02s
❌ rhoso-architecture-validate-uni04delta-ipv6 FAILURE in 4m 04s

softwarefactory-project-zuul · 2024-09-18T19:08:32Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/77b345f8b27c4b94894fb0ec2736c3a9

✔️ noop SUCCESS in 0s
✔️ rhoso-architecture-validate-uni04delta SUCCESS in 3m 45s
❌ rhoso-architecture-validate-uni04delta-ipv6 FAILURE in 4m 17s

softwarefactory-project-zuul · 2024-09-22T20:40:00Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/cb8f76a23e4040518b98dade07c4f325

✔️ noop SUCCESS in 0s
✔️ rhoso-architecture-validate-uni04delta SUCCESS in 3m 54s
❌ rhoso-architecture-validate-uni04delta-ipv6 FAILURE in 4m 18s

fultonj · 2024-09-23T12:49:32Z

rhoso-architecture-validate-uni04delta-ipv6 fails here

Sunday 22 September 2024  16:39:33 -0400 (0:00:00.050)       0:00:29.477 ****** 
fatal: [localhost]: FAILED! => changed=true 
  cmd:
  - oc
  - kustomize
  delta: '0:00:00.395164'
  end: '2024-09-22 16:39:34.530784'
  msg: non-zero return code
  rc: 1
  start: '2024-09-22 16:39:34.135620'
  stderr: 'error: fieldPath `data.node_3.internalapi_ip` is missing for replacement source ConfigMap.[noVer].[noGrp]/network-values.[noNs]'
  stderr_lines: <omitted>
  stdout: ''
  stdout_lines: <omitted>

https://softwarefactory-project.io/zuul/t/rdoproject.org/build/26179debe63644d6bff1fc3873e5d9dd

abays · 2024-10-21T09:15:38Z

Is this still a draft? Has it been tested upstream and downstream?

gmarcian · 2024-11-19T12:47:48Z

Tested upstream: https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/build/4702eb82b51945868ad8bbbbbd107dd8/logs

fultonj · 2024-11-19T16:03:36Z

https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/build/4702eb82b51945868ad8bbbbbd107dd8/logs

I see that test failed. Shouldn't it pass before we merge it?

fultonj · 2024-11-19T16:35:54Z

So all uni04delta-ipv6 deployments will be able to run with four OCP nodes; i.e. 3 masters and 1 new worker node introduced by this patch?

I see it's following the pattern used here:

426b290

@arxcruz I'm assuming our testing nodes have the resources to handle the exta node.

@tosky I'm assuming this won't cause problems for storage related tests in this DT.

I'm OK with it provided we have the resources to handle it and others who depend on uni04delta-ipv6 are OK with it.

abays · 2024-11-19T17:28:25Z

https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/build/4702eb82b51945868ad8bbbbbd107dd8/logs

I see that test failed. Shouldn't it pass before we merge it?

I also echo John's question.

gmarcian · 2024-11-19T19:23:46Z

@abays , @fultonj, a lot of tobiko tests failed, but IMO these failures are unrelated to this change. Part of them relate to some tobiko issue, another failures are since vms, created by these tests, are unreachable via their floating IPs, which is because of ipv6/ipv4 misconfiguration.
To be more confident about the patch, I will re-trigger the job with tempest tests, which are expected to work the same way as on the original uni04delta-ipv6 job.

gmarcian · 2024-11-21T03:52:54Z

So, it seems to be ready now.
A testproject job[1] passed the deployment;
Two Tempest tests that failed there[2], seem also to fail on the official uni04delta-ipv6 [3].
Tobiko ipv4/ipv6 connectivity issue was resolved in [4]
Remaining failed tests are related to [5] - need to be fixed in tobiko framework.
[1] https://gitlab.cee.redhat.com/ci-framework/ci-framework-testproject/-/merge_requests/698
[2] https://sf.hosted.upshift.rdu2.redhat.com/logs/98/698/25ca684db05f6156f2e656a99a1b273d47e63621/check-gitlab-cee/uni04delta-ipv6-rhel9-rhoso18.0-pidone-titan151/17f6a07/logs/controller-0/ci-framework-data/tests/test_operator/tempest-tests-multi-thread-testing-workflow-step-0/stestr_results.html

[3] https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-uni-update-jobs-periodic-integration-rhoso18.0-rhel9/gitlab.cee.redhat.com/ci-framework/ci-framework-jobs/main/uni04delta-ipv6-rhel9-rhoso18.0-update/2dc3eed/logs/controller-0/ci-framework-data/tests/test_operator/post-deployment-tempest-multi-thread-testing-workflow-step-0/stestr_results.html

[4] https://gitlab.cee.redhat.com/ci-framework/ci-framework-jobs/-/merge_requests/1187
[5] https://issues.redhat.com/browse/TOBIKO-126

abays

Given @gmarcian's recent update, I am good with this. But I will defer to @fultonj to have the final word for approval.

/lgtm

fultonj

/approve
/lgtm

openshift-ci · 2024-11-21T12:55:47Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: fultonj, gmarcian

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [fultonj]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

softwarefactory-project-zuul · 2024-11-21T13:01:43Z

Build succeeded (gate pipeline).
https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/36de8a683e4a4171be67efd1c141f0c4

✔️ noop SUCCESS in 0s
✔️ rhoso-architecture-validate-uni04delta SUCCESS in 5m 10s
✔️ rhoso-architecture-validate-uni04delta-ipv6 SUCCESS in 5m 06s

…er_unidelta_ipv6 Add worker node to run test operator on uni04delta-ipv6 deployments In order to run disruptive tests on the uni04delta env, We need to have a worker node dedicated for the test operator. Thus we can ensure that test pods won't be affected by the failures. Running disruptive tests is part of HA testing of the system. Please see OSPRH-7602 Reviewed-by: Andrew Bays <[email protected]> Reviewed-by: John Fulton <[email protected]>

Cherry-pick pull request #403 from gmarcian/add_worker_unidelta_ipv6 Add worker node to run test operator on uni04delta-ipv6 deployments In order to run disruptive tests on the uni04delta env, We need to have a worker node dedicated for the test operator. Thus we can ensure that test pods won't be affected by the failures. Running disruptive tests is part of HA testing of the system. Please see OSPRH-7602 Reviewed-by: Andrew Bays [email protected] Reviewed-by: John Fulton [email protected] Reviewed-by: Andrew Bays <[email protected]>

tosky · 2024-12-11T09:20:09Z

Please please next time squash all the changes, otherwise the git history becomes complicated to follow. This is a request to the reviewers too.

abays · 2024-12-11T09:39:03Z

Please please next time squash all the changes, otherwise the git history becomes complicated to follow. This is a request to the reviewers too.

Do you mean have Zuul squash everything when it merges, or ask the author to squash their commits?

tosky · 2024-12-11T10:05:58Z

Please please next time squash all the changes, otherwise the git history becomes complicated to follow. This is a request to the reviewers too.

Do you mean have Zuul squash everything when it merges, or ask the author to squash their commits?

I would be more for "have the author present a clean history", as there may be cases where the original commits need to be preserved. But that's up to you.

gmarcian · 2024-12-11T13:53:24Z

@tosky Ack. Can it be done from the github PR GUI?

tosky · 2024-12-11T14:48:33Z

@tosky Ack. Can it be done from the github PR GUI?

I'll leave the answer to others who know the github GUI, but I would say: no. Better rebase locally, you have more control (the operation is a local rebase of the history): https://about.gitlab.com/blog/2020/11/23/keep-git-history-clean-with-interactive-rebase/ (yes, the article is from gitlab.com, but it describes the step to execute with the git CLI tool).

openshift-ci bot requested review from karelyatin and leifmadsen September 18, 2024 18:50

openshift-ci bot added the needs-ok-to-test label Sep 18, 2024

gmarcian changed the title ~~[Draft] Add worke node to run test operator on uni04delta~~ [Draft] Add worker node to run test operator on uni04delta Sep 22, 2024

gmarcian force-pushed the add_worker_unidelta_ipv6 branch from 7fd9d21 to 04d698a Compare September 22, 2024 20:35

fultonj added the do-not-merge/work-in-progress label Oct 21, 2024

gmarcian force-pushed the add_worker_unidelta_ipv6 branch 2 times, most recently from 918e4d2 to 77ff71b Compare November 5, 2024 13:22

Julia Marciano added 4 commits November 14, 2024 04:37

[Draft] Add worke node to run test operator on uni04delta

f8d1259

Adjust iAd idddd AMove all changes under examples/dt/

dd08029

Add worker node to automation/net-env/uni04delta-ipv6.yaml

de4860a

Distinguish between 'name\ and 'node_name' when configure node-3 nncp

52f9c6c

gmarcian force-pushed the add_worker_unidelta_ipv6 branch from 77ff71b to 52f9c6c Compare November 14, 2024 02:38

gmarcian changed the title ~~[Draft] Add worker node to run test operator on uni04delta~~ Add worker node to run test operator on uni04delta-ipv6 deployments Nov 19, 2024

openshift-ci bot removed the do-not-merge/work-in-progress label Nov 19, 2024

Merge branch 'main' into add_worker_unidelta_ipv6

88e97a0

abays reviewed Nov 21, 2024

View reviewed changes

openshift-ci bot assigned abays Nov 21, 2024

openshift-ci bot added the lgtm label Nov 21, 2024

fultonj approved these changes Nov 21, 2024

View reviewed changes

openshift-ci bot assigned fultonj Nov 21, 2024

openshift-ci bot added the approved label Nov 21, 2024

softwarefactory-project-zuul bot merged commit 1100a50 into openstack-k8s-operators:main Nov 21, 2024
7 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add worker node to run test operator on uni04delta-ipv6 deployments #403

Add worker node to run test operator on uni04delta-ipv6 deployments #403

gmarcian commented Sep 18, 2024

openshift-ci bot commented Sep 18, 2024

softwarefactory-project-zuul bot commented Sep 18, 2024

softwarefactory-project-zuul bot commented Sep 18, 2024

softwarefactory-project-zuul bot commented Sep 22, 2024

fultonj commented Sep 23, 2024 •

edited

Loading

abays commented Oct 21, 2024

gmarcian commented Nov 19, 2024

fultonj commented Nov 19, 2024

fultonj commented Nov 19, 2024

abays commented Nov 19, 2024

gmarcian commented Nov 19, 2024

gmarcian commented Nov 21, 2024 •

edited

Loading

abays left a comment

fultonj left a comment

openshift-ci bot commented Nov 21, 2024

softwarefactory-project-zuul bot commented Nov 21, 2024

tosky commented Dec 11, 2024

abays commented Dec 11, 2024

tosky commented Dec 11, 2024

gmarcian commented Dec 11, 2024

tosky commented Dec 11, 2024

Add worker node to run test operator on uni04delta-ipv6 deployments #403

Add worker node to run test operator on uni04delta-ipv6 deployments #403

Conversation

gmarcian commented Sep 18, 2024

openshift-ci bot commented Sep 18, 2024

softwarefactory-project-zuul bot commented Sep 18, 2024

softwarefactory-project-zuul bot commented Sep 18, 2024

softwarefactory-project-zuul bot commented Sep 22, 2024

fultonj commented Sep 23, 2024 • edited Loading

abays commented Oct 21, 2024

gmarcian commented Nov 19, 2024

fultonj commented Nov 19, 2024

fultonj commented Nov 19, 2024

abays commented Nov 19, 2024

gmarcian commented Nov 19, 2024

gmarcian commented Nov 21, 2024 • edited Loading

abays left a comment

Choose a reason for hiding this comment

fultonj left a comment

Choose a reason for hiding this comment

openshift-ci bot commented Nov 21, 2024

softwarefactory-project-zuul bot commented Nov 21, 2024

tosky commented Dec 11, 2024

abays commented Dec 11, 2024

tosky commented Dec 11, 2024

gmarcian commented Dec 11, 2024

tosky commented Dec 11, 2024

fultonj commented Sep 23, 2024 •

edited

Loading

gmarcian commented Nov 21, 2024 •

edited

Loading