Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

change CKE to proceed rebooting immediately after draining of node is completed #707

Merged
merged 6 commits into from
May 7, 2024

Conversation

YZ775
Copy link
Contributor

@YZ775 YZ775 commented Mar 11, 2024

In the current CKE, even if max_concurrent_reboots is larger than 1, draining is conducted by serial and rebooting is conducted after draining of all drainable node is finished.
This behavior cause long down time of first drained node.

This PR changes the CKE to move rebooting immediately after draining of each node is finished.
In addition, this PR fixes behavior of canceled reboot queue entry.

@YZ775 YZ775 force-pushed the change-drain-behavior branch from d7f34a3 to 4f48336 Compare April 10, 2024 01:38
@YZ775 YZ775 marked this pull request as ready for review April 10, 2024 01:43
@YZ775 YZ775 requested a review from masa213f April 10, 2024 02:06
@YZ775 YZ775 force-pushed the change-drain-behavior branch from 8762f40 to f27329c Compare April 19, 2024 01:13
@YZ775 YZ775 force-pushed the change-drain-behavior branch from 08d1831 to 7eab07c Compare April 23, 2024 06:28
Comment on lines 916 to 918
if len(ops) > 0 {
phaseReboot = true
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for fixing phaseReboot!

I checked this variable carefully, and I was convinced that this function need not return phaseReboot.
This function should only determine ops from the given information.

The caller(DecideOps()) should determine the phase. And which can be determined only by the returned ops.

Could you change the return value of this function and change the caller as following?
It's okay to change it in another PR.

https://github.com/cybozu-go/cke/blob/v1.28.0/server/strategy.go#L106-L113

	// 11. Reboot nodes if reboot request has been arrived to the reboot queue, and the number of unreachable nodes is less than a threshold.
	if ops := rebootOps(c, constraints, rebootArgs, nf); len(ops) > 0 {
		if !nf.EtcdIsGood() {
			log.Warn("cannot reboot nodes because etcd cluster is not responding and in-sync", nil)
			return nil, cke.PhaseRebootNodes
		}
		return ops, cke.PhaseRebootNodes
	}

@YZ775 YZ775 requested a review from masa213f April 30, 2024 01:49
Copy link
Contributor

@masa213f masa213f left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@YZ775 YZ775 merged commit 9f43e0b into main May 7, 2024
7 checks passed
@YZ775 YZ775 deleted the change-drain-behavior branch May 7, 2024 00:14
@YZ775 YZ775 mentioned this pull request May 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants