Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Send publishes a new batch of events, but the thread hangs indefinitely when the load balancer (Envoy) distributes requests to a closed server. #40

Open
tuanhnguyen888 opened this issue Dec 8, 2024 · 0 comments

Comments

@tuanhnguyen888
Copy link

 // Send publishes a new batch of events by JSON-encoding given batch.
// Send blocks until the complete batch has been ACKed by lumberjack server or
// some error happened.
func (c *SyncClient) Send(data []interface{}) (int, error) {
	if err := c.cl.Send(data); err != nil {
		return 0, err
	}

	seq, err := c.cl.AwaitACK(uint32(len(data)))
	return int(seq), err
}

My team's project uses this function to send events to the server for centralized storage (the model uses Envoy v1.24 for request distribution).
When a request routed by Envoy is sent to a closed server, the thread hangs in the loop waiting for enough ACKs to be received.
tcpdump analysis reveals that Envoy continuously sends 6-byte data packets to the client. But the ReceiveACK function always returns ackSeq equal to 0 without incrementing, leading to infinite looping and never reaching the Count value.

// AwaitACK waits for count elements being ACKed. Returns last known ACK on error.
func (c *Client) AwaitACK(count uint32) (uint32, error) {
	var ackSeq uint32
	var err error

	// read until all ACKs
	for ackSeq < count {
		ackSeq, err = c.ReceiveACK()
		if err != nil {
			return ackSeq, err
		}
	}

	if ackSeq > count {
		return count, fmt.Errorf(
			"invalid sequence number received (seq=%v, expected=%v)", ackSeq, count)
	}
	return ackSeq, nil
}

In my opinion, a timeout mechanism must be implemented in this case to ensure that the loop does not run indefinitely if it fails to receive enough ACKs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant