-
Notifications
You must be signed in to change notification settings - Fork 15
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Jobs that timeout will never be able to run again #2
Comments
Upon further reading I'm less clear on how DEADLINE_SOON is meant to operate 😕 |
DEADLINE_SOON is sent to a client that is in a blocking The issue with racing beanstalkd to bury a task is that you miss out on the timed-out metadata. It's simply buried, if you beat the server to it. Perhaps it's just the fact that the job is buried on timeout? What should actually happen to timed-out jobs? If we just kick them at the minute then perhaps we should change the behaviour to Either way, seems like we haven't got it 100% right. Thoughts @pda @rbone? |
A longer backoff sounds like a reasonable change for the moment. It is tricky however, as some tasks may merit more aggressive burying strategies while others may be safe to retry very frequently. I'd say a longer backoff makes sense as a default, but it might be nice in the future to make this behaviour configurable, possibly even on a per tube basis. |
Should the backoff be proportional to the TTR? |
Honestly I can't make up my mind on what the default behaviour should be, so it probably doesn't matter too much what way you go. A proportional TTR should be fine. I think having it be configurable per tube will become pretty important however. |
I think a simple function of the try count PR #4 proposes 3 tries with |
When a job overruns it's TTR, beanstalkd will increment the job's timeout stat and put it back on the work queue for another worker to reserve.
In an effort to prevent pathological jobs from dog-piling all available workers,
cmdstalk
will bury a task it reserves that has timeouts greater than 1. This means that once a task is buried because of a timeout, it will always re-bury instantly each time it is kicked: the job becomes un-runnable.Using just the
buried
,kicked
andtimeout
counters, there does not appear to be a way to differentiate between "kicks due buries due to timeouts" in the way that would allowcmdstalk
to bury a job the next time it is reserved after a timeout.The
beanstalkd
protocol docs make mention of a one second grace period at the end of a reserve time - would it be possible to use this grace period to bury a timed out job in the "same run" as the timeout occurred?The text was updated successfully, but these errors were encountered: