Code

Batched job processing (opt-in)#474

Code

Open

benjie wants to merge 156 commits intomaingraphile/worker:mainfrom pool-centricgraphile/worker:pool-centricCopy head branch name to clipboard

+3,291-591

Commits

Commits on Nov 22, 2024

• Start work on batch fetching jobs

  benjie

  committed
  3b4092b

• Batched job fetching with watermark

  benjie

  committed
  ecaa062

• Fix getJob call to reflect changes in #469

  benjie

  committed
  9b2ba39

• Hoist completeJob and failJob

  benjie

  committed
  484e02c

• Refactor failJob/completeJob in preparation for batching

  benjie

  committed
  862251d

• Stub batch function

  benjie

  committed
  2382ed1

• Implement batching function

  benjie

  committed
  9538247

• Lint fix

  benjie

  committed
  3752cdc

• Tweak graceful/forceful shutdown handover

  benjie

  committed
  113ef71

• Evaluate envvar just once up top

  benjie

  committed
  1821691

• Release batches before releasing worker pool

  benjie

  committed
  c542aa3

• Refactor

  benjie

  committed
  93fb462

• Warn about bad batch settings

  benjie

  committed
  1cab674

• Refactor in preparation for new queue

  benjie

  committed
  fde4341

• Refactor again and explain the purpose of LocalQueue and how it works

  benjie

  committed
  64f6aab

• Implement localQueue

  benjie

  committed
  044aa5a

• Return to POLLING from WAITING; RELEASE mode.

  benjie

  committed
  33faf35

• Fix releasing from POLLING mode

  benjie

  committed
  8ac92e6

• Concurrency 24 works better on my new machine

  benjie

  committed
  d7e381e

• Wait for background tasks to complete before releasing

  benjie

  committed
  1f6a4b0

• Default settings

  benjie

  committed
  6b51f26

• Remove double map

  benjie

  committed
  016ac39

• More efficient implementation

  benjie

  committed
  59f4d99

• Neater API

  benjie

  committed
  52dd450

• Marginally more efficient

  benjie

  committed
  af7a26b

• Need to test more jobs because we're too fast now

  benjie

  committed
  d05c966

• Boolean comparison first

  benjie

  committed
  9da3ed8

• Single job different statement, plus disabled alternative approach

  benjie

  committed
  99e9231

• Listen with new LocalQueue too

  benjie

  committed
  f757f0d

• Lint fixes

  benjie

  committed
  37b5055

• Oops

  benjie

  committed
  222c979

• 0.17.0-canary.6c2c85c

  benjie

  committed
  96b2126

• 0.17.0-canary.67dbcb6

  benjie

  committed
  333c9c2

• [ci] bump

  benjie

  committed
  56ca277

• [ci] bump

  benjie

  committed
  afd7059

• [ci] bump

  benjie

  committed
  25f1fe8

• [ci] bump

  benjie

  committed
  307683e

• [ci] bump

  benjie

  committed
  28a173f

• [ci] bump

  benjie

  committed
  32f8299

• [ci] bump

  benjie

  committed
  a119a8b

• [ci] bump

  benjie

  committed
  2b7bd90

• [ci] bump

  benjie

  committed
  c43bdf9

• Update src/localQueue.ts

  benjie

  committed
  6d1cfa4

• Move localQueue options into their own object

  benjie

  committed
  8868d1f

• Introduce STARTING mode

  benjie

  committed
  9767d48

• Add localQueue refetchDelay feature: be kind on DB when queue is near-empty

  benjie

  committed
  6d19367

• Randomize the abort threshold

  benjie

  committed
  a087440

• Cleaner branching without setting fetchAgain by accident

  benjie

  committed
  e3ec005

• Lint

  benjie

  committed
  d98598d

• Don't trigger refetch once setting mode to released

  benjie

  committed
  8f72ece

• Fix bug and clarify variable name

  benjie

  committed
  098d203

• Comments and variable renames for clarity

  benjie

  committed
  b45b59d

• Clarify and fix behavior of refetch delay

  benjie

  committed
  835502f

• Reduce diff

  benjie

  committed
  abc16f8

• 0.17.0-canary.379fb2e

  benjie

  committed
  187c4e0

• Update performance results

  benjie

  committed
  f044bfe

• Beginnings of tower defence

  benjie

  committed
  da79374

• More waves and don't wait between waves

  benjie

  committed
  4b8462c

• Add some events so we can collect stats from tower defence

  benjie

  committed
  6c7d92d

• Easier to read results

  benjie

  committed
  24d51c1

• Extreme values demonstrate problem

  benjie

  committed
  e295c72

• Produce the problematic behavior I was worried about

  benjie

  committed
  5749bed

• Clarify doc

  benjie

  committed
  993cf44

• Rename abortThreshold to maxAbortThreshold and update docs and implementation

  benjie

  committed
  c0d7e9b

• More waves

  benjie

  committed
  6b9179c

• Shorter waves

  benjie

  committed
  5126d39

• Ive we've already been running jobs slowly, don't sleep so long

  benjie

  committed
  81bc1bb

• Track empty fetches

  benjie

  committed
  1308d84

• Use addJobs API and track latency

  benjie

  committed
  57b411e

• Can't delete DB if workerUtils is connected

  benjie

  committed
  d2e4f12

• Stupid mistake

  benjie

  committed
  daa2a62

• Fix division by zero error

  benjie

  committed
  ee97cde

• Add another wave, and increase local queue size by 1

  benjie

  committed
  f000ad0

• Use JSON rather than constructing tuples; batch at 1M

  benjie

  committed
  b445bf6

• 0.17.0-canary.6aeb577

  benjie

  committed
  1afae2c

• 0.17.0-canary.9817f67

  benjie

  committed
  51c5648

• "Breaking" change

  benjie

  committed
  6a7dd03

• Fix migration tests

  benjie

  committed
  c141856

• Fix all the issues from TypeScript strict mode

  benjie

  committed
  e7bd577

• Move release of batching of fail/complete to terminate

  benjie

  committed
  91f1a5e

• Test that graceful shutdown works in runOnce

  benjie

  committed
  f4b7c4a

• Implement middleware system

  benjie

  committed
  aa43296

• gracefulShutdown middleware

  benjie

  committed
  b746b4a

• Shutting down non-continuous worker should go via gracefulShutdown path

  benjie

  committed
  b57766a

• f

  benjie

  committed
  4e72225

• Add middleware for forcefulShutdown, guarantee event emitter, rework init middleware

  benjie

  committed
  17ccefd

• Clarify relationship between CompiledSharedOptions and WorkerPluginContext and share relevant types

  benjie

  committed
  49bcbee

• Fix lint issues

  benjie

  committed
  8035788

• Only shutdown if we're not already doing so

  benjie

  committed
  ddce61f

• Fix onTerminate handling

  benjie

  committed
  18835b1

• Forceful shutdown should error if something went wrong

  benjie

  committed
  c83069a

• Forceful shutdown should result in promise rejection

  benjie

  committed
  40afb61

• Refactoring to ensure all cases are handled

  benjie

  committed
  cf50392

• We don't release job releasers here any more.

  benjie

  committed
  6afe773

• Ensure that LocalQueue exits with the correct status (e.g. rejects if returning jobs fails)

  benjie

  committed
  4bb7227

• Fix types in a test

  benjie

  committed
  4b6b5dc

• I threw it myself, I know what it is

  benjie

  committed
  9848f8c

• Ensure deactivate happens at most once, and yields same errors if called a second time (e.g. from forcefulShutdown)

  benjie

  committed
  b62b8e3

• Ensure forcefulShutdown and gracefulShutdown yield the same promises when called a second time

  benjie

  committed
  0bdcd9f

• Graceful shutdown should not complete if forceful shutdown has begun

  benjie

  committed
  6bc93fe

• Cleanup gracefulShutdown handover to forcefulShutdown

  benjie

  committed
  b3150ac

• More consistently handle errors in forcefulShutdown

  benjie

  committed
  a561feb

• Move promise handling outside of the gracefulShutdown/forcefulShutdown hooks

  benjie

  committed
  3c3a237

• Only finish once

  benjie

  committed
  28f3b0e

• The _moment_ that deactivate is called, all future getJob calls should immediately return undefined

  benjie

  committed
  7aa5955

• Apply timeouts to forceful shutdown

  benjie

  committed
  174c76c

• forcefulShutdown can be successful; channel errors separately

  benjie

  committed
  afcaaed

• On unexpected worker exit, shut down entire pool (gracefully)

  benjie

  committed
  fdd041c

Commits on Dec 6, 2024

• Upgrade graphile-config, simplify middleware handlers

  benjie

  committed
  84fb704

• New event emitter each time

  benjie

  committed
  88dba0f

• Move to resolvePreset

  benjie

  committed
  fc3f4a5

• Rename completeJob -> completeJobs

  benjie

  committed
  d880d2f

• Rename getJob -> getJobs

  benjie

  committed
  3a6195e

• Rename failJob -> batchFailJobs

  benjie

  committed
  c3946cd

• Rename to batch for consistency

  benjie

  committed
  6b8cbab

Commits on Dec 7, 2024

• Add and correct LocalQueue comments

  benjie

  committed
  e922c13

• Overhaul error handling in returnJobs

  benjie

  committed
  4149e90

• Minor refactoring and more comments

  benjie

  committed
  0c71374

Commits on Dec 9, 2024

• Tidy

  benjie

  committed
  3a0161c

• Add to graceful shutdown errors

  benjie

  committed
  31fc5c1

• Make code clearer

  benjie

  committed
  16fb4cd

• Give 'batch' retry logic with backpressure

  benjie

  committed
  5a75d76

• Reason for internal shutdown trigger

  benjie

  committed
  eb9c69d

• Only trigger shutdown if not already shutting down

  benjie

  committed
  d7693bc

• Trigger gracefulShutdown on forcefulShutdown

  benjie

  committed
  53f4580

• Use correct variable

  benjie

  committed
  6d344f0

• Give users the ability to change how premature worker shutdown is handled

  benjie

  committed
  f43f47f

• Refactor

  benjie

  committed
  9c411dd

• Clarify message

  benjie

  committed
  37ea013

• Give pool error handlers

  benjie

  committed
  a5ff74d

Commits on Dec 10, 2024

• Merge branch 'main' into pool-centric

  benjie

  committed
  6e22aff

• Comments

  benjie

  committed
  01a083f

• Use new retry delay calculator

  benjie

  committed
  0bd0b77

• Ensure errors are escalated to batch() so backpressure applies

  benjie

  committed
  c695bcc

• Refactor

  benjie

  committed
  ad686d9

• Use delays from retryables in batch

  benjie

  committed
  e4545e1

• Handle our own errors via onError

  benjie

  committed
  2b250e9

• Move parallelism into config

  benjie

  committed
  abf2463

• Have tasks sleep for 250ms

  benjie

  committed
  79d2b72

• Centralize configuration of sleepTime

  benjie

  committed
  4905a93

• Add a base sleep duration estimate

  benjie

  committed
  ba43590

• Set threshold to one less than the local queue size

  benjie

  committed
  f6a6889

Commits on Dec 11, 2024

• 0.17.0-canary.f6a6889

  benjie

  committed
  698f522

• All event handlers should provide ctx

  benjie

  committed
  2ae2f2b

• Kill command

  benjie

  committed
  f676c3d

• Aggregate error messages

  benjie

  committed
  676f6e9

• .finally() promises still need error handling

  benjie

  committed
  1d9bb10

• .finally() promises still need error handling (more)

  benjie

  committed
  852b279

• Add migration table data to dump

  benjie

  committed
  2f8d378

• Upgrade to latest graphile-config

  benjie

  committed
  0e88c16

• 0.17.0-canary.0e88c16

  benjie

  committed
  e3fa698

• Move tsconfig to devDependencies

  benjie

  committed
  209dbf6

• Trim grant and stable timestamps

  benjie

  committed
  0103634

• Lint

  benjie

  committed
  6f1217a

• Enable more clash codes

  benjie

  committed
  1fcb2a0

• 0.17.0-canary.1fcb2a0

  benjie

  committed
  c665c80