Skip to content

Commit

Permalink
feat(queue): add new upsertJobScheduler, getJobSchedulers and removeJ…
Browse files Browse the repository at this point in the history
…obSchedulers methods

Additionally: do not remove delayed jobs that belongs to a job scheduler when using remove job, drain or clean delayed.
  • Loading branch information
manast authored Oct 6, 2024
1 parent 3ed47f5 commit dd6b6b2
Show file tree
Hide file tree
Showing 26 changed files with 2,644 additions and 353 deletions.
226 changes: 113 additions & 113 deletions docs/gitbook/SUMMARY.md
Original file line number Diff line number Diff line change
@@ -1,130 +1,130 @@
# Table of contents

* [What is BullMQ](README.md)
* [Quick Start](<README (1).md>)
* [API Reference](https://api.docs.bullmq.io)
* [Changelogs](changelog.md)
* [v4](changelogs/changelog-v4.md)
* [v3](changelogs/changelog-v3.md)
* [v2](changelogs/changelog-v2.md)
* [v1](changelogs/changelog-v1.md)
- [What is BullMQ](README.md)
- [Quick Start](<README (1).md>)
- [API Reference](https://api.docs.bullmq.io)
- [Changelogs](changelog.md)
- [v4](changelogs/changelog-v4.md)
- [v3](changelogs/changelog-v3.md)
- [v2](changelogs/changelog-v2.md)
- [v1](changelogs/changelog-v1.md)

## Guide

* [Introduction](guide/introduction.md)
* [Connections](guide/connections.md)
* [Queues](guide/queues/README.md)
* [Auto-removal of jobs](guide/queues/auto-removal-of-jobs.md)
* [Adding jobs in bulk](guide/queues/adding-bulks.md)
* [Global Concurrency](guide/queues/global-concurrency.md)
* [Removing Jobs](guide/queues/removing-jobs.md)
* [Workers](guide/workers/README.md)
* [Auto-removal of jobs](guide/workers/auto-removal-of-jobs.md)
* [Concurrency](guide/workers/concurrency.md)
* [Graceful shutdown](guide/workers/graceful-shutdown.md)
* [Stalled Jobs](guide/workers/stalled-jobs.md)
* [Sandboxed processors](guide/workers/sandboxed-processors.md)
* [Pausing queues](guide/workers/pausing-queues.md)
* [Jobs](guide/jobs/README.md)
* [FIFO](guide/jobs/fifo.md)
* [LIFO](guide/jobs/lifo.md)
* [Job Ids](guide/jobs/job-ids.md)
* [Job Data](guide/jobs/job-data.md)
* [Debouncing](guide/jobs/debouncing.md)
* [Delayed](guide/jobs/delayed.md)
* [Repeatable](guide/jobs/repeatable.md)
* [Prioritized](guide/jobs/prioritized.md)
* [Removing jobs](guide/jobs/removing-job.md)
* [Stalled](guide/jobs/stalled.md)
* [Getters](guide/jobs/getters.md)
* [Job Schedulers](guide/job-schedulers/README.md)
* [Repeat Strategies](guide/job-schedulers/repeat-strategies.md)
* [Repeat options](guide/job-schedulers/repeat-options.md)
* [Manage Job Schedulers](guide/job-schedulers/manage-job-schedulers.md)
* [Flows](guide/flows/README.md)
* [Adding flows in bulk](guide/flows/adding-bulks.md)
* [Get Flow Tree](guide/flows/get-flow-tree.md)
* [Fail Parent](guide/flows/fail-parent.md)
* [Remove Dependency](guide/flows/remove-dependency.md)
* [Ignore Dependency](guide/flows/ignore-dependency.md)
* [Remove Child Dependency](guide/flows/remove-child-dependency.md)
* [Metrics](guide/metrics/metrics.md)
* [Rate limiting](guide/rate-limiting.md)
* [Parallelism and Concurrency](guide/parallelism-and-concurrency.md)
* [Retrying failing jobs](guide/retrying-failing-jobs.md)
* [Returning job data](guide/returning-job-data.md)
* [Events](guide/events.md)
* [QueueScheduler](guide/queuescheduler.md)
* [Redis™ Compatibility](guide/redis-tm-compatibility/README.md)
* [Dragonfly](guide/redis-tm-compatibility/dragonfly.md)
* [Redis™ hosting](guide/redis-tm-hosting/README.md)
* [AWS MemoryDB](guide/redis-tm-hosting/aws-memorydb.md)
* [AWS Elasticache](guide/redis-tm-hosting/aws-elasticache.md)
* [Architecture](guide/architecture.md)
* [NestJs](guide/nestjs/README.md)
* [Producers](guide/nestjs/producers.md)
* [Queue Events Listeners](guide/nestjs/queue-events-listeners.md)
* [Going to production](guide/going-to-production.md)
* [Migration to newer versions](guide/migration-to-newer-versions.md)
* [Troubleshooting](guide/troubleshooting.md)
- [Introduction](guide/introduction.md)
- [Connections](guide/connections.md)
- [Queues](guide/queues/README.md)
- [Auto-removal of jobs](guide/queues/auto-removal-of-jobs.md)
- [Adding jobs in bulk](guide/queues/adding-bulks.md)
- [Global Concurrency](guide/queues/global-concurrency.md)
- [Removing Jobs](guide/queues/removing-jobs.md)
- [Workers](guide/workers/README.md)
- [Auto-removal of jobs](guide/workers/auto-removal-of-jobs.md)
- [Concurrency](guide/workers/concurrency.md)
- [Graceful shutdown](guide/workers/graceful-shutdown.md)
- [Stalled Jobs](guide/workers/stalled-jobs.md)
- [Sandboxed processors](guide/workers/sandboxed-processors.md)
- [Pausing queues](guide/workers/pausing-queues.md)
- [Jobs](guide/jobs/README.md)
- [FIFO](guide/jobs/fifo.md)
- [LIFO](guide/jobs/lifo.md)
- [Job Ids](guide/jobs/job-ids.md)
- [Job Data](guide/jobs/job-data.md)
- [Debouncing](guide/jobs/debouncing.md)
- [Delayed](guide/jobs/delayed.md)
- [Repeatable](guide/jobs/repeatable.md)
- [Prioritized](guide/jobs/prioritized.md)
- [Removing jobs](guide/jobs/removing-job.md)
- [Stalled](guide/jobs/stalled.md)
- [Getters](guide/jobs/getters.md)
- [Job Schedulers](guide/job-schedulers/README.md)
- [Repeat Strategies](guide/job-schedulers/repeat-strategies.md)
- [Repeat options](guide/job-schedulers/repeat-options.md)
- [Manage Job Schedulers](guide/job-schedulers/manage-job-schedulers.md)
- [Flows](guide/flows/README.md)
- [Adding flows in bulk](guide/flows/adding-bulks.md)
- [Get Flow Tree](guide/flows/get-flow-tree.md)
- [Fail Parent](guide/flows/fail-parent.md)
- [Remove Dependency](guide/flows/remove-dependency.md)
- [Ignore Dependency](guide/flows/ignore-dependency.md)
- [Remove Child Dependency](guide/flows/remove-child-dependency.md)
- [Metrics](guide/metrics/metrics.md)
- [Rate limiting](guide/rate-limiting.md)
- [Parallelism and Concurrency](guide/parallelism-and-concurrency.md)
- [Retrying failing jobs](guide/retrying-failing-jobs.md)
- [Returning job data](guide/returning-job-data.md)
- [Events](guide/events.md)
- [QueueScheduler](guide/queuescheduler.md)
- [Redis™ Compatibility](guide/redis-tm-compatibility/README.md)
- [Dragonfly](guide/redis-tm-compatibility/dragonfly.md)
- [Redis™ hosting](guide/redis-tm-hosting/README.md)
- [AWS MemoryDB](guide/redis-tm-hosting/aws-memorydb.md)
- [AWS Elasticache](guide/redis-tm-hosting/aws-elasticache.md)
- [Architecture](guide/architecture.md)
- [NestJs](guide/nestjs/README.md)
- [Producers](guide/nestjs/producers.md)
- [Queue Events Listeners](guide/nestjs/queue-events-listeners.md)
- [Going to production](guide/going-to-production.md)
- [Migration to newer versions](guide/migration-to-newer-versions.md)
- [Troubleshooting](guide/troubleshooting.md)

## Patterns

* [Adding jobs in bulk across different queues](patterns/adding-bulks.md)
* [Manually processing jobs](patterns/manually-fetching-jobs.md)
* [Named Processor](patterns/named-processor.md)
* [Flows](patterns/flows.md)
* [Idempotent jobs](patterns/idempotent-jobs.md)
* [Throttle jobs](patterns/throttle-jobs.md)
* [Process Step Jobs](patterns/process-step-jobs.md)
* [Failing fast when Redis is down](patterns/failing-fast-when-redis-is-down.md)
* [Stop retrying jobs](patterns/stop-retrying-jobs.md)
* [Timeout jobs](patterns/timeout-jobs.md)
* [Redis Cluster](patterns/redis-cluster.md)
- [Adding jobs in bulk across different queues](patterns/adding-bulks.md)
- [Manually processing jobs](patterns/manually-fetching-jobs.md)
- [Named Processor](patterns/named-processor.md)
- [Flows](patterns/flows.md)
- [Idempotent jobs](patterns/idempotent-jobs.md)
- [Throttle jobs](patterns/throttle-jobs.md)
- [Process Step Jobs](patterns/process-step-jobs.md)
- [Failing fast when Redis is down](patterns/failing-fast-when-redis-is-down.md)
- [Stop retrying jobs](patterns/stop-retrying-jobs.md)
- [Timeout jobs](patterns/timeout-jobs.md)
- [Redis Cluster](patterns/redis-cluster.md)

## BullMQ Pro

* [Introduction](bullmq-pro/introduction.md)
* [Install](bullmq-pro/install.md)
* [Observables](bullmq-pro/observables/README.md)
* [Cancelation](bullmq-pro/observables/cancelation.md)
* [Groups](bullmq-pro/groups/README.md)
* [Getters](bullmq-pro/groups/getters.md)
* [Rate limiting](bullmq-pro/groups/rate-limiting.md)
* [Concurrency](bullmq-pro/groups/concurrency.md)
* [Local group concurrency](bullmq-pro/groups/local-group-concurrency.md)
* [Max group size](bullmq-pro/groups/max-group-size.md)
* [Pausing groups](bullmq-pro/groups/pausing-groups.md)
* [Prioritized intra-groups](bullmq-pro/groups/prioritized.md)
* [Sandboxes for groups](bullmq-pro/groups/sandboxes-for-groups.md)
* [Batches](bullmq-pro/batches.md)
* [NestJs](bullmq-pro/nestjs/README.md)
* [Producers](bullmq-pro/nestjs/producers.md)
* [Queue Events Listeners](bullmq-pro/nestjs/queue-events-listeners.md)
* [API Reference](https://nestjs.bullmq.pro/)
* [Changelog](bullmq-pro/nestjs/changelog.md)
* [API Reference](https://api.bullmq.pro)
* [Changelog](bullmq-pro/changelog.md)
* [Support](bullmq-pro/support.md)
- [Introduction](bullmq-pro/introduction.md)
- [Install](bullmq-pro/install.md)
- [Observables](bullmq-pro/observables/README.md)
- [Cancelation](bullmq-pro/observables/cancelation.md)
- [Groups](bullmq-pro/groups/README.md)
- [Getters](bullmq-pro/groups/getters.md)
- [Rate limiting](bullmq-pro/groups/rate-limiting.md)
- [Concurrency](bullmq-pro/groups/concurrency.md)
- [Local group concurrency](bullmq-pro/groups/local-group-concurrency.md)
- [Max group size](bullmq-pro/groups/max-group-size.md)
- [Pausing groups](bullmq-pro/groups/pausing-groups.md)
- [Prioritized intra-groups](bullmq-pro/groups/prioritized.md)
- [Sandboxes for groups](bullmq-pro/groups/sandboxes-for-groups.md)
- [Batches](bullmq-pro/batches.md)
- [NestJs](bullmq-pro/nestjs/README.md)
- [Producers](bullmq-pro/nestjs/producers.md)
- [Queue Events Listeners](bullmq-pro/nestjs/queue-events-listeners.md)
- [API Reference](https://nestjs.bullmq.pro/)
- [Changelog](bullmq-pro/nestjs/changelog.md)
- [API Reference](https://api.bullmq.pro)
- [Changelog](bullmq-pro/changelog.md)
- [Support](bullmq-pro/support.md)

## Bull

* [Introduction](bull/introduction.md)
* [Install](bull/install.md)
* [Quick Guide](bull/quick-guide.md)
* [Important Notes](bull/important-notes.md)
* [Reference](https://github.com/OptimalBits/bull/blob/develop/REFERENCE.md)
* [Patterns](bull/patterns/README.md)
* [Persistent connections](bull/patterns/persistent-connections.md)
* [Message queue](bull/patterns/message-queue.md)
* [Returning Job Completions](bull/patterns/returning-job-completions.md)
* [Reusing Redis Connections](bull/patterns/reusing-redis-connections.md)
* [Redis cluster](bull/patterns/redis-cluster.md)
* [Custom backoff strategy](bull/patterns/custom-backoff-strategy.md)
* [Debugging](bull/patterns/debugging.md)
* [Manually fetching jobs](bull/patterns/manually-fetching-jobs.md)
- [Introduction](bull/introduction.md)
- [Install](bull/install.md)
- [Quick Guide](bull/quick-guide.md)
- [Important Notes](bull/important-notes.md)
- [Reference](https://github.com/OptimalBits/bull/blob/develop/REFERENCE.md)
- [Patterns](bull/patterns/README.md)
- [Persistent connections](bull/patterns/persistent-connections.md)
- [Message queue](bull/patterns/message-queue.md)
- [Returning Job Completions](bull/patterns/returning-job-completions.md)
- [Reusing Redis Connections](bull/patterns/reusing-redis-connections.md)
- [Redis cluster](bull/patterns/redis-cluster.md)
- [Custom backoff strategy](bull/patterns/custom-backoff-strategy.md)
- [Debugging](bull/patterns/debugging.md)
- [Manually fetching jobs](bull/patterns/manually-fetching-jobs.md)

## Python

* [Introduction](python/introduction.md)
* [Changelog](python/changelog.md)
- [Introduction](python/introduction.md)
- [Changelog](python/changelog.md)
24 changes: 13 additions & 11 deletions docs/gitbook/guide/job-schedulers/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,16 +12,18 @@ To create a scheduler, simply use the "upsertJobScheduler" method as demonstrate

```typescript
// Creates a new Job Scheduler that generates a job every 1000 milliseconds (1 second)
const firstJob = await queue.upsertJobScheduler("my-scheduler-id", { every: 1000 });
const firstJob = await queue.upsertJobScheduler('my-scheduler-id', {
every: 1000,
});
```

This example will create a new Job Scheduler that will produce a new job every second. It will also return the first job created for this Job Scheduler, which will be in "delayed" status waiting to be processed after 1 second.

Now there are also a few important considerations that need to be explained here.:

* **Upsert vs. Add:** the 'upsert' is used instead of 'add' to simplify management of recurring jobs, especially in production deployments. It ensures the scheduler is updated or created without duplications.
* **Job Production Rate:** The scheduler will only generate new jobs when the last job begins processing. Therefore, if your queue is very busy, or if you do not have enough workers or concurrency, it is possible that you will get the jobs less frequently than the specified repetition interval.
* **Job Status:** As long as a Job Scheduler is producing jobs, there will be always one job associated to the scheduler in the "Delayed" status.
- **Upsert vs. Add:** the 'upsert' is used instead of 'add' to simplify management of recurring jobs, especially in production deployments. It ensures the scheduler is updated or created without duplications.
- **Job Production Rate:** The scheduler will only generate new jobs when the last job begins processing. Therefore, if your queue is very busy, or if you do not have enough workers or concurrency, it is possible that you will get the jobs less frequently than the specified repetition interval.
- **Job Status:** As long as a Job Scheduler is producing jobs, there will be always one job associated to the scheduler in the "Delayed" status.

### Using Job Templates

Expand All @@ -30,21 +32,21 @@ You can also define a template with standard names, data, and options for jobs a
```typescript
// Create jobs every day at 3:15 (am)
const firstJob = await queue.upsertJobScheduler(
"my-scheduler-id",
'my-scheduler-id',
{ pattern: '0 15 3 * * *' },
{
name: "my-job-name",
data: { foo: "bar" },
name: 'my-job-name',
data: { foo: 'bar' },
opts: {
backoff: 3,
attempts: 5,
removeOnFail: 1000
removeOnFail: 1000,
},
});
},
);
```

All jobs produced by this scheduler will use the given settings. Note that in the future you could call "upsertJobScheduler" again with the given "my-scheduler-id" in order to update any settings of this particular job scheduler, such as the repeat options or/and the job's template settings.
All jobs produced by this scheduler will use the given settings. Note that in the future you could call "upsertJobScheduler" again with the given "my-scheduler-id" in order to update any settings of this particular job scheduler, such as the repeat options or/and the job's template settings.

{% hint style="info" %}
Since jobs produced by the Job Scheduler will get a special job ID in order to guarantee that jobs will never be created more often than the given repeat settings, you cannot choose a custom job id. However you can use the job's name if you need to discriminate these jobs from other jobs.
Expand Down
5 changes: 3 additions & 2 deletions docs/gitbook/guide/job-schedulers/manage-job-schedulers.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,9 @@ The removeJobScheduler method is designed to delete a specific job scheduler fro
```typescript
// Remove a job scheduler with ID 'scheduler-123'
const result = await queue.removeJobScheduler('scheduler-123');
console.log(result ? 'Scheduler removed successfully' : 'Failed to remove scheduler');
console.log(
result ? 'Scheduler removed successfully' : 'Failed to remove scheduler',
);
```

#### Get Job Schedulers
Expand All @@ -23,4 +25,3 @@ console.log('Current job schedulers:', schedulers);
```

This method can be particularly useful for generating reports or dashboards that provide insights into when jobs are scheduled to run, aiding in system monitoring and troubleshooting.

Loading

0 comments on commit dd6b6b2

Please sign in to comment.