Add ability to disable job scheduling of specified extension plugin

[dbbaughe]

From: opendistro-for-elasticsearch/job-scheduler#3

JobScheduler plugin maybe extended by multiple extension plugins. Extension plugins may support the configuration to allow users to disable the plugin at runtime, which requires all scheduled jobs of that plugin to be descheduled.

Let extension plugin to disable all jobs by updating all job documents by setting enabled flag to false is an option, but it is not ideal, as the enabled field is meant for extension plugins to control at job level to support their use cases.

Thus we may want to implement a new mechanism for extension plugins to disable all their jobs at one shot.

Comments:

From @zengyan-amazon
One potential option is to add a extension plugin blacklist setting for JobScheduler. If extension plugin wants to disable all their jobs, they can add itself to the blacklist. JobScheduler listens to this setting change and disable the jobs accordingly

---

From @sixdouglas
Hi,
Do you think I can involve for this improvement? If so, do you have more inputs/thoughts since the beginning of May?

---

From @zengyan-amazon
Hi @sixdouglas ,

Your involvement is welcomed. We didn't have a chance to finalized the approach we want to adopt since May due to limited bandwidth.

I believe the high level idea is allow extension plugin to update a setting to notify the JobScheduler to disable/re-enable its jobs. Then job scheduler can take care of the deschedule/schedule accordingly. However we are not sure whether is will be a cluster setting, or an index setting of extension plugins' job index, or some other type of setting.

Please let us know if you have any suggestions, and we would be happy to discuss any possibilities and move this forward.

Thanks

[qreshi]

Adding a note here that if this enhancement ends up adding a condition check where the sweeper is enabled or disabled for jobs (similar to what Alerting currently does) we could end up hitting a bug that Alerting currently has where disable jobs don't get rescheduled when re-enabled if the job doc version didn't change because of conditions sweeper() checks for (the related Alerting issue is: opensearch-project/alerting#89).

So whatever fix is applied to Alerting should be added to Job Scheduler as well to avoid that situation.