Add support for custom celery configs

[arorasachin9]

closes: #45037

Description:
Currently Airflow support limited celery options only. This PR adds the support for the additional celery config for celery workers.

1. Changes are completely backward compatible
2. No test cases found for these changes
3. If the config is not available then default value of {} is taken which will be same as earlier only
4. If config is available then it will be added to the celery config and applied to celery workers.

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

• Pay attention to the quality of your code (ruff, mypy and type annotations). Our pre-commits will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
• Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: [email protected]
  Slack: https://s.apache.org/airflow-slack

[jscheffl]

Thanks for the PR - the contribution looks really small and is efficient. Thanks for taking care for the backwards compatibility.

I assume the fallback handling in default_celery.py:72is not needed if the config entry is correctly registered as a default in providers/src/airflow/providers/celery/provider.yaml - There all configs for the provider should be registered. This also would add missing documentation.

[jedcunningham]

Can you also add a test for this?

[arorasachin9]

Addressed the comments. CI is failing not able to debug why CI pipeline is failing.

[jscheffl]

Except some minor comment I think it is good, let me check what is broken on CI. Nothing related to your code.
Ignore that it is "red" atm, will be better once you push the next change on the PR.

Then I think... LGTM!

[jscheffl]

Somthing is wrong with this PR and CI - I don't see the reason but I am very sure it is not related to the changes. I am on it...

[arorasachin9]

@jscheffl I have fixed the unit test. The CI is passing. and Thanks for the help in dubuging the CI issue.

[jscheffl]

LGTM!

THANKS for the patience, fixes getting this green!

[boring-cyborg]

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.