Airflow installation recommendation is incorrect

[dataders]

This repo should not rely upon Airflow's auto-generated contraint files. As maintainers of dbt, we have received several issues on our repo requesting that our dependencies be less strict. Come to find out, these users are encountering errors because they are relying on this repo's Dockerfile which provides ZERO dependency flexibility, because all dependencies are hard pinned to the patch version.

See these issues

• [ADAP-692] [Bug] snowflake-connector-python dependency version too strict dbt-labs/dbt-snowflake#687
• [ADAP-1087] [Regression] Loosen redshift-connector pin dbt-labs/dbt-redshift#700
• Installing 1.7.3 on Airflow 2.9 dbt-labs/dbt-snowflake#968

Airflow's guidance on the constraints file, apache/airflow/blob/main/constraints/README.md, says the following

For development use only, you can store your own version of constraint files used during CI image build here and refer to them during building of the image using --airflow-constraints location constraints/constraints.txt This allows you to iterate on dependencies without having to run --upgrade-to-newer-dependencies flag continuously.

Correct me if I'm wrong, but AWS MWAA local runner does NOT constitute "development use only"

See my response last year at dbt-labs/dbt-snowflake#687 (comment) for more detail

[mayushko26]

Correct me if I'm wrong, but AWS MWAA local runner does NOT constitute "development use only"

I would argue the contrary. The local runner is intended as a quick mechanism to test a user's Airflow setup before launching an MWAA environment. It is not intended to run any production workflows

That said, customizing the constraints is possible, by changing the local constraints.txt file, and referencing it by its path on the docker image. The same is true of MWAA environments; users can upload their own constraints file to S3 if their use-case is such that the official constraints published by Airflow are not suitable.

[dataders]

willing to concede that MWAA local runner can be development.

My issue here is really asking y'all for help. We've had three instances so far (e.g. dbt-labs/dbt-redshift#700) of users of this package's Dockerfile asking that dbt be more flexible, when they're using Airflow constraint files.

Perhaps this resolvable with documentation either in this repo or I can just quote your above guidance the next time someone opens a similar issue?

customizing the constraints is possible, by changing the local constraints.txt file, and referencing it by its path on the docker image. The same is true of MWAA environments; users can upload their own constraints file to S3 if their use-case is such that the official constraints published by Airflow are not suitable.

[mayushko26]

I will bring this up to the broader MWAA team to see what improvements we can make on our end. In the mean time, feel free to quote the guidance when an issue is raised in dbt.

[mayushko26]

I took a look at the dbt issues you linked here, and none of them are specific to MWAA. MWAA is a managed service for the open-source Apache Airflow software. The local runner, in turn, is a lightweight estimation of an actual MWAA environment, used for local testing of environment configurations.

Most people use the default constraints provided by Airflow because those are the set of constraints against which Airflow has tested the stability of their releases, and is consequently generally a good practice. That said MWAA, other managed services for Airflow, and self-managed Airflow all have mechanisms to supply your own constraints file instead of the (default) published constraints.

This is pointed out in Airflow's documentation here: https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html#using-your-own-constraints.