[bitnami/airflow] Add libraries libpq-dev python-dev gcc to support installation psycopg2 as dependency

[ghost]

Description of the change

The change introduces three new packages which will be installed in airflow pods to support installation of psycopg2 library from requirements.txt file.
Currently if we add psycopg2 library to requirements.txt we will get an error:
Error: pg_config executable not found.

Benefits

Possibility to use bitnami images and helm chart to install psycopg2 and other libraries which depend on it

Applicable issues

• fixes #48237
• fixes #19242

[fevisera]

Hi @oktayalizada,

I have built the image locally, and it appears to be quite large:

local/airflow                   latest                        5bc6e03c8bf9   14 minutes ago   3.28GB 
bitnami/airflow                 latest                        8255c5199bb5   3 days ago       1.77GB

I believe the intention in bitnami/charts#19242 was to create a fork rather than modify the bitnami airflow images. Adding these additional packages would need additional analysis, including an assessment of image size and the need for these packages. For instance, adding python-dev may include more packages than necessary.

Please let me know if there are any additional details I may be overlooking.

[ghost]

Hi @fevisera,
So you mean airflow will not support installation and usage of psycopg2 library in airflow pods? Can we have maybe a different tag for this specific build (e.g. debian-11-psycopg2)?

[fevisera]

Hi @oktayalizada,

Thank you for your question! I did not mean to imply that Airflow will not support the installation and usage of the psycopg2 library for bitnami/airflow. What I intended to convey is that before adding those new packages, it is essential to conduct a more thorough analysis to assess the image size impact and the actual necessity of these packages.

We already have an ongoing internal task to address adding dependencies feature, which has been created for #48258. As soon as this functionality becomes available, we will notify you here.

Thank you for proposing this change and bringing it to our attention. If you have any more questions or concerns, please feel free to ask. We appreciate your feedback.

[alekseyolg]

Any opinion?
If necessary, I can open a new pull request to add the functionality needed in the work, taking into account the changes that have appeared since the creation of this pull request.
Thank you!

[juan131]

Hi everyone!

These deps the PR is proposing to add (such as gcc) are required for compilation and they should never be included in the runtime image. We can use these libraries for compiling internally the different packages included in the container but they're useless at runtime.

IMHO this PR should be closed and an issue requesting psycopg2 should be created instead. That said, we are already including the psycopg2-binary pip package in the container image, see:

$ docker run --rm -it bitnami/airflow bash
(...)
airflow@8275af36d3d5:/$ . /opt/bitnami/airflow/venv/bin/activate
(venv) airflow@8275af36d3d5:/$ pip list | grep psycopg2
psycopg2-binary                           2.9.10

[github-actions]

This Pull Request has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thank you for your contribution.

[github-actions]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Pull Request. Do not hesitate to reopen it later if necessary.