Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Airflow tutorial (#1377 to #1382) #1404

Open
wants to merge 11 commits into
base: master
Choose a base branch
from
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
Expand Up @@ -80,8 +80,8 @@ curl -LfO 'https://airflow.apache.org/docs/apache-airflow/2.3.0/docker-compose.y
- ./dags:/opt/airflow/dags
- ./logs:/opt/airflow/logs
- ./plugins:/opt/airflow/plugins
- ./dbt:/dbt # add this in
- ./dags:/dags # add this in
- ${AIRFLOW_PROJ_DIR:-.}/dbt:/dbt # add this in
- ${AIRFLOW_PROJ_DIR:-.}/dags:/dags # add this in

```

Expand All @@ -90,7 +90,7 @@ curl -LfO 'https://airflow.apache.org/docs/apache-airflow/2.3.0/docker-compose.y
`.env`

```bash
_PIP_ADDITIONAL_REQUIREMENTS=dbt==0.19.0
_PIP_ADDITIONAL_REQUIREMENTS=dbt-snowflake==1.7.3
```

---
Expand Down Expand Up @@ -208,13 +208,15 @@ packages.yml

```yml
packages:
- package: fishtown-analytics/dbt_utils
version: 0.6.4
- package: dbt-labs/dbt_utils
version: 1.1.1
```

dbt_project.yml

```yml
name: example
profile: default
models:
my_new_project:
# Applies to all files under models/example/
Expand All @@ -224,6 +226,7 @@ models:
analysis:
schema: analysis
materialized: view
seed-paths: ["data"]
```

次に、`packages.yml`内に配置した`fishtown-analytics/dbt_utils`をインストールします。これを行うには、`dbt`フォルダから`dbt deps`コマンドを実行します。
Expand Down Expand Up @@ -265,7 +268,7 @@ dbt deps

ここまでで、次のようなフォルダ構造になります。

![airflow](assets/data_engineering_with_apache_airflow_3_dbt_structure.png)
![airflow](assets/data_engineering_with_apache_airflow_3_dbt_structure_fixed.png)

これでdbtの設定は完了です。次のセクションでは、csvファイルとdagsの作成に進みましょう。

Expand Down Expand Up @@ -455,6 +458,7 @@ from airflow.operators.python import PythonOperator, BranchPythonOperator
from airflow.operators.bash import BashOperator
from airflow.operators.dummy_operator import DummyOperator
from datetime import datetime
import os


default_args = {
Expand Down Expand Up @@ -555,4 +559,4 @@ duration:1
### ここまで学んだ内容:

- Airflow、dbt、Snowflakeの設定方法
- DAGを作成し、DAGからdbtを実行する方法
- DAGを作成し、DAGからdbtを実行する方法