You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This parallel execution isn't native to meltano and wouldn't be support yet by meltano run natively, but it would at least allow airflow users to leverage parallel subtasks.
The text was updated successfully, but these errors were encountered:
@aaronsteers@tayloramurphy might be nice to try to squeeze into a future iteration. This would be a pretty simple change (weight 2) but should teamed up with #17 (adding some test coverage).
Given a yaml such as:
Today we create one dag with two connected tasks:
But it should be "trivial" to parallelize the subtasks rather than flattening them into a single task:
To do so, we'd need just trigger subtask creation at this check:
files-airflow/bundle/orchestrate/dags/meltano.py
Lines 145 to 147 in ba72d83
This parallel execution isn't native to meltano and wouldn't be support yet by
meltano run
natively, but it would at least allow airflow users to leverage parallel subtasks.The text was updated successfully, but these errors were encountered: