Are we thinking about orchestrators the wrong way?

[astrojuanlu]

Let's talk about Airflow. How do people use it in the wild?

exhibit 1:

airflow is for orchestration, never use it to process data. 99% of the people I've talked to whose Airflow cluster is mess are using it like a data processing platform.. troubleshooting performance issues is a total nightmare.

exhibit 2:

Surprised to see so many answer with Airflow. Airflow is not an ETL tool, it is a scheduling tool.

exhibit 3

I would prefer to isolate these two environments so that I don't need to import Kedro in Airflow or Airflow in Kedro. As managing dependencies in Airflow is challenging, it would be better to avoid this problem altogether.
[...]
Thirdly, as the code would be executed by the same processes, it would need to be shared in the form of packages. In this setup Airflow runs in a docker image, so then I'd have to either re-build and re-run this image every time either the Airflow or Kedro project code changes, OR additionally manage lots of virtual python environments somewhere and ship the new versions of the micro-packaged Kedro pipelines there whenever the code changes.

📜

To unpack all this or try to explain it in my own words, if you run the Airflow control plane in the same process/machine as your data processing code, you risk killing Airflow because of a task gone rogue. This is more or less obvious in hindsight, but I've seen products make this mistake and it was extremely painful.

So then people realise that Airflow (or your orchestrator of choice) actually needs to live separately from the tasks, and coordinate those using some sort of queue like Celery, or use something like Kubernetes.

...at which point, why should Airflow take care of all that at all? Just make the tasks call an API, and design the backend of that API in any way you want.

🔮

Long story short: what if we should actually focus¹ on making serving Kedro pipelines as a web application flawless, and make orchestrators (including, but not limited to, Airflow) just call the corresponding REST API?²

(diagram courtesy of @datajoely)

Footnotes

1. "Focus" as in: among all the possible things we could be doing, and given time and resource constraints, what things do we say "yes" to at the cost of saying "no" to others? ↩

2. This might not apply to some platforms that are very idiosyncratic like Databricks ↩

[DimedS]

That's a really interesting idea, @astrojuanlu. I agree that separating orchestration from execution is a very good practice, and moving towards a microservices architecture would be a great step forward.

Does that mean the Kedro Server should be responsible for resource allocation? As I understand it, when Airflow runs on Kubernetes, it creates one Pod per task, and then Kubernetes is responsible for providing resources for all those Pods. In this context, the Kedro Server would need to always be running to be ready to respond, just like an Airflow scheduler. However, additional resources are only needed when executing tasks.

[astrojuanlu]

Thanks @DimedS!

I don't think kedro-server can necessarily be a single thing, because different teams will have different opinions on how to serve their pipelines. But the existence of https://github.com/takikadiri/kedro-boot/ proves that there is some friction when trying to use Kedro in that way. So this would be more about incorporating the learnings of kedro-boot into Kedro (#3540), and providing a blueprint for our recommended way of doing things (think of a "kedro-server" you can install, but assuming that most users will want to extend it)

[datajoely]

Yeah I think focus should be:

• Improve the session so that it works better for microservice applications and means things like kedro-boot don't have to do any creative workarounds.
• (Potentially) ship a very basic kedro-fastapi plug-in that lets people extend however they want

[astrojuanlu]

Would love to know @sbrugman thoughts on this given that you're heavy users of kedro-airflow

[sbrugman]

Separation of scheduling from execution is important, e.g. one can use Airflow to schedule Spark tasks (on k8s).
Note that it's not always possible to install Python dependencies on the Airflow scheduler!

Using Airflow + Kedro + Spark on k8s, we can use the SparkSubmitOperator.
This operator requires a Python script to start, so the kedro executable cannot be directly used.
Instead, we provide a kedro_launcher.py script that does just that + set some environment variables for the config location:

"""Helper script to run kedro via spark-submit"""
import os
import subprocess
import sys

if __name__ == "__main__":
    # config is placed in `src` for syncing
    os.environ["KEDRO_CONF_SOURCE"] = "src/conf/"
    project_source = "/workspace/"
    print(f"Project at {project_source}")
    KEDRO_COMMAND = ["kedro", "run"]
    args = KEDRO_COMMAND + sys.argv[1:]
    result = subprocess.run(args, cwd=project_source, check=True)
    exit(result.returncode)

Kedro only needs to be available in the environment inside the pods, not on the scheduler.
This means that at minimum, the DAG needs to be generated from the pipeline (like we do now, perhaps more generic)

[astrojuanlu]

Thanks @sbrugman, a follow-up if I may: what do you think are the advantages of doing subprocess.run(["kedro", "run", sys.argv[1:]) over instantiating a KedroSession as explained in https://docs.kedro.org/en/stable/kedro_project_setup/session.html ?

[takikadiri]

We're migrating our batch workfloads to an organization-wide Airflow offer. Here is some main desing decisions for our DS/ML batch workloads:

• No transaltion between Kedro objects and Airflow objects. Airflow would just run Kedro package with the given CLI args

• Since the DS/ML workload are compute and memory intensive, the compute would be performed outside of Airflow. Other reasons comforting us in that choice, among others: python dependencies are hard to manage in airflow, airflow workers resources sizing are restrictives in our offer, separation of concerns, ...
  The compute will be managed by either KubernetesPodOperatoror or by a custom airflow operator that run the apps in a remote VMs using SSH. In both cases, the compute resources are provided at runtime.

• Airflow take care of scheduling and tasks queuing (and maybe manage the configs). Airflow would be aware of the execution state and the logs as everything is triggered and managed under the airflow worker processes, and reported back to the executor/scheduler and to the the web server/UI.

The Kedro-Server proposition is interesting, it would however put a lot of burden into the application, as it should take care of a lot of things; The API would be a sort of a Gateway to a complex backend that do tasks queuing, resources managements, loggins managements, and more ... With a shared observability between the API and Airflow. If we push the idea further, we coud even embed a lighweight scheduler inside that API, and end up with an orchestration framework. This could eliminate the need of any orchestration platform. At least of some simple orchestations ...

[astrojuanlu]

Thanks @takikadiri. One follow-up question if I may:

No transaltion between Kedro objects and Airflow objects. Airflow would just run Kedro package with the given CLI args

Are your DSs okay with not seeing the full Kedro DAG in Airflow, hence behaving like a sort of "black box"?

---

Since the DS/ML workload are compute and memory intensive, the compute would be performed outside of Airflow. Other reasons comforting us in that choice, among others: python dependencies are hard to manage in airflow, airflow workers resources sizing are restrictives in our offer, separation of concerns, ...

That makes a lot of sense and it's one of the things that motivated me to open this discussion 👍🏼

Airflow take care of scheduling and tasks queuing (and maybe manage the configs). Airflow would be aware of the execution state and the logs as everything is triggered and managed under the airflow worker processes, and reported back to the executor/scheduler and to the the web server/UI.

Also 👍🏼

it would however put a lot of burden into the application, as it should take care of a lot of things; The API would be a sort of a Gateway to a complex backend that do tasks queuing, resources managements, loggins managements, and more ...

Not sure it has to be that complex, but I see your point. I guess different teams will have different preferences on how to serve Kedro pipelines. And I believe low-latency use cases would highly benefit from some pre-loading.