Overview

STACKIT Workflows is a managed service based on Apache Airflow 2. Apache Airflow 2 is an open-source platform designed to programmatically author, schedule, and monitor complex data engineering workflows. Airflow Workflows are defined in Python as Directed Acyclic Graphs (DAGs), which represent a collection of tasks and their dependencies.

The core building blocks of a DAG are Operators, which are pre-defined templates for individual tasks like running a SQL query or a Bash script. Airflow’s scheduler triggers these tasks based on time or external events, distributing the workload across multiple workers for scalability. STACKIT Workflows in particular distributes the workload, which is automatically encapsulated in pods, in a Kubernetes cluster. A comprehensive web interface provides an overview for monitoring, managing, and debugging these data pipelines.

For a more detailed overview, please refer to the official Airflow 2.10 documentation.