The text discusses how to coordinate two Airflow DAGs such that the hourly DAG runs only if the daily DAG has been successful on the same day. It outlines three different methods to achieve this: using the ExternalTaskSensor with execution_delta, using the ExternalTaskSensor with execution_date_fn, and using a customized approach with PythonOperator. The tutorial provides detailed insights on implementation for each method.
Using Airflow sensors to control the execution of DAGs on a different schedule
Recently, I’ve been trying to coordinate two Airflow DAGs such that one would only run — on its own hourly schedule — if the other DAG (running on a daily basis) has been successful.
Use Case: Running the hourly DAG only if the daily DAG succeeded
In our use case, we would like my_hourly_dag to run only if my_daily_dag has run successfully within the current date. If not, then my_hourly_dag should be skipped. It is important to mention here that we don’t want to trigger my_hourly_dag as soon as my_daily_dag succeeds. That would be achievable with TriggerDagRun operator. Instead, we want both DAGs to run on their own schedule but add a condition on the my_hourly_dag.
How to Skip Tasks in Airflow DAGs
In the next two sections we will discuss and demonstrate how to achieve this using a few different approaches.
Using ExternalTaskSensor
Our first option is the built-in ExternalTaskSensor operator. We can use this sensor in our my_hourly_dag that will essentially check if my_daily_dag has been successful in the specified interval.
Using PythonOperator
The second approach involves a more customized solution. More specifically, we can programmatically find the latest successful DagRun of our daily DAG and handle the behavior of the operator accordingly. In other words, if the latest successful DagRun of the daily DAG does not align with the execution date of our hourly DAG, the task will be skipped (as well as the downstream tasks).
Final Thoughts: In today’s tutorial we discussed how to handle dependencies between different DAGs when using Airflow. More specifically, we discussed how to run a DAG that is supposed to execute on an hourly basis, only if a different DAG, on a daily schedule, executes successfully within the day.
Evolve Your Company with AI
If you want to evolve your company with AI, stay competitive, use for your advantage Running Airflow DAG Only If Another DAG Is Successful.
Discover how AI can redefine your way of work. Identify Automation Opportunities, Define KPIs, Select an AI Solution, Implement Gradually. For AI KPI management advice, connect with us at hello@itinai.com. And for continuous insights into leveraging AI, stay tuned on our Telegram or Twitter.
Spotlight on a Practical AI Solution
Consider the AI Sales Bot from itinai.com/aisalesbot designed to automate customer engagement 24/7 and manage interactions across all customer journey stages.
Discover how AI can redefine your sales processes and customer engagement. Explore solutions at itinai.com.
List of Useful Links:
- AI Lab in Telegram @aiscrumbot – free consultation
- Running Airflow DAG Only If Another DAG Is Successful
- Towards Data Science – Medium
- Twitter – @itinaicom
