A framework proposal for Apache Airflow — This is the second article about why Data Engineers shouldn’t write Airflow DAGs. In this new article, we are going to introduce a framework proposal for Apache Airflow. This article aims to shed some light on how building a framework can help you solve some of the problems related to…

Testing is an integral part of any software system to build confidence and increase the reliability of the system. Recently, I joined Grab and here at Grab, we are using Airflow to create and manage pipelines. But, we were facing issues with Airflow. …

In the blog post, we will see some best practices for authoring DAGs. Let’s start. DAG as configuration file The Airflow scheduler scans and compiles DAG files at each heartbeat. If DAG files are heavy and a lot of top-level codes are present in them, the scheduler will consume a…

I have been developing on the Airflow platform for some time now, using it for scheduling and orchestrating data engineering workloads. …

I’ve recently been developing and operating a lot more on the Airflow platform. While doing this development, there was always the challenge of communicating the parts of your DAG effectively without the developer or operator having to open the DAG code to understand how it functions. Whilst trying to find…