Start a DAG run based on the status of some other DAG With the rise in Data Mesh adoptions, we are seeing decentralized ownership of data systems. At the same time, we also need to create a holistic view of the data. …

How to control concurrency Airflow allows us to run multiple tasks in parallel. At the same time, Airflow is highly configurable hence it exposes various configuration parameters to control the amount of parallelism. …

and ways to avoid them. Lack of data definition When a new data engineer joins a team, he/she struggles to find the meaning of the data under consideration. If there is a source to find the meaning of the data it helps them to correctly interpret the values. Single source of data definition also…

Metrics that we track for our data platform Tracking the health and success of a data platform as a whole can be challenging. KPIs for Data platforms are different from other software platforms. The value proposition and performance metrics for the Data platforms are distinct. …

Choosing the right database for your workload AWS offers 15+ different data storage options each one purpose-built and applicable to a range of use cases. Knowing which one to choose helps us in building a long-term DB solution. While deciding on the right database we should consider the following factors: …

Tools to build a data stack that will help your company become data driven In this blog we will discuss some flexible set of technologies to support the storage, management & consumption of data. These tools are inspired by modern data stack. Characteristics of MDS for an org that is…

Save resource by freeing workers while task is idle Airflow operates with concept of worker. To manage tasks it also provides concepts like Slots, Pools and priority. When a task is running it occupies a worker slot. Sensors and Operators take up a full worker slot for the entire time…

some ideas put together — In previous blogs on data engineering, I have discussed ETL, ELT, and CDC design patterns, in this blog I will discuss a few more design patterns that I have seen in the industry. Data Transfer Zero copy Data transfer System Level — Zero copy refers to copying the data directly from the disk file to the…

Lesser discussed features of Airflow In this blog, we will discuss the following features of Airflow Callbacks Cluster Policy Task Dependency Trigger Rules Let us roll! Callbacks The callback provides a way to act upon changes in the state of a given task or DAG. …