All you need to know for handling duplicates in SQL. Identifying the duplicate entries The first step should be to define which column or combination of columns forms the unique row. Once we have the set of columns that decide the uniqueness of the row we can use the following two strategies to find…

Lambda & Kappa Modern data processing platforms need to process real-time streaming events in addition to more traditional data pipelines. Deciding on the right architecture, Lambda or Kappa is an important step in the direction of a good platform strategy. Design goals for Data processing Architectures: Read consistent data Read incrementally from large tables/datasets Rollback Capability…

SQL on anything with Dremio Nowadays every company relies on data to make decisions. So every company needs a data infrastructure that will handle their use cases over time. And the use cases have evolved over time. With changing use cases we saw different solutions like a data warehouse, Hadoop…

A basic introduction to CDC CDC Change Data Capture (CDC) is a data integration design pattern that allows us to track row-level changes in database tables in response to CREATE, UPDATE and DELETE operations. CDC allows us to treat the changes in DB as events and respond to them. Using this…

Skills you need to become a data engineer Programming Language Data engineering leads to various roles in the future like Data Analyst, Data Scientist, ML Engineer, and BI Developer. So choosing the right language is important. Python is more diverse in the sense that it is used widely in all the previously…

Dissecting the window function Window functions compute the result based on a sliding window frame. A window frame is a set of rows that are related to the current row. The relation is that all the rows have the same value for all terms of the PARTITION BY clause in…

Part III of Graph traversal algorithm series. Read Part I and Part II In the last blog we learned about two Single Source Shortest Path algorithms, In this blog, we will see two algorithms for finding the shortest path among all node pairs. Floyd Warshall Algorithm This algorithm takes a dynamic programming approach…

Part II of Graph traversal algorithm series. Read Part I and Part III

Reusable Databricks notebook with parameters Input widgets are utility in Databricks that allow us to add parameters to our Dashboards and Notebooks. The Widget API provides methods for creating getting values and removing them. In this blog, we will see how we can use those APIs. Help All the APIs for…