Next installment of the interview series Part I | Part II | Part III | Part IV | Part V | Part VI | Part VII | Part VIII | Part IX | Part X What is Arrow & how does it improve Python UDF in Spark? Apache Arrow is a multi-language toolbox for accelerated data interchange and in-memory processing. Before Arrow…

Probabilistic data structures used in big data Read Part I here. 1. HyperLogLog Counting unique items usually requires the amount of memory proportional to the number of items we want to count because we need to remember the elements we have already seen in the past in order to avoid counting them…

Compilation of some good practices for DE Data engineering is a critical aspect of any data-driven organization. Building robust and efficient data pipelines requires a combination of expertise, well-defined processes, and best practices. In this blog, I will list the comprehensive set of best practices that help us streamline our…

next part of the interview series Part I | Part II | Part III | Part IV | Part V | Part VI | Part VII | Part VIII | Part IX | Part X What Aggregate Strategies are provided in Spark and how it chooses one? (Hash vs Sort aggregate) The Sort Aggregate requires the rows to be sorted by the grouping key so that…

Security consideration of spark cluster & its components The Spark ecosystem is an integral part of most of the analytics workloads at big companies. While generally this is deployed in private network zones but still its security sanitization is still needed to avoid any data breach. …

process streaming data using spark dataframe API Stream Processing can mean different things to different people. Some look at this with the lens of Real-time vs Schedule where they associate it with latency. Some people link this to continuous/ongoing processing of records. While a majority of people link streaming to…

Next blog in the spark interview series. Part I | Part II | Part III | Part IV | Part V | Part VI | Part VII | Part VIII | Part IX | Part X What are the different types of window operations available in Spark Streaming? There are four different types of window operations available in Spark Streaming: 1. Tumbling…

Another part of the Spark interview series. Part I | Part II | Part III | Part IV | Part V | Part VI | Part VII | Part VIII | Part IX | Part X What is the difference between Select vs SelectExpr in Spark? selectExpr() is a powerful method for column selection and transformation when you need to…

The next part of the series. Part I | Part II | Part III | Part IV | Part V | Part VI | Part VII | Part VIII | Part IX | Part X What happens when we give Join hints on both sides of join? When the hints are specified on both sides of the Join, Spark selects the hint…