Another set of questions on Spark Read Part I here. Read Part II here. Read Part IV here. What are the partitioning hints in Spark? There are four partitioning hints available in Spark. COALESCE — This hint is used to reduce the number of partitions REPARTITION — This hint is used to increase or decrease the number…

Issues faced in BAU Data Engineering On call 1 Data Engineering On call 2 Data Engineering On call 3 Data Engineering On call 4 Data Engineering On call 5 ISSUE 1 One of the Spark jobs failed with the following error. After spark application run for a period of time on spark…

Next blog in the Spark Question Interview Series Read Part I here. Read Part III here. What is the potential common issue with below two code snippets? ROW_NUMBER() OVER (order by column_x) repartitionByRange(col1, col2) The above two codes are prone to OOM. In the first code, we are not giving any partition by clause so it will bring everything into a…

Issues faced in daily BAU Data Engineering On call 1 Data Engineering On call 2 Data Engineering On call 3 Data Engineering On call 4 ISSUE 1 Spark fails to initialize with the actual database. df = spark.read.format("jdbc"). option("url", "<host url>"). option("dbtable", "CSDTL.MY_TABLE"). option("user", "postgres"). option("password", "<password>"). option("numPartitions", 50). option("fetchsize", 20). load()

what & why of Spark Scheduler delay In one of the apps in production, we saw an unusual delay in the job. When we looked at the Spark UI we noticed a huge scheduler delay. …

Manipulate data at its lowest level. In this blog we some key concepts about bitwise operators that every software engineer should be familiar with. Get ready to get into the world of bits & bytes. In any positional system to represent a number, we use base, which represents the number…

Series on spark question we should know before an interview Read Part II here. Read Part III here. Read Part IV here. How does limit work in Spark? How is different from take? The limit is implemented as LocalLimit (at partition) and then GlobalLimit. This limit is implemented incrementally. First, it tries limit on one partition and then the next (current*…

Another day, new learnings. Earlier parts of the series: Data Engineering On call 1 Data Engineering On call 2 Data Engineering On call 3 Issue 1 After an update of a spark job, the following exception occurred: java.lang.ClassCastException: cannot assign instance of scala.None$ to field org.apache.spark.scheduler.Task.appAttemptId of type scala.Option in instance of…

Using Blackbox-exporter to monitor service endpoints In a big data platform, there are many service endpoints (Hue, Livy, SHS, etc.), that need to be continuously monitored. This monitoring solution needs to have the following features: External probing Configurable parameters & check interval Prometheus format (As this is our monitoring solution) …