Understand how to work with intervals Interval is a period of time(can be data points as well) between events or states. There are many real-life problems that deal with intervals. 1. Scheduling and assigning rooms for classes or meetings. 2. Organizing data in memory and calculating allocated and free spaces…

Implementation of priority queue using heap Priority Queue is an abstract data structure. It supports the following operations: Check whether the queue is empty. Add an element to the queue. Pop an element with the highest priority. There are two ways to assign priority: The smallest element has the highest…

How joins are performed in Spark The Join operation is the most commonly used transformation in Spark. Behind the scene, spark uses some algorithms to decide how to implement the join. In this blog, we will see how spark decides which strategy is used for Join. Factors affecting Spark Join strategy Data Size Join Hint Type of…

7 ways of iterating over lists in Python. — In Python, there are multiple ways to iterate over a list. In this blog, we will see the 7 ways of doing so. For loop Index iteration — for_loop_idx for index in range(start, stop[, step]) Direct Item access — for_loop_item for item in input_list: Both index & Item for_loop_idx_item for_loop_enumerate enumerate(iterable, start_index): …

Good practices to follow while working with lambda A brief intro to lambda working. Lambda function components Handler —Entrypoint of the lambda function Event — JSON Payload received during lambda function invocation. AWS services generate their own JSON. Context — Representation of the runtime and execution environment. function_name , function_version , invoked_function_arn , memory_limit_in_mb …

First-hand experience of cost-saving for Spark on EMR — Recently I was engaged in a project which aimed at improving our ETL pipeline’s performance & bringing down the cost of running those Jobs. We were running most ETLs on long-running EMR clusters. Our upgrade was as follows:

try…except…else…finally — When a Python script raises a runtime exception if it is not handled program terminates and exits. If we want to control the flow of the program in those scenarios we should use exception handling. In this blog we use the following code block to understand the flow: import random try…

Embedding security into the lambda service offering In the last blog, Threat Modelling — EKS, we saw how to evaluate the security posture of EKS, in this blog, we will do a similar exercise for AWS Lambda. Shared responsibility Model Customer Function code, libraries, extension, layers Data in cloud Data in Transit Resource Configuration IAM Other Service configuration…

Understanding the Trie data structure — A Trie, (also known as a prefix tree) is a special type of tree used to store associative data structures (collection of nodes that point to other nodes). The term “trie” comes from the word retrieval as strings can be retrieved by traveling down a branch path of the tree…