Understand the basic networking of AWS Virtual Private Cloud
The above diagram depicts the major components of the AWS VPC service. Let us go through them one by one.
VPC
Virtual Private Cloud is a service from AWS which enables us to launch AWS resources in a logically isolated virtual network (similar to a local data center). It gives us…
Eventing, messaging or brokering — understand their facets.
In a distributed system, the exchange of information is very critical. Generally, those exchange happens between applications/services. Hence it becomes very critical to choose the right message/eventing platform. In this blog, we will go through some aspects of such a platform.
Unordered vs Strictly Ordered
…
Production, Detection, Asynchronous & Reactive processing of events
An event processing application design should have the following consideration at its core:
Decoupling — Event producer & consumer should be decoupled.
Asynchronous — The system in the process of event processing should be non-blocking (asynchronous).
Call Back — The event should contain a callback URL/endpoint for acknowledgment of the…
3 Billion SSM Calls in a Month: The hidden cost of using secrets backend in airflow, if not used correctly.
Recently I was involved in moving 180 DAGs from our own EC2 based airflow installation to EKS hosted airflow installation. Since we were very short on time we did a lift and shift move of DAGs on the new infrastructure. …
Adjusting the search space to reach the target quickly
Binary Search is a divide and conquer algorithm which breaks down the search space in two halves, after each iteration. In each iteration, it discards half of the search space.
Template:
- Initialize the search space (start and end)
- Find Mid of the search space
- Discard/select one half based on the specific…
Access files stored in S3 as a regular table in Snowflake
Snowflake integration objects enable us to connect with external systems from Snowflake. In my last blog, we went through how to create an API integration to invoke an AWS Lambda. …
Variables and connections in Airflow make the installation plug & play and also, these decide which system to talk to for various purposes. In a large system, these will grow very fast, and managing their lifecycle (creating, updating) becomes challenging. As a data platform provider, we allow each product team…
Auto Scaling is one of the prime features of the cloud. And AWS Elastic Map-reduce is no exception to that. With the mix of instances and purchasing options (on-demand and spot), we can achieve faster job turnarounds at a reasonable price.
EMR comes with Instance groups (IG) & Instance fleets…
Execute python code in lambda as external function in Snowflake
Recently I and Supreeth Chandrashekar collaborated on a task, where we wanted to run multiple inferences on some of data stored in Snowflake. We leveraged external function to trigger lambda which in turn got the inference from AWS Sagemaker endpoints(serving different models). …
How to use Pass, Continue, Break in Python
Python has three Jump operators which can move the program control from one place to another. In this blog, we go through these three jump operators.
Pass — The pass acts as a placeholder, a syntactic sugar. Whenever the python interpreter encounters pass it does nothing. …
