Authorization strategy that defines permissions based on attributes.
Attribute-Based Access Control evaluates and grants/denies access based on Attributes (tags, a custom attribute, in this case) attached to the caller entity and requested resources. Tags can be attached to the resource, passed in the request, or attached to the principal who…
In collaboration with Harshith Acharya.
With container support, we can run any runtime (within resource limitation) on AWS Lambda. In this blog, we will see how we can run a PySpark application on AWS Lambda.
The project structure for the experiment is as below:
├─spark-on-lambda
├───spark-class
├───spark-defaults.conf
├───Dockerfile
└───spark_lambda_demo.pyThis…
Role-based access control and IAM role for Service Accounts.
RBAC Stands for Role-based access control. RBAC is the method of regulating who can perform which actions on which resources? Let us understand the term used with RBAC.
An identity that wants to execute certain actions and requires permission to do…
The above diagram depicts the major components of the AWS VPC service. Let us go through them one by one.
Virtual Private Cloud is a service from AWS which enables us to launch AWS resources in a logically isolated virtual network (similar to a local data center). It gives us…
In a distributed system, the exchange of information is very critical. Generally, those exchange happens between applications/services. Hence it becomes very critical to choose the right message/eventing platform. In this blog, we will go through some aspects of such a platform.
…
An event processing application design should have the following consideration at its core:
Decoupling — Event producer & consumer should be decoupled.
Asynchronous — The system in the process of event processing should be non-blocking (asynchronous).
Call Back — The event should contain a callback URL/endpoint for acknowledgment of the…
Recently I was involved in moving 180 DAGs from our own EC2 based airflow installation to EKS hosted airflow installation. Since we were very short on time we did a lift and shift move of DAGs on the new infrastructure. …
Binary Search is a divide and conquer algorithm which breaks down the search space in two halves, after each iteration. In each iteration, it discards half of the search space.
Snowflake integration objects enable us to connect with external systems from Snowflake. In my last blog, we went through how to create an API integration to invoke an AWS Lambda. …
Variables and connections in Airflow make the installation plug & play and also, these decide which system to talk to for various purposes. In a large system, these will grow very fast, and managing their lifecycle (creating, updating) becomes challenging. As a data platform provider, we allow each product team…
