List: Data Engineering | Curated by Ani

Oct 16, 2024

30 stories

3 saves

Data EngineeringAll about data engineering
Ani
Kafka, Spark Streaming & Flink : Are they on War?Rivers, ponds, lakes and streams — they all have different names, but they all contain water. Just as religions do — they all contain…
Oct 16, 2024
1
Oct 16, 2024
1
Ani
Apache Spark Learning Path for a Budding Data EngineerIntroduction
Oct 15, 2024
Oct 15, 2024
Ani
The Bat Mobile : Fun of Tuning Spark JobLife is made up of sobs, sniffles, and smiles, with sniffles predominating. The magi, as you know, were wise men — wonderfully wise men —…
Sep 22, 2024
Sep 22, 2024
Ani
Partitioning & Bucketing — A Big Data EsThe most successful Subway customers, of course, are the ones who can’t keep their hands off their sandwich. Join your artist in the…
Sep 9, 2024
1
Sep 9, 2024
1
Ani
Write better SQLs — A top grade skill!“Creativity takes courage.” — Henri Matisse
Sep 5, 2024
Sep 5, 2024
Ani
Data Lake House ArchitectureWith a number of buzzes around lakehouses and it’s benefits let’s have a look at the basic architecture of a working lakehouse.
Nov 1, 2021
Nov 1, 2021
Ani
Adding Derived Column to a DataframeHow you can add a new derived column to a dataframe in spark?
Nov 1, 2021
Nov 1, 2021
Ani
Databricks Lakehouse solving big data problem in Health CareDo you know a single patient produces approximately 80 megabytes of medical data every year? Multiply that across thousands of patients…
Nov 1, 2021
Nov 1, 2021
Ani
SQL Join : A Common Mistake“Everything should be made as simple as possible, but not simpler.” 
— Sir Albert Einstein
Nov 17, 2021
2
Nov 17, 2021
2
Ani
Consuming Restful API data and store in Spark DataframeI would rather walk with a friend in the dark, than alone in the light. — Helen Keller
Mar 3, 2022
1
Mar 3, 2022
1
Ani
The Importance of Data Structures and Algorithms in the Life of a Data EngineerThere’s no shortcut to learning a craft; you just have to put the years in.
— Kylie Minogue
Sep 3, 2024
Sep 3, 2024
Ani
Apache Spark — Play with nested files“Setting goals give you a life to live. When you have zero goals its life that consumes you.” ― Thomas Vato
Mar 2, 2023
1
Mar 2, 2023
1
Ani
Spark 3.4 — Parameterised SQLCode reuse is the Holy Grail of Software Engineering.
Douglas Crockford
Aug 4, 2023
1
Aug 4, 2023
1
Ani
Understanding Spark Re-Partition“The bad news is time flies. The good news is you’re the pilot.” — Michael Altshuler
Jul 23, 2024
Jul 23, 2024
Farooq Qaiser
Manipulating Nested Data Just Got Easier in Apache Spark 3.1.1In previous versions of Apache Spark, adding or dropping deeply nested fields could be a challenging programming exercise. In this article…
Mar 8, 2021
4
Mar 8, 2021
4
In
SelectFrom
by
Ani
Spark Optimization : Reducing Shuffle“Shuffling is the only thing which Nature cannot undo.” — Arthur Eddington
Jul 30, 2022
6
Jul 30, 2022
6
Ani
Design a Scalable Data Solution : Know the requirementsA recipe has no soul. You as the cook must bring soul to the recipe.”
– Thomas Keller
Jul 17, 2022
Jul 17, 2022
The next generation write ups of data engineering, by a data engineer, for the data engineers.
Ani
Parquet : ZSTD vs GZIPWhat could be the best compression codec for your datalake? Most popular and optimised file format that is parquet which is also the…
Nov 1, 2021
1
Nov 1, 2021
1
Ani
Table Evolution in Apache IcebergIn you datalake table you want to make changes to the physical layout of the table but don’t want to rebuild the table with huge compute?
Nov 1, 2021
1
Nov 1, 2021
1
 This story is no longer available

Data Engineering

All about data engineering

Kafka, Spark Streaming & Flink : Are they on War?

Rivers, ponds, lakes and streams — they all have different names, but they all contain water. Just as religions do — they all contain…

Apache Spark Learning Path for a Budding Data Engineer

Introduction

The Bat Mobile : Fun of Tuning Spark Job

Life is made up of sobs, sniffles, and smiles, with sniffles predominating. The magi, as you know, were wise men — wonderfully wise men —…

Partitioning & Bucketing — A Big Data Es

The most successful Subway customers, of course, are the ones who can’t keep their hands off their sandwich. Join your artist in the…

Write better SQLs — A top grade skill!

“Creativity takes courage.” — Henri Matisse

Data Lake House Architecture

With a number of buzzes around lakehouses and it’s benefits let’s have a look at the basic architecture of a working lakehouse.

Adding Derived Column to a Dataframe

How you can add a new derived column to a dataframe in spark?

Databricks Lakehouse solving big data problem in Health Care

Do you know a single patient produces approximately 80 megabytes of medical data every year? Multiply that across thousands of patients…

SQL Join : A Common Mistake

“Everything should be made as simple as possible, but not simpler.” — Sir Albert Einstein

Consuming Restful API data and store in Spark Dataframe

I would rather walk with a friend in the dark, than alone in the light. — Helen Keller

The Importance of Data Structures and Algorithms in the Life of a Data Engineer

There’s no shortcut to learning a craft; you just have to put the years in. — Kylie Minogue

Apache Spark — Play with nested files

“Setting goals give you a life to live. When you have zero goals its life that consumes you.” ― Thomas Vato

Spark 3.4 — Parameterised SQL

Code reuse is the Holy Grail of Software Engineering. Douglas Crockford

Understanding Spark Re-Partition

“The bad news is time flies. The good news is you’re the pilot.” — Michael Altshuler

Manipulating Nested Data Just Got Easier in Apache Spark 3.1.1

In previous versions of Apache Spark, adding or dropping deeply nested fields could be a challenging programming exercise. In this article…

Spark Optimization : Reducing Shuffle

“Shuffling is the only thing which Nature cannot undo.” — Arthur Eddington

Design a Scalable Data Solution : Know the requirements

A recipe has no soul. You as the cook must bring soul to the recipe.” – Thomas Keller

Parquet : ZSTD vs GZIP

What could be the best compression codec for your datalake? Most popular and optimised file format that is parquet which is also the…

Table Evolution in Apache Iceberg

In you datalake table you want to make changes to the physical layout of the table but don’t want to rebuild the table with huge compute?

Ani