“An artist’s concern is to capture beauty wherever he finds it.” ― Kazuo Ishiguro, An Artist of the Floating World We are data engineers and Spark is our best friend and the natural choice when the job is massive parallel data processing. Many times a day we interact with spark…

Work hard and figure out how to be useful and don’t try to imitate anybody else’s success. Figure out how to do it for yourself with yourself. — Harrison Ford Being a data engineer, I try to find fun to create small utils to overcome repetitive work. …

Calculus, the electrical battery, the telephone, the steam engine, the radio — all these groundbreaking innovations were hit upon by multiple inventors working in parallel with no knowledge of one another. — Steven Johnson Spark, my all time favourite ETL framework and for all the reasons to be the best…

“If you are capable, but not available, nature will raise a person with lesser ability to replace you soon.” ― Israelmore Ayivor, Become a Better You No distributed system is safe from network failures, thus network partitioning generally has to be tolerated. In the presence of a partition, one is…

“It is far easier to design a class to be thread-safe than to retrofit it for thread safety later.” ― Brian Goetz, Java Concurrency in Practice The Success Story of RDBMS Handling transactions with ease is always the USP for RDBMS systems. We live in a world where every interaction is transactional in nature. A…

I would rather walk with a friend in the dark, than alone in the light. — Helen Keller Well apache spark is that friend, Helen was talking about in the big data world. Multiple ocassions when we deal with very low volume of data in some of the interfaces that…

There was something special and unique about the love triangle that existed between Clark Kent, Superman, and Lois Lane. — Jim Lee Motivation Well it is evident that we are talking about Kafka, Spark and Mongo DB together that means there was a story at the beginning. The use-case for this…

The acronym NoSQL was first used in 1998 by Carlo Strozzi while naming his lightweight, open-source “relational” database that did not use SQL. The name came up again in 2009 when Eric Evans and Johan Oskarsson used it to describe non-relational databases. Relational databases are often referred to as SQL…

“If you’ve found some way to educate yourself about engineering, stocks, or whatever it is, good employers will have some type of exam or interview and see a sample of your work.” — Bill Gates What it is ideally? Setting the tone with François Truffaut’s interviewing Alfred Hitchcock is my honest effort to assure…