“Setting goals give you a life to live. When you have zero goals its life that consumes you.” ― Thomas Vato In general when we try to read a directory in spark with say a pattern we do this. val path = "examples/src/main/resources/people.csv" val df = spark.read.csv(path) df.show() +------------------+ |…

“Shuffling is the only thing which Nature cannot undo.” — Arthur Eddington I used to see people playing cards and using the word “Shuffle” even before I knew how to play it. Shuffling in cards, play a very critical role to distribute “power”, adding weightage to a player’s hand. It…

A recipe has no soul. You as the cook must bring soul to the recipe.” – Thomas Keller We the data engineers for me, should be no different than a spiritual seeker. A seeker mind is always curious to get more clarity, more transparency. …

“An artist’s concern is to capture beauty wherever he finds it.” ― Kazuo Ishiguro, An Artist of the Floating World We are data engineers and Spark is our best friend and the natural choice when the job is massive parallel data processing. Many times a day we interact with spark…

Work hard and figure out how to be useful and don’t try to imitate anybody else’s success. Figure out how to do it for yourself with yourself. — Harrison Ford Being a data engineer, I try to find fun to create small utils to overcome repetitive work. …

Calculus, the electrical battery, the telephone, the steam engine, the radio — all these groundbreaking innovations were hit upon by multiple inventors working in parallel with no knowledge of one another. — Steven Johnson Spark, my all time favourite ETL framework and for all the reasons to be the best…

“If you are capable, but not available, nature will raise a person with lesser ability to replace you soon.” ― Israelmore Ayivor, Become a Better You No distributed system is safe from network failures, thus network partitioning generally has to be tolerated. In the presence of a partition, one is…

“It is far easier to design a class to be thread-safe than to retrofit it for thread safety later.” ― Brian Goetz, Java Concurrency in Practice The Success Story of RDBMS Handling transactions with ease is always the USP for RDBMS systems. We live in a world where every interaction is transactional in nature. A…

I would rather walk with a friend in the dark, than alone in the light. — Helen Keller Well apache spark is that friend, Helen was talking about in the big data world. Multiple ocassions when we deal with very low volume of data in some of the interfaces that…