The Hitchhikers guide to handle Big Data using Spark

Big Data has become synonymous with Data engineering.

But the line between Data Engineering and Data scientists is blurring day by day.

At this point in time, I think that Big Data must be in the repertoire of all data scientists.

Reason: Too much data is getting generated day by day

And that brings us to Spark.

Now most of the Spark documentation, while good, did not explain it from the perspective of a data scientist.

So I thought of giving it a shot.

This post is going to be about — “How to make Spark work?”

This post is going to be quite long. Actually my longest post on medium, so go pick up a Coffee.

How it all started?-MapReduce