Learning pyspark – Installation – Part 1

This is part one of a learning series of pyspark, which is a python binding to the spark program written in Scala. The installation is pretty simple. These steps were done on Mac OS Mavericks but should work for Linux too. Here are the steps for the installation: 1. Download the Binaries: Spark : Scala : Dont use Latest Version of Scala, Use Scala 2.10.x 2. Add these lines to your .

Hadoop, Mapreduce and More – Part 1

It has been some time since I was stalling learning Hadoop. Finally got some free time and realized that Hadoop may not be so difficult after all. What I understood finally is that Hadoop is basically comprised of 3 elements: A File System Map – Reduce Its many individual Components. Let’s go through each of them one by one. 1. Hadoop as a File System: One of the main things that Hadoop provides is cheap data storage.