Deep Learning, Data Science and NLP Enthusiast

Learning pyspark – Installation – Part 1

This is part one of a learning series of pyspark, which is a python binding to the spark program written in Scala.

The installation is pretty simple. These steps were done on Mac OS Mavericks but should work for Linux too. Here are the steps for the installation:

1. Download the Binaries:

Spark :
Scala :

Dont use Latest Version of Scala, Use Scala 2.10.x

2. Add these lines to your .bash_profile:

export SCALA_HOME=your_path_to_scala
export SPARK_HOME=your_path_to_spark

3. Build Spark(This will take time):

brew install sbt
sbt/sbt assembly

4. Start the Pyspark Shell:


And Voila. You are running pyspark on your Machine

To check that everything is properly installed, Lets run a simple program:

test = sc.parallelize([1,2,3])

This should return 3. So Now Just Run Hadoop On your Machine and then run pyspark Using:

cd /usr/local/hadoop/
Deep Learning Specialization on Coursera

pyspark learning
Advertiser Disclosure: All Amazon links are affiliate links, which means I receive compensation for any purchases through them. You do not have to purchase via my links, but you support me if you do.