Building Apache Spark 1.3 development branch

In this short post, I am documenting how to download and build Apache Spark development branch 1.3 in Linux Mint. Apache Spark 1.3 provides a new feature, the DataFrame API similar to that of R and to a database table. The official version 1.3 is due to be released in early March, however one can download and evaluate the development version.

To download and build Apache Spark 1.3, do the following:

* clone the 1.3 branch by running at the shell prompt

git clone -b branch-1.3 https://github.com/apache/spark.git

Once this step is done, there should be a folder called “spark”, cd into the folder then run the build command using sbt

sudo sbt/sbt assembly

After a while, if the build is successful, you’ll see

the Spark shell is started by running the following command

 bin/spark-shell