Getting Started With Apache Spark in Ubuntu

This is the first episode of the Apache Spark series that i wish to continue to up to advance Apache Spark Programming. 

What is Apache Spark

Apache Spark is a cluster computing platform. In simpler terms it will execute your instructions in a cluster of computer instead using a single one. By doing so it will produce fast results. Apache spark is most used in Big Data Analysis but it has features which can be used for some other areas such as Machine Learning and Graph Processing. The most satisfying thing about Apache Spark is that it supports Java, Scala and python which are well known languages.  

Downloading Apache Spark

 Please note this installation walk through is for installing Spark on a single node.  Download Apache from here.

Select Apache version and Package type. The package type is important if you have already using Apache Hadoop. Make sure you use the correct package type to avoid any collisions. But please note that Apache Hadoop is NOT a necessary per-requisite to install and use Spark.

Once download is complete you will have a tar file.
 Go to the directory where the tar file is. To unpack Spark open up terminal and use following commands us usual. 

..$ tar -xf spark-2.1.0-bin-hadoop2.7.tgz
..$ cd spark-2.1.0-bin-hadoop2.7
..$ ls

If all worked out well so far you should have following content in your directory.
bin - Contains the various executable files that we can use to interact with Spark in various ways. In the next step we will use use some of these.
examples - This directory contains some example Standalone Spark jobs that has been already done for you!. You go through this to get more familiar with Spark API.

Getting Started with Spark

As I said before there are few languages that we can use to work with Spark. Here I will use python in my Examples.
Lets  start with something like "Hello World" in Spark Universe which is counting words in a document.
Using terminal go to the directory where Spark is. 
and type bin/pyspark .
Then you will see that Spark is starting as below.


Bravo!. All is set for our first Spark job. You can use CTRL+D to exit this python shell. Only two lines of code is need for our first Spark job!

lines = sc.textFile("README.md")
lines.count()

This simple program will be our Driver Program. The Driver Program will access Spark through a SparkContext object. When you execute these two lines of code. Spark creates a RDD and perform Action (count) on that RDD to produce our result.
We have reached the end. Now you Know how to install Apache Spark and run a simple on that using python.

 I will explain SparkContext, RDDs, Actions and some more in our next episode. 
Please comment if you have any issues following these steps and feel free to point out errors I've made. Until next time adios amigos!!

 

 


Comments

Post a Comment

Popular posts from this blog

DDoS and DoS Attacks and Tools - Part 1

Image
Note: Following Content is only for educational and White Hat security purposes . History of DOS Attackers Back in 1974, a teenager named David Dennis created the first ever DoS attack. This student was a student at University High School, located across the street from thr University of Illinois at Urbana. He came to know about a command that could be run on PLATO terminals which was used at the Computer Based Learning Center (CERL) of University of Illinois. He was curious and wondered what will happen if a room full of people executed a command of PLATO terminal. When he wrote a program and executed he found out that his program has forced all 31 computers of the  CERL to turned off! After that DoS attacks has spread and DDoS, a new and powerful variant of DoS came in to play. Below are some noted cyber attacks from 1988 to near 2014.from year 2000 DDoS Tools were used for the attacks most of the time. The attacks were in scales from Small tickles to Geo-Political even
Read more

DDoS and DoS Attacks and Tools - Part 2

Image
In this we will go through the rest of the DoS attack types from the previous blog post. I n the latter part we will get to know few infamous DoS attack tools. Please note that how you use these is at you own risk and make sure you always adhere to laws from your governing body. Attacks Targeting Server Resources  In this type attacks the attacker will try to use as mush as server resources so that the server can't handle legitimate requests any more. This type of attacks will target servers memory, buffers and other resources. Attacker will exploit a weakness in communication protocol. There are attacks in this category which can result server to crash or otherwise require a manual reboot to come alive again. Server can be normal server or a state-full device like firewall or intrusion-detection system.  TCP/IP Weakness This is a connection-oriented protocol. One of main consents behind this protocol is that the three way handshake. which uses flag bits in
Read more

Apache Spark Small Talk

Image
Apache Spark is sub project of Apache Hadoop. It is developed in University of California, Berkeley. Apache Spark was build for "Lighting faster cluster computing" as it's official web site says. By using Spark some issues in Hadoop was addressed well therefore Spark was popular in no time. It was open sourced in 2010 under BSD license. Main Components in Apache Spark  Apache Spark has a few tightly integrated components as you can see here. As you may understand Spark Core has core functionalities like memory management, task scheduling, fault recovery etc. The main data abstraction Resilient Distributed Data set (RDD) is also defined in the Spark Core. Spark SQL, Spark Streaming Real Time, MLib(Machine Lerning Library), Graph X each component has unique and different functionalities. Since we focus more on Spark Core functionalities and concepts at this stage, we will dive into those in later episodes of this serise.  Resilient Distributed Dataset
Read more