Hello everyone and welcome to the spark overview lecture in this lecture.
We're just going to be taking an abstract overview of the spark ecosystem.
Discuss spark in general how spark is different than map reduce what spark already these are already
the operations are.
Don't worry about having to understand all the operations we go through.
We're going to be reviewing covering them all again when we actually begin to program with SPARK and
Python.
Let's talk about spark in general spark is one of the latest technologies being used to quickly and
easily handle big data.
It's an open source project on a Patchi and is first released in February 2013 and has exploded in popularity
due to its ease of use and speed.
It was created around 2009 at the amp lab at UC Berkeley so you can see that relatively speaking it's
a very new technology.
However it's been adopted by many many users.
You can think of Sparke as a flexible alternative to map reduce Sparke can run on top of existing Hadoop
distributed file system infrastructure to provide enhanced and additional functionality.
However SPARC can also use data stored in a variety of formats such as Cassandra Amazon Web Services
as Three-Eyes storage DFS as they just mentioned and more.
Let's talk about the differences between spark and map reduce.
We should really be looking at SPARC as an alternative to map reduce rather than a replacement for Hadoop
.
It's not intended to replace Hadoop but to provide a comprehensive and unified solution to manage different
big data use cases and requirements map reduce indifference to spark requires files to be stored specifically
into Hadoop distribution file system SPARC while it can run on top of HD offense does not actually require
those files to be stored in that manner.
More importantly SPARC can also perform operations up to 100 times faster than say map redos.
You may be wondering how does it actually achieve this huge increase in speed.
Well Mabry this writes most of its data to disk after each map and reduce operation spark on the other
hand keeps most the data in memory after each transformation spark can spill over the disk if the memory
is filled.
Basically the difference here as an abstract point of view is that map reduce has to write things to
disk or your hard drive all Sparke on the other hand can keep most of that data in memory or your RAM
so you can imagine that keeping it in RAM versus having to continually write to disk is much faster
.
At the core of SPARC is the idea of the resilient distributed data set or already resilient distributed
datasets already these have four main features they have a distributed collection of the data they are
fault tolerant.
They support parallel operations so they partition that data and they have the ability to use many data
sources.
As I mentioned earlier just sweet discussed in the last lecture about distributed systems SPARC operates
the same way you have a driver program which operates a spark context that communicates a fake cluster
manager which then communicates the worker nodes which actually execute those various tasks or whatever
computations you ask on that distributed data.
Here you can see we have the ardy the objects themselves.
And then you perform a bunch transformation actions on those objects.
You have the DHB scheduler which stands for the directed a cyclic graph and then you have a task scheduler
and then you actually have the worker nodes themselves and just in a general sense Sparke allows programmers
to develop complex multistep data pipelines using this directed acyclic graph pattern.
And it also supports in-memory data sharing across these DA GS so that different jobs can work the same
data.
As far as our work of SPARC in Python we're really only going to be concerned with these already D objects
although on the left hand side everything else will basically occur under the hood for us resilient
distributed data sets are immutable easily evaluated and cashable and there's basically two types of
RTT operations transformations and actions.
And this is the core of what we're going to be doing as we code with Python and Spahr will be coding
up transformations and actions on some sort of distributed dataset.
The basic actions we're going to take a look at is the first collect count and take actions collect
is going to return all the elements of the R D D as an array at the driver program.
Count is going to return a number of elements in the RTD.
First will return the first element in the RTT and take we'll return an array of the first and elements
of the RTT.
So you pass an end as an argument into take and basically these four actions are going to look like
methods were actually programming of Python on whatever Ardi the objects we create then we're going
to have transformations such as filter map and flat map ardy the filter applies a function to each element
and returns elements that evaluate to TRUE.
Just like Python's built in filter function ardy the map transforms each element and preserves the same
number of elements.
This is a really similar idea to those that apply you map some sort of function to every element in
the RTD already the flat map is going to transform each element into zero through and elements and will
actually probably change the number of elements from your original already the two after he performed
the flat map operation.
Sometimes there's some confusion between map versus flat map.
Let me go ahead and give you two examples to try to clarify this.
And don't worry if you don't understand yet you're going to show a lot more examples and we actually
program with PI SPARC.
But for instance a map operation could be something like this.
You could have an R D D with a list of names.
And then you want to map an operation that grabs the first letter of each name that way.
Before and After the map operation you're going to have the same number of elements one first letter
for each name elements.
In example a flat map could be transforming a corpus of text into a list of words.
So for instance in your first r d d each element is a line of text and then you can use that map to
actually convert that into a list of words.
Now each element is actually a singular word.
So you're going to have more elements after that flat map operation and again don't worry we're going
to show a lot more examples when we're actually programming of python and SPARC.
Now let's discuss where are the these often are the these will be holding their values in tuples which
will look something like this.
They'll have a key and a value.
This usually offers better partitioning of data and it leads to functionality based off of a reduction
.
So then we're going to have two other methods to use reduce and then reduce bike key reduce is going
to perform an action that will aggregate our DD elements using a function that returns a single elements
reduced by key is an action that will aggregate pair RDD elements using a function that returns a pair
RTT.
These ideas are really similar to a group by operation.
So I want you to keep that in mind when we actually perform the reduce and reduce by key operations
.
When we get to the coding section Okay.
One last note SPARC has continually developed and new releases come out really often.
The spark ecosystem now includes things such as SPARC sequel SPARC data frames and the Lib Grafx and
Sparke streaming.
So the spark ecosystem is expanding really quickly and there's a lot of exciting stuff happening in
it now that we've learned enough to get started.
What we're going to go ahead and do is show you how to set up an Amazon Web Services account to get
spark up and running on the web.
We also have a text article lecture for some other options in case you don't want to use Amazon Web
Services and would prefer to use Sparke on your local computer.
The reason we're going to show you how to get everything set up on Amazon Web Services is because the
whole idea of SPARC is that you can no longer use a local machine.
You need to use some sort of distributed set of machines that you access or some sort of cloud service
and Amazon Web services is by far the most popular cloud service.
If you're just using spark on a local computer you may as well just use something like sequel or just
Pancho's like we've already done throughout the course.
OK.
So coming up next is just a text article lecture with some other options in case you don't want to use
Amazon Web Services.
But after that we'll dive right back into the video lectures showing you how to set up an account and
get spark up and running.
It's a pretty complicated process so make sure to follow step by step in order to get everything set
up correctly.
Thanks everyone and I'll see you at the next lecture