Hello everyone and welcome to the Big Data overview lecture and this section of the course we're going
to start off with an explanation of Hadoop map reduce Sparc and PI SPARC.
We'll explain what a local system is a distributed system.
Afterwards we'll get an overview of the Hadoop ecosystem then we'll switch gears have a detailed overview
of Sparke and we'll show you how to set up Jupiter Sparc and everything you need on Amazon Web Services
will also give you some resources or other SPARC options in case you want to install SPARC locally or
on some other system.
Afterwards we'll dive right in into the Jupiter notebook with hands on code learning about pi Sparc
and Ardie these and we'll learn about these and what that stands for in the next lecture.
Let's talk about big data in general.
So far we've worked of data that can fit onto a local computer and that's usually in the scale of somewhere
between 0 and 8 gigabytes.
That's just data that can fit directly onto your RAM.
But what can we do if we have a larger set of data.
You could try using some sort of sequel database to move storage onto your hard drive instead of RAM
that will still work on a local computer that if you need to expand beyond what your local machine can
do you can use what's known as a distributed system that distributes the data to multiple machines or
computers.
Just to reiterate a local machine is basically what we've been using.
It's your own computer or a single machine where you're restricted to the RAM or hard drive of that
one machine on a distributed system.
You can have one machine controlling a distribution of multiple machines.
For instance if you have one local machine you may use all the cores on that machine.
However if you leverage the power of a distributed system you can have one smaller master node controlling
a distributed system of computers that allows you to take advantage of this distributed system and actually
retain more processing power and more memory than you would be able to on a single local machine.
And that's what we're going to be discussing as we talk about Hadoop and map reduce as I previously
mentioned a local process will use the computational resources of just the single machine a distributed
process will have access to the computational resources across a number of machines connected through
a network after a certain point.
It's much easier to scale out to many lower CPQ machines than it is to try to scale up to a single machine
with the high CPQ distributed machines also have the advantage of easily scaling.
You can just easily add more machines versus a single local machine.
It's much harder to actually upgrade the Sipa you and you're going to reach a certain limit.
Distributed machines can also include something known as fault tolerance which basically means if one
machine fails the whole network can still go on versus a local machine if that goes down your entire
work goes down.
Now let's go ahead and discuss the typical format of a distributed architecture that uses Hadoop as
a basic overview.
Hadoop is essentially just the way to distribute very large files across multiple machines.
It uses what is known as the Hadoop distributed file system or HFS HFS allows a user to work with large
data sets.
And that also duplicates blocks of data for fault tolerance.
It can then also use what's known as map reduce and map redos allows computations on that data.
Let's go in and take a look at what a distributed storage or HFS system would look like.
You would have your main name node with some CPQ and ram attached to it and then you would have several
other distributed machines as data nodes.
Will use blocks of data usually of a size of 128 megabytes by default.
And each of these blocks is replicated three times and then the blocks are distributed in a way support
fault tolerance meaning if one of these lower data nodes gets knocked out for some reason you have your
data replicated on the other nodes.
So you're not going to lose any information.
Smaller blocks provide more paralyzation during processing multiple copies of a block prevent loss of
data due to a failure of a node.
Now let's talk about map reduce map reduce as a way of splitting a computation task to a distributed
set of files such as the FS.
It consists of a job tracker and multiple task trackers.
The job tracker sends code to run on the task trackers those task trackers allocate CPQ and memory for
the tasks and monitor the tasks on the worker nodes.
What we covered can be thought of in two distinct parts using HFS to distribute large data sets and
then using map reduce to distribute a computational task to a distributed dataset.
Next we're going to be learning about is the latest technology in the space which is known as SPARK
spark improves on the concepts of using distribution.
OK.
Up next we're going to learn about SPARK spark can use Hadoop distributed file system but it has many
advantages to just using map reduce.
Thanks everyone and I'll see you at the next lecture where we learn more about Sparc and how we're going
to use it in this course.