to Backspace Academy, one area of AWS that has grown rapidly over the last 2
couple of years is out of big data in this lecture I'll go through the big 3
data services that are available with AWS I'll talk about Amazon redshift the 4
data warehouse service Amazon Elastic MapReduce the Hadoop service the elastic 5
search service which provides search engine capabilities the Amazon quick 6
sight which is a relatively new service that has come in for business 7
intelligence reporting of big data I look at machine learning for 8
predictive analysis of data and Amazon Kinesis for analysis of real-time 9
streams the big data services available to us in database and storage are 10
redshift DynamoDB Amazon s3 and of course RDS in analysis we have elastic 11
MapReduce the dupe framework as a service we have elastic search elastic 12
search search engine as a service quick site business intelligence reporting and 13
Amazon machine learning for predictive analysis of data and AWS lambda for 14
compute capacity for analysis of real-time data we have Kinesis streams 15
and we also have third-party applications that can be launched on ec2 16
instances Amazon redshift is a petabyte scale data warehousing service and it is 17
based upon PostgreSQL database engine and it's access to be standard business 18
intelligent reporting tools you can still use it with tableau or bird or 19
whatever you are using to to collect data from a PostgreSQL database you can 20
still use those tools all data is replicated between nodes in a cluster 21
and it is continuously backed up to Amazon s3 with snapshots and those 22
snapshots can be held for 1 to 35 days as is the case with RDS user-initiated 23
DB snapshots are retained upon cluster deletion and it also allows for quick 24
recovery from those snapshots 25
now it's one thing to have a big data warehouse but we need to be able to get 26
our big data that we've got already into that data warehouse so we need to think 27
carefully about migration as Big Data the services that are available for us 28
to use first off there we've got data base snapshots to s3 so we could create 29
an RDS snapshot about about data base and then use that snapshot to create a 30
replica of that data base we have the AWS database migration service and 31
that will allow us to transfer data from a source database through to a target 32
database over a connection we also have the AWS data pipeline and that again 33
will allow us to create a data pipeline job which can transfer and convert it 34
needed data from one data source through to a target database and finally there 35
we have the really the best solution for migrating as large amounts of data and 36
that is to use the AWS snowball and use that snowball device to import to Amazon 37
s3 and again like we explained in the introductory of this course the AWS 38
snowball device you can have it sent out to you on premises you can upload your 39
large amounts of data to that snowball device or devices if you've got even 40
more than the snowball device can handle and then you carry that off to AWS and 41
they will upload that for you into an s3 bucket all you need to worry about then 42
is getting it from your s3 bucket into your target database and that is where 43
the AWS data pipeline comes in so you cannot do that with the AWS data 44
migration service you can't go from s3 to a data base but you can create a job 45
in the data pipeline service to import from s3 into your database the reason we 46
would want to use snowball I've just got a table there even when we're looking at 47
a gigabit connection with high network utilization to transfer a hundred 48
terabyte of data across to AWS would take 12 days 49
so we would want to consider using AWS snowball when we have 60 terabytes or 50
more of data to transfer across when we start looking at lower quality 51
connections that are going to take about a year to upload certainly we want to 52
look at using snowball for two terabyte or more of data elastic MapReduce is a 53
WSS fully managed Hadoop service aperture hoop it's an open source 54
framework for the distributed processing of large data sets across clusters of 55
compute instances the AWS EMR service it provides clusters of ec2 instances that 56
analyze our data and those clusters can be automatically deleted upon task 57
completion so when the job's done we can have those those clusters 58
terminated that's what we want it's not suitable for small data sets and it's 59
not suitable for acid transaction requirements the data processing 60
frameworks available Hadoop MapReduce and Apache spark which simplifies the 61
process of creating MapReduce functions the storage options available to us 62
other Hadoop distributed file system which is ephemeral storage and we have 63
the EMR file system or EMR FS which allows you to access Amazon s3 as a file 64
system for elastic MapReduce and we also have the local file system which is out 65
on our ec2 clusters which is the ec2 instance or AWS elastic search is a 66
fully managed implementation of elastic codes elastic search framework elastic 67
search is a real-time distributed search and analytics engine it is the most 68
popular enterprise search engine it is used by Facebook github Stack Exchange 69
and Quora to name a few so it is a very popular analytics engine and used as a 70
popular search engine you can analyze data from Amazon s3 Amazon Kinesis 71
streams DynamoDB streams your cloud watch logs and cloud trail 72
it's suitable for clearing and searching large amounts of data as is any good 73
search engine and is not but it's not suitable for online transaction 74
processing and for petabytes storage so if you're looking for petabytes storage 75
in an elastic search framework then you should look at managing your that that's 76
that yourself by using an ami and launching it on ec2 instances yourself 77
Amazon quick sight is a relatively new business intelligence reporting service 78
from AWS very good service it provides a very good bi software at one tenth of 79
the cost of traditional software such as tableau it provides superfast it 80
provides a super fast parallel in-memory calculation engine or spice although it 81
is it says there one tenth of the traditional BI software of course there 82
are open source BI tools such as bird that will cost you nothing virtually 83
Amazon machine learning is for predictive analytics and machine 84
learning simplified with visualization tools and wizards so it has some very 85
very complex machine learning algorithms behind it and it and it makes it very 86
easy to use because of the wizards and these tools that are available to 87
visualize what you are at what you are trying to analyze the data sources for 88
input to AML include Amazon s3 redshift and RDS or the my sequel RDS it's very 89
suitable to flag suspicious transactions for forecasting product demand for 90
example for personalizing a play application content on a website for 91
predicting user activity and also for analyzing social media streams it's not 92
suitable for very large data sets and it's not suitable for unsupported 93
learning tasks so there are a library of learning tasks but if they're not 94
supported by the AML then it cannot do that so consider 95
using if you need something else that has more learning tasks and consider 96
using Elastic Map Reduce to run spark and the machine learning library for 97
elastic MapReduce Amazon Kinesis is very good for analyzing real-time data in the 98
format of a stream so data can be put into streams into a Kinesis stream using 99
API calls using one of the software development kits which has Kinesis 100
functions inside it and also with the Animas on Kinesis producing producer 101
library or kpl which is a c++ application that we can use to to 102
channel our data into Kinesis and also the Kinesis agent java application as 103
well once we have a Kinesis stream established we can use the Kinesis 104
client library or KCl to process that process that data in that stream that's 105
available for Java nodejs net Python and is currently being developed from what I 106
understand for Ruby Kinesis firehose can capture transform and load streaming 107
data into Amazon Kinesis analytics Amazon s3 Amazon redshift and also the 108
Amazon Elastic search service so that layer allows for real-time analysis 109
through that Kinesis of firehose of our stream data 110
Kinesis is obviously suitable for real-time data analytics it's suitable 111
for log and data feed intake and processing and for real-time metrics and 112
reporting it's not suitable for small-scale consistent throughput and 113
it's not suitable for long term data storage and a delayed exit it's 114
real-time data as a stream and it's not suitable for long-term data storage so 115
that is that concludes our lecture on Big Data and I'll see you in the next lecture