Lecturer: Okay, so we are going to practice
using Amazon S3 batch operations,
which is the left hand side.
But first we need to create a bucket.
So to create a bucket, I'll just go to
S3-batch-demo-stephane,
and I will create in this region,
then I will go ahead and click on Create Bucket.
Okay, this is good.
And next I'm going to just create another bucket
for the reporting.
So this one will be called s3-batch-reporting-stephane,
and we're good to go.
Okay, so we have two buckets now being created
and I'm going to open them in new tabs,
one and two,
and next we're going to create a batch operation.
Next one I'm going to do is I'm going to upload a few files
into my S3 buckets,
and I'm going to make sure that I upload them
being not encrypted.
So I'll upload my index, error.html,
coffee and beach.jpeg.
So four files are going to be uploaded in this bucket
and I'm going to perform the upload right now.
Okay, so the upload has succeeded.
And if you look at one of these files,
for example, beach.jpeg,
and we look at the encryption, as we can see
there is no encryption applied at all to this file.
So the goal is going to be,
to do a batch operation on S3
to encrypt all these four files, okay?
So to do so, I need to set up a batch operation now.
Just know that a batch operation will cost you
about 25 cents, it is not free to run.
So if you don't want to incur any cost,
please do not do what I'm doing,
but I'm going to show you how to create a job right now.
So create a job and you have to choose a region,
which is a region where your files are.
So this is the same as the region of this bucket,
so we're good to go.
Next, we need to choose a manifest.
So a Manifest is a way to reference your files
in your buckets, okay?
And to tell them which ones are going to be considered by
the S3 batch operation.
And there are the two ways of doing it.
You can use an S3 inventory report
and by innovating an S3 inventory
you get a manifest .json file.
And by referencing the manifest.json file,
you're good to go.
Or you can create your own csv to have two or three columns
in a following orders, bucket name, object key,
and optionally version ID.
Now to keep this simple,
I'm going to create a csv, okay?
But the more industrial way of doing things is obviously to
use a manifest file to using S3 inventory reports, okay?
So let's go ahead and create the csv.
So I'm going to create a file in S3Batch.csv,
and it has to contain two columns, okay?
As it contain the bucket name and the object key.
So super easy.
The bucket name is what I have right here.
So it's s3-batch-demo-stephane.
And then the object key is just a one right here.
So I will have s3-batch-demo-stephane
and then beach.jpeg.
And then the other files are coffee.jpeg
So coffee.jpeg.
And then we have index.html and error.html.
So here we go.
I have now referenced all my files within my batch.
Perfect.
So next, what I'm going to do is that I'm going to
upload this csv manifest, okay?
Directly in my bucket as well.
So let's upload this file,
Add a File
and then I will navigate to it.
So my S3Batch.csv is right here.
I will upload it.
It's successful.
It's right here.
So now I can find the manifest objects
by giving the entire format.
So s3:// and then the bucket name.
So this one, right here, slash and then S3Batch.csv.
Okay. So we can click on the view to see if that works.
Yes, my file is being found.
Okay.
Next I'm going to click on Next
and then we just choose an Operation type, okay?
So we have Copy, Invoke Lambda function,
Replace all object tags,
Delete all object tags,
Replace access control list,
Restore, if you want it to restore some file from Glacier.
Object Lock Retention or Object Lock legal hold.
So I'm going to do a copy operation, okay?
To do a copy operation on all objects listed in the manifest
and the copy destination could be a new bucket, for example.
So it must be in the EU Frankfurt region,
but you can use the same bucket that I had from before.
So we can for example, use the same buckets, okay?
And we're going to say s3 and then the bucket name,
slash and then we're going to do slash encrypted, okay?
And if you add a slash to the prefix,
it's going to appear as an extra folder in the S3 console.
Now if we do have the buckets
and we don't enable versioning, okay?
In that case, what's going to happen is that
the existing objects will be overwritten.
So if I didn't have the slash encrypted folder,
then it would have encrypted my objects in place.
So you would have replaced my objects in here.
So it's a warning that we know about, but we're good to go.
So we'll say, yes, I acknowledge this and this is good.
But if you wanted to encrypt the objects in place,
you should enable versioning to really make sure that you're
not overwriting your own objects, okay.
Next to the storage class.
So on top of copying the object somewhere we can specify
a different storage class, but I'll keep it as standard.
And here we can enable Server-side encryption,
so I'll enable it using Amazon S3 key,
so Amazon S3 (SSE-S3).
Okay. We'll going to copy of the existing tags,
copy the existing metadata.
So we can see a lot of options
and we can change the ACL's,
but for now we will not do it
and then click on Next.
Okay. So this is the description of this operation,
this is the priority.
And then we want to generate a Completion Report,
for all tasks.
And this Completion Report should go into a bucket.
It's going to be s3-batch-reporting-stephane.
So we're doing this, so s3-batch-reporting-stephane, okay?
And then I can just have a folder named reporting.
Okay, next we need permissions.
So this S3 batch job is going to have to
operate on my files
in this buckets to get them and then encrypt them.
And it is going to have to also send a report
to these buckets.
And so therefore I need to create IAM role for it.
Now there is an IAM role template
and IAM trust policy that we can use to make sure
that this is going to work.
So from my experience,
I will put this to be as permissive as possible right now,
because I've had issues in the past with this template.
So what I'm going to do is just go into the IAM,
and I'm not applying the least privilege because
this is not something I want to do right now.
But this template right here starts to give you the lead
onto how to do a least privilege.
But I want to make sure that my operations
are going to work.
So I'm going to give myself a lot of privilege.
So this is going to be a trusted entity of type S3.
So this one is S3, and this is an S3 Batch Operation.
And by specifying this, we are effectively applying this
IAM trust policy.
Next, we need to create a policy for it in json form
and based this one.
So let's have a look at how we can maybe modify this.
So we have an S3:PutObject and so on for this buckets.
And so we'll just have this S3 star in here,
just to allow anything to be done on my
s3-batch-demo-buckets by this role, okay?
And here, this source buckets in here
should be the same as the S3-batch-demo.
So I do not need this block anymore.
And this is to get the object s3-batch.csv.
So this is good enough.
I will keep it, even though this is again
covered by this statement right here,
that is very permissive.
And then finally for the reporting,
to put objects and get bucket location of this,
this is good enough as well.
So come Next:Tags.
Next and the policy is going to be s3-batch-demo-stephane.
Okay.
We create this policy
and now we need you to apply this policy in here.
So I'll have it S3Batch, and then I will refresh this.
And the s3-batch-demo-stephane is right here.
So I click on the Next: Tags,
Next: Review.
And I'll it DemoS3BatchRole,
create this role.
And now this role is created,
I can probably look for it in here.
So we'll choose some existing IAM role
and I will look for S3Batch demo role, here we go.
Next, and so now we can review the jobs.
So this is the path to the manifest file
to operate on a few files.
And then this is your opportunity to introduce a copy
operation into this encrypted folder.
So we're not doing in place where moving into a new folder
and we have server-side encryption
that's going to be enabled for my files.
So remember this is 25 cents if you enable this.
So just so you know, so I've created a job right here.
And as you can see, its Status New.
And what we need to do is click on it
and we need to validate this job.
So I will show you what I mean right now.
So if you refresh,
you see the Status is Awaiting your confirmation to run.
So once we've enabled it, okay,
it says that it has found four objects in the manifest.
And so it needs to get our confirmation to go ahead.
So I will scroll all the way down.
Oops, I will go up and click on Run Job.
Now the job is going to run and the encrypt operations is
going to happen for my full files.
So we get a Status and we get a percentage complete
and a percentage succeeded and a percentage fail.
So we can really track the progress of our S3 batch job,
okay?
But the cool thing about S3 batch in here,
is that it is going to give us a lot of information
around what is happening on our files.
So somehow that is not being run.
So let's do Run Job.
Oh, I need to scroll all the way down now, here we go.
And then click on Run Job.
This is why I wanted to scroll all the way down.
Okay, so now the job is ready
and I will wait until it is done.
So I just refreshed my page and the status is completing
and percentage complete is 100%,
total succeeded rate is four.
And total failed rate is zero.
So very cool. If I go to my bucket now and refresh this,
as we can see there's an encrypted folder right here
available to us.
And within it, we can find our full files.
Okay, but if we click on one of these files
and we have a look at the encryption status,
as we can see service-side encryption is enabled
for this file with Amazon SSE-S3 encryption,
which is really, really nice,
which is what we wanted.
And if we go into the batch reporting buckets
and we refresh this,
we can find a reporting folder with the job ID
and in it, we have a manifest file and a result that json,
and this results, sorry, that csv
is going to give us information around all the files
that have been processed and if it was successful or not.
And so, as you can see by opening this csv file
what we get is the bucket name,
the file name,
then whether or not it's succeeded,
the status code and successful.
And in case you had some error messages,
they would show up in here as well.
So that's it for S3 batches, as we can see,
we can find all the jobs here and it is completed.
So hopefully you see the power of it,
especially when you start running S3 batch on a lot of files
at the time, okay?
So I hope you liked this lecture and I will see you in the
next lecture.