Hey, Cloud Gurus,
welcome to our last video on Meeting the Tools of the Trade,
finishing up with Apache Spark.
In this lesson,
we'll have a bit of an overview and then we'll go straight
to demo and wrap everything up with a review.
This is a super, high-level look at using Apache Spark
in Azure Data Factory,
mostly focused with getting you familiar on how to access
the Apache Spark Transformations,
and be ready for exam questions around them.
To begin with,
let's talk a bit about Apache Spark in Azure.
This activity executes a Spark program on either your own
or an on-demand HDInsight cluster.
And if you remember a few videos back,
when we were looking at Azure Data Factory,
you could see the HDInsight Spark activity listed there.
One of its advantages is that
Spark jobs are more extensible,
allowing you to provide multiple files,
such as Python scripts, and jar packages.
It expects a certain folder structure in the Blob storage
referenced by the HDInsight linked service,
and so once you have all of that set up at runtime,
Data Factory will be able to use them if properly placed.
And so this really allows you to have very customized
transformations using one or more scripts,
and one or more jar files to support them.
With that brief intro, let's jump straight into the action
and see it in the portal.
Here we are back in Azure Data Factory Studio,
in the same Data Factory we've been working with
in other videos.
I have just a basic, blank pipeline opened up here.
And as you can see,
we have our HDInsight category.
Opening that up, we have our Spark activity.
Within here, we can specify how we want to handle
the HDI cluster.
If we have a link service specified,
we can certainly choose that, but we can create new,
and we have the option of bringing
our own HDInsight cluster,
if you've already got one set up, or using an on-demand one.
As you can see, we specify our Azure storage linked service,
and this is where the needed directory structure will be,
starting with the top-level Spark folder,
and then having your scripts
and jar package folders under that.
I won't take you through all of the options,
it will be very specific to each situation,
but just be aware that this is where you specify
which cluster type you want,
and the associated storage account.
I'll cancel out and discard those changes.
The other thing I wanted to show you is here
under the script/jar heading.
Here we give the specifics of our script's
name and location,
the link service will default to the one for the cluster,
unless you want to specify a different one,
and of course you provide the file path,
and any additional scripts.
Once you have everything set up, validated, and published,
you can trigger this just like any other pipeline activity
and the output will be stored
in that same Azure Blob storage account,
within that same Spark directory that you created earlier.
By way of review, the Spark activity in Azure Data Factory
allows you to execute Apache Spark programs
on either your own or on demand HDInsight cluster.
Spark job flexibility means you can provide
multiple dependencies, such as jar packages,
Python scripts, and other files.
I hope that you have enjoyed this brief look
at using Apache Spark Transformations,
this concludes our introduction of the tools of the trade.
You now have several options that you can use
in your arsenal for data ingestion and transformation.
From here, we'll move on in the next video
to creating data pipelines.
When you're ready, I'll see you there.