-: So in this lesson,
we are going to talk about "Designing Metastores".
No, I am not talking about mega stores,
we're going to talk about metastores.
And so we'll start off by talking about
what a metastore is in 5 minutes or less.
And then I'm going to take just a little bit
of time and we'll talk about cluster configuration
and how you select metastores
and get them configured properly.
So with that, let's start talking about what a metastore is.
So when you work in Databricks or Spark,
a metastore is created by default.
Now this metastore holds metadata about our data.
So this could information
like paths to the location to the data, right?
Or it could be formats of the data,
so this is in Parquet or CSV or whatever.
And when you create Databricks or Synapse Spark instances
you're going to have a metastore that's created;
however, those metastores, on their own,
are only accessible within that Databricks workspace
or within that Synapse instance.
So in this lesson, we're going to talk about how
we can take those metastores and we can combine them.
So instead of having our individual instance of Databricks,
or Spark, we can have multiple instances
that are feeding into one giant metastore
that we can then access about all of our workspaces.
Alright, so now let's talk a little bit about
cluster configuration.
So when we talk about cluster configuration
the first thing you need to understand
is this can be done in Synapse or Databricks,
but we you're going to focus on Databricks in this lesson.
There's 2 main configuration points.
There's the metastore version in JAR.
And then there is connecting to that external metastore.
So we'll briefly cover both of those sections now.
So to start off with cluster configuration
you need to understand that there's basically
5 steps to configuring your metastore version
in your JAR.
So we're going to start by creating a cluster
and we need to create that cluster
in order to take our notebook and grab those metastore JARs
and repoint them to the new metastore that we want to use.
Alright, so we're going to create a Databricks runtime
and this is going to be based upon the Databricks runtime
that we're using in the rest of our environment.
Once we have that done,
we are going to figure out the location of our JARs.
And so you can see this script here
and let me actually just jump over
and I'll show you we are in Databricks,
and I'll show you this is our actual same script here.
And when we run that,
it's going to give us the location of our JARs, okay?
So that is what that first line is doing.
This is pulling and giving us the location of our JARs.
Alright.
Now, once you have that done,
you're going to run this command.
And what this command is doing
this is actually going to copy your datastore JARs
and it's going to move them
to a hive/metastore_jar file.
And so that's what this is doing.
And so let's jump over and see that.
So we have this script here and I've replaced the path file
with the path that's located right here.
And then I'm copying that information
over to this hive_metastore_jar directory.
Alright, so that's the third step.
The fourth step is going to be creating an init script.
And this is going to be a little bit specific
to what you're wanting to do.
I've thrown a sample in here.
I'm not actually going to go through this
because it's not all that helpful for the DP-203.
So you just need to remember that you need to create
an init script that's going to copy that metastore JAR
that we just moved, this metastore_jar
and it's going to move it
to the local file system of the node that we're using.
And then finally for 5,
we are going to set our metastore JARs to use that new path.
And so basically for that one,
we are just simply taking that
spark.sql.hive.metastore.jars.
We're taking that and we are pointing it
at the new directory that we created.
So that's going to be our Databricks hive_metastore_jar
that we created in our init script section.
So, alright, I know that that seems like quite a bit
of information and quite a bit of depth
for metastore JARs.
Don't get too far in the weeds on this one.
We really need to understand the basics
of what a metastore JAR is
and the basics of what we're doing
to create and configure it.
We don't need to understand how to create the init script,
at least for the DP-203.
Alright, so let's move on to the next step,
which is connecting to our external metastore.
So what we would do now is we would set
our hive configuration options, and you can see here,
this is just simply a script that we're using
and it's basically just connection strings.
So connection string, username, password,
and then the driver class that we need.
So not a whole lot there, other than understanding
that there is a hive configuration option
that you need to set, and this is what it looks like.
Alright, so let's kind of sum all of this up.
The first part, don't be afraid to read the docs
but don't get lost in the weeds.
So this stuff can get very complicated and get very specific
to instances that you would be working with.
We don't need to know all of that information
for the DP-203.
The DP-203,
we just need to understand, we have cluster
configurations that involve configuring Spark,
which is the downloading the metastore JARs
and pointing them in the right location.
And then we have our hive configuration option
which is just simply telling hive the information
to connect to our external metastore.
Also, don't forget that this is in Databricks
and Synapse Analytics.
So it could be either one.
So again, DP-203,
you need to understand what a metastore is.
You need to understand
that it contains metadata and understand the basics
of how you could create an external metastore
and then point your instance to that new metastore.
Alright, that's it for this lesson
I'll see you in the next.