In this lesson,
we are going to talk about shuffling in a pipeline.
And we're going to start off
by talking about distributed tables,
and we're going to take a look at hash distribution
and round-robin distribution.
And then we're going to follow up
by taking a look at how we set partition types
in the portal.
So let's start by talking about table distribution.
There's several different types of table distribution.
There's round-robin, hash, dynamic range,
fixed range, and key.
Now, this is specific to Data Factory
which is the service that we're looking at here.
This is actually an image from Data Factory,
and we'll see this later in the portal.
But we're going to be talking through these partition types,
and it's also important
to talk a little bit about shuffling.
So shuffling is just whenever we need to move data
from point A to point B, or we need to change attributes.
We shuffle that data around,
and the way that we shuffle that data around,
that's our table distribution.
So let's start off by talking about hash distribution.
A hash function is used to assign data to rows.
Think about it kind of like a map.
So, it's the key or the information that you need
to figure out how to get from point A to point B.
When we're talking about data,
it's the hash function
is the information or the key
to help us figure out how to move data
from our starting point,
and then where we're going to put it
as we move data into the new table.
Essentially, it's a fancy math algorithm.
So let's take a look at how this works.
So we have 3 input sources,
and from those input sources, we are going to move our data.
And what we're going to do
is we're going to take those blue boxes there,
that's our input,
and we're going to move it into those 9 boxes
over there on the right.
Now, the purple box represents our hash function.
So as the data moves from the left to the right,
it's going to encounter that purple box.
And that is our math algorithm
that's going to tell us which box over there on the right
we need to move into.
Now, hash functions are really important
because they're going to give you fast querying.
It takes a little longer to load data
into a hash distribution, but it's going to be fast to query
because we know where the data is.
Because of that algorithm,
we know where that data is going to be located.
However, it's going to be take a little bit longer
because the input has to go through that hash
before it can come out and go to the right spot.
The other side of that
is looking at a round-robin distribution.
In a round-robin distribution,
we're going to have first random,
and then sequential movement of our data.
So you can see over there on the right,
we basically we put an X over our hash function.
There is no more hash function,
we're going to randomly assign the data
into those 9 buckets.
Now, when I say random,
we are going to have an even distribution,
so it's not like 1 bucket's going to get all of the data.
It's going to be evenly distributed
across all 9 of those boxes.
However, it's going to be randomly distributed at first.
Now, that is good because we're going to get faster loads,
because we don't have to move through that hash function.
However, it's going to take longer to query
because we don't really know where that data is exactly.
And so it's going to take longer to search through
to get to the data that we need when we're running queries.
So in addition to hash and round-robin,
there's a couple more distribution types.
There is fixed, dynamic, and key.
So fixed range
is just simply: we are going to set number of the partitions
that we want,
and we are going to use a custom expression,
and that custom expression
is going to be determining how that data gets loaded
into those partitions.
Now, dynamic is going to be very similar to that,
except you're going to have the system to determine
the ranges for your data,
rather than it being based upon conditions
that you're setting.
The last one down there is key.
Key distributions is just simply:
you're going to pick a key column,
and it's going to do distribution
based upon that key column.
So with that, let's go ahead and jump into the portal,
and I'll show you how we set distribution.
So here, we find ourselves in Data Factory,
in Data Factory flows, specifically.
And I have set up just a very simple flow;
we have our input source,
and then we have our output or our sync.
If we go to our sync and look down here at the bottom,
there is an Optimize tab,
and you'll see that I can choose the partition type
that I want.
So I just simply choose the current partitioning
or I set the partitioning as round-robin,
or key, or fixed, right?
That's literally it.
When I run this data flow,
it's going to change my partition type
or it's going to use the partition type
that is currently there from the source data.
All right, so with that,
let's go ahead and kind of wrap this lesson up
with a few key points.
You do need to know the basic distribution types,
especially round-robin and hash.
Those are very common across Azure,
so make sure that you know those.
And you need to understand that shuffling
has a very large effect on performance.
When you're moving data around,
that's going to affect the cost,
and that's going to affect your performance.
So don't neglect optimization.
It can make a huge difference
when you set your distribution appropriately.
All right, that's it for this lesson.
I'll see you in the next.