Hey, what's up Gurus.
In this lesson, we are going to be talking about
implementing version control for pipeline artifacts.
And in order to do that,
we need to talk about GitHub.
And so I'm going to provide you a very basic introduction
to GitHub and CI/CD.
However, CI/CD and GitHub,
we're going to talk about those at the 20,000 foot level,
because they're not really on the DP-203.
I mean, they kind of are
because we're talking about version control,
but just a little bit.
So we're not going to go very deep into quality assurance,
CI/CD, GitHub.
We're just going to talk in broad strokes
about what they actually are.
Then we're going to talk about source control
and Azure Data Factory,
and I'm going to talk to you about using GitHub
in Azure Data Factory, and the best way to do that
is to show it to you in the portal.
So we'll hop into the portal
and take a look at how everything functions.
With that, let's jump into the lesson.
So, start off with CI/CD.
Why do we care about CI/CD, and what even is CI/CD?
The thought process behind CI/CD
is to take small changes and continually push them
through the system rather than wait, store up,
and then do one massive change all at once.
That's the thought process behind CI/CD.
So CI/CD is important because it helps you to decide
how code is built, tested, and released.
It allows you to reduce downtime,
increases your efficiency,
and allows for parallel development.
How does it do that?
Well, through branches and commits.
And of course, again,
we're talking about this in very broad strokes,
but at the base level, branches and commits
are one way that CI/CD helps you
to do parallel development
and increases your efficiency, and et cetera.
So let's take a look at this picture over here on the right.
And what we can see here is we have a master branch
is what that's called. That's the blue circles.
And then we have 2 other branches.
We have an orange branch,
and then we have that green branch down there at the bottom.
So let's say that we are implementing some new code.
And we implement the new code,
and as soon as we've implemented the new code,
someone says, hey, you know what we really need?
It'd be really great if we had some orange cats
that just popped up on the screen
when you opened up your browser.
Okay.
So they sent you on the task of getting those orange cats
to pop up on the browser.
That may take a while.
So that represents our orange circles up there at the top.
That's you working through the changes
that need to happen to the code
in order to get those orange cats to pop up.
Well, while that's going on, that blue branch,
hey, it's still going.
And there's still some basic things that you need to do
in order to keep the code base moving forward
and error-free and et cetera, et cetera.
And so each one of those orange bubbles or blue bubbles
represents basically a save.
So we can work on our orange cats.
We can continue to iteratively work on that
until we're done,
those orange bubbles.
And once we're done,
we can request a pull.
Now that is this line right here.
And that is basically us saying hey, we're done.
We want to pull our work back into another branch.
So in this case,
we want to take our work and pull it
into this master branch.
And we want to combine those two together
once we're sure that everything is safe
and that our cats aren't going to break
the rest of the application.
So this is essentially how CI/CD branch and commits work
and GitHub is the repository that we use
to make that happen.
Again, keep in mind,
this is an extremely high-level look at CI/CD and GitHub.
The goal is just to get you to understand
a few of the basics of kind of what's happening here,
and then the real work is actually going to happen now,
as we start to talk about source control and Data Factory.
So where do we use this?
Well, for the DP-203, the answer is Data Factory.
Of course, there's other places we can use GitHub
and we can use CI/CD,
but for the DP-203,
you're going to use it in Data Factory.
The advantages again, source control, performance,
collaboration, and integration.
Those are really the reasons that we want to do CI/CD,
and we want to use source control.
Source control being those branches, right.
We're going to have the master branch,
and then we're going to build additional branches
in order to work on multiple projects
in parallel at the same time,
and then pull everything into that master
once we're sure that everything is safe.
So let's jump in and take a look at this in Data Factory.
So I'm going to switch over here to the portal.
The first thing we need to see
is under Creating a Data Factory.
So if I went in to start a brand new data factory,
I have an option to configure a GitHub.
So I can come in here and I can choose
and put all of my information in for my repository.
So I click on GitHub,
I put my repo name in,
I put the branch in that I'm working on,
all of the information as I create my Data Factory,
and then I can use that repository in Data Factory.
So now let's jump over and look in Data Factory Studio
by clicking here.
And first, we're going to go down to Manage,
and we're going to see here under Git Configuration
that this starts off in the master branch.
So if I do work in Data Factory right now,
it's going to save it to that master branch.
But if you remember, I am working on orange cats,
so I can click on this,
and I can choose my orange cat branch.
Hey, there we go.
Let's go ahead and discard changes,
and now we're in our orange cat branch.
So anything that I do now
is going to go to that orange cat branch.
So I can come in here and I can create a new pipeline.
So let's just start.
We're not going to build the whole thing,
but let's just say that we were working on our pipeline here
under this orange cat branch.
And we finished all of our work.
We've saved everything,
and we are ready to pull it into our master branch.
Well, I can click on that
and I can create a pull request,
which we talked about that,
and then I can choose to pull into our master branch.
So here's my orange cat moving into the master branch
and it'll look and compare any changes
between my version and the master version.
And we can talk about what kinds of things
we would need to do before we were comfortable
and ready to do that,
understanding that there's not going to be any bad effects.
Alright, so with that, I'm actually going to stop,
jump back over into the lesson,
and we are going to stop talking about this.
There's tons of things that we could talk about
with CI/CD and GitHub,
but for the DP-203, which is our focus,
we only need to understand those basics.
So just remember that.
So it's just the basics for CI/CD,
source control being really important,
which is why I spent so much time focusing on branches
and how that works.
And you need to understand the basic steps.
And most importantly,
you need to understand that everything lives
in Data Factory, at least for the DP-203
when we talk about CI/CD and version control.
Alright, that is it for this lesson.
I will see you in the next.