Hey, Cloud Gurus. Welcome to our lesson
on Designing and Creating Tests for Data Pipelines.
This type of automation
is something you would typically find
in more of an application development setting,
but we're going to look at how it applies
to creating data pipelines.
First off, we'll talk a little bit about
what automated testing is.
We'll then discuss unit tests versus functional tests
and give a little example of what we're talking about
and wrap everything up with a review.
So when we talk about automated testing,
you might think of little robot minions
going through your code and making sure it's okay,
but that isn't exactly what we mean.
You might say that it's the use of software
to execute your code or pipeline
and compare the results against what was expected.
This is different from exploratory testing,
where a human manually follows a set of instructions
to verify functionality.
So say your developers write out instructions
for how to test their code
and hand it over to your quality and testing department.
Manual testing has its benefits,
but as your code grows and gets more complex,
testing every little facet of the code
can become repetitive and tedious,
and quite honestly, prone to error.
Automated testing helps alleviate some of that
and it frees up your testers to focus more on making sure
the user experience of your software
is what you want it to be.
The less of their time we have to take up the better.
So what types of tests should you run and why?
First we have unit tests,
and this verifies the actions that are taken,
or in other words, is it doing the right things?
In general programming, this might look like
verifying that an individual line of code is executed,
but for our purposes,
the smallest unit of development is a pipeline.
And so you might be writing tests, for instance,
to verify that certain activities ran.
Unit testing is different from functional testing,
which verifies the effects caused,
or in other words, is it doing things right?
And so, whereas unit testing
is verifying the activities ran,
functional testing is making sure they produced
the right result.
Again, things are a little different with our pipelines,
as opposed to a traditional programming environment.
Ideally you would run the unit test
as part of your build process,
and if it failed, your build would not be deployed.
However, with pipelines,
they have to be published first before you can test them.
For this reason,
even though unit and functional tests
are not normally placed together,
with pipelines, we'd put them in the same testing fixture
when testing a specific scenario.
With those definitions in mind,
the workflow might look something like this.
First, you would use Visual Studio
or another integrated development environment
to write the unit and functional tests.
You would then publish your pipeline.
Before the test can be run,
the pipeline must be live and published.
That is unfortunate.
I really wish you could test it first
and only deploy what is successful,
but that's the brakes.
Third, you might perform unit tests,
such as counting activities,
verifying how many activities were executed,
and what status they ended with.
You can absolutely use
stored procedures within your pipelines,
to indicate certain statuses or log certain elements
and have your tests run against those
to keep track of how things went.
And you might have functional tests
to do things like check row counts:
verifying the output of activities
by counting the rows copied or transformed.
By way of review,
Visual Studio, or another integrated development
environment, or IDE can be used to write unit and functional
tests for data pipelines.
Unit tests allow you to verify
that pipelines are doing the right things,
such as that activities have fired as expected
and functional tests allow you to verify
that pipelines are doing things right,
that the correct results were produced.
That's it for this lesson.
I hope this gives you an understanding
of how you can go about designing
and creating tests for data pipelines.
It follows the same testing principles
as many other development situations,
but with a little bit of nuance,
and in the end can help ensure you have more reliable
and high-quality data workflows.
Thanks for joining me.
When you're ready, I'll see you in the next lesson.