Let's take a look at testing methodologies.
When we look at testing, there's a number of different tests we should do.
Obviously, you test the completed product, but we start the testing
long before then. We start by testing the individual units or modules
of code to make sure each one of those works.
This is very often where we can do white box or
static application security tests.
We can take a look at the code,
we can see whether or not there's problems in the way the code was actually
written, and that will reveal to us any problems there, hopefully, in logic,
for example. We can also do integrated tests.
We bring the modules together and see if data flows correctly between
them, and the processes are working in what we often would call a black
box or dynamic application security test.
This is something we often do when all we get is an executable of a
program. We don't see the source code. We can't read it.
All we can just do is test for functionality.
We put data in and see if we get the right result that comes out, for
example. Both of these are important types of tests. We want to examine
in where we can, but in the end, we want to make sure that we have the
required functionality as well.
Another type of test that's been gaining in popularity is what we call
IAST, or interactive application security testing.
We try different types of inputs and so on and
behaviors to see how the program works.
It's important when we have a program that's being modified that we make
sure we haven't overwritten earlier changes and we haven't regressed back
and reintroduced problems. And we've seen even very large companies do that
on an, unfortunately, semi‑regular basis.
They reintroduce a problem that they had fixed months earlier because the
development of this new release forgot to include some of the things that
were being done while that new release was being written, and these, of
course, are known as regression tests.
The other thing is we should do what we call sociability testing. How
will this product work in its operational environment?
I could write a really good program that just doesn't work because
we're not dealing with sophisticated users, or it's the old example.
I could have a piece of machinery that is very well built, but it has
to operate in a muddy area, and it just quite simply can't operate in
that environment.
And so our programs, we should do the same thing.
How will they work in the real world of heat and moisture and maybe, for
example, a number of users that are not really that computer literate, for
example? One of the things that can help us build a good product is to use
good products when we build the product.
We, for example, know that many of our applications today are built
of a number of different elements coming together.
And if we have a way to know that these elements,
many of which were then written or developed by a different
company, actually have been tested to make sure they have proper
security, then hopefully I know that I'll have that security in
my finished product as well.
I should not build a high quality product and include some
inferior quality pieces or elements of it.
And for this, the ISO 15408 is a very good standard.
It's known as the common criteria for the evaluation of our IT systems, and
it tests whether or not the various system components actually have
adequate and reliable security in them, according to what the vendor has
stated they are putting into that product.
So it tests the functionality of this program in relation to the rigor of
the test. So that, for example, if I want to have a very high level of
trust, I'm going to do a much more rigorous test then if it's something I
just want to make sure it works.
So a high level such as an evaluated assurance level of seven would have
a very in‑depth examination of the product, whereas something with only
evaluated assurance level of one is going to have a much more
superficial functionality type of test.
Now, all programs should meet certain security functional requirements, 11
major families of areas that get looked at. And how does the program fail, for
example? How easy is to maintain? And these are things that we'll look for when
we examine a product according to the common criteria. And the vendor will list
for us the security target of how they have intended to meet those security
functional requirements.
We then have seven levels of assurance.
The evaluation assurance levels go from the lowest level of one up to
the highest level of seven. Other types of tests we frequently will
perform, especially against a cloud‑based application, is a
vulnerability assessment and a penetration test.
In fact,
in many cases, a vulnerability assessment is one of the
precursors to a penetration test.
A vulnerability assessment will look for any vulnerabilities in the
entire program and then with a penetration test will try to exploit one
or more of those possible vulnerabilities.
This gives us the indication of whether or not there are weaknesses in the
program that could be exploited by an outside hacker.
In many cases,
a penetration test is a type of black box because the person who is
conducting it doesn't see how a program works.
Instead, they're testing for weaknesses in the functionality of the program.
There's some really good tools we have to help us with
guidance on how to test applications, for example,
from the payment card industry,
the Payment Application Data Security Standard says, here's all
the things you should test for. Make sure that this program does
not have any of these flaws in it.
We have from the CSA the Treacherous Twelve, a list of
common things that have been problems in applications we
should make sure we don't have.
NIST has a number of very good testing documents, and one of
them, a standard on this, especially looking at technical types
of tests, is the NIST SP800‑115.
That is a little more technical than the SP800‑53A,
which also tests security controls.
We've mentioned OWASP a few times, and the Open Web Application Security
Project comes out every few years, the list of some of the critical web
application security risks. We would be maybe a little bold in saying this,
but it would be almost inexcusable for any web application today to have one
of these security risks because they're well known, they're documented. And
any web applications should have been tested specifically for these before it
ever saw production.
The number one is broken access control, something we stressed
especially back in the last module. But other things, and this
one has gained in its ranking over the last few years, and that
is cryptographic failures.
People put in cryptography, but unfortunately they
don't put it in in a secure way.
Keys are visible, for example, or it's a poor choice of random number
generators, for example. The other number one flaw used to be injection
flaws, whether or not it was, should we say, operating system command
line injection or SQL injection.
It's dropped down to number three now, but the other is insecure design.
This system wasn't designed to operate in a secure,
reliable way. Security misconfiguration, the
controls were there, but misconfigured.
And we just talked about the common criteria using tested components. And
this is number six, the use of vulnerable and outdated components such as
APIs, for example, that are just not secure enough.
There's also the problem of identification and authentication
failures not being granular enough, and this is very much
related actually to number one as well, broken access control,
but not being able to identify,
especially when one program calls other programs or
processes, problems with software, data integrity failures,
problems with security,
logging, and monitoring. And of course, finally number
10, server‑side request forgery.
So these are the 10 that are very well described in the OWASP Top 10
document that we should test for and examine to see if any of our
programs have these vulnerabilities or security risks.
There are many benefits to doing cloud‑based development.
It's a versatile environment that we can set up pretty much to be almost
identical, at least similar, to a production environment.
We can have teams working together from all over the world
and less problems with things like passing modules back and
forth and version control. Better,
we can say here, therefore, for collaboration. And of
course, related to that are some of the risks,
both with development and testing.
The thing is that if I have a team working from all over the world,
it could be that I'm disclosing some of the sensitive data.
My testing data, for example, could have protected health
information or personal information. And I have to watch,
for example, if I'm using cryptography,
whether or not that cryptographic algorithm is allowed to
be exported to another jurisdiction.
So we do have to watch out for jurisdictional issues.
We also have to look for who has ownership of the code in the end.
If I've worked with several other companies and
subcontractors to develop a program, do they own it?
Do they have the rights to use it themselves after they're done? And of course,
if I bought a program that I don't have the source code for,
do I have access to that source code, say, in escrow so that
if I do need to maintain it in the future,
I would be able to if I could not work out an agreement with the
person who sold me that software? One of the things we should make
sure is always addressed in any contract is jurisdiction so that if
there's any type of contractual dispute,
will this case then be adjudicated in my country or another country?
I want to know because there are different legal systems, and
certainly I don't want to find that my case is going to be heard in
a country that has a completely different legal system than I have
if there was to be a dispute.
The other thing is to have good control over change management.
We've seen many programs fail quite simply because of scope creep.
There's been all kinds of changes made that now all of a sudden
we didn't have the budget or the schedule for, for example. It
could well be that a change is being made that we don't even know
who asked for that change.
So we should have a very formal change management process.
And this is a little more complex when we're dealing with maybe
different development teams from all over the world.
The key points review. Testing should be required. It should be a
requirement that all programs have been adequately tested before
they are able to go into production.
It's not an option or something that, well,
if we have time. The test should be thorough.
They should be thorough enough to provide adequate assurance that this
program is going to work in a hostile environment, and good testing
has to think as creatively as a hacker.
A hacker, for example, well, if they can't get in here,
they'll try something else. They'll try this. And so when we do testing,
we have to be just as creative.
So we can try to think of all the possible ways that maybe a hacker
would try to get in and find a vulnerability.