[OPENING MUSIC]
Web server enumeration.
In this video, we are going to be taking a look at how
to utilize various auxiliary modules to enumerate
as much information as possible from a web server.
So the first question that we need
to ask ourselves is, what is a web server?
Well simply put, a web server is a piece of software
that is used to serve website data on the web.
So whenever you purchase a domain
and you, essentially, set up your website
with a hosting company, the directory
under which you store your website files is then
hosted or is served by a web server technology.
An example of a web server is Apache.
You also have Nginx and Microsoft IaaS just
to name a few.
Now, web servers utilize HTTP, which stands for the hypertext
transfer protocol, as the underlying protocol
to facilitate the communication between clients and the web
server.
So whenever you want to access a website,
you'll typically open up your browser and then
type in HTTP or HTTPS and then the domain name,
and you hit Enter.
What happens when you hit Enter is your browser
will perform a DNS lookup in order
to resolve that domain to the web server IP address.
Once it gets the IP address, it'll send an HTTP request
to the web server.
The HTTP, or rather the web server,
will then send back an HTTP response to the browser
and this response will typically include or involve information
like the website data that will then be rendered back to you,
and you'll be able to interact with the website.
So that's how requests are sent between you, the browser,
and the web server.
So again, you send an HTTP request and you get a response.
And based on what you're trying to send,
that is going to, consequently, affect the response.
So if I hit Log In on a website, that request
is sent to the web server.
And whether or not we provided the correct credentials,
the web server will then provide an appropriate response.
So if we provided the correct credentials, for example,
we'll be able to log in.
If we don't, then the web server will, essentially, send back
a response saying, hey, the credentials are incorrect,
you might want to try again.
So let's talk a little bit about the HTTP protocol.
So HTTP is an application layer protocol
that typically utilizes port 80 for communication.
So HTTP uses port 80.
And when you then set up an SSL certificate for encryption,
it then uses port 443.
So irregardless of whether or not
we're dealing with a website with an SSL certificate,
all you need to know is that when you're talking about HTTP,
it's port 80.
When you're talking about HTTPS, it's port 443.
Now, we can utilize multiple auxiliary modules
to enumerate the information like the web server version.
So this information will typically
be what web server is running, the version of the web server
that's running, et cetera.
We can also obtain additional information
like the HTTP headers, and that information
can be used to inde--
furthermore identify what web server technology
is being used, what extensions, or what
programming language or stack is being used
on the web server, et cetera.
We can also perform additional functionality
like the ability to perform directory and file brute
forcing in order to identify hidden files that you may not
be able to find if you were just browsing
the website through your browser.
So we'll take a look at how to do all of this
as we move into the practical section.
And again, all we need to do is just set up the lab environment
and get started.
So I'll see you within the lab environment.
All right, I am back within the lab environment.
And the first thing we need to do
is, of course, identify the target IP address, which
we can by saying ifconfig.
So we just hit Enter, and the interface we're looking for
is ethernet1.
And this is our IP address.
So 192.140.160.2, which means the target IP address is going
to be 192.140.160.3.
And in your case, this will, of course, be different.
All you need to do is just change the two to a three,
and that is the target IP.
So I'm just going to copy this because we will
need this in a few seconds.
The next thing we want to do, of course,
is ensure that the PostgreSQL database server is started.
And we can do that by starting up this service.
So I'm going to say service postgresql.
We hit Enter, that's going to start the PostgreSQL database
server.
We can then launch the Metasploit framework console.
All right, the next thing we need to do
is, of course, set up a workspace.
So I'm going to say workspace, and we'll say add
and we'll just call this a web_enum, because we
are doing web enumeration.
So I'm just going to hit Enter, and there we are.
We have created the workspace, and it's currently active.
And we can finally get started.
So again, the first thing that we
want to do before we actually start
playing with any of the modules is
we need to set up a few global variables
so that we don't need to keep retrieving and obtaining
the target IP address and setting it over again.
So in order to do this, I'm just going
to say set G. So set global, and we'll set the R host value
to the target IP.
So I'm just going to paste in the IP we copied and of course,
change the two into a three.
Hit Enter, and we'll also do this for the RHOST variable.
So we are doing it for both RHOSTS and RHOST.
We hit Enter and that should, again,
save us from having to type in the IP address
every time we use a module.
So that's very, very helpful.
So we can now begin searching for HTTP modules,
because that's the protocol that we are
going to be communicating with.
And we can just hit a search HTTP.
And as you can see, we have quite a few modules here.
And of course, we need to narrow this down.
So I'm going to say search, and then we
can specify the type, which in this case is auxiliary,
and then the name, which we know is HTTP.
Hit Enter, and now that is limited to about 77.
And the one that we want to start out with
is going to be the HTTP version module right over here.
And this module will essentially enumerate
or will print out the version of the web server
that is running on the target system, as you can see.
HTTP version detection, that's what it does.
So again, I'm just going to copy that
and we can then say use, and then paste that in there.
And we can then clear out the terminal and say show options.
There we are.
And the options, as you can see in regards to RHOST variable,
have been set.
The RPORT variable is set to port 80.
So we are currently dealing with a web server that doesn't
have an SSL certificate.
So the R-- we don't need to change
the value of RPORTS to 443.
And of course, if you are dealing
with a website with an SSL certificate,
then you also want to change the SSL option here to true.
So whenever you're dealing with a Boolean setting,
all you need to do is say Set, and then SSL.
And then you want to set that to a default,
so true if you're dealing with a website within an SSL
certificate.
All you need to do is just type in true,
and that will be enabled.
However, in our case, we don't have a website
with an SSL certificate.
So we have everything ready.
So I'm just going to hit Run.
And as you can see, it enumerates the web server
version running on the target as well as the operating system.
So we can see that the target is running
Apache 2.4.18, which is great.
That gives us an idea of what we're dealing with.
And we also know where or on what operating system
this web server is being hosted on.
And in this case, you can see it's a Linux distribution.
And in this case, it's ubu--
in this case, it is Ubuntu.
So now that I've enumerated that information,
we can also perform some additional enumeration
in regards to the HTTP headers.
And that will also give us an idea of the web server version,
as well as some additional information.
And of course, this will all depend
on how the web server has been configured.
If it has been misconfigured to leak information
pertinent to the web server configuration,
then we will get quite a lot of useful information.
So we are going to search for the module and it's just--
the module name is simply called HTTP header.
Let me hit Enter.
And this is the module here, auxiliary scanner HTTP,
HTTP header.
And we're going to say use.
And we can clear that out.
So we're going to say show options.
And now, we need to configure this module.
By default, we only need to work or set the required options.
As you can see here, starting off with the HTTP method,
this simply refers to the HTTP method
that you're using to again get the HTTP headers
from the target's web server.
In this case, as you can see, we can either use HEAD or GET.
In this case, we're just going to leave it
with HEAD because that is appropriate in this case.
As for the IGN header, again, you
can specify a list of headers to ignore separated by the comma.
So you can unset this to get all the information like the date
content, length connection, e-tag, when
the cache expires, et cetera.
You can also specify a proxy.
The RHOST option has been set to the target IP.
Of course, we're not dealing with a web server with an SSL
certificate.
And finally, we can specify the target URI,
which simply refers to any specific directory
within the web server that we want
to get the HTTP headers off.
And in this case, we are targeting
the root of the web server.
So we're going to leave it as is.
And once we're ready to go, just type in Run.
And as you can see here, it gives us the content type,
which tells us it's text HTML.
So that pretty much gives us an indication
of what programming language has been
used to develop the website that is being hosted on this web
server with a few exceptions.
And in this case, it's text HTML,
when it was last modified, and then the server
is printed out here.
So that's similar information to what
we got when we, essentially, printed out the HTTP version
using the HTTP version module.
So again, we really not got any additional information
that's useful.
As I said, you can modify the IGN header or the ignore header
setting here, to, again, essentially display or exclude
specific HTTP headers.
So again, you can go through that description there.
So now, we've been able to enumerate the web server
version that's running on the target system.
Let's perform some additional enumeration
in regards to a few hidden directories
that we may not have access to.
And the way we're going to do this
is by downloading and analyzing the robots.txt file.
Now, if you're not familiar with how the web works
and how websites are configured, the robots.txt file
is a TXT file that stored at the root of a web server
and is used to prevent search engines from indexing
specific directories and files that
are hosted on that web server.
So for example, if I had a website
and I wanted to prevent Google and any other search engines
from indexing a specific directory within that web
server or within my website, I would add it
to the robots.txt file.
So you might be asking yourself, well, why is this important?
Well, this is important because the developers of website
may add directories there that they
don't want anyone else to find or they don't
want search engines to index.
So they could be potentially interesting, all right?
So again, this is a very important piece
of web server enumeration.
And in order to use the module, we need to search for it first.
And the name of the module is robots_txt.
We hit Enter, and as you can see, that's the only module.
So we're going to copy it and we're
going to say use and paste the name there.
And that is currently active.
If I show the options now, you can
see that we the first option we can set is the path.
And this is simply referring to the path on the web server
that you want to essentially find the robots.txt file.
As I said, the robots.txt file is typically
stored at the root of the web server.
So we don't need to change that.
They RHOST option has been set, RPORT has been configured.
We don't have a web server that has an SSL certificate,
so that is set to false.
And we're not dealing with an HTTP server virtual host.
If you are doing that, or if you are
dealing with an HTTP virtual host,
then you can change this value.
And again, modify the current setting.
However, in this case, we're not doing that.
So I'm going to hit Run.
And as you can see, it finds the robots.txt file.
And it says, right over here, contents of robots.txt.
And we get the disallowed directories here.
So we can see that the developer of this website,
for some reason, didn't want anyone
to access the data directory as well
as a directory called secure.
All right, so you can see that's some very interesting
information and some potentially damning information in terms
of exploitation.
Because again, as you can see, one of the directories
here is called data.
So that could potentially be used to store data.
And then the secure directory, we
are not really sure what that is for.
And let's actually find out.
So we can use the curk utility.
So I'm going to say curl, and I'll just copy the IP address
here of the target web server.
And we'll say curl HTTP.
And let's base that in there.
And let's see what's within the data directory.
So what we're doing is we're essentially pulling down
the actual index of that particular directory
to see what's going on there, what this particular directory
does or what it is used for.
So I hit Enter.
And as you can see, it downloads the HTML file.
And it says-- it essentially tells us that this is an index.
So what this means is that this directory has directory listing
enabled.
Now, directory listing is a configuration
that is native to Apache as well as a few other web servers.
And what that means is it essentially
allows you to store files within a directory.
And they can be indexed and accessed within that directory.
So you can actually just have a list of files.
And directory listing is typically
used for download pages.
So if you ever want to host files for download
to the public, you typically have them in a directory,
and then enable directory listing.
So we know that directory listing is enabled for the data
folder, which is interesting.
Let's do that to the secured directory.
Let's see what is actually stored within this directory.
So we hit Enter, and we can see we get a 401 response code,
unauthorized.
And that tells us that this server could not
verify that you are authorized to access
the document requested.
Either you supplied the wrong credentials, for example,
a bad password, or your browser doesn't
understand how to supply the credentials required.
And we get the web server banner right over here.
Now, what this means is that this particular directory
is protected by an authentication form.
So if I were to access this directory within my browser,
it would actually prompt me to Enter a username and a password
in order to gain access to this directory.
Now, within the lab environment, we
don't have access to a web browser.
But what we can do is we can actually try and perform
a brute force on this particular directory
to try and identify the credentials that
can give us access to this secure directory.
So we'll come back to that in a second.
So we've established quite a bit of information,
or we have essentially been able to enumerate
quite a bit of information regarding the target web server
and how it's been configured.
And as you've already seen, we've
been able to identify a secure directory
that we can target in a few seconds.
Before we do that, let's be thorough with our directory
search.
And in order to do that, we're going to perform a directory
brute force.
Now as I said, a directory brute force is essentially a method
of performing a brute force on a web server with the objective
of finding hidden directories that may not be accessed,
or that cannot be accessed, typically.
So it's a very useful way of finding out
what else is being hosted or what else is
being stored on the web server.
So in order to do this, we're going to search for the module.
And the name of the module is directory scanner.
So directory_scanner, and that's the one there.
So I'm going to say copy, and we're
going to say use and paste that in there.
There we are.
Show options.
And you can see that the first option is the dictionary file.
So that's the brute force dictionary
file that's going to be used.
And this file, essentially, contains a list of directories
that this module will perform the brute force with.
So it will go through each line within this text file.
And if we get a match, it'll actually be output here
and it'll tell us, hey, we've got a match for this directory.
So that directory probably exists on the web server.
We can then provide the path.
So again, based on what directory
you want to target on the web server.
So for example, if we wanted to perform a directory
brute force on this secure folder,
then we could specify secure here.
Or maybe the data folder, we can also do that.
But for now, let's just leave it at the root of the web server.
The target IP has been set, target port is set.
We don't have an SSL certificate,
so the only thing left to do is just hit Run.
All right, so it's going to say detecting the error code.
It's going to use the error code 404 not found,
and as you can see, it's already begun finding
various directories here.
So we're going to give that a few seconds.
All right so the module has completed running.
As you can see, it's found quite a few directories
on the web server.
And some of them we were able to find with this module
if we had not analyze the robots.txt file.
So for example, we can see the secure directory as well as
the data directory.
It looks like we also have a downloads directory
and a few other directories here.
So if you're performing a real penetration test,
then I would recommend, again, testing out
all of these URLs within a browser
so that you can confirm whether they exist.
And then, of course, it also gives us
an additional information regarding the directories
and whether or not they require authentication
in order to be accessed.
So for example, in the WebDAV directory,
you can see that it requires a WebDAV
authentication, and the sam for web mail and webdb.
So again, that gives us an idea of what
this web server is hosting.
So again, we can always use Curl to analyze
the contents of these directories
and see what they're storing.
But again, in this case, we already
know that really not much is stored.
This looks like a simple directory listing
for a very basic website.
However, we can also perform file brute forcing, right?
And file brute forcing is essentially
the opposite of directory brute forcing in that,
instead of performing a brute force to find directories,
we're performing a brute force to find files,
which is extremely important.
So in order to do that, we're simply going to say search.
And what we'll do is we'll type in search file, files
directory, underscore directory.
We hit Enter, and that's the module there.
As you can see, the description says
this is an HTTP interesting file scanner.
So again, we'll copy that, and we'll say use.
Paste that in there.
If we say Show Options, we can see that the dictionary
specified here.
By the way, you can modify this to whatever
you're comfortable with.
And again, you can typically find word lists
under the following directory.
So the user share Metasploit framework directory,
or you can also look under the--
if I was to set the dictionary here,
you can also look under the user share word lists directory.
The word list directory on Kali Linux
is where the word lists are stored.
So again, if I use the tab auto completion,
you can see you have various other word lists
here that you can use to perform your brute forcing.
Now, based on the type of brute forcing you're performing,
you, again, will need to use the appropriate word list
or dictionary file.
And in this case, it's utilizing WMAP,
which again is a module for the Metasploit framework
that is used to perform vulnerability scanning on a web
application.
And we'll be exploring it during the web--
we'll be exploring it during the vulnerability scanning section.
For now, we're just going to use that particular dictionary.
We can specify the extensions that we
want to limit our results to.
So if we only wanted to perform a file
brute forcing for PHP files, we can specify the extension
there.
However, in this case, because we really don't
know what extensions or files that we're going to find,
we're just going to leave that as blank.
We're going to specify the path as the root of the web
server, the target IP and port is set,
and the SSL certificate is set to false because the target web
server doesn't have one.
So I'm going to run.
And again, just give that a few seconds to complete.
All right, so the module is completed.
And as you can see, we're able to find
a few files here like code.c, index.html, tests.php, et
cetera, et cetera.
And it also brings-- or actually, outputs
the various directories.
And the reason that it's actually providing us
with a 301 HTTP response code is because it's redirecting.
So again, we can always test this manually.
But we're able to enumerate the list of directories
as well as the files on the web server.
And in this case, apart from tests.php and index.html,
there really isn't any other interesting file here that,
again, can give us an idea of finding--
or rather, give us an entry vector
into exploiting this web server.
But again, as I said, we'll be exploring
the concept of exploiting a web server as we progress.
For now, however, let's move on to targeting
the secure directory because that looks like it
might have something useful.
And in this case, we can see that that's the directory
we want we want to target.
Now, as I mentioned previously, the secure directory
has authentication enabled, which
means we need to provide a username and password
to successfully authenticate.
And because we don't have an idea of what
the credentials are, we will need
to perform a brute force similar to what we did with FTP.
And in order to do this, or what we'll need to do,
is just clear out the terminal here.
And we're just going to search for a module called http_login.
And we hit Enter.
And the one we're looking for is this one here.
This is an auxiliary module scanner, HTTP, HTTP login.
And it is an HTTP login utility, so it essentially
allows us to perform a brute force on an authentication
form.
So I'm going to say use, and we're
going to paste in the module name there.
And we'll show the options.
And we'll hit Enter.
So the first option or setting that we need to configure
is the authURI, or the directory that
has authentication enabled.
So that is the secure directory.
So we want to set that up first.
So I'm going to say set auth_uri,
and then we're going to set that to secure.
We hit Enter.
The next option that we can configure
is the brute force speed.
So again, you might want to tweak this.
And it's currently set to the fastest option.
So it's from zero to five, based on the web service
capability of handling traffic.
So again, you may want to be careful with that.
Again, I'm just going to leave it as the default.
We don't have a database of all credentials.
Instead, we're going to be utilizing a password
file and a user file here.
And again, as for the target IP, that's been set.
The target port has been set.
We don't have an SSL certificate,
so that's set correctly.
You can also change the amount of threads, or rather,
the number of concurrent threads if you
want to speed up the process.
For now, what I want to do is unset
the user pass file, because we really don't need it.
As you can see, this is a file containing
users and passwords separated by a space one per line.
And the reason we don't need it is because we already
have a user file specified and a pass file specified.
So the user file contains a list of usernames
that we're performing the brute force with,
and then the pass file, again, contains a list of passwords
that we're performing the brute force with.
So I'm just going to say unset user pass, user pass file.
And that's going to unset that option.
So we say show options again.
You can see that we have everything configured.
So I'm just going to hit Run, and let's
see whether we're able to find any credentials here
through the brute force.
So I'm going to hit Run.
All right, so it doesn't look like we
were able to find any legitimate credentials.
And the way the Metasploit framework modules work
is that whenever you see this red brackets here
with an hyphen within it, that essentially
means that that is failed.
If you see a blue bracket, and that essentially encapsulating
an asterisk, that is simply a banner or information
that's printed out.
If we do get a successful operation,
it'll be printed in green, or rather, you
should see a green set of brackets beside the results.
So what we want to do is we want to change our user list.
So I'm going to say Show Options.
We're going to change the user file as well as the password
file to something that's a little bit more powerful
or one that's a bit more robust, because it's currently
using HTTP default users.
And again, that might not be a good option for us.
So I'm going to say set user file right over here.
And we are going to use one under the Metasploit framework
directory.
So I'm to say user share Metasploit framework.
And then, we're going to go on the data, word lists,
and we're going to use the namelist.txt file.
So I'm going to hit Enter.
The next option we need to set is the pass file.
So I'm to say set pass file.
And we're going to change that to user share Metasploit
framework data word lists.
And the one we're going to use is,
we'll just use Unix passwords as that one
has been quite successful for us during our previous videos.
So I'm going to hit Enter, and we're just going to hit Run.
And let's see whether we able to enumerate any or find
the legitimate credentials here.
So it's going to go through all username and password
combinations.
And this, of course, will take a while.
And if I just terminate this, I can actually-- again,
let me just show the options here and take you
through one important option that will be very useful,
and that is the VERBOSE option.
We can set this to false to essentially prevent it
from outputting all failed requests
and only displaying the ones that are successful.
So we can say set VERBOSE to false and then hit Run.
And that is going to begin the brute force.
So again, give this a few seconds to a few minutes.
Remember, these user files and password files are quite large,
and it actually has to go through all combinations
of users and passwords.
So we want to give this a few seconds to a few minutes.
All right, so the module has been running for a few minutes,
and it looks like we get an error here for some reason.
All right, so the module has been running for quite a while
now and it hasn't returned any results.
So again, in this case, it might be a far-fetched
to actually get the legitimate credentials.
So we can also try other ways of enumerating users here
in order to identify specific usernames that will, again,
help us narrow down the brute force.
So what we can do is just terminate that there.
And we can search for another module that's
pertinent to Apache, and that is,
if I can just search for it here,
let me just type in the search term.
So that's the Apache user directory enum module,
which is right over here.
And I'll just copy it and let me tell you exactly what it does.
So if I show the options here, you can see--
let me type in info.
You can see that this will, essentially--
again, this module essentially utilizes the mod directory
functional-- or rather, the user directory directive that's
enabled, or rather, that is available with the Apache web
server.
And again, as you can see, Apache with the user directory
directive enables generated different error quotes
when a username exists, and there is no public HTML
directory.
And when the username does not exist,
which could allow remote attackers
to determine valid usernames on the server.
So this will help us identify valid users on the server.
So again, what we can do is specify
the target IP, which is done.
The target URI, which we will just leave it as
is as the root of the web server.
And we can then change the word list, probably,
that we're using here to maybe common users or something
like that.
So I'm going to do that right now.
So I'm going to say Set User, and we'll just
say Ser User file and say user share Metasploit framework data
word lists, and then commonusers.txt.
And then we just hit Enter.
And it looks like with we've been
able to find a few users here.
So for example, as you can see here, users found is rooty.
So we can use that information, now,
to try and narrow down our brute force.
So again, if I search for HTTP login, and we can then use it.
And we're just going to remember the user here,
which is just rooty.
So I'm going to say use and then paste that in there.
Show the options now.
We're going to use the default password list as is,
and we can then specify or set up the user
that we want to utilize.
So in order to do that, we'll need
to create our own user file.
And we can do that by simply saying echo,
and we'll just Enter the username
that we found, which was rooty.
And we'll export that into a file called user.txt.
And we just hit Enter.
And now, we can specify that as the user file.
So I'm going to say set user file, and that
is under the root directory of this system.
And we're just going to say user.txt, hit Enter, and now
let's see whether we're able to find the password for the user
rooty.
So again, just give that a few seconds.
All right, so it looks like we get an error here.
So what I'm going to do is I'm going
to set the verbosity option to true
so that we can see what's going wrong exactly.
So you can see it goes through all the combinations here.
And let's see where we have the error here.
So it probably might be a working password.
However, we can't legitimately log in or utilize it
for whatever reason.
So again, just give this a few seconds.
And it looks like we get an error when we, essentially,
reach one of the following passwords.
So again, in this case, we aren't
able to find the credentials to access the secure directory.
We can probably, again, play around with the various word
lists that we're using and the password
files that we're using, and again,
all I wanted to demonstrate within this particular module
or within this particular video is
that there are multiple auxiliary modules that you
can use and they can be very helpful in different contexts.
Now, all of these modules will work on any web server,
regardless of whether it's Apache, Nginx,
or Microsoft IaaS.
However, you also need to remember
to specify the correct port and enable SSL
if you're targeting a web server with the SSL certificate.
So that's going to conclude the lab demonstration.
In the next video, we're going to be
taking a look at how to perform enumeration on MySQL.
So we're going to be taking a look at how
to extract as much information as possible
from a MySQL database server.
So I'll see you there.