Website Reconnaissance and Footprinting, in this video,
we're going to be exploring the process of performing
a passive reconnaissance or passive information
gathering on a website.
And, of course, we're going to be exploring
footprinting a little bit.
What is footprinting?
Footprinting is essentially the same as reconnaissance.
The only difference is, we're identifying
more important information that's
pertinent to a particular target.
So again, in this case, we'll be taking a look
at a couple of websites.
One of them is my own.
I'm not giving you permission to do anything to it.
As I said, from a passive perspective,
you can go ahead and use this site.
But we'll be using others that you can also use.
In my case, you do have the permission
to do passive reconnaissance on the site, so no problem there.
So with regards to what we are going to be doing
or the information we are looking for,
this is it right over here.
So the first bit of information we are going to look for
is the IP address of the actual website or the web server
that's hosting that website, any directories that
might be hidden from search engines, any names
on the website, email addresses, phone numbers,
physical addresses, anything that
can give us a better understanding of our target,
and, of course, in the case of the website itself,
the web technologies being used on the website.
So we can actually get started with this.
I'm just going to switch over to my Kali Linux virtual machine.
As I said, this lab or this course
will not really use a lot of lab environments
because we're doing this, especially
in the case of passive information gathering,
we're doing this on the internet.
So I'm just going to switch over to my Kali VM
and we can actually get started.
So give me a moment.
All right so I'm back on my Kali VM.
And as you can see, I've just opened up Firefox here.
And in this case, we're going to be
utilizing my own website or my own blog,
which is hackersploit.org.
So we'll just open this up.
And for the sake of this video, this
is going to be the actual target website that we're
going to be performing passive reconnaissance on
or passive information gathering on.
So you can see, this is the website here.
This is a blog.
It's not really a company website.
But there we are.
We can see that it looks fairly simple enough.
But we really don't know anything about this site.
So what is our first order of business?
Well, the first thing we can do is
we can obtain the IP address of the web server
that's hosting this website.
Now, in some cases, or in most cases,
when you're dealing with production sites,
the website might be behind a proxy or a firewall
like Cloudflare or Sucuri.
So in order to do this, just open up a terminal.
And we're going to be utilizing the host command.
Now,if you want to learn more about a command on Linux,
you can always utilize the what is command and then type
the name of the command, so in this case, the host command.
You can see that this is a DNS lookup utility.
Now, we'll be covering DNS lookups when we get there.
But for now, all we are doing is we
are trying to resolve the domain name to an IP address.
So whenever you open up your browser
and you type in a website, the browser
really doesn't know what to do with that information.
And this is where DNS comes into place.
So whenever you type in a website into your browser,
your browser says, OK, I have this website.
I need to resolve it to an IP address
because that's really the only way I can communicate
with that web server because remember, on the internet,
systems are identified via the IP addresses
and not via domain names.
Domain names were essentially set
up so that we don't have to remember IP addresses.
Just imagine if you had to remember the IP address
of every website you visit.
So that's why they were created.
So whenever you type in an address in your browser,
your address will perform a lookup on your configured DNS
server and it'll tell it essentially,
hey, I have this website.
Could you please tell me the IP address of this website?
It'll look through.
And it'll say, OK, I have this website here or this domain.
It is mapped to the following IP address.
And here you go.
Here's the IP address.
Your browser then sends a get request.
And the server then responds and sends
back the website in HTML, CSS format.
And your browser then renders it to you.
And that happens within a couple of seconds,
which is really cool.
So I can type in host and hackersploit.org.
And you can see that will perform a DNS lookup.
And in this case, it tells us that hackersploit.org
has two addresses, two IPv4 addresses,
which is a little bit strange.
And the reason for that is because my website, as it were,
is behind Cloudflare, which is a firewall slash proxy.
So it's not giving us one IP, which,
again, can be quite confusing.
So if you ever see that, just know from that point
that you're dealing with some form of proxy.
Now, the host command also displays the IPv6 addresses
for that particular web server.
And in this case, as I said, because it's
going through Cloudflare, we have that there.
We also get the actual mail server.
And it provides us with the actual domain
for that mail server.
We'll be touching upon that as we proceed
or as we move along in this course.
But there we are.
We have identified the IP address of the actual target
website.
Now, on the website itself, there's
a couple of other checks that we can perform with regards
to identifying information.
As I said, one of the important things
is to try and see if we can identify any names
or any email addresses.
In this case, at the bottom, we can
see that we have a Facebook link, a Twitter
link, and a YouTube channel.
So that can be quite useful.
However, when we talk about scouring a website
or looking for information on a website,
the best place to start is the robots.txt file.
Now, I'll explain what this file is used for in a second.
But if we click on it, you can see
that it is a text file that contains a few entries.
Now, when you have a website, it's
pretty common that a search engine
like Google or Bing or DuckDuckGo
will crawl the website.
What does crawling mean?
It essentially scours the website
and then it indexes it on google.com
so that when someone searches hackersploit,
that website is brought up.
So when search engines do this, they
could potentially be revealing information that you do not
want to be made public.
And this is where the robots.txt file comes into place.
Almost every website has it.
If a website doesn't have it, then they're
really doing something wrong because it essentially
allows you to specify what folders
or what files you don't want search engines to index.
So in this case, you can see that it specifies the we have
a disallowed rule here.
So what does that mean?
It means that we are telling search engines, whenever you're
crawling through this website, please disregard
the following directory.
Now, in this case, the directory is WP Admin.
Now WP is in reference to WordPress.
So we have already identified that this website
is running WordPress.
And why would you not want this folder or this directory
leaked to or made available on Google?
Well, that's because this is a restricted directory.
This is the WordPress admin directory.
And it's only restricted to administrators.
And generally speaking, it's good security practice
to prevent this from being indexed
by a search engine like Google.
Now, just by viewing this file, which is publicly accessible,
it needs to be so that Google can actually go through it
and disallow the directories or the files that
have been specified by the website administrator.
We can identify that this website is running WordPress
because that directory tells us so.
And we have a couple of other directories right over here
that, essentially, are in reference
to a specific WordPress plugin.
So that's the first file.
The second one that's very important
is the sitemap.xml file.
So I'll type in sitemap.xml.
Now, this is typically called sitemap or sitemaps.xml.
So I'll just give this a couple of seconds.
There we are.
So what's a sitemap?
A sitemap is essentially a file, it's
typically XML, that is essentially
used to provide search engines, really, with an organized
way of indexing the website.
So you can see because this is a blog,
there's a post sitemap, a page sitemap, a category sitemap,
and an author the sitemap.
So already from this, you can pretty much
tell what type of information we can obtain.
So if we wanted to view authors--
and again, this is just used to tell search engines more
about the website itself.-- so in this case,
the author site map tells Google or any other search engine
about the authors on this blog.
In this case, we can see that the blog has
an author called hackersploit.
All right, that really doesn't tell us much.
It doesn't give us any concrete names or email addresses.
We then have the page site map, which
will give us a list of pages that can be publicly accessible
or that can be indexed by a search engine like Google.
So this is very useful because if a particular page was not
linked on the main website or the front end of a website,
then this could tell us or provide us with an idea
as to some of the other links or some of the other pages
that can be accessed publicly.
So we can see that we have all of these pages on the site.
And, of course, this really doesn't make any sense.
As I said, this is my own personal blog.
But you're starting to get an understanding.
We then have the category sitemap,
which, again, in the case of a blog,
just displays the list of categories.
This is very, very useful when you're performing a penetration
test on a WordPress website.
Always look out for categories because there
could be some categories that are hidden on the front end.
And then, of course, we have the post sitemap,
which just gives Google or search engines a list of posts
on that particular website.
So yeah, that is the robots.txt file and the sitemap file.
Now, there's a couple of other really cool resources
that I recommend taking a look at.
I'm just going to zoom out here.
And these are browser plug-ins.
So I'm currently using Firefox.
There are equivalents for Google Chrome or Chromium.
So I'll just perform a Google search for Mozilla add-ons
like so.
And I'll just open up the add-ons page here.
And the first one we're going to look for
is a add-on called BuiltWith.
This is a technology profiler that will essentially tell you
what is running on that website or what
web technology or content management system
is running on that website.
So I'll just add it to Firefox here.
There we are.
Add that there.
And we can head back over into my blog.
And I'll just refresh that there.
And we'll give this a couple of seconds.
There we go.
And we click on the BuiltWith icon here,
you can see that it will give you,
essentially, various categories.
But it'll tell you what's running on this website.
So let's take a look at the detailed view.
So you can see that this website has Google Analytics, which
means the analytics are being aggregated and analyzed
on the Google Analytics dashboard.
It also tells us that this site is
using a few widgets like Yoast, Mailchimp, WordPress plugins.
So we can pretty much tell that the site is running WordPress.
It's pretty self-explanatory by this point.
So this is one of the really cool things about BuiltWith.
In the context of content management systems,
it actually tells us what widgets
or what plugins are installed on this WordPress site.
And then, of course, if we take a look
at some of the other information,
we can see that these are the web technologies or web
frameworks being used.
So we can see that this website utilizes jQuery CDN or content
delivery network.
There's a YouTube plugin.
Let's see what else we have here.
These are the subdomains here, which I'll get to in a second.
This is very, very important.
But it essentially gives you a list
of subdomains associated with this particular domain.
We'll get into sub-domain enumeration later.
But you can take a look at BuiltWith
to learn more about a target site.
We can also perform this on ine.com,
just to have a different site to run this on as well.
So I'll click on BuiltWith here.
And if we take a look at the actual Detailed tab here,
you can see that my free lookups have ended.
So no problem there.
But if we click on this here, you can see,
it just gives you an idea as to what technologies
are being used on this site, fairly simple to understand.
So it just gives you a better idea
as to what website you're dealing with.
So yeah, that is how to use BuiltWith.
There are a couple of other add-ons that you can use,
one of which is, let me see if I can find it here,
is Wappalyzer.
Wappalyzer's also another web technology profiler.
So as you can see, it identifies technologies on websites.
So I'll just add that there.
And we'll give that a couple of seconds.
There we go.
And we'll go back to my blog here.
And I'll just click on that there.
And you can see, I'll just click on OK.
And in this case, it tells us that no technologies
were detected.
That's really weird.
So I'm just going to refresh that.
Hopefully this works here.
I think this requires me to actually sign in.
But this does, in certain cases, display the web technologies
being used.
Either way, you can use BuiltWith or Wappalyzer.
I personally prefer utilizing BuiltWith
because it works quite a lot.
And of course, you can also log in
and sign up for a subscription.
Another really, really cool utility
that you can use to perform web technology footprinting
for a particular website is the WhatWeb utility.
So WhatWeb, which comes prepackaged with Kali,
so there we are.
WhatWeb is fairly simple to use.
We can, in this case, perform a stealthy scan.
But it's very simple.
We just type in WhatWeb and hackersploit.org or the domain
that you're trying to check.
So I'll give this a couple of seconds.
As you can see here, it's going to provide you
with a somewhat comprehensive view of the actual web
technologies on the site.
So it's not going to display it very neatly.
But I'll just go through this here.
So you can see that there is a redirect, which
is where it tells you 301 moved permanently.
The HTTP server is going through Cloudflare,
then provides us with the IP.
The redirect is completed because it's
going through a proxy.
Remember that.
We then have the actual get request here.
So this was the redirect because it's going through Cloudflare.
So you can see that it's using jQuery, Modernizr.
These are the web frameworks, and then of course PHP.
This is very important.
It tells us the version of PHP that's
running on the web server, and then, of course, the script.
In this case, that really isn't useful.
It provides us with the actual site title,
and then, of course, the HTTP headers.
So these are all the headers that
are passed with each request or with each response, which
may give you an idea as to what WordPress plug-ins are being
used.
And then, of course, it tells us that it's running WordPress
and it's powered by PHP 7.4.29.
So you can always utilize WhatWeb because it's free.
And if you don't want to go through
or if you don't utilize any of the built-in, browser-based
technology profilers, then you can always rely on WhatWeb.
Now, the final technique that I'll
be highlighting or covering here is
the process of downloading an entire website.
All right, so what does this mean?
Well, let's say you are analyzing website
and maybe you want to analyze the source code of a website.
You can actually download the entire website
as is through the use of a tool called HTTrack or HT Track,
however you want to pronounce it.
So you can see, it's called Website Copier.
This is fairly simple to understand.
It allows you to download a website
from the internet to a local directory, building
recursively, all the directories,
getting HTML images and other files from the server
onto your computer.
This works on both Windows and Linux.
And you can install it on Linux by installing the actual Web
HTTrack command or the installing the Web HTTrack
package, so sudo apt-get install webhttrack.
I believe I already have it installed.
I'll just provide my password for the Kali user.
And if I open up my Start menu here,
I can just search for it, so Web HTTrack Website Copier.
And I'll just open this up.
This will open up the web instance.
So it's going to start the server on port 8080
on your local host or on your Kali Linux system.
My language preference is English.
And then next, you can specify an existing project name.
Or in this case, I can just type in hackersploit,
although I may want to run this on another website.
But we'll just do it on hackersploit here.
Project category, we don't have one.
We can save this under /home/kali/websites.
That's perfectly fine.
And then next, we can download the website.
We can also get individual files,
download all the sites in pages, or we can just
go for the default, which is download the website.
So we can add the URL, so hackersploit.org.
If there is authentication, you can provide a username
and password.
So I'll just hit OK.
Or you can also specify a list of URLs
if you want to download multiple websites.
And hit next.
And in this case, we can hit start.
All right, so it's going to begin downloading the website.
Now, in my case, you can see that, in this case, it failed.
And of course, I knew that it was going to fail because--
or maybe it didn't fail.
We can actually check the directory
because I'm not sure whether the proxy would allow that.
So if I click on websites, you can
see we have hackersploit.org.
If we click on that there, yeah, it
doesn't look like it's saved it or downloaded it successfully.
Just trying to see if I have a website that we can actually
perform this on, although we really
don't need to go into that.
This can be very useful if you're
trying to analyze the source code of a website to learn more
about the website, whether you're
trying to identify vulnerabilities
or you're trying to get an understanding
of the actual structure of the website.
So that is going to conclude the practical demonstration
site of this video.
All right, so we've been able to perform passive reconnaissance
and footprinting on a website.
We've taken a look at the various tools
that can be used to do it.
In the next video, we're going to be taking a look at how
to enumerate whois information.
So this will essentially entail into learning more
about a particular domain name, who
it was purchased by, who owns it, when it expires, et cetera.
So I'll be seeing you in the next video.