Inspecting HTTP Response Headers and Sitemaps
We can also search server responses for additional information.
There are two types of tools we can use to accomplish this task.
The first type is a proxy, like Burp Suite,
which intercepts requests and responses between a client and a web server,
and the other is the browser’s own Network tool.
We will explore both approaches in this Topic,
but let’s begin by demonstrating the Network tool.
We can launch it from the Firefox Web Developer menu
to review HTTP requests and responses.
This tool shows network activity that occurs after it launches,
so we must refresh the page to display traffic.
We can click on a request to get more details about it.
In this case, we want to inspect response headers.
Response headers are a subset of HTTP headers
that are sent in response to an HTTP request.
The Server header will often reveal at least the name of the web server software.
In many default configurations, it also reveals the version number.
The names or values in the response header often reveal additional information
about the technology stack used by the application.
Sitemaps are another important element we should take into consideration
when enumerating web applications.
Web applications can include sitemap files
to help search engine bots crawl and index their sites.
These files also include directives of which URLs not to crawl
typically sensitive pages or administrative consoles,
which are exactly the sort of pages we are interested in.
Inclusive directives are performed with the sitemaps protocol,
while robots.txt excludes URLs from being crawled.
For example,
we can retrieve the robots.txt file from www.google.com with curl:
Allow and Disallow are directives for web crawlers indicating pages or directories
that “polite” web crawlers may or may not access, respectively.
In most cases, the listed pages and directories may not be interesting,
and some may even be invalid.
Nevertheless, sitemap files should not be overlooked
because they may contain clues about the website layout
or other interesting information,
such as yet-unexplored portions of the target.