There are multiple tools at your disposal to find web directories and pages that haven’t been indexed by the search engines or cannot be found from a sitemap.
Just as a basic example, if you were enumerating a WordPress site, there’s a good chance of finding a /wp-admin/ directory within the site, even if you cannot find a direct link to it.
So how do you find “private” directories. Well the simplest answer is to try all the common ones. If they load, they exist….if they don’t load or they return some form of error code or message, then they either do not exist (Error 404 ring any bells?) or you are simply not allowed to view them.
In a nutshell, most directory fuzzers or “busters” simply fire off a ton of guesses, and take note of the error codes they return. If they get a positive hit (code 200 for example) then they’ve found a match, and the directory exists. If they get a 404, then they move on.
So what tools do we have at our disposal? Well the five main tools I personally use are dirb, nikto, wfuzz, dirbuster and gobuster. All do similar things, but in slightly different ways. I won’t go into detail about each and every one of them, but will give you an example of how to use dirb.
Firstly, to seek out directories:
# dirb http://xxx.xxx.xxx.xxx wordlists/big.txt | tee dirb_xxx.xxx.xxx.xxx_big.txt
dirb obviously starts dirb, and you then replace the xxx’s with the IP address (or website) you are targeting. You will be using the “big.txt” wordlist to try various common directories. You can then pipe the output to a file with the tee command.