A frequent question the web analyst has to answer is “what is in my site?” If you have access to the developer or the site’s server, you can easily create an outline of the site’s content.
Often, however, you’ll be limited to analyzing the site from a visitor’s standpoint. That’s where spider software comes in.
There are many good spiders available, but the most reliable and helpful is “Xenu Link Sleuth.” Despite the odd name, Xenu is a solid tool that spiders the whole site and creates useful reports for site structure and errors.
Tip: the report in HTML can take a long time to write and load in a browser (for large sites). I usually skip this step, and export as csv instead. Then I manipulate the csv results in Excel.