Some search engines won’t look beyond the home page when indexing your site. Others will delve a little deeper but still won’t index the entire site. It can be really frustrating when your best content goes un-spidered.
To give your content the best chance of being indexed, it’s good practice to make every page in your site accessible within three clicks (preferably two) of the home page. If you bury your content deeper, then it may not be found.
A great way to do this is to employ a HTML Site Map page that contains links to all pages in your website. As long as your Site Map is accessible from your website’s home page, then your pages should be discoverable by the search engines.
Another alternative is to publish an XML sitemap (I use XML-Sitemaps.com to build these on the server) of your content and point to the sitemap using Google Webmaster Tools, Windows Live Search Webmaster Centre and Yahoo! Site Explorer. This will ensure that information about content is ‘pushed’ to the major search engines, giving it a better chance of discovery when the search engine bots and spiders come crawling. Be sure to update your sitemap regularly, particularly if you run a content site that is frequently updated.
