Some search engines won’t look beyond the home page when indexing your site. Others will delve a little deeper but still won’t index the entire site. It can be really frustrating when your best content goes un-spidered.
To give your content the best chance of being indexed, it’s good practice to make every page in your site accessible within three clicks (preferably two) of the home page. If you bury your content deeper, then it may not be found.
A great way to do this is to employ a HTML Site Map page that contains links to all pages in your website. As long as your Site Map is accessible from your website’s home page, then your pages should be discoverable by the search engines.
Another alternative is to publish an XML sitemap (I use XML-Sitemaps.com to build these on the server) of your content and point to the sitemap using Google Webmaster Tools, Windows Live Search Webmaster Centre and Yahoo! Site Explorer. This will ensure that information about content is ‘pushed’ to the major search engines, giving it a better chance of discovery when the search engine bots and spiders come crawling. Be sure to update your sitemap regularly, particularly if you run a content site that is frequently updated.
NineMSN / Windows Live Search is one of the major search engines in Australia – it’s not as big as Google or Yahoo!, but it’s still important. If you don’t currently appear in NineMSN / Windows Live Search, then you can submit your URL to the search engine.
