Some search engines won’t look beyond the home page when indexing your site.  Others will delve a little deeper but still won’t index the entire site.   It can be really frustrating when your best content goes un-spidered.

To give your content  the best chance of being indexed, it’s good practice to make every page in your site accessible within three clicks (preferably two) of the home page.   If you bury your content deeper, then it may not be found.

A great way to do this is to employ a HTML Site Map page that contains links to all pages in your website.  As long as your Site Map is accessible from your website’s home page, then your pages should be discoverable by the search engines.

Another alternative is to publish an XML sitemap (I use XML-Sitemaps.com to build these on the server) of your content and point to the sitemap using Google Webmaster Tools, Windows Live Search Webmaster Centre and Yahoo! Site Explorer.  This will ensure that information about content is ‘pushed’ to the major search engines, giving it a better chance of discovery when the search engine bots and spiders come crawling.  Be sure to update your sitemap regularly, particularly if you run a content site that is frequently updated.

Google is the king of the search engines. So when Google offers you a set of webmaster tools to assist you with your listings in Google, you would be silly not to use them, wouldn’t you?

With Google Webmaster Central and Webmaster Tools you can learn how Google ranks websites, how to submit all of your content to Google and how to improve traffic to your site.

Webmaster Central can be found at http://www.google.com/webmasters/. There are a bunch of excellent resources for every webmaster on this page.

Google’s Webmaster Tools can be found at Webmaster Central. These tools can assist you with statistics, diagnostics and management of Google’s indexing of your website, including sitemap submission and reporting.

To get started you will need to specify your website URL and then verify that you are the owner of that site (you will be required to add some code or a file to your website for recognition by Google). From there, you can diagnose how well your site is being indexed by Google, view statistics, learn which sites link to yours, get information about keywords used to find your site, specify sitemaps, and generate a robots.txt indexing file.

Sitemaps are relatively new to the web world. These are XML files that reside on the top level of your domain and which contain a listing of all pages in your site. These files make life easier for Google’s indexing spiders because they tell the spiders where to look and what to index. They are definitely worth setting up and there are a number of tools that can assist you to do so.   Try XML Sitemaps Generator, a server based generator that can be set up to auto-run periodically on your host.

Similarly, the robots.txt file sits at the top level of your website and directs Google’s spiders to indexable content. Google has a robots.txt generator tool here – https://www.google.com/webmasters/tools/robotsgen.

Windows Live Search has recently launched a Webmaster Tools portal, to match similar tools offered by Google and Yahoo!  Make sure that your website is recognised and indexed by Windows Live Search and the NineMSN search engine by making use of these tools today.

Like Google Webmaster Tools and Yahoo! Site Explorer, Windows Live Search Webmaster Tools provides you with indexing statistics, information about keyword searches, inbound/outbound links and your domain rank.

Importantly, it also offers you the opportunity to identify a Sitemap for regular crawling by the Windows Live Search search engine – this is very important if you want to have as many pages appear in the search engine results pages as possible.  You can create a Sitemap using one of the many sitemap tools available on the web – I like XML-Sitemaps.com.

Access Windows Live Search Webmaster Tools here:

http://webmaster.live.com/

Here are some of the things that you should be doing on a regular basis – schedule it into your diary!

• Check for broken links
• Check for redundant or out of date information
• Check your competition – what are they up to?
• Check for site errors via your statistics package and rectify
• Check, record and analyse your site statistics – unique visitors, repeat visitors, page views, page views per session, time on the site, most popular pages, etc.
• Check for web accessibility and standards - does it meet Web Accessibility Initiative and W3C guidelines?
• Check your search engine listings – where do you appear in the search engine results pages?  Has there been an improvement over time?
• Check that all pages have correct metadata tags
• Check that you have an XML sitemap and that the information is being sent to Google Webmaster Tools and Yahoo Site Explorer
• Check your site navigation to ensure that it is logical
• Check any search engine marketing programs (eg Google Adwords) that you are running to make sure that you are getting a good return on investment