Could you imagine what it would be like if you could make Google bow to your every whim and desire? What if you could be at the top of the search engines for every search term that you wanted to be on top for - without spending a dime on PPC? You could rule the business and informational world! Alas, we all bow down to Mother Google and Queen Yahoo!, but there are some ways that we can direct the search giants (including Bing) in how they view your website and what functions to perform while visiting. This article will discuss the fine points of your robots.txt file, your robots meta tag, and even the new nofollow HTML attribute.

1. Don't use the Whitehouse.gov's website as an example for your robots.txt file

Now-a-days, if you don't disallow the search engines from indexing your website, they will default to indexing it. There's no real need to "allow" Google, Yahoo!, or Bing to access your website - the will by default.

If, however, you wish to keep the robots off of your website (maybe during development, a redesign, or whatever), use the following in your robots.txt file:

User-agent: Googlebot
Disallow: /

This will block Google from indexing your entire website.

To make all of the search engines blocked:

User-agent: *
Disallow: /

Now, in some cases you'll want to block the search engines from indexing only a specific directory or subdirectory on your website. Here's what that'll look like in your robots.txt file:

User-agent: *
Disallow: /yoursubdirectorynamehere

Make sense?

Now, here's a hint - if I were you, I'd make folders on my server before the development of my application that were for "allow" and "no_allow" folders. This doesn't necessarily benefit the search engines, but the alternative is having a 200 line robots.txt file that mirrors the stupid robots.txt file found on the previous Whitehouse.gov robots.txt file. (http://web.archive.org/web/20070217205444/www.whitehouse.gov/robots.txt)

2. Disallowing a cached version of your website

If you don't want the search engines to send a searcher to the search engine's stored cached version of your website, you would use the following meta tag:

Generally, webmasters use this in cases in which a page is constantly updated and refreshed, and the previous copy that might have been stores on the search engine is no longer applicable to the searcher.

3. Controlling Snippets
I bet you don't know what a snippet is, do you? A snippet is the couple of lines of words that are displayed under your website's title on the search results. Sometimes it comes from your meta description tag, but often times, it comes from a snippet of your page that Google deems relevant. It's been proven that when Google produces these snippets, people actually click your link more often because they see how relevant the website will be.

However, there are times when you simply want Google to obey - and to display exactly what is contained in your search engine results snippet. For example, if you are a newspaper and your website is updated several times a day, Google may not have time to re-index your website, and so your snippets would be detrimental to your click through rates.

In that case, here's what meta tag would look like: