How (and why) to hide your webpages from search engines

There are two main reasons to hide some of your webpages from Search Engine Results Pages (SERPs)

1. You have information that you do not want turning up in search results: private data, family pictures, coupon codes, limited offers, test/demo pages, etc.
2. You have duplicate content: landing pages for e-mail campaigns, highly-templated pages, etc.

Extensive duplicate content will hurt your search engine rankings. Search engines are looking for relevant pages. Duplicate content reduces your relevancy. There are two ways you can fix this:

1. Use the robots.txt exclusion file to tell the search engine spiders crawling your site not to index particular pages or folders
2. Add appropriate metatags to your pages which will then tell search engine spiders to exclude those particular pages from their results

Creating the robots.txt file is generally more efficient than crafting individual metatatags. Information on how to properly create your robots.txt file can be found here. Remember to create only one file for your website.

You may chose to use the metatags if you do not have the ability to upload files to the root of your webserver (for example pages on a shared site like Yahoo! Geocities). The code you should use is:



This code should be pasted into the content of your HTML.