{"id":772,"date":"2007-04-17T22:42:45","date_gmt":"2007-04-17T21:42:45","guid":{"rendered":"http:\/\/www.markwilson.co.uk\/blog\/2007\/04\/removing-duplicate-search-engine-content-using-robotstxt.htm"},"modified":"2012-04-02T23:48:24","modified_gmt":"2012-04-02T22:48:24","slug":"removing-duplicate-search-engine-content-using-robotstxt","status":"publish","type":"post","link":"https:\/\/www.markwilson.co.uk\/blog\/2007\/04\/removing-duplicate-search-engine-content-using-robotstxt.htm","title":{"rendered":"Removing duplicate search engine content using robots.txt"},"content":{"rendered":"

Here’s something that no webmaster wants to see:<\/p>\n

\"Screenshot<\/p>\n

It’s part of a screenshot from the Google Webmaster Tools<\/a> that says “[Google] can’t current access your home page because of a robots.txt restriction”. Arghh!<\/p>\n

This came about because, a couple of nights back, I made some changes to the website in order to remove the duplicate content in Google. Google (and other search engines) don’t like duplicate content, so by removing the archive pages, categories, feeds, etc. from their indexes, I ought to be able to reduce the overall number of pages from this site that are listed and at the same time increase the quality of the results (and hopefully my position in the index). Ideally, I can direct the major search engines to only index the home page and individual item pages.<\/p>\n

I based my changes on some information on the web that caused me a few issues – so this is what I did and by following these notes, hopefully others won’t repeat my mistakes; however, there is a caveat – use this advice with care – I’m not responsible for other people’s sites dropping out of the Google index (or other such catastrophes).<\/p>\n

Firstly, I made some changes to the <\/code> section in my WordPress template:<\/p>\n


\n
\n
\n
\n
\n
\n<\/dirtycode><\/p>\n

Because WordPress content is generated dynamically, this tells the search engines which pages should be in, and which should be out, based on the type of page. So, basically, if this is an post page, another single page, or the home page then go for it; otherwise follow the appropriate rule for Google, MSN or other spiders (Yahoo! and Ask will follow the standard robots directive) telling them not to index or archive the page but to follow any links and additionally, for Google not to include any open directory information<\/a>. This was based on advice from askapache.com<\/a> but amended because the default indexing behaviour for spiders is to index<\/code>, follow<\/code> or all<\/code><\/a> so I didn’t need to specify specific rules for Google and MSN as in the original example (but did need something there otherwise the logic reads “if condition is met donothing<\/em> else dosomething<\/em>” and the donothing<\/em> could be problematic) .<\/p>\n

Next, following fiLi’s advice for using robots.txt to avoid content duplication<\/a>, I started to edit my robots.txt file<\/a>. I won’t list the file contents here – suffice to say that the final result is visible on my web server and for those who think that publishing the location of robots.txt is a bad idea (because the contents are effectively a list of places that I don’t want people to go to), then think of it this way: robots.txt is a standard file on many web servers, which by necessity needs to be readable and therefore should not be used for security purposes – that’s what file permissions are for (one useful analogy refers to robots.txt as a “no entry” sign – not a locked door<\/a>)!<\/p>\n

The main changes that I made were to block certain folders:<\/p>\n

Disallow: \/blog\/page
\nDisallow: \/blog\/tags
\nDisallow: \/blog\/wp-admin
\nDisallow: \/blog\/wp-content
\nDisallow: \/blog\/wp-includes
\nDisallow: \/*\/feed
\nDisallow: \/*\/trackback<\/dirtycode><\/p>\n

(the trailing slash is significant – if it is missing then the directory itself is blocked, but if it is present then only the files within the directory are affected, including subdirectories).<\/p>\n

I also blocked certain file extensions:<\/p>\n

Disallow: \/*.css$
\nDisallow: \/*.html$
\nDisallow: \/*.js$
\nDisallow: \/*.ico$
\nDisallow: \/*.opml$
\nDisallow: \/*.php$
\nDisallow: \/*.shtml$
\nDisallow: \/*.xml$<\/dirtycode><\/p>\n

Then, I blocked URLs that include ? except those that end with ?<\/a>:<\/p>\n

Allow: \/*?$
\nDisallow: \/*?<\/dirtycode><\/p>\n

The problem at the head of this post came about because I blocked all .php files using<\/p>\n

Disallow: \/*.php$<\/dirtycode><\/p>\n

As https:\/\/www.markwilson.co.uk\/blog\/ is equivalent to https:\/\/www.markwilson.co.uk\/blog\/index.php then I was effectively stopping spiders from accessing the home page. I’m not sure how to get around that as both URLs are serving the same content, but in a site of about 1500 URLs at the time of writing, I’m not particularly worried about a single duplicate instance (although I would like to know how to work around the issue). I resolved this by explicitly allowing access to index.php (and another important file – sitemaps.xml) using:<\/p>\n

Allow: \/blog\/index.php
\nAllow: \/sitemap.xml<\/dirtycode><\/p>\n

It’s also worth noting that neither wildcards (*<\/code>, ?<\/code>) nor allow<\/code> are valid robots.txt directives and so the file will fail validation<\/a>. After a bit of research I found that the major search engines have each added support for their own enhancements to the robots.txt specification:<\/p>\n