How to avoid wordpress duplicate content issue using robots.txt?

by Sathishkumar on June 28, 2010

In my previous poston robots.txt, I have explained what is robots.txt and how usingrobots.txt controls the search engine spiders . Here’s a quick recap on what is meant by robots.txt. Robots.txt is a text file, which is used to keep a complete control over the crawling of your blog.Using robots.txt you can allow particular bots to crawl your blog and disallow some other bots.

avoid duplicate content

Duplicate content is a killer issue for WordPress blog and if you solve this issue, there are better chances that your blog or site can get ranked higher in search engines.

What are the Common Duplicate Content Issues on a WordPress blog?

The common duplicate content issues in a WordPress blog are listed below:

  1. Same content in your Index pages and post pages.
  2. Same content in your Category pages and post pages.
  3. Same content in www and non-www version of your blog.
  4. Permalinks with and without trailing slash.
  5. Duplicate content in post feed.

Thanks to Freakitude for providing this details.

Using robots.txt you can avoid the above said WordPress duplicate content issue.

How to avoid wordpress duplicate content issue using robots.txt?

To avoid duplicate content issue follow the robots.txt file give below and make adjustments to your robots.txt file similar to this structure.

User-agent: *
Disallow: /wp-
Disallow: /feed/
Disallow: /trackback/
Disallow: /comments/feed/
Disallow: /page/
Disallow: /comments/

This robots.txt file will disallow bots to crawl your admin folder, feeds, trackbacks, comments, pages and comments. You can also check my robots.txt file for your reference.

If you find this tutorial useful, give us a thumbs up.

View all posts by
Blog →

Follow Us On

@techiemania facebook

Article by Sathishkumar Varatharajan

Sathishkumar is an Online Marketer and a full-time Blogger.

Sathishkumar Has Written 634 Articles For TechieMania.com

  • Techshares

    thanks to satish for sharing this . i use WP and i don’t know these terms, and i have just in build site map .

  • TechGopal

    Thanxx for nice info…

    • http://www.techiemania.com Sathish

      You are always welcome Gopal

  • http://www.spidermyweb.com raghunath pillai

    please also confirm about categories and labels too….won’t that also result in duplicate content.

Previous post:

Next post: