In my previous poston robots.txt, I have explained what is robots.txt and how usingrobots.txt controls the search engine spiders . Here’s a quick recap on what is meant by robots.txt. Robots.txt is a text file, which is used to keep a complete control over the crawling of your blog.Using robots.txt you can allow particular bots to crawl your blog and disallow some other bots.
Duplicate content is a killer issue for WordPress blog and if you solve this issue, there are better chances that your blog or site can get ranked higher in search engines.
What are the Common Duplicate Content Issues on a WordPress blog?
The common duplicate content issues in a WordPress blog are listed below:
- Same content in your Index pages and post pages.
- Same content in your Category pages and post pages.
- Same content in www and non-www version of your blog.
- Permalinks with and without trailing slash.
- Duplicate content in post feed.
Thanks to Freakitude for providing this details.
Using robots.txt you can avoid the above said WordPress duplicate content issue.
How to avoid wordpress duplicate content issue using robots.txt?
To avoid duplicate content issue follow the robots.txt file give below and make adjustments to your robots.txt file similar to this structure.
This robots.txt file will disallow bots to crawl your admin folder, feeds, trackbacks, comments, pages and comments. You can also check my robots.txt file for your reference.
If you find this tutorial useful, give us a thumbs up.