Pages

Saturday, October 4, 2014

Add Custom Robots.txt File in Blogger

Custom robots.txt file in blogger is most important file in terms of your blog indexing in search engines, Lots of beginner blogger have more doubts and confusion regarding blogger custom robots.txt file so in this post I am going to explain you each and every thing and how to add custom.txt file to blogger blog but before let me tell you that custom robots.txt file controls how search crawlers crawl your blog or website, You can restrict or allow search crawlers to index certain pages or posts in blogger and it also helps to SEO optimize your blogspot blogs Earlier I have written post for adding custom header tags which is also very important So this is another cool trick to have complete control over your blogger blog

Add Custom Robots.txt in Blogger
How To Add Custom Robots.txt in Blogger
Since search engines don't know what all pages to index so by default they index all the posts and pages from your blog including labels which can create duplicate content issue in eyes of Google and other search engines since you tag your blog posts with different labels So it make it more essential for you to add custom robots.txt file to your blogger blog also there are some instances where you want that a particular post or page should not be indexed that thing you can easily implement here So lets see how to add custom robots.txt fle in blogger


What is Custom robot.txt File

By default blogger adds Robots.txt File to your blog which looks like below

User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Allow: /

Sitemap: http://example.blogspot.com/feeds/posts/default?orderby=UPDATED


Let me first explain the terms included in it since it is very important to know what are you doing to your blog and if you add anything wrong in this then your blog rankings can significantly reduced or may be completely igonored by search engines so its very essential to know what these terms means


1) User-agent: Mediaparters-Google - It is used for Google media partners like Google adsense and it helps Google adsense to serve more relevant ads on your blog


2) Disallow: - It restricts search crawlers to restrict certain posts and pages for example if you don't want to index and crawl certain post then use below syntax


Disallow: /yyyy/mm/post-url.html


And to restrict particular pages from being crawled and indexed by search engines


Disallow: /p/page-url.html


You can restrict any no of posts and pages by doing this for making your job easy simply copy the URL of posts or pages and paste here without your blog domain name for eg If I want to about me page to not be accessed by search crawlers then I will use the below syntax


 Disallow: /p/about-me.html


 3) User-agent: * - It is applicable to all robots and crawlers


4) Disallow: /search - It is used to restrict those URL which contain the keyword search after your domain name like your label pages and search pages 


5) Allow: / - It is used for homepage of blog and it allows search crawlers to crawl and index our blog homepage


So now that you know what exactly is there in Custom Robots.txt File so carefully add only those elements which is required and now lets see how to add Custom Robots.txt File in your blogger blog

Adding Custom Robots.txt File In Blogger

 

1) Login to your blogger dashboard and go to Settings then Search preferences

2) Under Crawlers and Indexing section you will see Custom Robots.txt by default it is disabled so click on Edit link

Add Custom Robots.txt in Blogger
Add Custom Robots.txt File Blogger

3) Now select the Yes radio button and add the below code in the input box

User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Allow: /

Sitemap: http://domainname.com/atom.xml?redirect=false&start-index=1&max-results=500

 
Replace domainname.com with your domain name and note this sitemap works for first 500 posts only if you have more than 500 posts then again add the line as below

Sitemap: http://domainname.com/atom.xml?redirect=false&start-index=500&max-results=1000

Add Custom Robots.txt Blogger
Add Custom Robots.txt File in Blogger


4) After adding the above code click on Save changes and you are done


So friends this was how to add custom Robots.txt File in blogger and if you like this post please share with your friends and don't forget to subscribe to get more such interesting blogger tips and tricks in future straight to your inbox ! Keep Blogging
Read more ...