What is Robots.txt and how to optimize WordPress Robot.txt File For Search engine Bots

WordPress robots.txt file plays a very important role in Wordpress SEO and search engine raking. It also help you to block search engine bots to crawl and index non-important part of blog. But if you configure Robot.txt in wrong way then it can remove your site from search engine results also. So make sure when you write or change Robots.txt file, you should not block and important page of your blog and its well optimized.

What is Robots.txt file and how to optimize for WordPress blog : eAskme

Other people are reading : How to Optimize Post title For Search Engines and for Readers and Search Engines

WordPress robots file

Usually people get confuse about how to use Robots.txt, so we will answer it here.

Robots.txt is one of the hundreds of factors of SEO. By adding this small file you can seriously optimize your Website. usually blogger don`t try to edit it, but its an easy process. You just need to knwo some basic and you can easily edit Robots file, if you are new to this concept then read this post throughly.

• Image optimization

Learn how to add Robots.txt file and how to optimize Robots.txt file.

What is WordPress Robots.txt and why to use it?

Robots.txt file always help search engines to know which part of site they should crawl and which part they should avoid. When Search engine spider  of bot comes to your blog to index your blog, first of all they follow Robots.txt file.

• OnPage Optimization

You can easily find Robots.txt in root file of WordPress installation.

http://www.eAskme.com/robots.txt

How to create robots.txt file?

Robots.txt file is a text file. You can simply create this file on notepad and then upload to root directory for your site. Now let`s discuss records of Robots.txt:

User-agent: googlebot

Disallow: /cgi-bin

These above lines shows that Google is allowed here to crawl and index site but But it is not allowing cgi-bin folder for indexing. This means Goggle will never index /cgi-bin directory.

Disallow option help you to stop any search engine to indexing party of your site. Usually sites use Disallow for archive folders and searches.

Where Can You Get names of Search bot?

You can easily find names of search bots on your site directory, But I recommend that you should allow every search bot. for this purpose you should write User-agent: *. Let`s see an example:

User-agent: *

Disallow: /cgi-bin

This allow all search engines to index your site.

• What is Anchor Text

What You Should not do on Robots.txt file?

Never use comments in Robots.txt file.

Never make ordinary space in file and never keep space in the beginning of any line. Now see some good Robots.txt and bad Robots.txt practices. Example:

• Never change rules of command.

• Don`t write names of directories together to disallow:

• Use small and capital letters properly. 
• To index everything of your site:

• No to index anything of from your site:

Now upload your Robots.txt file in root folder of your site.

Robots.Txt for WordPress:

On wordpress you can use Robots meta or you can edit Robots.txt file by logging in FTP account. You should also add your sitemap URL in robots.txt file. It will help search engines to find your sitemap and index faster.

See below example:

• What is backlink

How to check no content is affected by new Robots.txt file?

After making changes in Robots.txt file now you need to check the impact by updating robots.txt file. Use Google Webmaster tool  "Fetch as bot tool" to check if you can access your content with Robots.txt or not.

Also check for the crawl errors under Crawl error.  Select Restricted by Robots.txt to see what are the links denied by Robots.txt on your blog.

Final Words:

So these are the tips to optimize your Robots.txt file. do share how you add more insight to Robots.txt file? and How Are you optimizing WordPress Robots.txt? If you have any question, feel free to ask me in comments. Don’t forget to subscribe to our e-mail newsletter to keep receiving more SEO tips.