Optimized Robots.txt strategy improves SEO. Blocking unnecessary URLs is one of the most critical steps in this strategy.
Robotx.txt plays an essential role in SEO strategy. Beginners tend to make mistakes when they do not understand the use of Robots.txt on websites.
It is responsible for your website’s crawlability and indexability.
An optimized Robots.txt file can significantly improve your website’s crawling and indexing.
Google also told us to use Robots.txt to block action URLs such as login, signup, checkout, add-to-cark, etc.
But how to do it the right way.
Here is everything!
What is Robots.txt?
The robots.txt file is a code that you place in your website’s root folder. It is responsible for allowing crawlers to crawl your website.
Robots.txt contains 4 critical directives:
- User-agent: It tells that if you allow every crawler or a few targeted crawlers.
- Disallow: Pages you do not want search engines to crawl.
- Allow: Pages or part of the website that you want to allow for crawling.
- Sitemap: your XML sitemap link.
Robots.txt file is case sensitive.
Robots.txt Hierarchy:
Robots.txt should be in an optimized format.
The most common robots.txt order is as follows:
- User-agent: *
- Disallow: /login/
- Allow: /login/registration/
The first line allows search engines to crawl everything.
The second line disallows search bots from crawling login pages or URLs.
The third line allows the registration page to be crawled.
Simple Robots.txt rule:
User-agent: *
Disallow: /login/
Allow: /login/
In this format, the search engine will access the Login URL.
Importance of Robots.txt:
Robots.txt helps optimize your crawl budget. When you block unimportant pages, Googlebot spends its crawl budget only on relevant pages.
Search engines prefer an optimized crawl budget. Robotx.txt makes it possible.
For example, you may have an eCommerce website where check-in, add-to-cart, filter, and category pages do not offer unique value. It is often considered as duplicate content. It is best to save your crawl budget on such pages.
Robots.txt is the best tool for this job.
When You Must Use Robots.txt?
It is always necessary to use Robots.txt on your website.
- Block unnecessary URLs such as categories, filters, internal search, cart, etc.
- Block private pages.
- Block JavaScript.
- Block AI Chatbots and content scrapers.
How to Use Robots.txt to Block Specific Pages?
Block Internal Search Results:
You want to avoid indexing your internal search results. It is pretty easy to block action URLs.
Just go to your robotx.txt file and add the following code:
Disallow: *s=*
This line will disallow search engines from crawling internal search URLs.
Block Custom Navigation:
Custom navigation is a feature that you add to your website for users.
Most e-commerce websites allow users to create “Favorite” lists, which are displayed as navigation in the sidebar.
Users can also create Faceted navigation using sorted lists.
Just go to your robotx.txt file and add the following code:
Disallow: *sortby=*
Disallow: *favorite=*
Disallow: *color=*
Disallow: *price=*
Block Doc/PDF URLs:
Some websites upload documents in PDF or .doc formats.
You do not want them to be crawled by Google.
Here is the code to block doc/pdf URLs:
Disallow: /*.pdf$
Disallow: /*.doc$
Block a Website Directory:
You can also block website directories such as forms.
Disallow: /form/
Block User Accounts:
You do not want to index user pages in search results.
Add this code in Robots.txt:
Disallow: /myaccount/
Block Irrelevant JavaScript:
Add a simple line of code to block non-relevant JavaScript files.
Disallow: /assets/js/pixels.js
Block Scrapers and AI Chatbots:
The Google.com/robots.txt file says that you should block AI chatbots and scrapers.
#ai chatbots
User-agent: anthropic-ai
User-agent: Applebot-Extended
User-agent: Bytespider
User-agent: CCBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: cohere-ai
User-agent: Diffbot
User-agent: FacebookBot
User-agent: GPTBot
User-agent: ImagesiftBot
User-agent: Meta-ExternalAgent
User-agent: Meta-ExternalFetcher
User-agent: Omgilibot
User-agent: PerplexityBot
User-agent: Timpibot
Disallow: /
#scrapers
User-agent: magpie-crawler
User-Agent: omgilibot
User-agent: Node/simplecrawler
User-agent: Scrapy
User-agent: CCBot
User-Agent: omgili
Disallow: /
Allow Sitemap URLs:
Add sitemap URLs to be crawled using robots.txt.
- Sitemap: https://www.newexample.com/sitemap/articlesurl.xml
- Sitemap: https://www.newexample.com/sitemap/newsurl.xml
- Sitemap: https://www.newexample.com/sitemap/videourl.xml
Crawl Delay:
Crawl-delay works only for some search bots other than Google. You can set it to tell the bot to crawl the next page after a specific number of seconds.
Google Search Console Robots.txt Validator
- Go to Google Search Console.
- Click on “Settings.”
- Go to “robots.txt.”
- Click on “Request to Crawl.”
It will crawl and validate your robots.txt file.
Conclusion:
Robots.txt is an important tool for optimizing the crawl budget. It impacts your website’s crawlability, which in turn impacts the indexing in search results.
Block unnecessary pages to allow Googlebot to spend time on valuable pages.
Save resources with optimized robots.txt file.
Other People Are reading:
