Hey readers! I bet you like hacks as everyone do. What are we going to discuss today? The Robots.txt, sound familiar? Yes, we have discussed the basic of Robots.txt in the previous blog and in this blog, we are going to discuss how to create a perfect robots.txt file for your SEO strategy. This file is an undetectable part of every business website but many do not talk about it. In previous blog we have given you a brief introduction of Robots.txt. This file was designed to work along with the search engines guiding them through the web pages. In this blog post, we are going to discuss how to set-up the perfect Robots.txt file for your SEO campaign.

Where You Can Find Your Existing Robots.Txt File 


Now, it is your time to examine the result. If you find the empty file or 404 error then, you need to fix this issue. It is good if you find a valid file then, it must be due to the default setting you did while creating the website. If you have found the valid robots.txt file, it is time to make necessary changes.

Related Post: How to Fix Broken Links of Your Website

What If You Don’t Have a Robots.Txt File?

As we mentioned in the previous blog post that robots.txt file can be used to submit the sitemap, to stop the search engine to crawl and index specific pages and guide them through each page. There are certain things that you should hide in a robots.txt like:

  • Pages with duplicate content
  • Account pages
  • Pagination pages
  • Admin pages
  • Shopping cart
  • Dynamic product and service page

You can also hire SEO service to make these changes.


As you can see in the above image I have disallowed several pages like account, basket, price, tag, etc. Once I have disallowed them, the Google bot will not crawl or index these pages. You can easily access, hide, and secure the data in your robots.txt. There is one thing, you should always keep in mind that your robots.txt are only directories which means search engines can access them even if you instruct them not. The good news is the most popular search engines like Google, Yahoo, Bing and Yandex follows the robots.txt.

Qdexi Technology is an experienced digital marketing company who offer search engine optimization service. They have a team of professional SEO who can optimize the robots.txt to generate a higher ranking for the website.