Will be robots.txt archive necessary
Hi
Is it essential to put bots.txt file from the root index of web page, i know it’s helpful to overpower crawlers, but easily not put this file while in the website, really does this induce some trouble, please advice
Thanks to your support
Not having an example may be never a dilemma.
http://www.robotstxt.org/faq.html
God bots like the Google grinding bot are always interested in the softwares.txt protected bad bots ignore them.
Its just better to get one in the root In my opinion.
you really should have one when you need one, nevertheless the average innovative minisite is not going to need one.
As stated by way of few people today.If you have no content that needs to be blocked from serps then no abdominal muscles it (also remember if you choose block search engines using it serps do not always take see also when you’ve got something private within a specific area humans can still manage to find the item and robot’s can make it easier).
Some persons do own an vacant robots.txt file since it can make it easier to test which web sites show 404 faults (makes lifestyle difficult whenever looking by raw logs and you keep finding a 404 for a robots.txt request).
Robots.txt file seriously isn’t always necessary for just a website, nevertheless its better make use of in the certain circumstances.