Spiders.txt – Requesting confirmation
Hi all,
I get some sites which can be being built off from a structure I’ve personalized.The problem is the fact means many of the " documents" will probably be very equivalent, only changing inside the places the location where the name can be referenced (dynamic).
Please ok , i’ll know whether you think you will have any problems using the following that will block robot admission to those specific pages, which are however directory, while not restricting entry to all other pages.
Policy: User-Agent:*
Disallow:/privacy.php
Disallow:/contact.php
Disallow:/terms.php
Allow:/ Regards for what.
Regards,
Josh.
4 Responses to Robots.txtRequesting confirmation