Spiders.txt – Requesting confirmation
Hi all,
I get some sites which can be being built off from a structure I’ve personalized.The problem is the fact means many of the " documents" will probably be very equivalent, only changing inside the places the location where the name can be referenced (dynamic).
Please ok , i’ll know whether you think you will have any problems using the following that will block robot admission to those specific pages, which are however directory, while not restricting entry to all other pages.
Policy: User-Agent:*
Disallow:/privacy.php
Disallow:/contact.php
Disallow:/terms.php
Allow:/ Regards for what.
Regards,
Josh.
Aloha dude! I quite agree with your thoughts. I’ve just shared it on the social networks.
The blog is cool
Very interesting details you have noted, thank you for putting up.
You points about the topic discussed can’t be thought doubtful! Keep on writing!