Daily Archives: July 16, 2011

Spiders.txt – Requesting confirmation Hi all, I get some sites which can be being built off from a structure I’ve personalized.The problem is the fact means many of the " documents" will probably be very equivalent, only changing inside the places the location where the name can be referenced (dynamic). Please ok , i’ll know whether you think you will have any problems using the following that will block robot admission to those specific pages, which are however directory, while not restricting entry to all other pages. Policy: User-Agent:*Disallow:/privacy.phpDisallow:/contact.phpDisallow:/terms.phpAllow:/ Regards for what. Regards, Josh.