is there methods to keep your site posted on the sub directory w/o the actual search spiders crawling it
You can place a file in your root directory known as robots. txt, and use it to specify which subdirectories you desire spiders (robots) to steer clear of. The major se spiders usually esteem whatever’s in software. txt.
Here’s one of these of the file’s articles (Any directory not necessarily mentioned is assumed that they are allowed). The remarks (i. e. that # and all after) are optional:
User-agent: * # apply rule to all or any bots Disallow: /cgi-bin/ # stay right out the /cgi-bin directory Disallow: /develop/ # stay right out the /develop directory
There is also the meta tag route.
< meta name=" robots" content=" noindex, nofollow" />
Useful in the event you cannot create a robots. txt report, although you would need that line throughout every file.
thankyou quite definitely guys
does creating a robots. txt report also help seo
I don’t believe it does, but at the least it’ll keep individual files from being read*. Since WHEN I last posted, I read articles stating to stay your robots. txt file as simple and well-formed as they can (leaving out responses, for example). You will discover more tips on practices avoiding here.
*From just what I’ve heard, those files may certainly indexed if the public page on your site links to help it; but the details of it shouldn’t get read. There’s more of an explanation here.
Spiders always love to see the automations. txt file whether or not they are " allowed" to index all the site.
You should actually notice errors of your useage log when spiders crawl your site and don’t come across this file.