Tag Archives: Search Spiders

is there methods to keep your site posted on the sub directory w/o the actual search spiders crawling it You can place a file in your root directory known as robots. txt, and use it to specify which subdirectories you desire spiders (robots) to steer clear of. The major se spiders usually esteem whatever’s in software. txt. Here’s one of these of the file’s articles (Any directory not necessarily mentioned is assumed that they are allowed). The remarks (i. e. that # and all after) are optional: Code: User-agent: * # apply rule to all or any bots Disallow: /cgi-bin/ # stay right out the /cgi-bin directory Disallow: /develop/ # stay right out the /develop directory There is also the meta tag route. HTML: < meta name=" robots" content=" noindex, nofollow" /> Useful in the event you cannot create a robots. txt report, although you would need that line throughout every file. thankyou quite definitely guys does creating a robots. txt report also help seo I don’t believe it does, but at the least it’ll keep individual files from being read*. Since WHEN I last posted, I read articles stating to stay your robots. txt file as simple and well-formed … Continue reading →