Hello. I never got all over to posting a robots. txt archive at my site. I hoped to uncover a " exactly how to" discussion these…. also a beneficial discussion about what exactly is best to consist of, what not in adding….
But after hunting for robots, robots. txt, software….
All I can find is a lot of messages saying, " You have to have a robots. txt file! "…. however no instructions….
Except I located this link of hospitality attire discussion:
http: //www. robotstxt. org
In this article I found these helpful pages:
I attempted to adhere to the instructions right now there. Here below may be the result: here is the ENTIRE CONTENTS of an robots. txt file i will post on my site.
The intention of this file should be to exclude robots via my image record named PIX.
User-agent: * Disallow: /pix/
More disallows might be added. If one example is I also required specifically to disallow robots from my directory page—but NOT from any other page in your WEB-DESIGN-TIPS directory….. then seems like I would make it happen:
User-agent: * Disallow: /pix/ Disallow: /web-design-tips/index. shtml
Is this correct Perhaps there is anything more to incorporate, that might ensure it is better Thank a person anybody.
Paperwork:
1. Only one software. txt file each site. It must be however directory, with your property page.
2. For more options, you can utilize robots meta label protocol. The robots meta label is non-essential and not yet viewed by means of most robots. Some say it’s useless. If anyone likes make use of robots meta tags, make sure you post your favorites below. (Also please words your onions about if or not it should become used. )
A FEW. To allow use of all files—just abandon the robots. txt archive totally empty.
FIVE. Why is the item standard practice in order to exclude the picture directory The robots will use up bandwidth and also can be utilized by people searching for images to rob.
FIVE. Another file to exclude: if you might have any very similar your website itself, you do certainly not want robots to consider you have reflector pages. So in which case you exclude them.
SOME. Also if you then have a PDF document this repeats pages in the site—then exclude the particular PDF!
SIX. Any other beneficial reasons to exclude (or not to ever exclude) files in the robots. txt Make sure you reply here! Thanks a ton.
It appears you merely need to produce the name on the file you intend to exclude. i. at the. if you create:
Disallow: /web-design-tips/index. shtml
It can also exclude all files inside the ‘web-design-tips folder.
Disallow: directory. shtml
Will exclude only this file, regardless that folder it lives in.
(meaning you it is fair to rename the record to something besides index, if you might have an in index-file around each sub list. (Otherwise all index. shtml files is excluded)
ref: http: //www. searchengineworld. com/robots/robots_tutorial. htm.