Tag Archives: Robots Txt

How to for robots.txt or robot exclusion file…a basic essential (& robots meta tag)

Posted on November 21, 2012 by admin564 views

Hello. I never got all over to posting a robots. txt archive at my site. I hoped to uncover a " exactly how to" discussion these…. also a beneficial discussion about what exactly is best to consist of, what not in adding…. But after hunting for robots, robots. txt, software…. All I can find is a lot of messages saying, " You have to have a robots. txt file! "…. however no instructions…. Except I located this link of hospitality attire discussion: http: //www. robotstxt. org In this article I found these helpful pages: I attempted to adhere to the instructions right now there. Here below may be the result: here is the ENTIRE CONTENTS of an robots. txt file i will post on my site. The intention of this file should be to exclude robots via my image record named PIX. Code: User-agent: * Disallow: /pix/ More disallows might be added. If one example is I also required specifically to disallow robots from my directory page—but NOT from any other page in your WEB-DESIGN-TIPS directory….. then seems like I would make it happen: Code: User-agent: * Disallow: /pix/ Disallow: /web-design-tips/index. shtml Is this correct Perhaps there is anything more … Continue reading →

Posted in Web Design | Tagged , , , , , , , , , , , , , , , , , , , | Leave a comment

A password protected site with photos?

Posted on November 19, 2012 by admin597 views

I’m kind of only starting, and I might make some VERY fundamental HTML sites, but what I need to try is really a site with photos that we can allow family to view, whenever they have a password, rather than allow anyone with not a password access. Something akin to how www. webshots. com is effective. I have an account there (webshots) we pay for, but seeing that I have a website name and hosting, I would like to make my own ring personal site Could this be performed with just HTML DOCUMENT No. You’ll need some form of server side program code that checks with regard to login information, when i. e. PHP or ASP. Wired can be somewhat correct; that’s the best way to achieve this. However, but if your family or whoever is looking at your site features a Google toolbar or something similar mounted, make sure you’ve got a robots. txt file of which tells spiders and crawlers to not ever index your products. What you may be able to get away together with (cheaper alternative) is definitely getting your coordinator to assign password permissions to a directory, so you will get that username/password prompt whenever … Continue reading →

Posted in Web Design | Tagged , , , , , , , , , , , , , , , , , , , | Leave a comment

robots.txt: what is (~)? Can "images" disallow "images" folders on all levels? Etc.

Posted on October 5, 2012 by admin568 views

POST thought I find out about the robots. txt from not too long ago. However, this specific key article retains me scratching my head, now i always re-read it: what’s meant by a new squiggle () within this article In particular, this example: () = replace this squiggle by using any letters because of this example Or… () = actually utilize this squiggle in any robots. txt file to be a wildcard Or… () = several URLs actually contain a squiggle… can a particular directive exclude several " image" versions on multiple levels With all the standard method: Disallow: /images/ Disallow: forum/images/ Disallow: clients/joes-diner/images/ Disallow: clients/hotel-albion/images/ Can doing this and more be replaced because of the following single series… or something same… Is it advisable to place your blank index. htm report in every " disallowed" folder (such because the images folders) to discourage snooping But will it discourage snooping POST. e., if a folder includes no index file–anyone whom types that folder throughout his address pub is fed a convenient report on everything in in which folder. I have read that this sort of snoop particularly likes to visit the robots. txt report for directions wherever to snoop. Which … Continue reading →

Posted in Web Design | Tagged , , , , , , , , , , , , , | Leave a comment

Best resources for content entry

Posted on September 11, 2012 by admin576 views

I wanted to require a poll and ask what everyone’s favorite resources they need for shortcuts together with webdesign. My biggest question is Needed to have resources so other, not web savy individuals could input content for websites that I’ve made. I just don’t plan to be bugged every 10 minutes to include something. Resources that I’m looking for is a good calendar, and just main content input. Also some other great shortcuts could well be welcome. I’m okay at HTML and flash, however use dreamweaver a lot just to create things quicker. However always interested in the next significant shortcut. SOME SORT OF lame answer, however, Google. I use Google constantly to quickly find things I would like. Instance, it’s hard to have people to key in a date accurately, so I would need a " date-picker"… http: //www. google. com/searchhl=en& q=javascript day picker& btnG=Google Search Or even, I don’t keep in mind how PHP displays the afternoon of year variety: http: //www. google. com/searchhl=en& safe=off& q=php date& btnG=Search For just about any PHP snippets as well as " I aren’t able to remember how"…. I use Google. Here i will discuss one I applied today: http: //www. google. … Continue reading →

Posted in Web Design | Tagged , , , , , , , , , , , , , , , , , , , | Leave a comment

Developing Sites on sub directory?

Posted on September 2, 2012 by admin613 views

is there methods to keep your site posted on the sub directory w/o the actual search spiders crawling it You can place a file in your root directory known as robots. txt, and use it to specify which subdirectories you desire spiders (robots) to steer clear of. The major se spiders usually esteem whatever’s in software. txt. Here’s one of these of the file’s articles (Any directory not necessarily mentioned is assumed that they are allowed). The remarks (i. e. that # and all after) are optional: Code: User-agent: * # apply rule to all or any bots Disallow: /cgi-bin/ # stay right out the /cgi-bin directory Disallow: /develop/ # stay right out the /develop directory There is also the meta tag route. HTML: < meta name=" robots" content=" noindex, nofollow" /> Useful in the event you cannot create a robots. txt report, although you would need that line throughout every file. thankyou quite definitely guys does creating a robots. txt report also help seo I don’t believe it does, but at the least it’ll keep individual files from being read*. Since WHEN I last posted, I read articles stating to stay your robots. txt file as simple and well-formed … Continue reading →

Posted in Web Design | Tagged , , , , , , , , , | Leave a comment

robots.txt on a different server?

Posted on August 10, 2012 by admin428 views

Acceptable so here the item goes… We’ve recently acquired a website lindaruderman. com via another company. As of now we can’t do half of your monthly reports on the webpage because google (and each of our other utilities) are telling us the robots. txt isn’t allowing for moving. Enter lindaruderman. com/robots. txt and see that this indeed phony. And so my question is definitely… Can additional company have your robots. txt file within the old lindaruderman. com site that may be effecting us I’ve also noticed that every the pages google mentions which might be being block are in reference to your old site. For example there’s no contact. php file about our iteration on the site. We have a very /contact/ page being that him and i utilize friendly URLs. Any kind of suggestions are liked…. Only thing I could say is who’s will take your next crawl from Google the get your web blog indexed correctly. The old site must be removed from that server. Eventually these and other conflicts will become ironed out.

Posted in Web Design | Tagged , , , , , , | Leave a comment

safeguarding website from tresspassing

Posted on June 19, 2012 by admin486 views

I needed to know if you have any way an online site can be shielded from general tresspassing in general. If there are numerous sensitive data within website, what need to be the way onward Simply by little googling, I have so far found out that most of the popular websites have a very " " file inside their home directory containing instructions on accesses in addition to permissions. Nevertheless, at the very same time, I also think is often fooled as it only instructs with no imposing.. Is the simply alternative I will need to pay more pertaining to certificates then.. I desired to prevent value base in very few directories, though their functionality will a lot be visible.. This hosting platform is definitely linux without database support.. Thanks! Robots. txt usually are only requested by legitimate search engines Should your host allows setting permissions about the folder level.. That should utilize the computers security to who gets admission to files inside of which folder… Some level of access control is possible with scripting words where you create a login and password gain access to a section… But this really is really only usable on pages you may include … Continue reading →

Posted in Web Design | Tagged , , , , , , , , , , , , , , , , , , , | Leave a comment