11 Aug
2002
11 Aug
'02
11:09 p.m.
On Mon, 12 Aug 2002, Beck, Mike wrote:
we could detect if HTTP_AUTH_USER is set
That sounds reasonable, but will it work on all servers?
also: is a robots.txt file only searched for in the main directory of a domain or can it also be in subdirectories? if it can be then we could also put a robots.txt file in the dist that disallows robots to go there.
It applies if it is in subdirectories as well, and again a robot can ignore it if it wishes to. (I've written a few bots that do ignore it previously).
--
Robin Hugh Johnson
E-Mail : robbat2@orbis-terrarum.net
Home Page : http://www.orbis-terrarum.net/?l=people.robbat2
ICQ# : 30269588 or 41961639