12 Aug
2002
12 Aug
'02
4:09 a.m.
On Mon, 12 Aug 2002, Beck, Mike wrote:
we could detect if HTTP_AUTH_USER is set That sounds reasonable, but will it work on all servers?
also: is a robots.txt file only searched for in the main directory of a domain or can it also be in subdirectories? if it can be then we could also put a robots.txt file in the dist that disallows robots to go there. It applies if it is in subdirectories as well, and again a robot can ignore it if it wishes to. (I've written a few bots that do ignore it previously).
-- Robin Hugh Johnson E-Mail : robbat2@orbis-terrarum.net Home Page : http://www.orbis-terrarum.net/?l=people.robbat2 ICQ# : 30269588 or 41961639