Could we detect a .htaccess protection? If so, let's display a big red warning if someone uses the config auth mode without a .htaccess protection...
Alexander
we could detect if HTTP_AUTH_USER is set also: is a robots.txt file only searched for in the main directory of a domain or can it also be in subdirectories? if it can be then we could also put a robots.txt file in the dist that disallows robots to go there.
On Mon, 12 Aug 2002, Beck, Mike wrote:
we could detect if HTTP_AUTH_USER is set
That sounds reasonable, but will it work on all servers?
also: is a robots.txt file only searched for in the main directory of a domain or can it also be in subdirectories? if it can be then we could also put a robots.txt file in the dist that disallows robots to go there.
It applies if it is in subdirectories as well, and again a robot can ignore it if it wishes to. (I've written a few bots that do ignore it previously).
-
Beck, Mike -
Robin Johnson