# this is a generic robots.txt file that keeps web spiders, robots, etc. away. # note that entries here are *requests* that a robot stay away - only a # well-designed robot will honor these requests. # # a description can be found at: # http://info.webcrawler.com/mak/projects/robots/norobots.html User-agent: * Disallow: /tmp/ Disallow: /cgi-bin/ Disallow: /bin/ Disallow: /sbin/ Disallow: /private/ Disallow: /lib/ User-agent: Uptimebot Disallow: / # User-agent: freds-web-searcher # Disallow: /