id summary reporter owner description type status priority milestone component version resolution keywords cc launchpad_bug 823 WUI server should have a disallow-all robots.txt davidsarah "Currently, if a web crawler gets access to a Tahoe WUI gateway server then it will crawl all reachable links. This is probably undesirable, or at least not a sensible default (even though it is understood that {{{robots.txt}}} is not meant as a security mechanism). WUI servers should have a disallow-all {{{robots.txt}}}: {{{ User-agent: * Disallow: / }}} The robots.txt specification is at http://www.robotstxt.org/orig.html" defect new major undecided code-frontend-web 1.5.0 privacy