id,summary,reporter,owner,description,type,status,priority,milestone,component,version,resolution,keywords,cc,launchpad_bug 823,WUI server should have a disallow-all robots.txt,davidsarah,,"Currently, if a web crawler gets access to a Tahoe WUI gateway server then it will crawl all reachable links. This is probably undesirable, or at least not a sensible default (even though it is understood that {{{robots.txt}}} is not meant as a security mechanism). WUI servers should have a disallow-all {{{robots.txt}}}: {{{ User-agent: * Disallow: / }}} The robots.txt specification is at http://www.robotstxt.org/orig.html",defect,new,major,undecided,code-frontend-web,1.5.0,,privacy,,