[tahoe-lafs-trac-stream] [tahoe-lafs] #1719: Improve google search results for phrases like "tahoe file storage"

[tahoe-lafs]

#1719: Improve google search results for phrases like "tahoe file storage"
-------------------------+------------------------------------
     Reporter:  amiller  |      Owner:
         Type:  defect   |     Status:  new
     Priority:  normal   |  Milestone:  undecided
    Component:  website  |    Version:  n/a
   Resolution:           |   Keywords:  transparency usability
Launchpad Bug:           |
-------------------------+------------------------------------

Comment (by zooko):

 Some of our content, such as https://tahoe-lafs.org/trac/tahoe-
 lafs/browser/docs/about.rst for example, is served up directly from the
 trac source browser. To let that stuff be indexable, at Tony Arcieri's
 suggestion, I removed the exclusion of trac from robots.txt. It now looks
 like this:

 {{{
 User-agent: *
 Disallow: /source/
 Disallow: /buildbot-tahoe-lafs
 Disallow: /buildbot-zfec
 Disallow: /buildbot-pycryptopp
 Crawl-Delay: 60
 }}}

 This might impose too much CPU and disk-IO load on our server. We'll see.

-- 
Ticket URL: <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1719#comment:2>
tahoe-lafs <https://tahoe-lafs.org>
secure decentralized storage