[tahoe-dev] Modifying the robots.txt file on allmydata.org

[Kevin Reid]

On Feb 23, 2010, at 21:52, Peter Secor wrote:

> Hi everyone (sorry for the slightly operational message),
>
>   There is currently a robots.txt[1] file which blocks crawlers from a
> few of the projects on the site, specifically everything under / 
> trac. In
> the interest of getting the information from allmydata.org present in
> searches for it, I propose we change this to allow crawly spiders to  
> be
> able to index all of our projects.
>
>   Please let me know any issues or suggestions you may have with this,
> I'm planning to make the change within the next few days barring
> compelling reasons not to.

I agree that the Trac content should be indexable.

I suggest ensuring that all links to historical wiki page revisions  
have rel="nofollow" or are otherwise hidden, to ensure that they do  
not appear in search engines before the proper versions.

(MediaWiki does this instead by segregating historical pages under /w/  
instead of /wiki/, and having robots.txt exclude the former. But that  
would be a large change to Trac.)

-- 
Kevin Reid                                  <http://switchb.org/kpreid/>