[tahoe-dev] Modifying the robots.txt file on allmydata.org

[Aleksandr Milewski]

On 2/23/10 6:52 PM, Peter Secor wrote:

>     Please let me know any issues or suggestions you may have with this,
> I'm planning to make the change within the next few days barring
> compelling reasons not to.

Keep that Crawl-delay line in there, though, or MSNbot will render your 
server unusable.

Trac is actually pretty expensive to serve, so if you don't have 
something slowing down the bots, they'll crater your machines.

BTDT. This week, in fact.

-Zandr