[tahoe-lafs-trac-stream] [Tahoe-LAFS] #1886: cloud backend: crawlers are running too often

[Tahoe-LAFS]

#1886: cloud backend: crawlers are running too often
-------------------------+-------------------------------------------------
     Reporter:           |      Owner:  daira
  davidsarah             |     Status:  assigned
         Type:  defect   |  Milestone:  1.12.0
     Priority:  normal   |    Version:  1.9.2
    Component:  code-    |   Keywords:  cloud-backend crawler performance
  storage                |  regression blocks-cloud-merge
   Resolution:           |
Launchpad Bug:           |
-------------------------+-------------------------------------------------

Comment (by dquintela):

 Hello, first time user tahoe here,

 I've testing cloud storage for 3 or 4 days (branch 2237-cloud-backend-s4)
 into my raspberry-pi, running raspbian.
 Despite it having very long startup times, I noticed
 .tahoe/logs/twistd.log filled up of lines like this,
 that seems indicating the shares crawler is being run to often. This has
 the nasty side effecting that on amazon billing page
 I am already with 80000 requests. Rough estimate this could mean 3 to 5
 USD per month on an idle storage node alone.

 This seems to be related to #1835 and #1886 - sorry for the crossposting.

 2014-12-22 15:48:37+0000 [-] Starting factory <HTTPClientFactory:
 http://bucket_identifier.s3.amazonaws.com/?prefix=shares%2Fod%2F>
 2014-12-22 15:48:37+0000 [HTTPPageGetter,client] Stopping factory
 <HTTPClientFactory:
 http://bucket_identifier.s3.amazonaws.com/?prefix=shares%2Fod%2F>
 2014-12-22 15:48:37+0000 [-] Starting factory <HTTPClientFactory:
 http://bucket_identifier.s3.amazonaws.com/?prefix=shares%2Foe%2F>
 2014-12-22 15:48:38+0000 [HTTPPageGetter,client] Stopping factory
 <HTTPClientFactory:
 http://bucket_identifier.s3.amazonaws.com/?prefix=shares%2Foe%2F>
 2014-12-22 15:48:38+0000 [-] Starting factory <HTTPClientFactory:
 http://bucket_identifier.s3.amazonaws.com/?prefix=shares%2Fof%2F>
 2014-12-22 15:48:38+0000 [HTTPPageGetter,client] Stopping factory
 <HTTPClientFactory:
 http://bucket_identifier.s3.amazonaws.com/?prefix=shares%2Fof%2F>
 2014-12-22 15:48:38+0000 [-] Starting factory <HTTPClientFactory:
 http://bucket_identifier.s3.amazonaws.com/?prefix=shares%2Fog%2F>
 2014-12-22 15:48:38+0000 [HTTPPageGetter,client] Stopping factory
 <HTTPClientFactory:
 http://bucket_identifier.s3.amazonaws.com/?prefix=shares%2Fog%2F>
 ...

--
Ticket URL: <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1886#comment:8>
Tahoe-LAFS <https://Tahoe-LAFS.org>
secure decentralized storage