Ticket #999: pluggable-backends-davidsarah-v7.darcs.patch

File pluggable-backends-davidsarah-v7.darcs.patch, 359.6 KB (added by davidsarah, at 2011-09-21T18:54:37Z)

Latest snapshot, more tests passing.

Line 
19 patches for repository http://tahoe-lafs.org/source/tahoe/trunk:
2
3Thu Aug 25 01:32:17 BST 2011  david-sarah@jacaranda.org
4  * interfaces.py: 'which -> that' grammar cleanup.
5
6Tue Sep 20 00:29:26 BST 2011  david-sarah@jacaranda.org
7  * Pluggable backends -- new and moved files, changes to moved files. refs #999
8
9Tue Sep 20 00:32:56 BST 2011  david-sarah@jacaranda.org
10  * Pluggable backends -- all other changes. refs #999
11
12Tue Sep 20 04:38:03 BST 2011  david-sarah@jacaranda.org
13  * Work-in-progress, includes fix to bug involving BucketWriter. refs #999
14
15Tue Sep 20 18:17:37 BST 2011  david-sarah@jacaranda.org
16  * docs/backends: document the configuration options for the pluggable backends scheme. refs #999
17
18Wed Sep 21 04:12:07 BST 2011  david-sarah@jacaranda.org
19  * Fix some incorrect attribute accesses. refs #999
20
21Wed Sep 21 04:16:25 BST 2011  david-sarah@jacaranda.org
22  * docs/backends/S3.rst: remove Issues section. refs #999
23
24Wed Sep 21 04:17:05 BST 2011  david-sarah@jacaranda.org
25  * docs/backends/S3.rst, disk.rst: describe type of space settings as 'quantity of space', not 'str'. refs #999
26
27Wed Sep 21 19:46:49 BST 2011  david-sarah@jacaranda.org
28  * More fixes to tests needed for pluggable backends. refs #999
29
30New patches:
31
32[interfaces.py: 'which -> that' grammar cleanup.
33david-sarah@jacaranda.org**20110825003217
34 Ignore-this: a3e15f3676de1b346ad78aabdfb8cac6
35] {
36hunk ./src/allmydata/interfaces.py 38
37     the StubClient. This object doesn't actually offer any services, but the
38     announcement helps the Introducer keep track of which clients are
39     subscribed (so the grid admin can keep track of things like the size of
40-    the grid and the client versions in use. This is the (empty)
41+    the grid and the client versions in use). This is the (empty)
42     RemoteInterface for the StubClient."""
43 
44 class RIBucketWriter(RemoteInterface):
45hunk ./src/allmydata/interfaces.py 276
46         (binary) storage index string, and 'shnum' is the integer share
47         number. 'reason' is a human-readable explanation of the problem,
48         probably including some expected hash values and the computed ones
49-        which did not match. Corruption advisories for mutable shares should
50+        that did not match. Corruption advisories for mutable shares should
51         include a hash of the public key (the same value that appears in the
52         mutable-file verify-cap), since the current share format does not
53         store that on disk.
54hunk ./src/allmydata/interfaces.py 413
55           remote_host: the IAddress, if connected, otherwise None
56 
57         This method is intended for monitoring interfaces, such as a web page
58-        which describes connecting and connected peers.
59+        that describes connecting and connected peers.
60         """
61 
62     def get_all_peerids():
63hunk ./src/allmydata/interfaces.py 515
64 
65     # TODO: rename to get_read_cap()
66     def get_readonly():
67-        """Return another IURI instance, which represents a read-only form of
68+        """Return another IURI instance that represents a read-only form of
69         this one. If is_readonly() is True, this returns self."""
70 
71     def get_verify_cap():
72hunk ./src/allmydata/interfaces.py 542
73         passing into init_from_string."""
74 
75 class IDirnodeURI(Interface):
76-    """I am a URI which represents a dirnode."""
77+    """I am a URI that represents a dirnode."""
78 
79 class IFileURI(Interface):
80hunk ./src/allmydata/interfaces.py 545
81-    """I am a URI which represents a filenode."""
82+    """I am a URI that represents a filenode."""
83     def get_size():
84         """Return the length (in bytes) of the file that I represent."""
85 
86hunk ./src/allmydata/interfaces.py 553
87     pass
88 
89 class IMutableFileURI(Interface):
90-    """I am a URI which represents a mutable filenode."""
91+    """I am a URI that represents a mutable filenode."""
92     def get_extension_params():
93         """Return the extension parameters in the URI"""
94 
95hunk ./src/allmydata/interfaces.py 856
96         """
97 
98 class IFileNode(IFilesystemNode):
99-    """I am a node which represents a file: a sequence of bytes. I am not a
100+    """I am a node that represents a file: a sequence of bytes. I am not a
101     container, like IDirectoryNode."""
102     def get_best_readable_version():
103         """Return a Deferred that fires with an IReadable for the 'best'
104hunk ./src/allmydata/interfaces.py 905
105     multiple versions of a file present in the grid, some of which might be
106     unrecoverable (i.e. have fewer than 'k' shares). These versions are
107     loosely ordered: each has a sequence number and a hash, and any version
108-    with seqnum=N was uploaded by a node which has seen at least one version
109+    with seqnum=N was uploaded by a node that has seen at least one version
110     with seqnum=N-1.
111 
112     The 'servermap' (an instance of IMutableFileServerMap) is used to
113hunk ./src/allmydata/interfaces.py 1014
114         as a guide to where the shares are located.
115 
116         I return a Deferred that fires with the requested contents, or
117-        errbacks with UnrecoverableFileError. Note that a servermap which was
118+        errbacks with UnrecoverableFileError. Note that a servermap that was
119         updated with MODE_ANYTHING or MODE_READ may not know about shares for
120         all versions (those modes stop querying servers as soon as they can
121         fulfil their goals), so you may want to use MODE_CHECK (which checks
122hunk ./src/allmydata/interfaces.py 1073
123     """Upload was unable to satisfy 'servers_of_happiness'"""
124 
125 class UnableToFetchCriticalDownloadDataError(Exception):
126-    """I was unable to fetch some piece of critical data which is supposed to
127+    """I was unable to fetch some piece of critical data that is supposed to
128     be identically present in all shares."""
129 
130 class NoServersError(Exception):
131hunk ./src/allmydata/interfaces.py 1085
132     exists, and overwrite= was set to False."""
133 
134 class NoSuchChildError(Exception):
135-    """A directory node was asked to fetch a child which does not exist."""
136+    """A directory node was asked to fetch a child that does not exist."""
137 
138 class ChildOfWrongTypeError(Exception):
139     """An operation was attempted on a child of the wrong type (file or directory)."""
140hunk ./src/allmydata/interfaces.py 1403
141         if you initially thought you were going to use 10 peers, started
142         encoding, and then two of the peers dropped out: you could use
143         desired_share_ids= to skip the work (both memory and CPU) of
144-        producing shares for the peers which are no longer available.
145+        producing shares for the peers that are no longer available.
146 
147         """
148 
149hunk ./src/allmydata/interfaces.py 1478
150         if you initially thought you were going to use 10 peers, started
151         encoding, and then two of the peers dropped out: you could use
152         desired_share_ids= to skip the work (both memory and CPU) of
153-        producing shares for the peers which are no longer available.
154+        producing shares for the peers that are no longer available.
155 
156         For each call, encode() will return a Deferred that fires with two
157         lists, one containing shares and the other containing the shareids.
158hunk ./src/allmydata/interfaces.py 1535
159         required to be of the same length.  The i'th element of their_shareids
160         is required to be the shareid of the i'th buffer in some_shares.
161 
162-        This returns a Deferred which fires with a sequence of buffers. This
163+        This returns a Deferred that fires with a sequence of buffers. This
164         sequence will contain all of the segments of the original data, in
165         order. The sum of the lengths of all of the buffers will be the
166         'data_size' value passed into the original ICodecEncode.set_params()
167hunk ./src/allmydata/interfaces.py 1582
168         Encoding parameters can be set in three ways. 1: The Encoder class
169         provides defaults (3/7/10). 2: the Encoder can be constructed with
170         an 'options' dictionary, in which the
171-        needed_and_happy_and_total_shares' key can be a (k,d,n) tuple. 3:
172+        'needed_and_happy_and_total_shares' key can be a (k,d,n) tuple. 3:
173         set_params((k,d,n)) can be called.
174 
175         If you intend to use set_params(), you must call it before
176hunk ./src/allmydata/interfaces.py 1780
177         produced, so that the segment hashes can be generated with only a
178         single pass.
179 
180-        This returns a Deferred which fires with a sequence of hashes, using:
181+        This returns a Deferred that fires with a sequence of hashes, using:
182 
183          tuple(segment_hashes[first:last])
184 
185hunk ./src/allmydata/interfaces.py 1796
186     def get_plaintext_hash():
187         """OBSOLETE; Get the hash of the whole plaintext.
188 
189-        This returns a Deferred which fires with a tagged SHA-256 hash of the
190+        This returns a Deferred that fires with a tagged SHA-256 hash of the
191         whole plaintext, obtained from hashutil.plaintext_hash(data).
192         """
193 
194hunk ./src/allmydata/interfaces.py 1856
195         be used to encrypt the data. The key will also be hashed to derive
196         the StorageIndex.
197 
198-        Uploadables which want to achieve convergence should hash their file
199+        Uploadables that want to achieve convergence should hash their file
200         contents and the serialized_encoding_parameters to form the key
201         (which of course requires a full pass over the data). Uploadables can
202         use the upload.ConvergentUploadMixin class to achieve this
203hunk ./src/allmydata/interfaces.py 1862
204         automatically.
205 
206-        Uploadables which do not care about convergence (or do not wish to
207+        Uploadables that do not care about convergence (or do not wish to
208         make multiple passes over the data) can simply return a
209         strongly-random 16 byte string.
210 
211hunk ./src/allmydata/interfaces.py 1872
212 
213     def read(length):
214         """Return a Deferred that fires with a list of strings (perhaps with
215-        only a single element) which, when concatenated together, contain the
216+        only a single element) that, when concatenated together, contain the
217         next 'length' bytes of data. If EOF is near, this may provide fewer
218         than 'length' bytes. The total number of bytes provided by read()
219         before it signals EOF must equal the size provided by get_size().
220hunk ./src/allmydata/interfaces.py 1919
221 
222     def read(length):
223         """
224-        Returns a list of strings which, when concatenated, are the next
225+        Returns a list of strings that, when concatenated, are the next
226         length bytes of the file, or fewer if there are fewer bytes
227         between the current location and the end of the file.
228         """
229hunk ./src/allmydata/interfaces.py 1932
230 
231 class IUploadResults(Interface):
232     """I am returned by upload() methods. I contain a number of public
233-    attributes which can be read to determine the results of the upload. Some
234+    attributes that can be read to determine the results of the upload. Some
235     of these are functional, some are timing information. All of these may be
236     None.
237 
238hunk ./src/allmydata/interfaces.py 1965
239 
240 class IDownloadResults(Interface):
241     """I am created internally by download() methods. I contain a number of
242-    public attributes which contain details about the download process.::
243+    public attributes that contain details about the download process.::
244 
245      .file_size : the size of the file, in bytes
246      .servers_used : set of server peerids that were used during download
247hunk ./src/allmydata/interfaces.py 1991
248 class IUploader(Interface):
249     def upload(uploadable):
250         """Upload the file. 'uploadable' must impement IUploadable. This
251-        returns a Deferred which fires with an IUploadResults instance, from
252+        returns a Deferred that fires with an IUploadResults instance, from
253         which the URI of the file can be obtained as results.uri ."""
254 
255     def upload_ssk(write_capability, new_version, uploadable):
256hunk ./src/allmydata/interfaces.py 2041
257         kind of lease that is obtained (which account number to claim, etc).
258 
259         TODO: any problems seen during checking will be reported to the
260-        health-manager.furl, a centralized object which is responsible for
261+        health-manager.furl, a centralized object that is responsible for
262         figuring out why files are unhealthy so corrective action can be
263         taken.
264         """
265hunk ./src/allmydata/interfaces.py 2056
266         will be put in the check-and-repair results. The Deferred will not
267         fire until the repair is complete.
268 
269-        This returns a Deferred which fires with an instance of
270+        This returns a Deferred that fires with an instance of
271         ICheckAndRepairResults."""
272 
273 class IDeepCheckable(Interface):
274hunk ./src/allmydata/interfaces.py 2141
275                               that was found to be corrupt. Each share
276                               locator is a list of (serverid, storage_index,
277                               sharenum).
278-         count-incompatible-shares: the number of shares which are of a share
279+         count-incompatible-shares: the number of shares that are of a share
280                                     format unknown to this checker
281          list-incompatible-shares: a list of 'share locators', one for each
282                                    share that was found to be of an unknown
283hunk ./src/allmydata/interfaces.py 2148
284                                    format. Each share locator is a list of
285                                    (serverid, storage_index, sharenum).
286          servers-responding: list of (binary) storage server identifiers,
287-                             one for each server which responded to the share
288+                             one for each server that responded to the share
289                              query (even if they said they didn't have
290                              shares, and even if they said they did have
291                              shares but then didn't send them when asked, or
292hunk ./src/allmydata/interfaces.py 2345
293         will use the data in the checker results to guide the repair process,
294         such as which servers provided bad data and should therefore be
295         avoided. The ICheckResults object is inside the
296-        ICheckAndRepairResults object, which is returned by the
297+        ICheckAndRepairResults object that is returned by the
298         ICheckable.check() method::
299 
300          d = filenode.check(repair=False)
301hunk ./src/allmydata/interfaces.py 2436
302         methods to create new objects. I return synchronously."""
303 
304     def create_mutable_file(contents=None, keysize=None):
305-        """I create a new mutable file, and return a Deferred which will fire
306+        """I create a new mutable file, and return a Deferred that will fire
307         with the IMutableFileNode instance when it is ready. If contents= is
308         provided (a bytestring), it will be used as the initial contents of
309         the new file, otherwise the file will contain zero bytes. keysize= is
310hunk ./src/allmydata/interfaces.py 2444
311         usual."""
312 
313     def create_new_mutable_directory(initial_children={}):
314-        """I create a new mutable directory, and return a Deferred which will
315+        """I create a new mutable directory, and return a Deferred that will
316         fire with the IDirectoryNode instance when it is ready. If
317         initial_children= is provided (a dict mapping unicode child name to
318         (childnode, metadata_dict) tuples), the directory will be populated
319hunk ./src/allmydata/interfaces.py 2452
320 
321 class IClientStatus(Interface):
322     def list_all_uploads():
323-        """Return a list of uploader objects, one for each upload which
324+        """Return a list of uploader objects, one for each upload that
325         currently has an object available (tracked with weakrefs). This is
326         intended for debugging purposes."""
327     def list_active_uploads():
328hunk ./src/allmydata/interfaces.py 2462
329         started uploads."""
330 
331     def list_all_downloads():
332-        """Return a list of downloader objects, one for each download which
333+        """Return a list of downloader objects, one for each download that
334         currently has an object available (tracked with weakrefs). This is
335         intended for debugging purposes."""
336     def list_active_downloads():
337hunk ./src/allmydata/interfaces.py 2689
338 
339     def provide(provider=RIStatsProvider, nickname=str):
340         """
341-        @param provider: a stats collector instance which should be polled
342+        @param provider: a stats collector instance that should be polled
343                          periodically by the gatherer to collect stats.
344         @param nickname: a name useful to identify the provided client
345         """
346hunk ./src/allmydata/interfaces.py 2722
347 
348 class IValidatedThingProxy(Interface):
349     def start():
350-        """ Acquire a thing and validate it. Return a deferred which is
351+        """ Acquire a thing and validate it. Return a deferred that is
352         eventually fired with self if the thing is valid or errbacked if it
353         can't be acquired or validated."""
354 
355}
356[Pluggable backends -- new and moved files, changes to moved files. refs #999
357david-sarah@jacaranda.org**20110919232926
358 Ignore-this: ec5d2d1362a092d919e84327d3092424
359] {
360adddir ./src/allmydata/storage/backends
361adddir ./src/allmydata/storage/backends/disk
362move ./src/allmydata/storage/immutable.py ./src/allmydata/storage/backends/disk/immutable.py
363move ./src/allmydata/storage/mutable.py ./src/allmydata/storage/backends/disk/mutable.py
364adddir ./src/allmydata/storage/backends/null
365addfile ./src/allmydata/storage/backends/__init__.py
366addfile ./src/allmydata/storage/backends/base.py
367hunk ./src/allmydata/storage/backends/base.py 1
368+
369+from twisted.application import service
370+
371+from allmydata.storage.common import si_b2a
372+from allmydata.storage.lease import LeaseInfo
373+from allmydata.storage.bucket import BucketReader
374+
375+
376+class Backend(service.MultiService):
377+    def __init__(self):
378+        service.MultiService.__init__(self)
379+
380+
381+class ShareSet(object):
382+    """
383+    This class implements shareset logic that could work for all backends, but
384+    might be useful to override for efficiency.
385+    """
386+
387+    def __init__(self, storageindex):
388+        self.storageindex = storageindex
389+
390+    def get_storage_index(self):
391+        return self.storageindex
392+
393+    def get_storage_index_string(self):
394+        return si_b2a(self.storageindex)
395+
396+    def renew_lease(self, renew_secret, new_expiration_time):
397+        found_shares = False
398+        for share in self.get_shares():
399+            found_shares = True
400+            share.renew_lease(renew_secret, new_expiration_time)
401+
402+        if not found_shares:
403+            raise IndexError("no such lease to renew")
404+
405+    def get_leases(self):
406+        # Since all shares get the same lease data, we just grab the leases
407+        # from the first share.
408+        try:
409+            sf = self.get_shares().next()
410+            return sf.get_leases()
411+        except StopIteration:
412+            return iter([])
413+
414+    def add_or_renew_lease(self, lease_info):
415+        # This implementation assumes that lease data is duplicated in
416+        # all shares of a shareset, which might not be true for all backends.
417+        for share in self.get_shares():
418+            share.add_or_renew_lease(lease_info)
419+
420+    def make_bucket_reader(self, storageserver, share):
421+        return BucketReader(storageserver, share)
422+
423+    def testv_and_readv_and_writev(self, storageserver, secrets,
424+                                   test_and_write_vectors, read_vector,
425+                                   expiration_time):
426+        # The implementation here depends on the following helper methods,
427+        # which must be provided by subclasses:
428+        #
429+        # def _clean_up_after_unlink(self):
430+        #     """clean up resources associated with the shareset after some
431+        #     shares might have been deleted"""
432+        #
433+        # def _create_mutable_share(self, storageserver, shnum, write_enabler):
434+        #     """create a mutable share with the given shnum and write_enabler"""
435+
436+        # secrets might be a triple with cancel_secret in secrets[2], but if
437+        # so we ignore the cancel_secret.
438+        write_enabler = secrets[0]
439+        renew_secret = secrets[1]
440+
441+        si_s = self.get_storage_index_string()
442+        shares = {}
443+        for share in self.get_shares():
444+            # XXX is it correct to ignore immutable shares? Maybe get_shares should
445+            # have a parameter saying what type it's expecting.
446+            if share.sharetype == "mutable":
447+                share.check_write_enabler(write_enabler, si_s)
448+                shares[share.get_shnum()] = share
449+
450+        # write_enabler is good for all existing shares
451+
452+        # now evaluate test vectors
453+        testv_is_good = True
454+        for sharenum in test_and_write_vectors:
455+            (testv, datav, new_length) = test_and_write_vectors[sharenum]
456+            if sharenum in shares:
457+                if not shares[sharenum].check_testv(testv):
458+                    self.log("testv failed: [%d]: %r" % (sharenum, testv))
459+                    testv_is_good = False
460+                    break
461+            else:
462+                # compare the vectors against an empty share, in which all
463+                # reads return empty strings
464+                if not EmptyShare().check_testv(testv):
465+                    self.log("testv failed (empty): [%d] %r" % (sharenum,
466+                                                                testv))
467+                    testv_is_good = False
468+                    break
469+
470+        # gather the read vectors, before we do any writes
471+        read_data = {}
472+        for shnum, share in shares.items():
473+            read_data[shnum] = share.readv(read_vector)
474+
475+        ownerid = 1 # TODO
476+        lease_info = LeaseInfo(ownerid, renew_secret,
477+                               expiration_time, storageserver.get_serverid())
478+
479+        if testv_is_good:
480+            # now apply the write vectors
481+            for shnum in test_and_write_vectors:
482+                (testv, datav, new_length) = test_and_write_vectors[shnum]
483+                if new_length == 0:
484+                    if shnum in shares:
485+                        shares[shnum].unlink()
486+                else:
487+                    if shnum not in shares:
488+                        # allocate a new share
489+                        share = self._create_mutable_share(storageserver, shnum, write_enabler)
490+                        shares[shnum] = share
491+                    shares[shnum].writev(datav, new_length)
492+                    # and update the lease
493+                    shares[shnum].add_or_renew_lease(lease_info)
494+
495+            if new_length == 0:
496+                self._clean_up_after_unlink()
497+
498+        return (testv_is_good, read_data)
499+
500+    def readv(self, wanted_shnums, read_vector):
501+        """
502+        Read a vector from the numbered shares in this shareset. An empty
503+        shares list means to return data from all known shares.
504+
505+        @param wanted_shnums=ListOf(int)
506+        @param read_vector=ReadVector
507+        @return DictOf(int, ReadData): shnum -> results, with one key per share
508+        """
509+        datavs = {}
510+        for share in self.get_shares():
511+            shnum = share.get_shnum()
512+            if not wanted_shnums or shnum in wanted_shnums:
513+                datavs[shnum] = share.readv(read_vector)
514+
515+        return datavs
516+
517+
518+def testv_compare(a, op, b):
519+    assert op in ("lt", "le", "eq", "ne", "ge", "gt")
520+    if op == "lt":
521+        return a < b
522+    if op == "le":
523+        return a <= b
524+    if op == "eq":
525+        return a == b
526+    if op == "ne":
527+        return a != b
528+    if op == "ge":
529+        return a >= b
530+    if op == "gt":
531+        return a > b
532+    # never reached
533+
534+
535+class EmptyShare:
536+    def check_testv(self, testv):
537+        test_good = True
538+        for (offset, length, operator, specimen) in testv:
539+            data = ""
540+            if not testv_compare(data, operator, specimen):
541+                test_good = False
542+                break
543+        return test_good
544+
545addfile ./src/allmydata/storage/backends/disk/__init__.py
546addfile ./src/allmydata/storage/backends/disk/disk_backend.py
547hunk ./src/allmydata/storage/backends/disk/disk_backend.py 1
548+
549+import re
550+
551+from twisted.python.filepath import UnlistableError
552+
553+from zope.interface import implements
554+from allmydata.interfaces import IStorageBackend, IShareSet
555+from allmydata.util import fileutil, log, time_format
556+from allmydata.storage.common import si_b2a, si_a2b
557+from allmydata.storage.bucket import BucketWriter
558+from allmydata.storage.backends.base import Backend, ShareSet
559+from allmydata.storage.backends.disk.immutable import ImmutableDiskShare
560+from allmydata.storage.backends.disk.mutable import MutableDiskShare, create_mutable_disk_share
561+
562+# storage/
563+# storage/shares/incoming
564+#   incoming/ holds temp dirs named $START/$STORAGEINDEX/$SHARENUM which will
565+#   be moved to storage/shares/$START/$STORAGEINDEX/$SHARENUM upon success
566+# storage/shares/$START/$STORAGEINDEX
567+# storage/shares/$START/$STORAGEINDEX/$SHARENUM
568+
569+# Where "$START" denotes the first 10 bits worth of $STORAGEINDEX (that's 2
570+# base-32 chars).
571+# $SHARENUM matches this regex:
572+NUM_RE=re.compile("^[0-9]+$")
573+
574+
575+def si_si2dir(startfp, storageindex):
576+    sia = si_b2a(storageindex)
577+    newfp = startfp.child(sia[:2])
578+    return newfp.child(sia)
579+
580+
581+def get_share(fp):
582+    f = fp.open('rb')
583+    try:
584+        prefix = f.read(32)
585+    finally:
586+        f.close()
587+
588+    if prefix == MutableDiskShare.MAGIC:
589+        return MutableDiskShare(fp)
590+    else:
591+        # assume it's immutable
592+        return ImmutableDiskShare(fp)
593+
594+
595+class DiskBackend(Backend):
596+    implements(IStorageBackend)
597+
598+    def __init__(self, storedir, readonly=False, reserved_space=0, discard_storage=False):
599+        Backend.__init__(self)
600+        self._setup_storage(storedir, readonly, reserved_space, discard_storage)
601+        self._setup_corruption_advisory()
602+
603+    def _setup_storage(self, storedir, readonly, reserved_space, discard_storage):
604+        self._storedir = storedir
605+        self._readonly = readonly
606+        self._reserved_space = int(reserved_space)
607+        self._discard_storage = discard_storage
608+        self._sharedir = self._storedir.child("shares")
609+        fileutil.fp_make_dirs(self._sharedir)
610+        self._incomingdir = self._sharedir.child('incoming')
611+        self._clean_incomplete()
612+        if self._reserved_space and (self.get_available_space() is None):
613+            log.msg("warning: [storage]reserved_space= is set, but this platform does not support an API to get disk statistics (statvfs(2) or GetDiskFreeSpaceEx), so this reservation cannot be honored",
614+                    umid="0wZ27w", level=log.UNUSUAL)
615+
616+    def _clean_incomplete(self):
617+        fileutil.fp_remove(self._incomingdir)
618+        fileutil.fp_make_dirs(self._incomingdir)
619+
620+    def _setup_corruption_advisory(self):
621+        # we don't actually create the corruption-advisory dir until necessary
622+        self._corruption_advisory_dir = self._storedir.child("corruption-advisories")
623+
624+    def _make_shareset(self, sharehomedir):
625+        return self.get_shareset(si_a2b(sharehomedir.basename()))
626+
627+    def get_sharesets_for_prefix(self, prefix):
628+        prefixfp = self._sharedir.child(prefix)
629+        try:
630+            sharesets = map(self._make_shareset, prefixfp.children())
631+            def _by_base32si(b):
632+                return b.get_storage_index_string()
633+            sharesets.sort(key=_by_base32si)
634+        except EnvironmentError:
635+            sharesets = []
636+        return sharesets
637+
638+    def get_shareset(self, storageindex):
639+        sharehomedir = si_si2dir(self._sharedir, storageindex)
640+        incominghomedir = si_si2dir(self._incomingdir, storageindex)
641+        return DiskShareSet(storageindex, sharehomedir, incominghomedir, discard_storage=self._discard_storage)
642+
643+    def fill_in_space_stats(self, stats):
644+        stats['storage_server.reserved_space'] = self._reserved_space
645+        try:
646+            disk = fileutil.get_disk_stats(self._sharedir, self._reserved_space)
647+            writeable = disk['avail'] > 0
648+
649+            # spacetime predictors should use disk_avail / (d(disk_used)/dt)
650+            stats['storage_server.disk_total'] = disk['total']
651+            stats['storage_server.disk_used'] = disk['used']
652+            stats['storage_server.disk_free_for_root'] = disk['free_for_root']
653+            stats['storage_server.disk_free_for_nonroot'] = disk['free_for_nonroot']
654+            stats['storage_server.disk_avail'] = disk['avail']
655+        except AttributeError:
656+            writeable = True
657+        except EnvironmentError:
658+            log.msg("OS call to get disk statistics failed", level=log.UNUSUAL)
659+            writeable = False
660+
661+        if self._readonly:
662+            stats['storage_server.disk_avail'] = 0
663+            writeable = False
664+
665+        stats['storage_server.accepting_immutable_shares'] = int(writeable)
666+
667+    def get_available_space(self):
668+        if self._readonly:
669+            return 0
670+        return fileutil.get_available_space(self._sharedir, self._reserved_space)
671+
672+    def advise_corrupt_share(self, sharetype, storageindex, shnum, reason):
673+        fileutil.fp_make_dirs(self._corruption_advisory_dir)
674+        now = time_format.iso_utc(sep="T")
675+        si_s = si_b2a(storageindex)
676+
677+        # Windows can't handle colons in the filename.
678+        name = ("%s--%s-%d" % (now, si_s, shnum)).replace(":", "")
679+        f = self._corruption_advisory_dir.child(name).open("w")
680+        try:
681+            f.write("report: Share Corruption\n")
682+            f.write("type: %s\n" % sharetype)
683+            f.write("storage_index: %s\n" % si_s)
684+            f.write("share_number: %d\n" % shnum)
685+            f.write("\n")
686+            f.write(reason)
687+            f.write("\n")
688+        finally:
689+            f.close()
690+
691+        log.msg(format=("client claims corruption in (%(share_type)s) " +
692+                        "%(si)s-%(shnum)d: %(reason)s"),
693+                share_type=sharetype, si=si_s, shnum=shnum, reason=reason,
694+                level=log.SCARY, umid="SGx2fA")
695+
696+
697+class DiskShareSet(ShareSet):
698+    implements(IShareSet)
699+
700+    def __init__(self, storageindex, sharehomedir, incominghomedir=None, discard_storage=False):
701+        ShareSet.__init__(self, storageindex)
702+        self._sharehomedir = sharehomedir
703+        self._incominghomedir = incominghomedir
704+        self._discard_storage = discard_storage
705+
706+    def get_overhead(self):
707+        return (fileutil.get_disk_usage(self._sharehomedir) +
708+                fileutil.get_disk_usage(self._incominghomedir))
709+
710+    def get_shares(self):
711+        """
712+        Generate IStorageBackendShare objects for shares we have for this storage index.
713+        ("Shares we have" means completed ones, excluding incoming ones.)
714+        """
715+        try:
716+            for fp in self._sharehomedir.children():
717+                shnumstr = fp.basename()
718+                if not NUM_RE.match(shnumstr):
719+                    continue
720+                sharehome = self._sharehomedir.child(shnumstr)
721+                yield self.get_share(sharehome)
722+        except UnlistableError:
723+            # There is no shares directory at all.
724+            pass
725+
726+    def has_incoming(self, shnum):
727+        if self._incominghomedir is None:
728+            return False
729+        return self._incominghomedir.child(str(shnum)).exists()
730+
731+    def make_bucket_writer(self, storageserver, shnum, max_space_per_bucket, lease_info, canary):
732+        sharehome = self._sharehomedir.child(str(shnum))
733+        incominghome = self._incominghomedir.child(str(shnum))
734+        immsh = ImmutableDiskShare(self.get_storage_index(), shnum, sharehome, incominghome,
735+                                   max_size=max_space_per_bucket, create=True)
736+        bw = BucketWriter(storageserver, immsh, max_space_per_bucket, lease_info, canary)
737+        if self._discard_storage:
738+            bw.throw_out_all_data = True
739+        return bw
740+
741+    def _create_mutable_share(self, storageserver, shnum, write_enabler):
742+        fileutil.fp_make_dirs(self._sharehomedir)
743+        sharehome = self._sharehomedir.child(str(shnum))
744+        serverid = storageserver.get_serverid()
745+        return create_mutable_disk_share(sharehome, serverid, write_enabler, storageserver)
746+
747+    def _clean_up_after_unlink(self):
748+        fileutil.fp_rmdir_if_empty(self._sharehomedir)
749+
750hunk ./src/allmydata/storage/backends/disk/immutable.py 1
751-import os, stat, struct, time
752 
753hunk ./src/allmydata/storage/backends/disk/immutable.py 2
754-from foolscap.api import Referenceable
755+import struct
756 
757 from zope.interface import implements
758hunk ./src/allmydata/storage/backends/disk/immutable.py 5
759-from allmydata.interfaces import RIBucketWriter, RIBucketReader
760-from allmydata.util import base32, fileutil, log
761+
762+from allmydata.interfaces import IStoredShare
763+from allmydata.util import fileutil
764 from allmydata.util.assertutil import precondition
765hunk ./src/allmydata/storage/backends/disk/immutable.py 9
766+from allmydata.util.fileutil import fp_make_dirs
767 from allmydata.util.hashutil import constant_time_compare
768hunk ./src/allmydata/storage/backends/disk/immutable.py 11
769+from allmydata.util.encodingutil import quote_filepath
770+from allmydata.storage.common import si_b2a, UnknownImmutableContainerVersionError, DataTooLargeError
771 from allmydata.storage.lease import LeaseInfo
772hunk ./src/allmydata/storage/backends/disk/immutable.py 14
773-from allmydata.storage.common import UnknownImmutableContainerVersionError, \
774-     DataTooLargeError
775+
776 
777 # each share file (in storage/shares/$SI/$SHNUM) contains lease information
778 # and share data. The share data is accessed by RIBucketWriter.write and
779hunk ./src/allmydata/storage/backends/disk/immutable.py 41
780 # then the value stored in this field will be the actual share data length
781 # modulo 2**32.
782 
783-class ShareFile:
784-    LEASE_SIZE = struct.calcsize(">L32s32sL")
785+class ImmutableDiskShare(object):
786+    implements(IStoredShare)
787+
788     sharetype = "immutable"
789hunk ./src/allmydata/storage/backends/disk/immutable.py 45
790+    LEASE_SIZE = struct.calcsize(">L32s32sL")
791+
792 
793hunk ./src/allmydata/storage/backends/disk/immutable.py 48
794-    def __init__(self, filename, max_size=None, create=False):
795-        """ If max_size is not None then I won't allow more than max_size to be written to me. If create=True and max_size must not be None. """
796+    def __init__(self, storageindex, shnum, finalhome=None, incominghome=None, max_size=None, create=False):
797+        """ If max_size is not None then I won't allow more than
798+        max_size to be written to me. If create=True then max_size
799+        must not be None. """
800         precondition((max_size is not None) or (not create), max_size, create)
801hunk ./src/allmydata/storage/backends/disk/immutable.py 53
802-        self.home = filename
803+        self._storageindex = storageindex
804         self._max_size = max_size
805hunk ./src/allmydata/storage/backends/disk/immutable.py 55
806+        self._incominghome = incominghome
807+        self._home = finalhome
808+        self._shnum = shnum
809         if create:
810             # touch the file, so later callers will see that we're working on
811             # it. Also construct the metadata.
812hunk ./src/allmydata/storage/backends/disk/immutable.py 61
813-            assert not os.path.exists(self.home)
814-            fileutil.make_dirs(os.path.dirname(self.home))
815-            f = open(self.home, 'wb')
816+            assert not finalhome.exists()
817+            fp_make_dirs(self._incominghome.parent())
818             # The second field -- the four-byte share data length -- is no
819             # longer used as of Tahoe v1.3.0, but we continue to write it in
820             # there in case someone downgrades a storage server from >=
821hunk ./src/allmydata/storage/backends/disk/immutable.py 72
822             # the largest length that can fit into the field. That way, even
823             # if this does happen, the old < v1.3.0 server will still allow
824             # clients to read the first part of the share.
825-            f.write(struct.pack(">LLL", 1, min(2**32-1, max_size), 0))
826-            f.close()
827+            self._incominghome.setContent(struct.pack(">LLL", 1, min(2**32-1, max_size), 0) )
828             self._lease_offset = max_size + 0x0c
829             self._num_leases = 0
830         else:
831hunk ./src/allmydata/storage/backends/disk/immutable.py 76
832-            f = open(self.home, 'rb')
833-            filesize = os.path.getsize(self.home)
834-            (version, unused, num_leases) = struct.unpack(">LLL", f.read(0xc))
835-            f.close()
836+            f = self._home.open(mode='rb')
837+            try:
838+                (version, unused, num_leases) = struct.unpack(">LLL", f.read(0xc))
839+            finally:
840+                f.close()
841+            filesize = self._home.getsize()
842             if version != 1:
843                 msg = "sharefile %s had version %d but we wanted 1" % \
844hunk ./src/allmydata/storage/backends/disk/immutable.py 84
845-                      (filename, version)
846+                      (self._home, version)
847                 raise UnknownImmutableContainerVersionError(msg)
848             self._num_leases = num_leases
849             self._lease_offset = filesize - (num_leases * self.LEASE_SIZE)
850hunk ./src/allmydata/storage/backends/disk/immutable.py 90
851         self._data_offset = 0xc
852 
853+    def __repr__(self):
854+        return ("<ImmutableDiskShare %s:%r at %s>"
855+                % (si_b2a(self._storageindex), self._shnum, quote_filepath(self._home)))
856+
857+    def close(self):
858+        fileutil.fp_make_dirs(self._home.parent())
859+        self._incominghome.moveTo(self._home)
860+        try:
861+            # self._incominghome is like storage/shares/incoming/ab/abcde/4 .
862+            # We try to delete the parent (.../ab/abcde) to avoid leaving
863+            # these directories lying around forever, but the delete might
864+            # fail if we're working on another share for the same storage
865+            # index (like ab/abcde/5). The alternative approach would be to
866+            # use a hierarchy of objects (PrefixHolder, BucketHolder,
867+            # ShareWriter), each of which is responsible for a single
868+            # directory on disk, and have them use reference counting of
869+            # their children to know when they should do the rmdir. This
870+            # approach is simpler, but relies on os.rmdir refusing to delete
871+            # a non-empty directory. Do *not* use fileutil.fp_remove() here!
872+            fileutil.fp_rmdir_if_empty(self._incominghome.parent())
873+            # we also delete the grandparent (prefix) directory, .../ab ,
874+            # again to avoid leaving directories lying around. This might
875+            # fail if there is another bucket open that shares a prefix (like
876+            # ab/abfff).
877+            fileutil.fp_rmdir_if_empty(self._incominghome.parent().parent())
878+            # we leave the great-grandparent (incoming/) directory in place.
879+        except EnvironmentError:
880+            # ignore the "can't rmdir because the directory is not empty"
881+            # exceptions, those are normal consequences of the
882+            # above-mentioned conditions.
883+            pass
884+        pass
885+
886+    def get_used_space(self):
887+        return (fileutil.get_used_space(self._home) +
888+                fileutil.get_used_space(self._incominghome))
889+
890+    def get_storage_index(self):
891+        return self._storageindex
892+
893+    def get_shnum(self):
894+        return self._shnum
895+
896     def unlink(self):
897hunk ./src/allmydata/storage/backends/disk/immutable.py 134
898-        os.unlink(self.home)
899+        self._home.remove()
900+
901+    def get_size(self):
902+        return self._home.getsize()
903+
904+    def get_data_length(self):
905+        return self._lease_offset - self._data_offset
906+
907+    #def readv(self, read_vector):
908+    #    ...
909 
910     def read_share_data(self, offset, length):
911         precondition(offset >= 0)
912hunk ./src/allmydata/storage/backends/disk/immutable.py 147
913-        # reads beyond the end of the data are truncated. Reads that start
914+
915+        # Reads beyond the end of the data are truncated. Reads that start
916         # beyond the end of the data return an empty string.
917         seekpos = self._data_offset+offset
918         actuallength = max(0, min(length, self._lease_offset-seekpos))
919hunk ./src/allmydata/storage/backends/disk/immutable.py 154
920         if actuallength == 0:
921             return ""
922-        f = open(self.home, 'rb')
923-        f.seek(seekpos)
924-        return f.read(actuallength)
925+        f = self._home.open(mode='rb')
926+        try:
927+            f.seek(seekpos)
928+            sharedata = f.read(actuallength)
929+        finally:
930+            f.close()
931+        return sharedata
932 
933     def write_share_data(self, offset, data):
934         length = len(data)
935hunk ./src/allmydata/storage/backends/disk/immutable.py 167
936         precondition(offset >= 0, offset)
937         if self._max_size is not None and offset+length > self._max_size:
938             raise DataTooLargeError(self._max_size, offset, length)
939-        f = open(self.home, 'rb+')
940-        real_offset = self._data_offset+offset
941-        f.seek(real_offset)
942-        assert f.tell() == real_offset
943-        f.write(data)
944-        f.close()
945+        f = self._incominghome.open(mode='rb+')
946+        try:
947+            real_offset = self._data_offset+offset
948+            f.seek(real_offset)
949+            assert f.tell() == real_offset
950+            f.write(data)
951+        finally:
952+            f.close()
953 
954     def _write_lease_record(self, f, lease_number, lease_info):
955         offset = self._lease_offset + lease_number * self.LEASE_SIZE
956hunk ./src/allmydata/storage/backends/disk/immutable.py 184
957 
958     def _read_num_leases(self, f):
959         f.seek(0x08)
960-        (num_leases,) = struct.unpack(">L", f.read(4))
961+        ro = f.read(4)
962+        (num_leases,) = struct.unpack(">L", ro)
963         return num_leases
964 
965     def _write_num_leases(self, f, num_leases):
966hunk ./src/allmydata/storage/backends/disk/immutable.py 195
967     def _truncate_leases(self, f, num_leases):
968         f.truncate(self._lease_offset + num_leases * self.LEASE_SIZE)
969 
970+    # These lease operations are intended for use by disk_backend.py.
971+    # Other clients should not depend on the fact that the disk backend
972+    # stores leases in share files.
973+
974     def get_leases(self):
975         """Yields a LeaseInfo instance for all leases."""
976hunk ./src/allmydata/storage/backends/disk/immutable.py 201
977-        f = open(self.home, 'rb')
978-        (version, unused, num_leases) = struct.unpack(">LLL", f.read(0xc))
979-        f.seek(self._lease_offset)
980-        for i in range(num_leases):
981-            data = f.read(self.LEASE_SIZE)
982-            if data:
983-                yield LeaseInfo().from_immutable_data(data)
984+        f = self._home.open(mode='rb')
985+        try:
986+            (version, unused, num_leases) = struct.unpack(">LLL", f.read(0xc))
987+            f.seek(self._lease_offset)
988+            for i in range(num_leases):
989+                data = f.read(self.LEASE_SIZE)
990+                if data:
991+                    yield LeaseInfo().from_immutable_data(data)
992+        finally:
993+            f.close()
994 
995     def add_lease(self, lease_info):
996hunk ./src/allmydata/storage/backends/disk/immutable.py 213
997-        f = open(self.home, 'rb+')
998-        num_leases = self._read_num_leases(f)
999-        self._write_lease_record(f, num_leases, lease_info)
1000-        self._write_num_leases(f, num_leases+1)
1001-        f.close()
1002+        f = self._incominghome.open(mode='rb')
1003+        try:
1004+            num_leases = self._read_num_leases(f)
1005+        finally:
1006+            f.close()
1007+        f = self._home.open(mode='wb+')
1008+        try:
1009+            self._write_lease_record(f, num_leases, lease_info)
1010+            self._write_num_leases(f, num_leases+1)
1011+        finally:
1012+            f.close()
1013 
1014     def renew_lease(self, renew_secret, new_expire_time):
1015hunk ./src/allmydata/storage/backends/disk/immutable.py 226
1016-        for i,lease in enumerate(self.get_leases()):
1017-            if constant_time_compare(lease.renew_secret, renew_secret):
1018-                # yup. See if we need to update the owner time.
1019-                if new_expire_time > lease.expiration_time:
1020-                    # yes
1021-                    lease.expiration_time = new_expire_time
1022-                    f = open(self.home, 'rb+')
1023-                    self._write_lease_record(f, i, lease)
1024-                    f.close()
1025-                return
1026+        try:
1027+            for i, lease in enumerate(self.get_leases()):
1028+                if constant_time_compare(lease.renew_secret, renew_secret):
1029+                    # yup. See if we need to update the owner time.
1030+                    if new_expire_time > lease.expiration_time:
1031+                        # yes
1032+                        lease.expiration_time = new_expire_time
1033+                        f = self._home.open('rb+')
1034+                        try:
1035+                            self._write_lease_record(f, i, lease)
1036+                        finally:
1037+                            f.close()
1038+                    return
1039+        except IndexError, e:
1040+            raise Exception("IndexError: %s" % (e,))
1041         raise IndexError("unable to renew non-existent lease")
1042 
1043     def add_or_renew_lease(self, lease_info):
1044hunk ./src/allmydata/storage/backends/disk/immutable.py 249
1045                              lease_info.expiration_time)
1046         except IndexError:
1047             self.add_lease(lease_info)
1048-
1049-
1050-    def cancel_lease(self, cancel_secret):
1051-        """Remove a lease with the given cancel_secret. If the last lease is
1052-        cancelled, the file will be removed. Return the number of bytes that
1053-        were freed (by truncating the list of leases, and possibly by
1054-        deleting the file. Raise IndexError if there was no lease with the
1055-        given cancel_secret.
1056-        """
1057-
1058-        leases = list(self.get_leases())
1059-        num_leases_removed = 0
1060-        for i,lease in enumerate(leases):
1061-            if constant_time_compare(lease.cancel_secret, cancel_secret):
1062-                leases[i] = None
1063-                num_leases_removed += 1
1064-        if not num_leases_removed:
1065-            raise IndexError("unable to find matching lease to cancel")
1066-        if num_leases_removed:
1067-            # pack and write out the remaining leases. We write these out in
1068-            # the same order as they were added, so that if we crash while
1069-            # doing this, we won't lose any non-cancelled leases.
1070-            leases = [l for l in leases if l] # remove the cancelled leases
1071-            f = open(self.home, 'rb+')
1072-            for i,lease in enumerate(leases):
1073-                self._write_lease_record(f, i, lease)
1074-            self._write_num_leases(f, len(leases))
1075-            self._truncate_leases(f, len(leases))
1076-            f.close()
1077-        space_freed = self.LEASE_SIZE * num_leases_removed
1078-        if not len(leases):
1079-            space_freed += os.stat(self.home)[stat.ST_SIZE]
1080-            self.unlink()
1081-        return space_freed
1082-
1083-
1084-class BucketWriter(Referenceable):
1085-    implements(RIBucketWriter)
1086-
1087-    def __init__(self, ss, incominghome, finalhome, max_size, lease_info, canary):
1088-        self.ss = ss
1089-        self.incominghome = incominghome
1090-        self.finalhome = finalhome
1091-        self._max_size = max_size # don't allow the client to write more than this
1092-        self._canary = canary
1093-        self._disconnect_marker = canary.notifyOnDisconnect(self._disconnected)
1094-        self.closed = False
1095-        self.throw_out_all_data = False
1096-        self._sharefile = ShareFile(incominghome, create=True, max_size=max_size)
1097-        # also, add our lease to the file now, so that other ones can be
1098-        # added by simultaneous uploaders
1099-        self._sharefile.add_lease(lease_info)
1100-
1101-    def allocated_size(self):
1102-        return self._max_size
1103-
1104-    def remote_write(self, offset, data):
1105-        start = time.time()
1106-        precondition(not self.closed)
1107-        if self.throw_out_all_data:
1108-            return
1109-        self._sharefile.write_share_data(offset, data)
1110-        self.ss.add_latency("write", time.time() - start)
1111-        self.ss.count("write")
1112-
1113-    def remote_close(self):
1114-        precondition(not self.closed)
1115-        start = time.time()
1116-
1117-        fileutil.make_dirs(os.path.dirname(self.finalhome))
1118-        fileutil.rename(self.incominghome, self.finalhome)
1119-        try:
1120-            # self.incominghome is like storage/shares/incoming/ab/abcde/4 .
1121-            # We try to delete the parent (.../ab/abcde) to avoid leaving
1122-            # these directories lying around forever, but the delete might
1123-            # fail if we're working on another share for the same storage
1124-            # index (like ab/abcde/5). The alternative approach would be to
1125-            # use a hierarchy of objects (PrefixHolder, BucketHolder,
1126-            # ShareWriter), each of which is responsible for a single
1127-            # directory on disk, and have them use reference counting of
1128-            # their children to know when they should do the rmdir. This
1129-            # approach is simpler, but relies on os.rmdir refusing to delete
1130-            # a non-empty directory. Do *not* use fileutil.rm_dir() here!
1131-            os.rmdir(os.path.dirname(self.incominghome))
1132-            # we also delete the grandparent (prefix) directory, .../ab ,
1133-            # again to avoid leaving directories lying around. This might
1134-            # fail if there is another bucket open that shares a prefix (like
1135-            # ab/abfff).
1136-            os.rmdir(os.path.dirname(os.path.dirname(self.incominghome)))
1137-            # we leave the great-grandparent (incoming/) directory in place.
1138-        except EnvironmentError:
1139-            # ignore the "can't rmdir because the directory is not empty"
1140-            # exceptions, those are normal consequences of the
1141-            # above-mentioned conditions.
1142-            pass
1143-        self._sharefile = None
1144-        self.closed = True
1145-        self._canary.dontNotifyOnDisconnect(self._disconnect_marker)
1146-
1147-        filelen = os.stat(self.finalhome)[stat.ST_SIZE]
1148-        self.ss.bucket_writer_closed(self, filelen)
1149-        self.ss.add_latency("close", time.time() - start)
1150-        self.ss.count("close")
1151-
1152-    def _disconnected(self):
1153-        if not self.closed:
1154-            self._abort()
1155-
1156-    def remote_abort(self):
1157-        log.msg("storage: aborting sharefile %s" % self.incominghome,
1158-                facility="tahoe.storage", level=log.UNUSUAL)
1159-        if not self.closed:
1160-            self._canary.dontNotifyOnDisconnect(self._disconnect_marker)
1161-        self._abort()
1162-        self.ss.count("abort")
1163-
1164-    def _abort(self):
1165-        if self.closed:
1166-            return
1167-
1168-        os.remove(self.incominghome)
1169-        # if we were the last share to be moved, remove the incoming/
1170-        # directory that was our parent
1171-        parentdir = os.path.split(self.incominghome)[0]
1172-        if not os.listdir(parentdir):
1173-            os.rmdir(parentdir)
1174-        self._sharefile = None
1175-
1176-        # We are now considered closed for further writing. We must tell
1177-        # the storage server about this so that it stops expecting us to
1178-        # use the space it allocated for us earlier.
1179-        self.closed = True
1180-        self.ss.bucket_writer_closed(self, 0)
1181-
1182-
1183-class BucketReader(Referenceable):
1184-    implements(RIBucketReader)
1185-
1186-    def __init__(self, ss, sharefname, storage_index=None, shnum=None):
1187-        self.ss = ss
1188-        self._share_file = ShareFile(sharefname)
1189-        self.storage_index = storage_index
1190-        self.shnum = shnum
1191-
1192-    def __repr__(self):
1193-        return "<%s %s %s>" % (self.__class__.__name__,
1194-                               base32.b2a_l(self.storage_index[:8], 60),
1195-                               self.shnum)
1196-
1197-    def remote_read(self, offset, length):
1198-        start = time.time()
1199-        data = self._share_file.read_share_data(offset, length)
1200-        self.ss.add_latency("read", time.time() - start)
1201-        self.ss.count("read")
1202-        return data
1203-
1204-    def remote_advise_corrupt_share(self, reason):
1205-        return self.ss.remote_advise_corrupt_share("immutable",
1206-                                                   self.storage_index,
1207-                                                   self.shnum,
1208-                                                   reason)
1209hunk ./src/allmydata/storage/backends/disk/mutable.py 1
1210-import os, stat, struct
1211 
1212hunk ./src/allmydata/storage/backends/disk/mutable.py 2
1213-from allmydata.interfaces import BadWriteEnablerError
1214-from allmydata.util import idlib, log
1215+import struct
1216+
1217+from zope.interface import implements
1218+
1219+from allmydata.interfaces import IStoredMutableShare, BadWriteEnablerError
1220+from allmydata.util import fileutil, idlib, log
1221 from allmydata.util.assertutil import precondition
1222 from allmydata.util.hashutil import constant_time_compare
1223hunk ./src/allmydata/storage/backends/disk/mutable.py 10
1224-from allmydata.storage.lease import LeaseInfo
1225-from allmydata.storage.common import UnknownMutableContainerVersionError, \
1226+from allmydata.util.encodingutil import quote_filepath
1227+from allmydata.storage.common import si_b2a, UnknownMutableContainerVersionError, \
1228      DataTooLargeError
1229hunk ./src/allmydata/storage/backends/disk/mutable.py 13
1230+from allmydata.storage.lease import LeaseInfo
1231+from allmydata.storage.backends.base import testv_compare
1232 
1233hunk ./src/allmydata/storage/backends/disk/mutable.py 16
1234-# the MutableShareFile is like the ShareFile, but used for mutable data. It
1235-# has a different layout. See docs/mutable.txt for more details.
1236+
1237+# The MutableDiskShare is like the ImmutableDiskShare, but used for mutable data.
1238+# It has a different layout. See docs/mutable.rst for more details.
1239 
1240 # #   offset    size    name
1241 # 1   0         32      magic verstr "tahoe mutable container v1" plus binary
1242hunk ./src/allmydata/storage/backends/disk/mutable.py 31
1243 #                        4    4   expiration timestamp
1244 #                        8   32   renewal token
1245 #                        40  32   cancel token
1246-#                        72  20   nodeid which accepted the tokens
1247+#                        72  20   nodeid that accepted the tokens
1248 # 7   468       (a)     data
1249 # 8   ??        4       count of extra leases
1250 # 9   ??        n*92    extra leases
1251hunk ./src/allmydata/storage/backends/disk/mutable.py 37
1252 
1253 
1254-# The struct module doc says that L's are 4 bytes in size., and that Q's are
1255+# The struct module doc says that L's are 4 bytes in size, and that Q's are
1256 # 8 bytes in size. Since compatibility depends upon this, double-check it.
1257 assert struct.calcsize(">L") == 4, struct.calcsize(">L")
1258 assert struct.calcsize(">Q") == 8, struct.calcsize(">Q")
1259hunk ./src/allmydata/storage/backends/disk/mutable.py 42
1260 
1261-class MutableShareFile:
1262+
1263+class MutableDiskShare(object):
1264+    implements(IStoredMutableShare)
1265 
1266     sharetype = "mutable"
1267     DATA_LENGTH_OFFSET = struct.calcsize(">32s20s32s")
1268hunk ./src/allmydata/storage/backends/disk/mutable.py 54
1269     assert LEASE_SIZE == 92
1270     DATA_OFFSET = HEADER_SIZE + 4*LEASE_SIZE
1271     assert DATA_OFFSET == 468, DATA_OFFSET
1272+
1273     # our sharefiles share with a recognizable string, plus some random
1274     # binary data to reduce the chance that a regular text file will look
1275     # like a sharefile.
1276hunk ./src/allmydata/storage/backends/disk/mutable.py 63
1277     MAX_SIZE = 2*1000*1000*1000 # 2GB, kind of arbitrary
1278     # TODO: decide upon a policy for max share size
1279 
1280-    def __init__(self, filename, parent=None):
1281-        self.home = filename
1282-        if os.path.exists(self.home):
1283+    def __init__(self, storageindex, shnum, home, parent=None):
1284+        self._storageindex = storageindex
1285+        self._shnum = shnum
1286+        self._home = home
1287+        if self._home.exists():
1288             # we don't cache anything, just check the magic
1289hunk ./src/allmydata/storage/backends/disk/mutable.py 69
1290-            f = open(self.home, 'rb')
1291-            data = f.read(self.HEADER_SIZE)
1292-            (magic,
1293-             write_enabler_nodeid, write_enabler,
1294-             data_length, extra_least_offset) = \
1295-             struct.unpack(">32s20s32sQQ", data)
1296-            if magic != self.MAGIC:
1297-                msg = "sharefile %s had magic '%r' but we wanted '%r'" % \
1298-                      (filename, magic, self.MAGIC)
1299-                raise UnknownMutableContainerVersionError(msg)
1300+            f = self._home.open('rb')
1301+            try:
1302+                data = f.read(self.HEADER_SIZE)
1303+                (magic,
1304+                 write_enabler_nodeid, write_enabler,
1305+                 data_length, extra_least_offset) = \
1306+                 struct.unpack(">32s20s32sQQ", data)
1307+                if magic != self.MAGIC:
1308+                    msg = "sharefile %s had magic '%r' but we wanted '%r'" % \
1309+                          (quote_filepath(self._home), magic, self.MAGIC)
1310+                    raise UnknownMutableContainerVersionError(msg)
1311+            finally:
1312+                f.close()
1313         self.parent = parent # for logging
1314 
1315     def log(self, *args, **kwargs):
1316hunk ./src/allmydata/storage/backends/disk/mutable.py 87
1317         return self.parent.log(*args, **kwargs)
1318 
1319-    def create(self, my_nodeid, write_enabler):
1320-        assert not os.path.exists(self.home)
1321+    def create(self, serverid, write_enabler):
1322+        assert not self._home.exists()
1323         data_length = 0
1324         extra_lease_offset = (self.HEADER_SIZE
1325                               + 4 * self.LEASE_SIZE
1326hunk ./src/allmydata/storage/backends/disk/mutable.py 95
1327                               + data_length)
1328         assert extra_lease_offset == self.DATA_OFFSET # true at creation
1329         num_extra_leases = 0
1330-        f = open(self.home, 'wb')
1331-        header = struct.pack(">32s20s32sQQ",
1332-                             self.MAGIC, my_nodeid, write_enabler,
1333-                             data_length, extra_lease_offset,
1334-                             )
1335-        leases = ("\x00"*self.LEASE_SIZE) * 4
1336-        f.write(header + leases)
1337-        # data goes here, empty after creation
1338-        f.write(struct.pack(">L", num_extra_leases))
1339-        # extra leases go here, none at creation
1340-        f.close()
1341+        f = self._home.open('wb')
1342+        try:
1343+            header = struct.pack(">32s20s32sQQ",
1344+                                 self.MAGIC, serverid, write_enabler,
1345+                                 data_length, extra_lease_offset,
1346+                                 )
1347+            leases = ("\x00"*self.LEASE_SIZE) * 4
1348+            f.write(header + leases)
1349+            # data goes here, empty after creation
1350+            f.write(struct.pack(">L", num_extra_leases))
1351+            # extra leases go here, none at creation
1352+        finally:
1353+            f.close()
1354+
1355+    def __repr__(self):
1356+        return ("<MutableDiskShare %s:%r at %s>"
1357+                % (si_b2a(self._storageindex), self._shnum, quote_filepath(self._home)))
1358+
1359+    def get_used_space(self):
1360+        return fileutil.get_used_space(self._home)
1361+
1362+    def get_storage_index(self):
1363+        return self._storageindex
1364+
1365+    def get_shnum(self):
1366+        return self._shnum
1367 
1368     def unlink(self):
1369hunk ./src/allmydata/storage/backends/disk/mutable.py 123
1370-        os.unlink(self.home)
1371+        self._home.remove()
1372 
1373     def _read_data_length(self, f):
1374         f.seek(self.DATA_LENGTH_OFFSET)
1375hunk ./src/allmydata/storage/backends/disk/mutable.py 291
1376 
1377     def get_leases(self):
1378         """Yields a LeaseInfo instance for all leases."""
1379-        f = open(self.home, 'rb')
1380-        for i, lease in self._enumerate_leases(f):
1381-            yield lease
1382-        f.close()
1383+        f = self._home.open('rb')
1384+        try:
1385+            for i, lease in self._enumerate_leases(f):
1386+                yield lease
1387+        finally:
1388+            f.close()
1389 
1390     def _enumerate_leases(self, f):
1391         for i in range(self._get_num_lease_slots(f)):
1392hunk ./src/allmydata/storage/backends/disk/mutable.py 303
1393             try:
1394                 data = self._read_lease_record(f, i)
1395                 if data is not None:
1396-                    yield i,data
1397+                    yield i, data
1398             except IndexError:
1399                 return
1400 
1401hunk ./src/allmydata/storage/backends/disk/mutable.py 307
1402+    # These lease operations are intended for use by disk_backend.py.
1403+    # Other non-test clients should not depend on the fact that the disk
1404+    # backend stores leases in share files.
1405+
1406     def add_lease(self, lease_info):
1407         precondition(lease_info.owner_num != 0) # 0 means "no lease here"
1408hunk ./src/allmydata/storage/backends/disk/mutable.py 313
1409-        f = open(self.home, 'rb+')
1410-        num_lease_slots = self._get_num_lease_slots(f)
1411-        empty_slot = self._get_first_empty_lease_slot(f)
1412-        if empty_slot is not None:
1413-            self._write_lease_record(f, empty_slot, lease_info)
1414-        else:
1415-            self._write_lease_record(f, num_lease_slots, lease_info)
1416-        f.close()
1417+        f = self._home.open('rb+')
1418+        try:
1419+            num_lease_slots = self._get_num_lease_slots(f)
1420+            empty_slot = self._get_first_empty_lease_slot(f)
1421+            if empty_slot is not None:
1422+                self._write_lease_record(f, empty_slot, lease_info)
1423+            else:
1424+                self._write_lease_record(f, num_lease_slots, lease_info)
1425+        finally:
1426+            f.close()
1427 
1428     def renew_lease(self, renew_secret, new_expire_time):
1429         accepting_nodeids = set()
1430hunk ./src/allmydata/storage/backends/disk/mutable.py 326
1431-        f = open(self.home, 'rb+')
1432-        for (leasenum,lease) in self._enumerate_leases(f):
1433-            if constant_time_compare(lease.renew_secret, renew_secret):
1434-                # yup. See if we need to update the owner time.
1435-                if new_expire_time > lease.expiration_time:
1436-                    # yes
1437-                    lease.expiration_time = new_expire_time
1438-                    self._write_lease_record(f, leasenum, lease)
1439-                f.close()
1440-                return
1441-            accepting_nodeids.add(lease.nodeid)
1442-        f.close()
1443+        f = self._home.open('rb+')
1444+        try:
1445+            for (leasenum, lease) in self._enumerate_leases(f):
1446+                if constant_time_compare(lease.renew_secret, renew_secret):
1447+                    # yup. See if we need to update the owner time.
1448+                    if new_expire_time > lease.expiration_time:
1449+                        # yes
1450+                        lease.expiration_time = new_expire_time
1451+                        self._write_lease_record(f, leasenum, lease)
1452+                    return
1453+                accepting_nodeids.add(lease.nodeid)
1454+        finally:
1455+            f.close()
1456         # Return the accepting_nodeids set, to give the client a chance to
1457hunk ./src/allmydata/storage/backends/disk/mutable.py 340
1458-        # update the leases on a share which has been migrated from its
1459+        # update the leases on a share that has been migrated from its
1460         # original server to a new one.
1461         msg = ("Unable to renew non-existent lease. I have leases accepted by"
1462                " nodeids: ")
1463hunk ./src/allmydata/storage/backends/disk/mutable.py 357
1464         except IndexError:
1465             self.add_lease(lease_info)
1466 
1467-    def cancel_lease(self, cancel_secret):
1468-        """Remove any leases with the given cancel_secret. If the last lease
1469-        is cancelled, the file will be removed. Return the number of bytes
1470-        that were freed (by truncating the list of leases, and possibly by
1471-        deleting the file. Raise IndexError if there was no lease with the
1472-        given cancel_secret."""
1473-
1474-        accepting_nodeids = set()
1475-        modified = 0
1476-        remaining = 0
1477-        blank_lease = LeaseInfo(owner_num=0,
1478-                                renew_secret="\x00"*32,
1479-                                cancel_secret="\x00"*32,
1480-                                expiration_time=0,
1481-                                nodeid="\x00"*20)
1482-        f = open(self.home, 'rb+')
1483-        for (leasenum,lease) in self._enumerate_leases(f):
1484-            accepting_nodeids.add(lease.nodeid)
1485-            if constant_time_compare(lease.cancel_secret, cancel_secret):
1486-                self._write_lease_record(f, leasenum, blank_lease)
1487-                modified += 1
1488-            else:
1489-                remaining += 1
1490-        if modified:
1491-            freed_space = self._pack_leases(f)
1492-            f.close()
1493-            if not remaining:
1494-                freed_space += os.stat(self.home)[stat.ST_SIZE]
1495-                self.unlink()
1496-            return freed_space
1497-
1498-        msg = ("Unable to cancel non-existent lease. I have leases "
1499-               "accepted by nodeids: ")
1500-        msg += ",".join([("'%s'" % idlib.nodeid_b2a(anid))
1501-                         for anid in accepting_nodeids])
1502-        msg += " ."
1503-        raise IndexError(msg)
1504-
1505-    def _pack_leases(self, f):
1506-        # TODO: reclaim space from cancelled leases
1507-        return 0
1508-
1509     def _read_write_enabler_and_nodeid(self, f):
1510         f.seek(0)
1511         data = f.read(self.HEADER_SIZE)
1512hunk ./src/allmydata/storage/backends/disk/mutable.py 369
1513 
1514     def readv(self, readv):
1515         datav = []
1516-        f = open(self.home, 'rb')
1517-        for (offset, length) in readv:
1518-            datav.append(self._read_share_data(f, offset, length))
1519-        f.close()
1520+        f = self._home.open('rb')
1521+        try:
1522+            for (offset, length) in readv:
1523+                datav.append(self._read_share_data(f, offset, length))
1524+        finally:
1525+            f.close()
1526         return datav
1527 
1528hunk ./src/allmydata/storage/backends/disk/mutable.py 377
1529-#    def remote_get_length(self):
1530-#        f = open(self.home, 'rb')
1531-#        data_length = self._read_data_length(f)
1532-#        f.close()
1533-#        return data_length
1534+    def get_size(self):
1535+        return self._home.getsize()
1536+
1537+    def get_data_length(self):
1538+        f = self._home.open('rb')
1539+        try:
1540+            data_length = self._read_data_length(f)
1541+        finally:
1542+            f.close()
1543+        return data_length
1544 
1545     def check_write_enabler(self, write_enabler, si_s):
1546hunk ./src/allmydata/storage/backends/disk/mutable.py 389
1547-        f = open(self.home, 'rb+')
1548-        (real_write_enabler, write_enabler_nodeid) = \
1549-                             self._read_write_enabler_and_nodeid(f)
1550-        f.close()
1551+        f = self._home.open('rb+')
1552+        try:
1553+            (real_write_enabler, write_enabler_nodeid) = self._read_write_enabler_and_nodeid(f)
1554+        finally:
1555+            f.close()
1556         # avoid a timing attack
1557         #if write_enabler != real_write_enabler:
1558         if not constant_time_compare(write_enabler, real_write_enabler):
1559hunk ./src/allmydata/storage/backends/disk/mutable.py 410
1560 
1561     def check_testv(self, testv):
1562         test_good = True
1563-        f = open(self.home, 'rb+')
1564-        for (offset, length, operator, specimen) in testv:
1565-            data = self._read_share_data(f, offset, length)
1566-            if not testv_compare(data, operator, specimen):
1567-                test_good = False
1568-                break
1569-        f.close()
1570+        f = self._home.open('rb+')
1571+        try:
1572+            for (offset, length, operator, specimen) in testv:
1573+                data = self._read_share_data(f, offset, length)
1574+                if not testv_compare(data, operator, specimen):
1575+                    test_good = False
1576+                    break
1577+        finally:
1578+            f.close()
1579         return test_good
1580 
1581     def writev(self, datav, new_length):
1582hunk ./src/allmydata/storage/backends/disk/mutable.py 422
1583-        f = open(self.home, 'rb+')
1584-        for (offset, data) in datav:
1585-            self._write_share_data(f, offset, data)
1586-        if new_length is not None:
1587-            cur_length = self._read_data_length(f)
1588-            if new_length < cur_length:
1589-                self._write_data_length(f, new_length)
1590-                # TODO: if we're going to shrink the share file when the
1591-                # share data has shrunk, then call
1592-                # self._change_container_size() here.
1593-        f.close()
1594-
1595-def testv_compare(a, op, b):
1596-    assert op in ("lt", "le", "eq", "ne", "ge", "gt")
1597-    if op == "lt":
1598-        return a < b
1599-    if op == "le":
1600-        return a <= b
1601-    if op == "eq":
1602-        return a == b
1603-    if op == "ne":
1604-        return a != b
1605-    if op == "ge":
1606-        return a >= b
1607-    if op == "gt":
1608-        return a > b
1609-    # never reached
1610+        f = self._home.open('rb+')
1611+        try:
1612+            for (offset, data) in datav:
1613+                self._write_share_data(f, offset, data)
1614+            if new_length is not None:
1615+                cur_length = self._read_data_length(f)
1616+                if new_length < cur_length:
1617+                    self._write_data_length(f, new_length)
1618+                    # TODO: if we're going to shrink the share file when the
1619+                    # share data has shrunk, then call
1620+                    # self._change_container_size() here.
1621+        finally:
1622+            f.close()
1623 
1624hunk ./src/allmydata/storage/backends/disk/mutable.py 436
1625-class EmptyShare:
1626+    def close(self):
1627+        pass
1628 
1629hunk ./src/allmydata/storage/backends/disk/mutable.py 439
1630-    def check_testv(self, testv):
1631-        test_good = True
1632-        for (offset, length, operator, specimen) in testv:
1633-            data = ""
1634-            if not testv_compare(data, operator, specimen):
1635-                test_good = False
1636-                break
1637-        return test_good
1638 
1639hunk ./src/allmydata/storage/backends/disk/mutable.py 440
1640-def create_mutable_sharefile(filename, my_nodeid, write_enabler, parent):
1641-    ms = MutableShareFile(filename, parent)
1642-    ms.create(my_nodeid, write_enabler)
1643+def create_mutable_disk_share(fp, serverid, write_enabler, parent):
1644+    ms = MutableDiskShare(fp, parent)
1645+    ms.create(serverid, write_enabler)
1646     del ms
1647hunk ./src/allmydata/storage/backends/disk/mutable.py 444
1648-    return MutableShareFile(filename, parent)
1649-
1650+    return MutableDiskShare(fp, parent)
1651addfile ./src/allmydata/storage/backends/null/__init__.py
1652addfile ./src/allmydata/storage/backends/null/null_backend.py
1653hunk ./src/allmydata/storage/backends/null/null_backend.py 2
1654 
1655+import os, struct
1656+
1657+from zope.interface import implements
1658+
1659+from allmydata.interfaces import IStorageBackend, IShareSet, IStoredShare, IStoredMutableShare
1660+from allmydata.util.assertutil import precondition
1661+from allmydata.util.hashutil import constant_time_compare
1662+from allmydata.storage.backends.base import Backend, ShareSet
1663+from allmydata.storage.bucket import BucketWriter
1664+from allmydata.storage.common import si_b2a
1665+from allmydata.storage.lease import LeaseInfo
1666+
1667+
1668+class NullBackend(Backend):
1669+    implements(IStorageBackend)
1670+
1671+    def __init__(self):
1672+        Backend.__init__(self)
1673+
1674+    def get_available_space(self, reserved_space):
1675+        return None
1676+
1677+    def get_sharesets_for_prefix(self, prefix):
1678+        pass
1679+
1680+    def get_shareset(self, storageindex):
1681+        return NullShareSet(storageindex)
1682+
1683+    def fill_in_space_stats(self, stats):
1684+        pass
1685+
1686+    def set_storage_server(self, ss):
1687+        self.ss = ss
1688+
1689+    def advise_corrupt_share(self, sharetype, storageindex, shnum, reason):
1690+        pass
1691+
1692+
1693+class NullShareSet(ShareSet):
1694+    implements(IShareSet)
1695+
1696+    def __init__(self, storageindex):
1697+        self.storageindex = storageindex
1698+
1699+    def get_overhead(self):
1700+        return 0
1701+
1702+    def get_incoming_shnums(self):
1703+        return frozenset()
1704+
1705+    def get_shares(self):
1706+        pass
1707+
1708+    def get_share(self, shnum):
1709+        return None
1710+
1711+    def get_storage_index(self):
1712+        return self.storageindex
1713+
1714+    def get_storage_index_string(self):
1715+        return si_b2a(self.storageindex)
1716+
1717+    def make_bucket_writer(self, storageserver, shnum, max_space_per_bucket, lease_info, canary):
1718+        immutableshare = ImmutableNullShare()
1719+        return BucketWriter(self.ss, immutableshare, max_space_per_bucket, lease_info, canary)
1720+
1721+    def _create_mutable_share(self, storageserver, shnum, write_enabler):
1722+        return MutableNullShare()
1723+
1724+    def _clean_up_after_unlink(self):
1725+        pass
1726+
1727+
1728+class ImmutableNullShare:
1729+    implements(IStoredShare)
1730+    sharetype = "immutable"
1731+
1732+    def __init__(self):
1733+        """ If max_size is not None then I won't allow more than
1734+        max_size to be written to me. If create=True then max_size
1735+        must not be None. """
1736+        pass
1737+
1738+    def get_shnum(self):
1739+        return self.shnum
1740+
1741+    def unlink(self):
1742+        os.unlink(self.fname)
1743+
1744+    def read_share_data(self, offset, length):
1745+        precondition(offset >= 0)
1746+        # Reads beyond the end of the data are truncated. Reads that start
1747+        # beyond the end of the data return an empty string.
1748+        seekpos = self._data_offset+offset
1749+        fsize = os.path.getsize(self.fname)
1750+        actuallength = max(0, min(length, fsize-seekpos)) # XXX #1528
1751+        if actuallength == 0:
1752+            return ""
1753+        f = open(self.fname, 'rb')
1754+        f.seek(seekpos)
1755+        return f.read(actuallength)
1756+
1757+    def write_share_data(self, offset, data):
1758+        pass
1759+
1760+    def _write_lease_record(self, f, lease_number, lease_info):
1761+        offset = self._lease_offset + lease_number * self.LEASE_SIZE
1762+        f.seek(offset)
1763+        assert f.tell() == offset
1764+        f.write(lease_info.to_immutable_data())
1765+
1766+    def _read_num_leases(self, f):
1767+        f.seek(0x08)
1768+        (num_leases,) = struct.unpack(">L", f.read(4))
1769+        return num_leases
1770+
1771+    def _write_num_leases(self, f, num_leases):
1772+        f.seek(0x08)
1773+        f.write(struct.pack(">L", num_leases))
1774+
1775+    def _truncate_leases(self, f, num_leases):
1776+        f.truncate(self._lease_offset + num_leases * self.LEASE_SIZE)
1777+
1778+    def get_leases(self):
1779+        """Yields a LeaseInfo instance for all leases."""
1780+        f = open(self.fname, 'rb')
1781+        (version, unused, num_leases) = struct.unpack(">LLL", f.read(0xc))
1782+        f.seek(self._lease_offset)
1783+        for i in range(num_leases):
1784+            data = f.read(self.LEASE_SIZE)
1785+            if data:
1786+                yield LeaseInfo().from_immutable_data(data)
1787+
1788+    def add_lease(self, lease):
1789+        pass
1790+
1791+    def renew_lease(self, renew_secret, new_expire_time):
1792+        for i,lease in enumerate(self.get_leases()):
1793+            if constant_time_compare(lease.renew_secret, renew_secret):
1794+                # yup. See if we need to update the owner time.
1795+                if new_expire_time > lease.expiration_time:
1796+                    # yes
1797+                    lease.expiration_time = new_expire_time
1798+                    f = open(self.fname, 'rb+')
1799+                    self._write_lease_record(f, i, lease)
1800+                    f.close()
1801+                return
1802+        raise IndexError("unable to renew non-existent lease")
1803+
1804+    def add_or_renew_lease(self, lease_info):
1805+        try:
1806+            self.renew_lease(lease_info.renew_secret,
1807+                             lease_info.expiration_time)
1808+        except IndexError:
1809+            self.add_lease(lease_info)
1810+
1811+
1812+class MutableNullShare:
1813+    implements(IStoredMutableShare)
1814+    sharetype = "mutable"
1815+
1816+    """ XXX: TODO """
1817addfile ./src/allmydata/storage/bucket.py
1818hunk ./src/allmydata/storage/bucket.py 1
1819+
1820+import time
1821+
1822+from foolscap.api import Referenceable
1823+
1824+from zope.interface import implements
1825+from allmydata.interfaces import RIBucketWriter, RIBucketReader
1826+from allmydata.util import base32, log
1827+from allmydata.util.assertutil import precondition
1828+
1829+
1830+class BucketWriter(Referenceable):
1831+    implements(RIBucketWriter)
1832+
1833+    def __init__(self, ss, immutableshare, max_size, lease_info, canary):
1834+        self.ss = ss
1835+        self._max_size = max_size # don't allow the client to write more than this
1836+        self._canary = canary
1837+        self._disconnect_marker = canary.notifyOnDisconnect(self._disconnected)
1838+        self.closed = False
1839+        self.throw_out_all_data = False
1840+        self._share = immutableshare
1841+        # also, add our lease to the file now, so that other ones can be
1842+        # added by simultaneous uploaders
1843+        self._share.add_lease(lease_info)
1844+
1845+    def allocated_size(self):
1846+        return self._max_size
1847+
1848+    def remote_write(self, offset, data):
1849+        start = time.time()
1850+        precondition(not self.closed)
1851+        if self.throw_out_all_data:
1852+            return
1853+        self._share.write_share_data(offset, data)
1854+        self.ss.add_latency("write", time.time() - start)
1855+        self.ss.count("write")
1856+
1857+    def remote_close(self):
1858+        precondition(not self.closed)
1859+        start = time.time()
1860+
1861+        self._share.close()
1862+        filelen = self._share.stat()
1863+        self._share = None
1864+
1865+        self.closed = True
1866+        self._canary.dontNotifyOnDisconnect(self._disconnect_marker)
1867+
1868+        self.ss.bucket_writer_closed(self, filelen)
1869+        self.ss.add_latency("close", time.time() - start)
1870+        self.ss.count("close")
1871+
1872+    def _disconnected(self):
1873+        if not self.closed:
1874+            self._abort()
1875+
1876+    def remote_abort(self):
1877+        log.msg("storage: aborting write to share %r" % self._share,
1878+                facility="tahoe.storage", level=log.UNUSUAL)
1879+        if not self.closed:
1880+            self._canary.dontNotifyOnDisconnect(self._disconnect_marker)
1881+        self._abort()
1882+        self.ss.count("abort")
1883+
1884+    def _abort(self):
1885+        if self.closed:
1886+            return
1887+        self._share.unlink()
1888+        self._share = None
1889+
1890+        # We are now considered closed for further writing. We must tell
1891+        # the storage server about this so that it stops expecting us to
1892+        # use the space it allocated for us earlier.
1893+        self.closed = True
1894+        self.ss.bucket_writer_closed(self, 0)
1895+
1896+
1897+class BucketReader(Referenceable):
1898+    implements(RIBucketReader)
1899+
1900+    def __init__(self, ss, share):
1901+        self.ss = ss
1902+        self._share = share
1903+        self.storageindex = share.storageindex
1904+        self.shnum = share.shnum
1905+
1906+    def __repr__(self):
1907+        return "<%s %s %s>" % (self.__class__.__name__,
1908+                               base32.b2a_l(self.storageindex[:8], 60),
1909+                               self.shnum)
1910+
1911+    def remote_read(self, offset, length):
1912+        start = time.time()
1913+        data = self._share.read_share_data(offset, length)
1914+        self.ss.add_latency("read", time.time() - start)
1915+        self.ss.count("read")
1916+        return data
1917+
1918+    def remote_advise_corrupt_share(self, reason):
1919+        return self.ss.remote_advise_corrupt_share("immutable",
1920+                                                   self.storageindex,
1921+                                                   self.shnum,
1922+                                                   reason)
1923addfile ./src/allmydata/test/test_backends.py
1924hunk ./src/allmydata/test/test_backends.py 1
1925+import os, stat
1926+from twisted.trial import unittest
1927+from allmydata.util.log import msg
1928+from allmydata.test.common_util import ReallyEqualMixin
1929+import mock
1930+
1931+# This is the code that we're going to be testing.
1932+from allmydata.storage.server import StorageServer
1933+from allmydata.storage.backends.disk.disk_backend import DiskBackend, si_si2dir
1934+from allmydata.storage.backends.null.null_backend import NullBackend
1935+
1936+# The following share file content was generated with
1937+# storage.immutable.ShareFile from Tahoe-LAFS v1.8.2
1938+# with share data == 'a'. The total size of this input
1939+# is 85 bytes.
1940+shareversionnumber = '\x00\x00\x00\x01'
1941+sharedatalength = '\x00\x00\x00\x01'
1942+numberofleases = '\x00\x00\x00\x01'
1943+shareinputdata = 'a'
1944+ownernumber = '\x00\x00\x00\x00'
1945+renewsecret  = 'x'*32
1946+cancelsecret = 'y'*32
1947+expirationtime = '\x00(\xde\x80'
1948+nextlease = ''
1949+containerdata = shareversionnumber + sharedatalength + numberofleases
1950+client_data = shareinputdata + ownernumber + renewsecret + \
1951+    cancelsecret + expirationtime + nextlease
1952+share_data = containerdata + client_data
1953+testnodeid = 'testnodeidxxxxxxxxxx'
1954+
1955+
1956+class MockFileSystem(unittest.TestCase):
1957+    """ I simulate a filesystem that the code under test can use. I simulate
1958+    just the parts of the filesystem that the current implementation of Disk
1959+    backend needs. """
1960+    def setUp(self):
1961+        # Make patcher, patch, and effects for disk-using functions.
1962+        msg( "%s.setUp()" % (self,))
1963+        self.mockedfilepaths = {}
1964+        # keys are pathnames, values are MockFilePath objects. This is necessary because
1965+        # MockFilePath behavior sometimes depends on the filesystem. Where it does,
1966+        # self.mockedfilepaths has the relevant information.
1967+        self.storedir = MockFilePath('teststoredir', self.mockedfilepaths)
1968+        self.basedir = self.storedir.child('shares')
1969+        self.baseincdir = self.basedir.child('incoming')
1970+        self.sharedirfinalname = self.basedir.child('or').child('orsxg5dtorxxeylhmvpws3temv4a')
1971+        self.sharedirincomingname = self.baseincdir.child('or').child('orsxg5dtorxxeylhmvpws3temv4a')
1972+        self.shareincomingname = self.sharedirincomingname.child('0')
1973+        self.sharefinalname = self.sharedirfinalname.child('0')
1974+
1975+        # FIXME: these patches won't work; disk_backend no longer imports FilePath, BucketCountingCrawler,
1976+        # or LeaseCheckingCrawler.
1977+
1978+        self.FilePathFake = mock.patch('allmydata.storage.backends.disk.disk_backend.FilePath', new = MockFilePath)
1979+        self.FilePathFake.__enter__()
1980+
1981+        self.BCountingCrawler = mock.patch('allmydata.storage.backends.disk.disk_backend.BucketCountingCrawler')
1982+        FakeBCC = self.BCountingCrawler.__enter__()
1983+        FakeBCC.side_effect = self.call_FakeBCC
1984+
1985+        self.LeaseCheckingCrawler = mock.patch('allmydata.storage.backends.disk.disk_backend.LeaseCheckingCrawler')
1986+        FakeLCC = self.LeaseCheckingCrawler.__enter__()
1987+        FakeLCC.side_effect = self.call_FakeLCC
1988+
1989+        self.get_available_space = mock.patch('allmydata.util.fileutil.get_available_space')
1990+        GetSpace = self.get_available_space.__enter__()
1991+        GetSpace.side_effect = self.call_get_available_space
1992+
1993+        self.statforsize = mock.patch('allmydata.storage.backends.disk.core.filepath.stat')
1994+        getsize = self.statforsize.__enter__()
1995+        getsize.side_effect = self.call_statforsize
1996+
1997+    def call_FakeBCC(self, StateFile):
1998+        return MockBCC()
1999+
2000+    def call_FakeLCC(self, StateFile, HistoryFile, ExpirationPolicy):
2001+        return MockLCC()
2002+
2003+    def call_get_available_space(self, storedir, reservedspace):
2004+        # The input vector has an input size of 85.
2005+        return 85 - reservedspace
2006+
2007+    def call_statforsize(self, fakefpname):
2008+        return self.mockedfilepaths[fakefpname].fileobject.size()
2009+
2010+    def tearDown(self):
2011+        msg( "%s.tearDown()" % (self,))
2012+        self.FilePathFake.__exit__()
2013+        self.mockedfilepaths = {}
2014+
2015+
2016+class MockFilePath:
2017+    def __init__(self, pathstring, ffpathsenvironment, existence=False):
2018+        #  I can't just make the values MockFileObjects because they may be directories.
2019+        self.mockedfilepaths = ffpathsenvironment
2020+        self.path = pathstring
2021+        self.existence = existence
2022+        if not self.mockedfilepaths.has_key(self.path):
2023+            #  The first MockFilePath object is special
2024+            self.mockedfilepaths[self.path] = self
2025+            self.fileobject = None
2026+        else:
2027+            self.fileobject = self.mockedfilepaths[self.path].fileobject
2028+        self.spawn = {}
2029+        self.antecedent = os.path.dirname(self.path)
2030+
2031+    def setContent(self, contentstring):
2032+        # This method rewrites the data in the file that corresponds to its path
2033+        # name whether it preexisted or not.
2034+        self.fileobject = MockFileObject(contentstring)
2035+        self.existence = True
2036+        self.mockedfilepaths[self.path].fileobject = self.fileobject
2037+        self.mockedfilepaths[self.path].existence = self.existence
2038+        self.setparents()
2039+
2040+    def create(self):
2041+        # This method chokes if there's a pre-existing file!
2042+        if self.mockedfilepaths[self.path].fileobject:
2043+            raise OSError
2044+        else:
2045+            self.existence = True
2046+            self.mockedfilepaths[self.path].fileobject = self.fileobject
2047+            self.mockedfilepaths[self.path].existence = self.existence
2048+            self.setparents()
2049+
2050+    def open(self, mode='r'):
2051+        # XXX Makes no use of mode.
2052+        if not self.mockedfilepaths[self.path].fileobject:
2053+            # If there's no fileobject there already then make one and put it there.
2054+            self.fileobject = MockFileObject()
2055+            self.existence = True
2056+            self.mockedfilepaths[self.path].fileobject = self.fileobject
2057+            self.mockedfilepaths[self.path].existence = self.existence
2058+        else:
2059+            # Otherwise get a ref to it.
2060+            self.fileobject = self.mockedfilepaths[self.path].fileobject
2061+            self.existence = self.mockedfilepaths[self.path].existence
2062+        return self.fileobject.open(mode)
2063+
2064+    def child(self, childstring):
2065+        arg2child = os.path.join(self.path, childstring)
2066+        child = MockFilePath(arg2child, self.mockedfilepaths)
2067+        return child
2068+
2069+    def children(self):
2070+        childrenfromffs = [ffp for ffp in self.mockedfilepaths.values() if ffp.path.startswith(self.path)]
2071+        childrenfromffs = [ffp for ffp in childrenfromffs if not ffp.path.endswith(self.path)]
2072+        childrenfromffs = [ffp for ffp in childrenfromffs if ffp.exists()]
2073+        self.spawn = frozenset(childrenfromffs)
2074+        return self.spawn
2075+
2076+    def parent(self):
2077+        if self.mockedfilepaths.has_key(self.antecedent):
2078+            parent = self.mockedfilepaths[self.antecedent]
2079+        else:
2080+            parent = MockFilePath(self.antecedent, self.mockedfilepaths)
2081+        return parent
2082+
2083+    def parents(self):
2084+        antecedents = []
2085+        def f(fps, antecedents):
2086+            newfps = os.path.split(fps)[0]
2087+            if newfps:
2088+                antecedents.append(newfps)
2089+                f(newfps, antecedents)
2090+        f(self.path, antecedents)
2091+        return antecedents
2092+
2093+    def setparents(self):
2094+        for fps in self.parents():
2095+            if not self.mockedfilepaths.has_key(fps):
2096+                self.mockedfilepaths[fps] = MockFilePath(fps, self.mockedfilepaths, exists=True)
2097+
2098+    def basename(self):
2099+        return os.path.split(self.path)[1]
2100+
2101+    def moveTo(self, newffp):
2102+        #  XXX Makes no distinction between file and directory arguments, this is deviation from filepath.moveTo
2103+        if self.mockedfilepaths[newffp.path].exists():
2104+            raise OSError
2105+        else:
2106+            self.mockedfilepaths[newffp.path] = self
2107+            self.path = newffp.path
2108+
2109+    def getsize(self):
2110+        return self.fileobject.getsize()
2111+
2112+    def exists(self):
2113+        return self.existence
2114+
2115+    def isdir(self):
2116+        return True
2117+
2118+    def makedirs(self):
2119+        # XXX These methods assume that fp_<FOO> functions in fileutil will be tested elsewhere!
2120+        pass
2121+
2122+    def remove(self):
2123+        pass
2124+
2125+
2126+class MockFileObject:
2127+    def __init__(self, contentstring=''):
2128+        self.buffer = contentstring
2129+        self.pos = 0
2130+    def open(self, mode='r'):
2131+        return self
2132+    def write(self, instring):
2133+        begin = self.pos
2134+        padlen = begin - len(self.buffer)
2135+        if padlen > 0:
2136+            self.buffer += '\x00' * padlen
2137+        end = self.pos + len(instring)
2138+        self.buffer = self.buffer[:begin]+instring+self.buffer[end:]
2139+        self.pos = end
2140+    def close(self):
2141+        self.pos = 0
2142+    def seek(self, pos):
2143+        self.pos = pos
2144+    def read(self, numberbytes):
2145+        return self.buffer[self.pos:self.pos+numberbytes]
2146+    def tell(self):
2147+        return self.pos
2148+    def size(self):
2149+        # XXX This method A: Is not to be found in a real file B: Is part of a wild-mung-up of filepath.stat!
2150+        # XXX Finally we shall hopefully use a getsize method soon, must consult first though.
2151+        # Hmmm...  perhaps we need to sometimes stat the address when there's not a mockfileobject present?
2152+        return {stat.ST_SIZE:len(self.buffer)}
2153+    def getsize(self):
2154+        return len(self.buffer)
2155+
2156+class MockBCC:
2157+    def setServiceParent(self, Parent):
2158+        pass
2159+
2160+
2161+class MockLCC:
2162+    def setServiceParent(self, Parent):
2163+        pass
2164+
2165+
2166+class TestServerWithNullBackend(unittest.TestCase, ReallyEqualMixin):
2167+    """ NullBackend is just for testing and executable documentation, so
2168+    this test is actually a test of StorageServer in which we're using
2169+    NullBackend as helper code for the test, rather than a test of
2170+    NullBackend. """
2171+    def setUp(self):
2172+        self.ss = StorageServer(testnodeid, NullBackend())
2173+
2174+    @mock.patch('os.mkdir')
2175+    @mock.patch('__builtin__.open')
2176+    @mock.patch('os.listdir')
2177+    @mock.patch('os.path.isdir')
2178+    def test_write_share(self, mockisdir, mocklistdir, mockopen, mockmkdir):
2179+        """
2180+        Write a new share. This tests that StorageServer's remote_allocate_buckets
2181+        generates the correct return types when given test-vector arguments. That
2182+        bs is of the correct type is verified by attempting to invoke remote_write
2183+        on bs[0].
2184+        """
2185+        alreadygot, bs = self.ss.remote_allocate_buckets('teststorage_index', 'x'*32, 'y'*32, set((0,)), 1, mock.Mock())
2186+        bs[0].remote_write(0, 'a')
2187+        self.failIf(mockisdir.called)
2188+        self.failIf(mocklistdir.called)
2189+        self.failIf(mockopen.called)
2190+        self.failIf(mockmkdir.called)
2191+
2192+
2193+class TestServerConstruction(MockFileSystem, ReallyEqualMixin):
2194+    def test_create_server_disk_backend(self):
2195+        """ This tests whether a server instance can be constructed with a
2196+        filesystem backend. To pass the test, it mustn't use the filesystem
2197+        outside of its configured storedir. """
2198+        StorageServer(testnodeid, DiskBackend(self.storedir))
2199+
2200+
2201+class TestServerAndDiskBackend(MockFileSystem, ReallyEqualMixin):
2202+    """ This tests both the StorageServer and the Disk backend together. """
2203+    def setUp(self):
2204+        MockFileSystem.setUp(self)
2205+        try:
2206+            self.backend = DiskBackend(self.storedir)
2207+            self.ss = StorageServer(testnodeid, self.backend)
2208+
2209+            self.backendwithreserve = DiskBackend(self.storedir, reserved_space = 1)
2210+            self.sswithreserve = StorageServer(testnodeid, self.backendwithreserve)
2211+        except:
2212+            MockFileSystem.tearDown(self)
2213+            raise
2214+
2215+    @mock.patch('time.time')
2216+    @mock.patch('allmydata.util.fileutil.get_available_space')
2217+    def test_out_of_space(self, mockget_available_space, mocktime):
2218+        mocktime.return_value = 0
2219+
2220+        def call_get_available_space(dir, reserve):
2221+            return 0
2222+
2223+        mockget_available_space.side_effect = call_get_available_space
2224+        alreadygotc, bsc = self.sswithreserve.remote_allocate_buckets('teststorage_index', 'x'*32, 'y'*32, set((0,)), 1, mock.Mock())
2225+        self.failUnlessReallyEqual(bsc, {})
2226+
2227+    @mock.patch('time.time')
2228+    def test_write_and_read_share(self, mocktime):
2229+        """
2230+        Write a new share, read it, and test the server's (and disk backend's)
2231+        handling of simultaneous and successive attempts to write the same
2232+        share.
2233+        """
2234+        mocktime.return_value = 0
2235+        # Inspect incoming and fail unless it's empty.
2236+        incomingset = self.ss.backend.get_incoming_shnums('teststorage_index')
2237+
2238+        self.failUnlessReallyEqual(incomingset, frozenset())
2239+
2240+        # Populate incoming with the sharenum: 0.
2241+        alreadygot, bs = self.ss.remote_allocate_buckets('teststorage_index', 'x'*32, 'y'*32, frozenset((0,)), 1, mock.Mock())
2242+
2243+        # This is a transparent-box test: Inspect incoming and fail unless the sharenum: 0 is listed there.
2244+        self.failUnlessReallyEqual(self.ss.backend.get_incoming_shnums('teststorage_index'), frozenset((0,)))
2245+
2246+
2247+
2248+        # Attempt to create a second share writer with the same sharenum.
2249+        alreadygota, bsa = self.ss.remote_allocate_buckets('teststorage_index', 'x'*32, 'y'*32, frozenset((0,)), 1, mock.Mock())
2250+
2251+        # Show that no sharewriter results from a remote_allocate_buckets
2252+        # with the same si and sharenum, until BucketWriter.remote_close()
2253+        # has been called.
2254+        self.failIf(bsa)
2255+
2256+        # Test allocated size.
2257+        spaceint = self.ss.allocated_size()
2258+        self.failUnlessReallyEqual(spaceint, 1)
2259+
2260+        # Write 'a' to shnum 0. Only tested together with close and read.
2261+        bs[0].remote_write(0, 'a')
2262+
2263+        # Preclose: Inspect final, failUnless nothing there.
2264+        self.failUnlessReallyEqual(len(list(self.backend.get_shares('teststorage_index'))), 0)
2265+        bs[0].remote_close()
2266+
2267+        # Postclose: (Omnibus) failUnless written data is in final.
2268+        sharesinfinal = list(self.backend.get_shares('teststorage_index'))
2269+        self.failUnlessReallyEqual(len(sharesinfinal), 1)
2270+        contents = sharesinfinal[0].read_share_data(0, 73)
2271+        self.failUnlessReallyEqual(contents, client_data)
2272+
2273+        # Exercise the case that the share we're asking to allocate is
2274+        # already (completely) uploaded.
2275+        self.ss.remote_allocate_buckets('teststorage_index', 'x'*32, 'y'*32, set((0,)), 1, mock.Mock())
2276+
2277+
2278+    def test_read_old_share(self):
2279+        """ This tests whether the code correctly finds and reads
2280+        shares written out by old (Tahoe-LAFS <= v1.8.2)
2281+        servers. There is a similar test in test_download, but that one
2282+        is from the perspective of the client and exercises a deeper
2283+        stack of code. This one is for exercising just the
2284+        StorageServer object. """
2285+        # Contruct a file with the appropriate contents in the mockfilesystem.
2286+        datalen = len(share_data)
2287+        finalhome = si_si2dir(self.basedir, 'teststorage_index').child(str(0))
2288+        finalhome.setContent(share_data)
2289+
2290+        # Now begin the test.
2291+        bs = self.ss.remote_get_buckets('teststorage_index')
2292+
2293+        self.failUnlessEqual(len(bs), 1)
2294+        b = bs['0']
2295+        # These should match by definition, the next two cases cover cases without (completely) unambiguous behaviors.
2296+        self.failUnlessReallyEqual(b.remote_read(0, datalen), client_data)
2297+        # If you try to read past the end you get the as much data as is there.
2298+        self.failUnlessReallyEqual(b.remote_read(0, datalen+20), client_data)
2299+        # If you start reading past the end of the file you get the empty string.
2300+        self.failUnlessReallyEqual(b.remote_read(datalen+1, 3), '')
2301}
2302[Pluggable backends -- all other changes. refs #999
2303david-sarah@jacaranda.org**20110919233256
2304 Ignore-this: 1a77b6b5d178b32a9b914b699ba7e957
2305] {
2306hunk ./src/allmydata/client.py 245
2307             sharetypes.append("immutable")
2308         if self.get_config("storage", "expire.mutable", True, boolean=True):
2309             sharetypes.append("mutable")
2310-        expiration_sharetypes = tuple(sharetypes)
2311 
2312hunk ./src/allmydata/client.py 246
2313+        expiration_policy = {
2314+            'enabled': expire,
2315+            'mode': mode,
2316+            'override_lease_duration': o_l_d,
2317+            'cutoff_date': cutoff_date,
2318+            'sharetypes': tuple(sharetypes),
2319+        }
2320         ss = StorageServer(storedir, self.nodeid,
2321                            reserved_space=reserved,
2322                            discard_storage=discard,
2323hunk ./src/allmydata/client.py 258
2324                            readonly_storage=readonly,
2325                            stats_provider=self.stats_provider,
2326-                           expiration_enabled=expire,
2327-                           expiration_mode=mode,
2328-                           expiration_override_lease_duration=o_l_d,
2329-                           expiration_cutoff_date=cutoff_date,
2330-                           expiration_sharetypes=expiration_sharetypes)
2331+                           expiration_policy=expiration_policy)
2332         self.add_service(ss)
2333 
2334         d = self.when_tub_ready()
2335hunk ./src/allmydata/immutable/offloaded.py 306
2336         if os.path.exists(self._encoding_file):
2337             self.log("ciphertext already present, bypassing fetch",
2338                      level=log.UNUSUAL)
2339+            # XXX the following comment is probably stale, since
2340+            # LocalCiphertextReader.get_plaintext_hashtree_leaves does not exist.
2341+            #
2342             # we'll still need the plaintext hashes (when
2343             # LocalCiphertextReader.get_plaintext_hashtree_leaves() is
2344             # called), and currently the easiest way to get them is to ask
2345hunk ./src/allmydata/immutable/upload.py 765
2346             self._status.set_progress(1, progress)
2347         return cryptdata
2348 
2349-
2350     def get_plaintext_hashtree_leaves(self, first, last, num_segments):
2351hunk ./src/allmydata/immutable/upload.py 766
2352+        """OBSOLETE; Get the leaf nodes of a merkle hash tree over the
2353+        plaintext segments, i.e. get the tagged hashes of the given segments.
2354+        The segment size is expected to be generated by the
2355+        IEncryptedUploadable before any plaintext is read or ciphertext
2356+        produced, so that the segment hashes can be generated with only a
2357+        single pass.
2358+
2359+        This returns a Deferred that fires with a sequence of hashes, using:
2360+
2361+         tuple(segment_hashes[first:last])
2362+
2363+        'num_segments' is used to assert that the number of segments that the
2364+        IEncryptedUploadable handled matches the number of segments that the
2365+        encoder was expecting.
2366+
2367+        This method must not be called until the final byte has been read
2368+        from read_encrypted(). Once this method is called, read_encrypted()
2369+        can never be called again.
2370+        """
2371         # this is currently unused, but will live again when we fix #453
2372         if len(self._plaintext_segment_hashes) < num_segments:
2373             # close out the last one
2374hunk ./src/allmydata/immutable/upload.py 803
2375         return defer.succeed(tuple(self._plaintext_segment_hashes[first:last]))
2376 
2377     def get_plaintext_hash(self):
2378+        """OBSOLETE; Get the hash of the whole plaintext.
2379+
2380+        This returns a Deferred that fires with a tagged SHA-256 hash of the
2381+        whole plaintext, obtained from hashutil.plaintext_hash(data).
2382+        """
2383+        # this is currently unused, but will live again when we fix #453
2384         h = self._plaintext_hasher.digest()
2385         return defer.succeed(h)
2386 
2387hunk ./src/allmydata/interfaces.py 29
2388 Number = IntegerConstraint(8) # 2**(8*8) == 16EiB ~= 18e18 ~= 18 exabytes
2389 Offset = Number
2390 ReadSize = int # the 'int' constraint is 2**31 == 2Gib -- large files are processed in not-so-large increments
2391-WriteEnablerSecret = Hash # used to protect mutable bucket modifications
2392-LeaseRenewSecret = Hash # used to protect bucket lease renewal requests
2393-LeaseCancelSecret = Hash # used to protect bucket lease cancellation requests
2394+WriteEnablerSecret = Hash # used to protect mutable share modifications
2395+LeaseRenewSecret = Hash # used to protect lease renewal requests
2396+LeaseCancelSecret = Hash # used to protect lease cancellation requests
2397 
2398 class RIStubClient(RemoteInterface):
2399     """Each client publishes a service announcement for a dummy object called
2400hunk ./src/allmydata/interfaces.py 106
2401                          sharenums=SetOf(int, maxLength=MAX_BUCKETS),
2402                          allocated_size=Offset, canary=Referenceable):
2403         """
2404-        @param storage_index: the index of the bucket to be created or
2405+        @param storage_index: the index of the shareset to be created or
2406                               increfed.
2407         @param sharenums: these are the share numbers (probably between 0 and
2408                           99) that the sender is proposing to store on this
2409hunk ./src/allmydata/interfaces.py 111
2410                           server.
2411-        @param renew_secret: This is the secret used to protect bucket refresh
2412+        @param renew_secret: This is the secret used to protect lease renewal.
2413                              This secret is generated by the client and
2414                              stored for later comparison by the server. Each
2415                              server is given a different secret.
2416hunk ./src/allmydata/interfaces.py 115
2417-        @param cancel_secret: Like renew_secret, but protects bucket decref.
2418-        @param canary: If the canary is lost before close(), the bucket is
2419+        @param cancel_secret: ignored
2420+        @param canary: If the canary is lost before close(), the allocation is
2421                        deleted.
2422         @return: tuple of (alreadygot, allocated), where alreadygot is what we
2423                  already have and allocated is what we hereby agree to accept.
2424hunk ./src/allmydata/interfaces.py 129
2425                   renew_secret=LeaseRenewSecret,
2426                   cancel_secret=LeaseCancelSecret):
2427         """
2428-        Add a new lease on the given bucket. If the renew_secret matches an
2429+        Add a new lease on the given shareset. If the renew_secret matches an
2430         existing lease, that lease will be renewed instead. If there is no
2431hunk ./src/allmydata/interfaces.py 131
2432-        bucket for the given storage_index, return silently. (note that in
2433+        shareset for the given storage_index, return silently. (Note that in
2434         tahoe-1.3.0 and earlier, IndexError was raised if there was no
2435hunk ./src/allmydata/interfaces.py 133
2436-        bucket)
2437+        shareset.)
2438         """
2439         return Any() # returns None now, but future versions might change
2440 
2441hunk ./src/allmydata/interfaces.py 139
2442     def renew_lease(storage_index=StorageIndex, renew_secret=LeaseRenewSecret):
2443         """
2444-        Renew the lease on a given bucket, resetting the timer to 31 days.
2445-        Some networks will use this, some will not. If there is no bucket for
2446+        Renew the lease on a given shareset, resetting the timer to 31 days.
2447+        Some networks will use this, some will not. If there is no shareset for
2448         the given storage_index, IndexError will be raised.
2449 
2450         For mutable shares, if the given renew_secret does not match an
2451hunk ./src/allmydata/interfaces.py 146
2452         existing lease, IndexError will be raised with a note listing the
2453         server-nodeids on the existing leases, so leases on migrated shares
2454-        can be renewed or cancelled. For immutable shares, IndexError
2455-        (without the note) will be raised.
2456+        can be renewed. For immutable shares, IndexError (without the note)
2457+        will be raised.
2458         """
2459         return Any()
2460 
2461hunk ./src/allmydata/interfaces.py 154
2462     def get_buckets(storage_index=StorageIndex):
2463         return DictOf(int, RIBucketReader, maxKeys=MAX_BUCKETS)
2464 
2465-
2466-
2467     def slot_readv(storage_index=StorageIndex,
2468                    shares=ListOf(int), readv=ReadVector):
2469         """Read a vector from the numbered shares associated with the given
2470hunk ./src/allmydata/interfaces.py 163
2471 
2472     def slot_testv_and_readv_and_writev(storage_index=StorageIndex,
2473                                         secrets=TupleOf(WriteEnablerSecret,
2474-                                                        LeaseRenewSecret,
2475-                                                        LeaseCancelSecret),
2476+                                                        LeaseRenewSecret),
2477                                         tw_vectors=TestAndWriteVectorsForShares,
2478                                         r_vector=ReadVector,
2479                                         ):
2480hunk ./src/allmydata/interfaces.py 167
2481-        """General-purpose test-and-set operation for mutable slots. Perform
2482-        a bunch of comparisons against the existing shares. If they all pass,
2483-        then apply a bunch of write vectors to those shares. Then use the
2484-        read vectors to extract data from all the shares and return the data.
2485+        """
2486+        General-purpose atomic test-read-and-set operation for mutable slots.
2487+        Perform a bunch of comparisons against the existing shares. If they
2488+        all pass: use the read vectors to extract data from all the shares,
2489+        then apply a bunch of write vectors to those shares. Return the read
2490+        data, which does not include any modifications made by the writes.
2491 
2492         This method is, um, large. The goal is to allow clients to update all
2493         the shares associated with a mutable file in a single round trip.
2494hunk ./src/allmydata/interfaces.py 177
2495 
2496-        @param storage_index: the index of the bucket to be created or
2497+        @param storage_index: the index of the shareset to be created or
2498                               increfed.
2499         @param write_enabler: a secret that is stored along with the slot.
2500                               Writes are accepted from any caller who can
2501hunk ./src/allmydata/interfaces.py 183
2502                               present the matching secret. A different secret
2503                               should be used for each slot*server pair.
2504-        @param renew_secret: This is the secret used to protect bucket refresh
2505+        @param renew_secret: This is the secret used to protect lease renewal.
2506                              This secret is generated by the client and
2507                              stored for later comparison by the server. Each
2508                              server is given a different secret.
2509hunk ./src/allmydata/interfaces.py 187
2510-        @param cancel_secret: Like renew_secret, but protects bucket decref.
2511+        @param cancel_secret: ignored
2512 
2513hunk ./src/allmydata/interfaces.py 189
2514-        The 'secrets' argument is a tuple of (write_enabler, renew_secret,
2515-        cancel_secret). The first is required to perform any write. The
2516-        latter two are used when allocating new shares. To simply acquire a
2517-        new lease on existing shares, use an empty testv and an empty writev.
2518+        The 'secrets' argument is a tuple with (write_enabler, renew_secret).
2519+        The write_enabler is required to perform any write. The renew_secret
2520+        is used when allocating new shares.
2521 
2522         Each share can have a separate test vector (i.e. a list of
2523         comparisons to perform). If all vectors for all shares pass, then all
2524hunk ./src/allmydata/interfaces.py 280
2525         store that on disk.
2526         """
2527 
2528-class IStorageBucketWriter(Interface):
2529+
2530+class IStorageBackend(Interface):
2531     """
2532hunk ./src/allmydata/interfaces.py 283
2533-    Objects of this kind live on the client side.
2534+    Objects of this kind live on the server side and are used by the
2535+    storage server object.
2536     """
2537hunk ./src/allmydata/interfaces.py 286
2538-    def put_block(segmentnum=int, data=ShareData):
2539-        """@param data: For most segments, this data will be 'blocksize'
2540-        bytes in length. The last segment might be shorter.
2541-        @return: a Deferred that fires (with None) when the operation completes
2542+    def get_available_space():
2543+        """
2544+        Returns available space for share storage in bytes, or
2545+        None if this information is not available or if the available
2546+        space is unlimited.
2547+
2548+        If the backend is configured for read-only mode then this will
2549+        return 0.
2550+        """
2551+
2552+    def get_sharesets_for_prefix(prefix):
2553+        """
2554+        Generates IShareSet objects for all storage indices matching the
2555+        given prefix for which this backend holds shares.
2556+        """
2557+
2558+    def get_shareset(storageindex):
2559+        """
2560+        Get an IShareSet object for the given storage index.
2561+        """
2562+
2563+    def advise_corrupt_share(storageindex, sharetype, shnum, reason):
2564+        """
2565+        Clients who discover hash failures in shares that they have
2566+        downloaded from me will use this method to inform me about the
2567+        failures. I will record their concern so that my operator can
2568+        manually inspect the shares in question.
2569+
2570+        'sharetype' is either 'mutable' or 'immutable'. 'shnum' is the integer
2571+        share number. 'reason' is a human-readable explanation of the problem,
2572+        probably including some expected hash values and the computed ones
2573+        that did not match. Corruption advisories for mutable shares should
2574+        include a hash of the public key (the same value that appears in the
2575+        mutable-file verify-cap), since the current share format does not
2576+        store that on disk.
2577+
2578+        @param storageindex=str
2579+        @param sharetype=str
2580+        @param shnum=int
2581+        @param reason=str
2582+        """
2583+
2584+
2585+class IShareSet(Interface):
2586+    def get_storage_index():
2587+        """
2588+        Returns the storage index for this shareset.
2589+        """
2590+
2591+    def get_storage_index_string():
2592+        """
2593+        Returns the base32-encoded storage index for this shareset.
2594+        """
2595+
2596+    def get_overhead():
2597+        """
2598+        Returns the storage overhead, in bytes, of this shareset (exclusive
2599+        of the space used by its shares).
2600+        """
2601+
2602+    def get_shares():
2603+        """
2604+        Generates the IStoredShare objects held in this shareset.
2605+        """
2606+
2607+    def has_incoming(shnum):
2608+        """
2609+        Returns True if this shareset has an incoming (partial) share with this number, otherwise False.
2610+        """
2611+
2612+    def make_bucket_writer(storageserver, shnum, max_space_per_bucket, lease_info, canary):
2613+        """
2614+        Create a bucket writer that can be used to write data to a given share.
2615+
2616+        @param storageserver=RIStorageServer
2617+        @param shnum=int: A share number in this shareset
2618+        @param max_space_per_bucket=int: The maximum space allocated for the
2619+                 share, in bytes
2620+        @param lease_info=LeaseInfo: The initial lease information
2621+        @param canary=Referenceable: If the canary is lost before close(), the
2622+                 bucket is deleted.
2623+        @return an IStorageBucketWriter for the given share
2624+        """
2625+
2626+    def make_bucket_reader(storageserver, share):
2627+        """
2628+        Create a bucket reader that can be used to read data from a given share.
2629+
2630+        @param storageserver=RIStorageServer
2631+        @param share=IStoredShare
2632+        @return an IStorageBucketReader for the given share
2633+        """
2634+
2635+    def readv(wanted_shnums, read_vector):
2636+        """
2637+        Read a vector from the numbered shares in this shareset. An empty
2638+        wanted_shnums list means to return data from all known shares.
2639+
2640+        @param wanted_shnums=ListOf(int)
2641+        @param read_vector=ReadVector
2642+        @return DictOf(int, ReadData): shnum -> results, with one key per share
2643+        """
2644+
2645+    def testv_and_readv_and_writev(storageserver, secrets, test_and_write_vectors, read_vector, expiration_time):
2646+        """
2647+        General-purpose atomic test-read-and-set operation for mutable slots.
2648+        Perform a bunch of comparisons against the existing shares in this
2649+        shareset. If they all pass: use the read vectors to extract data from
2650+        all the shares, then apply a bunch of write vectors to those shares.
2651+        Return the read data, which does not include any modifications made by
2652+        the writes.
2653+
2654+        See the similar method in RIStorageServer for more detail.
2655+
2656+        @param storageserver=RIStorageServer
2657+        @param secrets=TupleOf(WriteEnablerSecret, LeaseRenewSecret[, ...])
2658+        @param test_and_write_vectors=TestAndWriteVectorsForShares
2659+        @param read_vector=ReadVector
2660+        @param expiration_time=int
2661+        @return TupleOf(bool, DictOf(int, ReadData))
2662+        """
2663+
2664+    def add_or_renew_lease(lease_info):
2665+        """
2666+        Add a new lease on the shares in this shareset. If the renew_secret
2667+        matches an existing lease, that lease will be renewed instead. If
2668+        there are no shares in this shareset, return silently.
2669+
2670+        @param lease_info=LeaseInfo
2671+        """
2672+
2673+    def renew_lease(renew_secret, new_expiration_time):
2674+        """
2675+        Renew a lease on the shares in this shareset, resetting the timer
2676+        to 31 days. Some grids will use this, some will not. If there are no
2677+        shares in this shareset, IndexError will be raised.
2678+
2679+        For mutable shares, if the given renew_secret does not match an
2680+        existing lease, IndexError will be raised with a note listing the
2681+        server-nodeids on the existing leases, so leases on migrated shares
2682+        can be renewed. For immutable shares, IndexError (without the note)
2683+        will be raised.
2684+
2685+        @param renew_secret=LeaseRenewSecret
2686+        """
2687+
2688+
2689+class IStoredShare(Interface):
2690+    """
2691+    This object contains as much as all of the share data.  It is intended
2692+    for lazy evaluation, such that in many use cases substantially less than
2693+    all of the share data will be accessed.
2694+    """
2695+    def close():
2696+        """
2697+        Complete writing to this share.
2698+        """
2699+
2700+    def get_storage_index():
2701+        """
2702+        Returns the storage index.
2703+        """
2704+
2705+    def get_shnum():
2706+        """
2707+        Returns the share number.
2708+        """
2709+
2710+    def get_data_length():
2711+        """
2712+        Returns the data length in bytes.
2713+        """
2714+
2715+    def get_size():
2716+        """
2717+        Returns the size of the share in bytes.
2718+        """
2719+
2720+    def get_used_space():
2721+        """
2722+        Returns the amount of backend storage including overhead, in bytes, used
2723+        by this share.
2724+        """
2725+
2726+    def unlink():
2727+        """
2728+        Signal that this share can be removed from the backend storage. This does
2729+        not guarantee that the share data will be immediately inaccessible, or
2730+        that it will be securely erased.
2731+        """
2732+
2733+    def readv(read_vector):
2734+        """
2735+        XXX
2736+        """
2737+
2738+
2739+class IStoredMutableShare(IStoredShare):
2740+    def check_write_enabler(write_enabler, si_s):
2741+        """
2742+        XXX
2743         """
2744 
2745hunk ./src/allmydata/interfaces.py 489
2746-    def put_plaintext_hashes(hashes=ListOf(Hash)):
2747+    def check_testv(test_vector):
2748+        """
2749+        XXX
2750+        """
2751+
2752+    def writev(datav, new_length):
2753+        """
2754+        XXX
2755+        """
2756+
2757+
2758+class IStorageBucketWriter(Interface):
2759+    """
2760+    Objects of this kind live on the client side.
2761+    """
2762+    def put_block(segmentnum, data):
2763         """
2764hunk ./src/allmydata/interfaces.py 506
2765+        @param segmentnum=int
2766+        @param data=ShareData: For most segments, this data will be 'blocksize'
2767+        bytes in length. The last segment might be shorter.
2768         @return: a Deferred that fires (with None) when the operation completes
2769         """
2770 
2771hunk ./src/allmydata/interfaces.py 512
2772-    def put_crypttext_hashes(hashes=ListOf(Hash)):
2773+    def put_crypttext_hashes(hashes):
2774         """
2775hunk ./src/allmydata/interfaces.py 514
2776+        @param hashes=ListOf(Hash)
2777         @return: a Deferred that fires (with None) when the operation completes
2778         """
2779 
2780hunk ./src/allmydata/interfaces.py 518
2781-    def put_block_hashes(blockhashes=ListOf(Hash)):
2782+    def put_block_hashes(blockhashes):
2783         """
2784hunk ./src/allmydata/interfaces.py 520
2785+        @param blockhashes=ListOf(Hash)
2786         @return: a Deferred that fires (with None) when the operation completes
2787         """
2788 
2789hunk ./src/allmydata/interfaces.py 524
2790-    def put_share_hashes(sharehashes=ListOf(TupleOf(int, Hash))):
2791+    def put_share_hashes(sharehashes):
2792         """
2793hunk ./src/allmydata/interfaces.py 526
2794+        @param sharehashes=ListOf(TupleOf(int, Hash))
2795         @return: a Deferred that fires (with None) when the operation completes
2796         """
2797 
2798hunk ./src/allmydata/interfaces.py 530
2799-    def put_uri_extension(data=URIExtensionData):
2800+    def put_uri_extension(data):
2801         """This block of data contains integrity-checking information (hashes
2802         of plaintext, crypttext, and shares), as well as encoding parameters
2803         that are necessary to recover the data. This is a serialized dict
2804hunk ./src/allmydata/interfaces.py 535
2805         mapping strings to other strings. The hash of this data is kept in
2806-        the URI and verified before any of the data is used. All buckets for
2807-        a given file contain identical copies of this data.
2808+        the URI and verified before any of the data is used. All share
2809+        containers for a given file contain identical copies of this data.
2810 
2811         The serialization format is specified with the following pseudocode:
2812         for k in sorted(dict.keys()):
2813hunk ./src/allmydata/interfaces.py 543
2814             assert re.match(r'^[a-zA-Z_\-]+$', k)
2815             write(k + ':' + netstring(dict[k]))
2816 
2817+        @param data=URIExtensionData
2818         @return: a Deferred that fires (with None) when the operation completes
2819         """
2820 
2821hunk ./src/allmydata/interfaces.py 558
2822 
2823 class IStorageBucketReader(Interface):
2824 
2825-    def get_block_data(blocknum=int, blocksize=int, size=int):
2826+    def get_block_data(blocknum, blocksize, size):
2827         """Most blocks will be the same size. The last block might be shorter
2828         than the others.
2829 
2830hunk ./src/allmydata/interfaces.py 562
2831+        @param blocknum=int
2832+        @param blocksize=int
2833+        @param size=int
2834         @return: ShareData
2835         """
2836 
2837hunk ./src/allmydata/interfaces.py 573
2838         @return: ListOf(Hash)
2839         """
2840 
2841-    def get_block_hashes(at_least_these=SetOf(int)):
2842+    def get_block_hashes(at_least_these=()):
2843         """
2844hunk ./src/allmydata/interfaces.py 575
2845+        @param at_least_these=SetOf(int)
2846         @return: ListOf(Hash)
2847         """
2848 
2849hunk ./src/allmydata/interfaces.py 579
2850-    def get_share_hashes(at_least_these=SetOf(int)):
2851+    def get_share_hashes():
2852         """
2853         @return: ListOf(TupleOf(int, Hash))
2854         """
2855hunk ./src/allmydata/interfaces.py 611
2856         @return: unicode nickname, or None
2857         """
2858 
2859-    # methods moved from IntroducerClient, need review
2860-    def get_all_connections():
2861-        """Return a frozenset of (nodeid, service_name, rref) tuples, one for
2862-        each active connection we've established to a remote service. This is
2863-        mostly useful for unit tests that need to wait until a certain number
2864-        of connections have been made."""
2865-
2866-    def get_all_connectors():
2867-        """Return a dict that maps from (nodeid, service_name) to a
2868-        RemoteServiceConnector instance for all services that we are actively
2869-        trying to connect to. Each RemoteServiceConnector has the following
2870-        public attributes::
2871-
2872-          service_name: the type of service provided, like 'storage'
2873-          announcement_time: when we first heard about this service
2874-          last_connect_time: when we last established a connection
2875-          last_loss_time: when we last lost a connection
2876-
2877-          version: the peer's version, from the most recent connection
2878-          oldest_supported: the peer's oldest supported version, same
2879-
2880-          rref: the RemoteReference, if connected, otherwise None
2881-          remote_host: the IAddress, if connected, otherwise None
2882-
2883-        This method is intended for monitoring interfaces, such as a web page
2884-        that describes connecting and connected peers.
2885-        """
2886-
2887-    def get_all_peerids():
2888-        """Return a frozenset of all peerids to whom we have a connection (to
2889-        one or more services) established. Mostly useful for unit tests."""
2890-
2891-    def get_all_connections_for(service_name):
2892-        """Return a frozenset of (nodeid, service_name, rref) tuples, one
2893-        for each active connection that provides the given SERVICE_NAME."""
2894-
2895-    def get_permuted_peers(service_name, key):
2896-        """Returns an ordered list of (peerid, rref) tuples, selecting from
2897-        the connections that provide SERVICE_NAME, using a hash-based
2898-        permutation keyed by KEY. This randomizes the service list in a
2899-        repeatable way, to distribute load over many peers.
2900-        """
2901-
2902 
2903 class IMutableSlotWriter(Interface):
2904     """
2905hunk ./src/allmydata/interfaces.py 616
2906     The interface for a writer around a mutable slot on a remote server.
2907     """
2908-    def set_checkstring(checkstring, *args):
2909+    def set_checkstring(seqnum_or_checkstring, root_hash=None, salt=None):
2910         """
2911         Set the checkstring that I will pass to the remote server when
2912         writing.
2913hunk ./src/allmydata/interfaces.py 640
2914         Add a block and salt to the share.
2915         """
2916 
2917-    def put_encprivey(encprivkey):
2918+    def put_encprivkey(encprivkey):
2919         """
2920         Add the encrypted private key to the share.
2921         """
2922hunk ./src/allmydata/interfaces.py 645
2923 
2924-    def put_blockhashes(blockhashes=list):
2925+    def put_blockhashes(blockhashes):
2926         """
2927hunk ./src/allmydata/interfaces.py 647
2928+        @param blockhashes=list
2929         Add the block hash tree to the share.
2930         """
2931 
2932hunk ./src/allmydata/interfaces.py 651
2933-    def put_sharehashes(sharehashes=dict):
2934+    def put_sharehashes(sharehashes):
2935         """
2936hunk ./src/allmydata/interfaces.py 653
2937+        @param sharehashes=dict
2938         Add the share hash chain to the share.
2939         """
2940 
2941hunk ./src/allmydata/interfaces.py 739
2942     def get_extension_params():
2943         """Return the extension parameters in the URI"""
2944 
2945-    def set_extension_params():
2946+    def set_extension_params(params):
2947         """Set the extension parameters that should be in the URI"""
2948 
2949 class IDirectoryURI(Interface):
2950hunk ./src/allmydata/interfaces.py 879
2951         writer-visible data using this writekey.
2952         """
2953 
2954-    # TODO: Can this be overwrite instead of replace?
2955-    def replace(new_contents):
2956-        """Replace the contents of the mutable file, provided that no other
2957+    def overwrite(new_contents):
2958+        """Overwrite the contents of the mutable file, provided that no other
2959         node has published (or is attempting to publish, concurrently) a
2960         newer version of the file than this one.
2961 
2962hunk ./src/allmydata/interfaces.py 1346
2963         is empty, the metadata will be an empty dictionary.
2964         """
2965 
2966-    def set_uri(name, writecap, readcap=None, metadata=None, overwrite=True):
2967+    def set_uri(name, writecap, readcap, metadata=None, overwrite=True):
2968         """I add a child (by writecap+readcap) at the specific name. I return
2969         a Deferred that fires when the operation finishes. If overwrite= is
2970         True, I will replace any existing child of the same name, otherwise
2971hunk ./src/allmydata/interfaces.py 1745
2972     Block Hash, and the encoding parameters, both of which must be included
2973     in the URI.
2974 
2975-    I do not choose shareholders, that is left to the IUploader. I must be
2976-    given a dict of RemoteReferences to storage buckets that are ready and
2977-    willing to receive data.
2978+    I do not choose shareholders, that is left to the IUploader.
2979     """
2980 
2981     def set_size(size):
2982hunk ./src/allmydata/interfaces.py 1752
2983         """Specify the number of bytes that will be encoded. This must be
2984         peformed before get_serialized_params() can be called.
2985         """
2986+
2987     def set_params(params):
2988         """Override the default encoding parameters. 'params' is a tuple of
2989         (k,d,n), where 'k' is the number of required shares, 'd' is the
2990hunk ./src/allmydata/interfaces.py 1848
2991     download, validate, decode, and decrypt data from them, writing the
2992     results to an output file.
2993 
2994-    I do not locate the shareholders, that is left to the IDownloader. I must
2995-    be given a dict of RemoteReferences to storage buckets that are ready to
2996-    send data.
2997+    I do not locate the shareholders, that is left to the IDownloader.
2998     """
2999 
3000     def setup(outfile):
3001hunk ./src/allmydata/interfaces.py 1950
3002         resuming an interrupted upload (where we need to compute the
3003         plaintext hashes, but don't need the redundant encrypted data)."""
3004 
3005-    def get_plaintext_hashtree_leaves(first, last, num_segments):
3006-        """OBSOLETE; Get the leaf nodes of a merkle hash tree over the
3007-        plaintext segments, i.e. get the tagged hashes of the given segments.
3008-        The segment size is expected to be generated by the
3009-        IEncryptedUploadable before any plaintext is read or ciphertext
3010-        produced, so that the segment hashes can be generated with only a
3011-        single pass.
3012-
3013-        This returns a Deferred that fires with a sequence of hashes, using:
3014-
3015-         tuple(segment_hashes[first:last])
3016-
3017-        'num_segments' is used to assert that the number of segments that the
3018-        IEncryptedUploadable handled matches the number of segments that the
3019-        encoder was expecting.
3020-
3021-        This method must not be called until the final byte has been read
3022-        from read_encrypted(). Once this method is called, read_encrypted()
3023-        can never be called again.
3024-        """
3025-
3026-    def get_plaintext_hash():
3027-        """OBSOLETE; Get the hash of the whole plaintext.
3028-
3029-        This returns a Deferred that fires with a tagged SHA-256 hash of the
3030-        whole plaintext, obtained from hashutil.plaintext_hash(data).
3031-        """
3032-
3033     def close():
3034         """Just like IUploadable.close()."""
3035 
3036hunk ./src/allmydata/interfaces.py 2144
3037         returns a Deferred that fires with an IUploadResults instance, from
3038         which the URI of the file can be obtained as results.uri ."""
3039 
3040-    def upload_ssk(write_capability, new_version, uploadable):
3041-        """TODO: how should this work?"""
3042-
3043 class ICheckable(Interface):
3044     def check(monitor, verify=False, add_lease=False):
3045         """Check up on my health, optionally repairing any problems.
3046hunk ./src/allmydata/interfaces.py 2505
3047 
3048 class IRepairResults(Interface):
3049     """I contain the results of a repair operation."""
3050-    def get_successful(self):
3051+    def get_successful():
3052         """Returns a boolean: True if the repair made the file healthy, False
3053         if not. Repair failure generally indicates a file that has been
3054         damaged beyond repair."""
3055hunk ./src/allmydata/interfaces.py 2577
3056     Tahoe process will typically have a single NodeMaker, but unit tests may
3057     create simplified/mocked forms for testing purposes.
3058     """
3059-    def create_from_cap(writecap, readcap=None, **kwargs):
3060+    def create_from_cap(writecap, readcap=None, deep_immutable=False, name=u"<unknown name>"):
3061         """I create an IFilesystemNode from the given writecap/readcap. I can
3062         only provide nodes for existing file/directory objects: use my other
3063         methods to create new objects. I return synchronously."""
3064hunk ./src/allmydata/monitor.py 30
3065 
3066     # the following methods are provided for the operation code
3067 
3068-    def is_cancelled(self):
3069+    def is_cancelled():
3070         """Returns True if the operation has been cancelled. If True,
3071         operation code should stop creating new work, and attempt to stop any
3072         work already in progress."""
3073hunk ./src/allmydata/monitor.py 35
3074 
3075-    def raise_if_cancelled(self):
3076+    def raise_if_cancelled():
3077         """Raise OperationCancelledError if the operation has been cancelled.
3078         Operation code that has a robust error-handling path can simply call
3079         this periodically."""
3080hunk ./src/allmydata/monitor.py 40
3081 
3082-    def set_status(self, status):
3083+    def set_status(status):
3084         """Sets the Monitor's 'status' object to an arbitrary value.
3085         Different operations will store different sorts of status information
3086         here. Operation code should use get+modify+set sequences to update
3087hunk ./src/allmydata/monitor.py 46
3088         this."""
3089 
3090-    def get_status(self):
3091+    def get_status():
3092         """Return the status object. If the operation failed, this will be a
3093         Failure instance."""
3094 
3095hunk ./src/allmydata/monitor.py 50
3096-    def finish(self, status):
3097+    def finish(status):
3098         """Call this when the operation is done, successful or not. The
3099         Monitor's lifetime is influenced by the completion of the operation
3100         it is monitoring. The Monitor's 'status' value will be set with the
3101hunk ./src/allmydata/monitor.py 63
3102 
3103     # the following methods are provided for the initiator of the operation
3104 
3105-    def is_finished(self):
3106+    def is_finished():
3107         """Return a boolean, True if the operation is done (whether
3108         successful or failed), False if it is still running."""
3109 
3110hunk ./src/allmydata/monitor.py 67
3111-    def when_done(self):
3112+    def when_done():
3113         """Return a Deferred that fires when the operation is complete. It
3114         will fire with the operation status, the same value as returned by
3115         get_status()."""
3116hunk ./src/allmydata/monitor.py 72
3117 
3118-    def cancel(self):
3119+    def cancel():
3120         """Cancel the operation as soon as possible. is_cancelled() will
3121         start returning True after this is called."""
3122 
3123hunk ./src/allmydata/mutable/filenode.py 753
3124         self._writekey = writekey
3125         self._serializer = defer.succeed(None)
3126 
3127-
3128     def get_sequence_number(self):
3129         """
3130         Get the sequence number of the mutable version that I represent.
3131hunk ./src/allmydata/mutable/filenode.py 759
3132         """
3133         return self._version[0] # verinfo[0] == the sequence number
3134 
3135+    def get_servermap(self):
3136+        return self._servermap
3137 
3138hunk ./src/allmydata/mutable/filenode.py 762
3139-    # TODO: Terminology?
3140     def get_writekey(self):
3141         """
3142         I return a writekey or None if I don't have a writekey.
3143hunk ./src/allmydata/mutable/filenode.py 768
3144         """
3145         return self._writekey
3146 
3147-
3148     def set_downloader_hints(self, hints):
3149         """
3150         I set the downloader hints.
3151hunk ./src/allmydata/mutable/filenode.py 776
3152 
3153         self._downloader_hints = hints
3154 
3155-
3156     def get_downloader_hints(self):
3157         """
3158         I return the downloader hints.
3159hunk ./src/allmydata/mutable/filenode.py 782
3160         """
3161         return self._downloader_hints
3162 
3163-
3164     def overwrite(self, new_contents):
3165         """
3166         I overwrite the contents of this mutable file version with the
3167hunk ./src/allmydata/mutable/filenode.py 791
3168 
3169         return self._do_serialized(self._overwrite, new_contents)
3170 
3171-
3172     def _overwrite(self, new_contents):
3173         assert IMutableUploadable.providedBy(new_contents)
3174         assert self._servermap.last_update_mode == MODE_WRITE
3175hunk ./src/allmydata/mutable/filenode.py 797
3176 
3177         return self._upload(new_contents)
3178 
3179-
3180     def modify(self, modifier, backoffer=None):
3181         """I use a modifier callback to apply a change to the mutable file.
3182         I implement the following pseudocode::
3183hunk ./src/allmydata/mutable/filenode.py 841
3184 
3185         return self._do_serialized(self._modify, modifier, backoffer)
3186 
3187-
3188     def _modify(self, modifier, backoffer):
3189         if backoffer is None:
3190             backoffer = BackoffAgent().delay
3191hunk ./src/allmydata/mutable/filenode.py 846
3192         return self._modify_and_retry(modifier, backoffer, True)
3193 
3194-
3195     def _modify_and_retry(self, modifier, backoffer, first_time):
3196         """
3197         I try to apply modifier to the contents of this version of the
3198hunk ./src/allmydata/mutable/filenode.py 878
3199         d.addErrback(_retry)
3200         return d
3201 
3202-
3203     def _modify_once(self, modifier, first_time):
3204         """
3205         I attempt to apply a modifier to the contents of the mutable
3206hunk ./src/allmydata/mutable/filenode.py 913
3207         d.addCallback(_apply)
3208         return d
3209 
3210-
3211     def is_readonly(self):
3212         """
3213         I return True if this MutableFileVersion provides no write
3214hunk ./src/allmydata/mutable/filenode.py 921
3215         """
3216         return self._writekey is None
3217 
3218-
3219     def is_mutable(self):
3220         """
3221         I return True, since mutable files are always mutable by
3222hunk ./src/allmydata/mutable/filenode.py 928
3223         """
3224         return True
3225 
3226-
3227     def get_storage_index(self):
3228         """
3229         I return the storage index of the reference that I encapsulate.
3230hunk ./src/allmydata/mutable/filenode.py 934
3231         """
3232         return self._storage_index
3233 
3234-
3235     def get_size(self):
3236         """
3237         I return the length, in bytes, of this readable object.
3238hunk ./src/allmydata/mutable/filenode.py 940
3239         """
3240         return self._servermap.size_of_version(self._version)
3241 
3242-
3243     def download_to_data(self, fetch_privkey=False):
3244         """
3245         I return a Deferred that fires with the contents of this
3246hunk ./src/allmydata/mutable/filenode.py 951
3247         d.addCallback(lambda mc: "".join(mc.chunks))
3248         return d
3249 
3250-
3251     def _try_to_download_data(self):
3252         """
3253         I am an unserialized cousin of download_to_data; I am called
3254hunk ./src/allmydata/mutable/filenode.py 963
3255         d.addCallback(lambda mc: "".join(mc.chunks))
3256         return d
3257 
3258-
3259     def read(self, consumer, offset=0, size=None, fetch_privkey=False):
3260         """
3261         I read a portion (possibly all) of the mutable file that I
3262hunk ./src/allmydata/mutable/filenode.py 971
3263         return self._do_serialized(self._read, consumer, offset, size,
3264                                    fetch_privkey)
3265 
3266-
3267     def _read(self, consumer, offset=0, size=None, fetch_privkey=False):
3268         """
3269         I am the serialized companion of read.
3270hunk ./src/allmydata/mutable/filenode.py 981
3271         d = r.download(consumer, offset, size)
3272         return d
3273 
3274-
3275     def _do_serialized(self, cb, *args, **kwargs):
3276         # note: to avoid deadlock, this callable is *not* allowed to invoke
3277         # other serialized methods within this (or any other)
3278hunk ./src/allmydata/mutable/filenode.py 999
3279         self._serializer.addErrback(log.err)
3280         return d
3281 
3282-
3283     def _upload(self, new_contents):
3284         #assert self._pubkey, "update_servermap must be called before publish"
3285         p = Publish(self._node, self._storage_broker, self._servermap)
3286hunk ./src/allmydata/mutable/filenode.py 1009
3287         d.addCallback(self._did_upload, new_contents.get_size())
3288         return d
3289 
3290-
3291     def _did_upload(self, res, size):
3292         self._most_recent_size = size
3293         return res
3294hunk ./src/allmydata/mutable/filenode.py 1029
3295         """
3296         return self._do_serialized(self._update, data, offset)
3297 
3298-
3299     def _update(self, data, offset):
3300         """
3301         I update the mutable file version represented by this particular
3302hunk ./src/allmydata/mutable/filenode.py 1058
3303         d.addCallback(self._build_uploadable_and_finish, data, offset)
3304         return d
3305 
3306-
3307     def _do_modify_update(self, data, offset):
3308         """
3309         I perform a file update by modifying the contents of the file
3310hunk ./src/allmydata/mutable/filenode.py 1073
3311             return new
3312         return self._modify(m, None)
3313 
3314-
3315     def _do_update_update(self, data, offset):
3316         """
3317         I start the Servermap update that gets us the data we need to
3318hunk ./src/allmydata/mutable/filenode.py 1108
3319         return self._update_servermap(update_range=(start_segment,
3320                                                     end_segment))
3321 
3322-
3323     def _decode_and_decrypt_segments(self, ignored, data, offset):
3324         """
3325         After the servermap update, I take the encrypted and encoded
3326hunk ./src/allmydata/mutable/filenode.py 1148
3327         d3 = defer.succeed(blockhashes)
3328         return deferredutil.gatherResults([d1, d2, d3])
3329 
3330-
3331     def _build_uploadable_and_finish(self, segments_and_bht, data, offset):
3332         """
3333         After the process has the plaintext segments, I build the
3334hunk ./src/allmydata/mutable/filenode.py 1163
3335         p = Publish(self._node, self._storage_broker, self._servermap)
3336         return p.update(u, offset, segments_and_bht[2], self._version)
3337 
3338-
3339     def _update_servermap(self, mode=MODE_WRITE, update_range=None):
3340         """
3341         I update the servermap. I return a Deferred that fires when the
3342hunk ./src/allmydata/storage/common.py 1
3343-
3344-import os.path
3345 from allmydata.util import base32
3346 
3347 class DataTooLargeError(Exception):
3348hunk ./src/allmydata/storage/common.py 5
3349     pass
3350+
3351 class UnknownMutableContainerVersionError(Exception):
3352     pass
3353hunk ./src/allmydata/storage/common.py 8
3354+
3355 class UnknownImmutableContainerVersionError(Exception):
3356     pass
3357 
3358hunk ./src/allmydata/storage/common.py 18
3359 
3360 def si_a2b(ascii_storageindex):
3361     return base32.a2b(ascii_storageindex)
3362-
3363-def storage_index_to_dir(storageindex):
3364-    sia = si_b2a(storageindex)
3365-    return os.path.join(sia[:2], sia)
3366hunk ./src/allmydata/storage/crawler.py 2
3367 
3368-import os, time, struct
3369+import time, struct
3370 import cPickle as pickle
3371 from twisted.internet import reactor
3372 from twisted.application import service
3373hunk ./src/allmydata/storage/crawler.py 6
3374+
3375+from allmydata.util.assertutil import precondition
3376+from allmydata.interfaces import IStorageBackend
3377 from allmydata.storage.common import si_b2a
3378hunk ./src/allmydata/storage/crawler.py 10
3379-from allmydata.util import fileutil
3380+
3381 
3382 class TimeSliceExceeded(Exception):
3383     pass
3384hunk ./src/allmydata/storage/crawler.py 15
3385 
3386+
3387 class ShareCrawler(service.MultiService):
3388hunk ./src/allmydata/storage/crawler.py 17
3389-    """A ShareCrawler subclass is attached to a StorageServer, and
3390-    periodically walks all of its shares, processing each one in some
3391-    fashion. This crawl is rate-limited, to reduce the IO burden on the host,
3392-    since large servers can easily have a terabyte of shares, in several
3393-    million files, which can take hours or days to read.
3394+    """
3395+    An instance of a subclass of ShareCrawler is attached to a storage
3396+    backend, and periodically walks the backend's shares, processing them
3397+    in some fashion. This crawl is rate-limited to reduce the I/O burden on
3398+    the host, since large servers can easily have a terabyte of shares in
3399+    several million files, which can take hours or days to read.
3400 
3401     Once the crawler starts a cycle, it will proceed at a rate limited by the
3402     allowed_cpu_percentage= and cpu_slice= parameters: yielding the reactor
3403hunk ./src/allmydata/storage/crawler.py 33
3404     long enough to ensure that 'minimum_cycle_time' elapses between the start
3405     of two consecutive cycles.
3406 
3407-    We assume that the normal upload/download/get_buckets traffic of a tahoe
3408+    We assume that the normal upload/download/DYHB traffic of a Tahoe-LAFS
3409     grid will cause the prefixdir contents to be mostly cached in the kernel,
3410hunk ./src/allmydata/storage/crawler.py 35
3411-    or that the number of buckets in each prefixdir will be small enough to
3412-    load quickly. A 1TB allmydata.com server was measured to have 2.56M
3413-    buckets, spread into the 1024 prefixdirs, with about 2500 buckets per
3414+    or that the number of sharesets in each prefixdir will be small enough to
3415+    load quickly. A 1TB allmydata.com server was measured to have 2.56 million
3416+    sharesets, spread into the 1024 prefixdirs, with about 2500 sharesets per
3417     prefix. On this server, each prefixdir took 130ms-200ms to list the first
3418     time, and 17ms to list the second time.
3419 
3420hunk ./src/allmydata/storage/crawler.py 41
3421-    To use a crawler, create a subclass which implements the process_bucket()
3422-    method. It will be called with a prefixdir and a base32 storage index
3423-    string. process_bucket() must run synchronously. Any keys added to
3424-    self.state will be preserved. Override add_initial_state() to set up
3425-    initial state keys. Override finished_cycle() to perform additional
3426-    processing when the cycle is complete. Any status that the crawler
3427-    produces should be put in the self.state dictionary. Status renderers
3428-    (like a web page which describes the accomplishments of your crawler)
3429-    will use crawler.get_state() to retrieve this dictionary; they can
3430-    present the contents as they see fit.
3431+    To implement a crawler, create a subclass that implements the
3432+    process_shareset() method. It will be called with a prefixdir and an
3433+    object providing the IShareSet interface. process_shareset() must run
3434+    synchronously. Any keys added to self.state will be preserved. Override
3435+    add_initial_state() to set up initial state keys. Override
3436+    finished_cycle() to perform additional processing when the cycle is
3437+    complete. Any status that the crawler produces should be put in the
3438+    self.state dictionary. Status renderers (like a web page describing the
3439+    accomplishments of your crawler) will use crawler.get_state() to retrieve
3440+    this dictionary; they can present the contents as they see fit.
3441 
3442hunk ./src/allmydata/storage/crawler.py 52
3443-    Then create an instance, with a reference to a StorageServer and a
3444-    filename where it can store persistent state. The statefile is used to
3445-    keep track of how far around the ring the process has travelled, as well
3446-    as timing history to allow the pace to be predicted and controlled. The
3447-    statefile will be updated and written to disk after each time slice (just
3448-    before the crawler yields to the reactor), and also after each cycle is
3449-    finished, and also when stopService() is called. Note that this means
3450-    that a crawler which is interrupted with SIGKILL while it is in the
3451-    middle of a time slice will lose progress: the next time the node is
3452-    started, the crawler will repeat some unknown amount of work.
3453+    Then create an instance, with a reference to a backend object providing
3454+    the IStorageBackend interface, and a filename where it can store
3455+    persistent state. The statefile is used to keep track of how far around
3456+    the ring the process has travelled, as well as timing history to allow
3457+    the pace to be predicted and controlled. The statefile will be updated
3458+    and written to disk after each time slice (just before the crawler yields
3459+    to the reactor), and also after each cycle is finished, and also when
3460+    stopService() is called. Note that this means that a crawler that is
3461+    interrupted with SIGKILL while it is in the middle of a time slice will
3462+    lose progress: the next time the node is started, the crawler will repeat
3463+    some unknown amount of work.
3464 
3465     The crawler instance must be started with startService() before it will
3466hunk ./src/allmydata/storage/crawler.py 65
3467-    do any work. To make it stop doing work, call stopService().
3468+    do any work. To make it stop doing work, call stopService(). A crawler
3469+    is usually a child service of a StorageServer, although it should not
3470+    depend on that.
3471+
3472+    For historical reasons, some dictionary key names use the term "bucket"
3473+    for what is now preferably called a "shareset" (the set of shares that a
3474+    server holds under a given storage index).
3475     """
3476 
3477     slow_start = 300 # don't start crawling for 5 minutes after startup
3478hunk ./src/allmydata/storage/crawler.py 80
3479     cpu_slice = 1.0 # use up to 1.0 seconds before yielding
3480     minimum_cycle_time = 300 # don't run a cycle faster than this
3481 
3482-    def __init__(self, server, statefile, allowed_cpu_percentage=None):
3483+    def __init__(self, backend, statefp, allowed_cpu_percentage=None):
3484+        precondition(IStorageBackend.providedBy(backend), backend)
3485         service.MultiService.__init__(self)
3486hunk ./src/allmydata/storage/crawler.py 83
3487+        self.backend = backend
3488+        self.statefp = statefp
3489         if allowed_cpu_percentage is not None:
3490             self.allowed_cpu_percentage = allowed_cpu_percentage
3491hunk ./src/allmydata/storage/crawler.py 87
3492-        self.server = server
3493-        self.sharedir = server.sharedir
3494-        self.statefile = statefile
3495         self.prefixes = [si_b2a(struct.pack(">H", i << (16-10)))[:2]
3496                          for i in range(2**10)]
3497         self.prefixes.sort()
3498hunk ./src/allmydata/storage/crawler.py 91
3499         self.timer = None
3500-        self.bucket_cache = (None, [])
3501+        self.shareset_cache = (None, [])
3502         self.current_sleep_time = None
3503         self.next_wake_time = None
3504         self.last_prefix_finished_time = None
3505hunk ./src/allmydata/storage/crawler.py 154
3506                 left = len(self.prefixes) - self.last_complete_prefix_index
3507                 remaining = left * self.last_prefix_elapsed_time
3508                 # TODO: remainder of this prefix: we need to estimate the
3509-                # per-bucket time, probably by measuring the time spent on
3510-                # this prefix so far, divided by the number of buckets we've
3511+                # per-shareset time, probably by measuring the time spent on
3512+                # this prefix so far, divided by the number of sharesets we've
3513                 # processed.
3514             d["estimated-cycle-complete-time-left"] = remaining
3515             # it's possible to call get_progress() from inside a crawler's
3516hunk ./src/allmydata/storage/crawler.py 175
3517         state dictionary.
3518 
3519         If we are not currently sleeping (i.e. get_state() was called from
3520-        inside the process_prefixdir, process_bucket, or finished_cycle()
3521+        inside the process_prefixdir, process_shareset, or finished_cycle()
3522         methods, or if startService has not yet been called on this crawler),
3523         these two keys will be None.
3524 
3525hunk ./src/allmydata/storage/crawler.py 188
3526     def load_state(self):
3527         # we use this to store state for both the crawler's internals and
3528         # anything the subclass-specific code needs. The state is stored
3529-        # after each bucket is processed, after each prefixdir is processed,
3530+        # after each shareset is processed, after each prefixdir is processed,
3531         # and after a cycle is complete. The internal keys we use are:
3532         #  ["version"]: int, always 1
3533         #  ["last-cycle-finished"]: int, or None if we have not yet finished
3534hunk ./src/allmydata/storage/crawler.py 202
3535         #                            are sleeping between cycles, or if we
3536         #                            have not yet finished any prefixdir since
3537         #                            a cycle was started
3538-        #  ["last-complete-bucket"]: str, base32 storage index bucket name
3539-        #                            of the last bucket to be processed, or
3540-        #                            None if we are sleeping between cycles
3541+        #  ["last-complete-bucket"]: str, base32 storage index of the last
3542+        #                            shareset to be processed, or None if we
3543+        #                            are sleeping between cycles
3544         try:
3545hunk ./src/allmydata/storage/crawler.py 206
3546-            f = open(self.statefile, "rb")
3547-            state = pickle.load(f)
3548-            f.close()
3549+            state = pickle.loads(self.statefp.getContent())
3550         except EnvironmentError:
3551             state = {"version": 1,
3552                      "last-cycle-finished": None,
3553hunk ./src/allmydata/storage/crawler.py 242
3554         else:
3555             last_complete_prefix = self.prefixes[lcpi]
3556         self.state["last-complete-prefix"] = last_complete_prefix
3557-        tmpfile = self.statefile + ".tmp"
3558-        f = open(tmpfile, "wb")
3559-        pickle.dump(self.state, f)
3560-        f.close()
3561-        fileutil.move_into_place(tmpfile, self.statefile)
3562+        self.statefp.setContent(pickle.dumps(self.state))
3563 
3564     def startService(self):
3565         # arrange things to look like we were just sleeping, so
3566hunk ./src/allmydata/storage/crawler.py 284
3567         sleep_time = (this_slice / self.allowed_cpu_percentage) - this_slice
3568         # if the math gets weird, or a timequake happens, don't sleep
3569         # forever. Note that this means that, while a cycle is running, we
3570-        # will process at least one bucket every 5 minutes, no matter how
3571-        # long that bucket takes.
3572+        # will process at least one shareset every 5 minutes, no matter how
3573+        # long that shareset takes.
3574         sleep_time = max(0.0, min(sleep_time, 299))
3575         if finished_cycle:
3576             # how long should we sleep between cycles? Don't run faster than
3577hunk ./src/allmydata/storage/crawler.py 315
3578         for i in range(self.last_complete_prefix_index+1, len(self.prefixes)):
3579             # if we want to yield earlier, just raise TimeSliceExceeded()
3580             prefix = self.prefixes[i]
3581-            prefixdir = os.path.join(self.sharedir, prefix)
3582-            if i == self.bucket_cache[0]:
3583-                buckets = self.bucket_cache[1]
3584+            if i == self.shareset_cache[0]:
3585+                sharesets = self.shareset_cache[1]
3586             else:
3587hunk ./src/allmydata/storage/crawler.py 318
3588-                try:
3589-                    buckets = os.listdir(prefixdir)
3590-                    buckets.sort()
3591-                except EnvironmentError:
3592-                    buckets = []
3593-                self.bucket_cache = (i, buckets)
3594-            self.process_prefixdir(cycle, prefix, prefixdir,
3595-                                   buckets, start_slice)
3596+                sharesets = self.backend.get_sharesets_for_prefix(prefix)
3597+                self.shareset_cache = (i, sharesets)
3598+            self.process_prefixdir(cycle, prefix, sharesets, start_slice)
3599             self.last_complete_prefix_index = i
3600 
3601             now = time.time()
3602hunk ./src/allmydata/storage/crawler.py 345
3603         self.finished_cycle(cycle)
3604         self.save_state()
3605 
3606-    def process_prefixdir(self, cycle, prefix, prefixdir, buckets, start_slice):
3607-        """This gets a list of bucket names (i.e. storage index strings,
3608+    def process_prefixdir(self, cycle, prefix, sharesets, start_slice):
3609+        """
3610+        This gets a list of shareset names (i.e. storage index strings,
3611         base32-encoded) in sorted order.
3612 
3613         You can override this if your crawler doesn't care about the actual
3614hunk ./src/allmydata/storage/crawler.py 352
3615         shares, for example a crawler which merely keeps track of how many
3616-        buckets are being managed by this server.
3617+        sharesets are being managed by this server.
3618 
3619hunk ./src/allmydata/storage/crawler.py 354
3620-        Subclasses which *do* care about actual bucket should leave this
3621-        method along, and implement process_bucket() instead.
3622+        Subclasses which *do* care about actual shareset should leave this
3623+        method alone, and implement process_shareset() instead.
3624         """
3625 
3626hunk ./src/allmydata/storage/crawler.py 358
3627-        for bucket in buckets:
3628-            if bucket <= self.state["last-complete-bucket"]:
3629+        for shareset in sharesets:
3630+            base32si = shareset.get_storage_index_string()
3631+            if base32si <= self.state["last-complete-bucket"]:
3632                 continue
3633hunk ./src/allmydata/storage/crawler.py 362
3634-            self.process_bucket(cycle, prefix, prefixdir, bucket)
3635-            self.state["last-complete-bucket"] = bucket
3636+            self.process_shareset(cycle, prefix, shareset)
3637+            self.state["last-complete-bucket"] = base32si
3638             if time.time() >= start_slice + self.cpu_slice:
3639                 raise TimeSliceExceeded()
3640 
3641hunk ./src/allmydata/storage/crawler.py 370
3642     # the remaining methods are explictly for subclasses to implement.
3643 
3644     def started_cycle(self, cycle):
3645-        """Notify a subclass that the crawler is about to start a cycle.
3646+        """
3647+        Notify a subclass that the crawler is about to start a cycle.
3648 
3649         This method is for subclasses to override. No upcall is necessary.
3650         """
3651hunk ./src/allmydata/storage/crawler.py 377
3652         pass
3653 
3654-    def process_bucket(self, cycle, prefix, prefixdir, storage_index_b32):
3655-        """Examine a single bucket. Subclasses should do whatever they want
3656+    def process_shareset(self, cycle, prefix, shareset):
3657+        """
3658+        Examine a single shareset. Subclasses should do whatever they want
3659         to do to the shares therein, then update self.state as necessary.
3660 
3661         If the crawler is never interrupted by SIGKILL, this method will be
3662hunk ./src/allmydata/storage/crawler.py 383
3663-        called exactly once per share (per cycle). If it *is* interrupted,
3664+        called exactly once per shareset (per cycle). If it *is* interrupted,
3665         then the next time the node is started, some amount of work will be
3666         duplicated, according to when self.save_state() was last called. By
3667         default, save_state() is called at the end of each timeslice, and
3668hunk ./src/allmydata/storage/crawler.py 391
3669 
3670         To reduce the chance of duplicate work (i.e. to avoid adding multiple
3671         records to a database), you can call save_state() at the end of your
3672-        process_bucket() method. This will reduce the maximum duplicated work
3673-        to one bucket per SIGKILL. It will also add overhead, probably 1-20ms
3674-        per bucket (and some disk writes), which will count against your
3675-        allowed_cpu_percentage, and which may be considerable if
3676-        process_bucket() runs quickly.
3677+        process_shareset() method. This will reduce the maximum duplicated
3678+        work to one shareset per SIGKILL. It will also add overhead, probably
3679+        1-20ms per shareset (and some disk writes), which will count against
3680+        your allowed_cpu_percentage, and which may be considerable if
3681+        process_shareset() runs quickly.
3682 
3683         This method is for subclasses to override. No upcall is necessary.
3684         """
3685hunk ./src/allmydata/storage/crawler.py 402
3686         pass
3687 
3688     def finished_prefix(self, cycle, prefix):
3689-        """Notify a subclass that the crawler has just finished processing a
3690-        prefix directory (all buckets with the same two-character/10bit
3691+        """
3692+        Notify a subclass that the crawler has just finished processing a
3693+        prefix directory (all sharesets with the same two-character/10-bit
3694         prefix). To impose a limit on how much work might be duplicated by a
3695         SIGKILL that occurs during a timeslice, you can call
3696         self.save_state() here, but be aware that it may represent a
3697hunk ./src/allmydata/storage/crawler.py 415
3698         pass
3699 
3700     def finished_cycle(self, cycle):
3701-        """Notify subclass that a cycle (one complete traversal of all
3702+        """
3703+        Notify subclass that a cycle (one complete traversal of all
3704         prefixdirs) has just finished. 'cycle' is the number of the cycle
3705         that just finished. This method should perform summary work and
3706         update self.state to publish information to status displays.
3707hunk ./src/allmydata/storage/crawler.py 433
3708         pass
3709 
3710     def yielding(self, sleep_time):
3711-        """The crawler is about to sleep for 'sleep_time' seconds. This
3712+        """
3713+        The crawler is about to sleep for 'sleep_time' seconds. This
3714         method is mostly for the convenience of unit tests.
3715 
3716         This method is for subclasses to override. No upcall is necessary.
3717hunk ./src/allmydata/storage/crawler.py 443
3718 
3719 
3720 class BucketCountingCrawler(ShareCrawler):
3721-    """I keep track of how many buckets are being managed by this server.
3722-    This is equivalent to the number of distributed files and directories for
3723-    which I am providing storage. The actual number of files+directories in
3724-    the full grid is probably higher (especially when there are more servers
3725-    than 'N', the number of generated shares), because some files+directories
3726-    will have shares on other servers instead of me. Also note that the
3727-    number of buckets will differ from the number of shares in small grids,
3728-    when more than one share is placed on a single server.
3729+    """
3730+    I keep track of how many sharesets, each corresponding to a storage index,
3731+    are being managed by this server. This is equivalent to the number of
3732+    distributed files and directories for which I am providing storage. The
3733+    actual number of files and directories in the full grid is probably higher
3734+    (especially when there are more servers than 'N', the number of generated
3735+    shares), because some files and directories will have shares on other
3736+    servers instead of me. Also note that the number of sharesets will differ
3737+    from the number of shares in small grids, when more than one share is
3738+    placed on a single server.
3739     """
3740 
3741     minimum_cycle_time = 60*60 # we don't need this more than once an hour
3742hunk ./src/allmydata/storage/crawler.py 457
3743 
3744-    def __init__(self, server, statefile, num_sample_prefixes=1):
3745-        ShareCrawler.__init__(self, server, statefile)
3746+    def __init__(self, backend, statefp, num_sample_prefixes=1):
3747+        ShareCrawler.__init__(self, backend, statefp)
3748         self.num_sample_prefixes = num_sample_prefixes
3749 
3750     def add_initial_state(self):
3751hunk ./src/allmydata/storage/crawler.py 471
3752         self.state.setdefault("last-complete-bucket-count", None)
3753         self.state.setdefault("storage-index-samples", {})
3754 
3755-    def process_prefixdir(self, cycle, prefix, prefixdir, buckets, start_slice):
3756+    def process_prefixdir(self, cycle, prefix, sharesets, start_slice):
3757         # we override process_prefixdir() because we don't want to look at
3758hunk ./src/allmydata/storage/crawler.py 473
3759-        # the individual buckets. We'll save state after each one. On my
3760+        # the individual sharesets. We'll save state after each one. On my
3761         # laptop, a mostly-empty storage server can process about 70
3762         # prefixdirs in a 1.0s slice.
3763         if cycle not in self.state["bucket-counts"]:
3764hunk ./src/allmydata/storage/crawler.py 478
3765             self.state["bucket-counts"][cycle] = {}
3766-        self.state["bucket-counts"][cycle][prefix] = len(buckets)
3767+        self.state["bucket-counts"][cycle][prefix] = len(sharesets)
3768         if prefix in self.prefixes[:self.num_sample_prefixes]:
3769hunk ./src/allmydata/storage/crawler.py 480
3770-            self.state["storage-index-samples"][prefix] = (cycle, buckets)
3771+            self.state["storage-index-samples"][prefix] = (cycle, sharesets)
3772 
3773     def finished_cycle(self, cycle):
3774         last_counts = self.state["bucket-counts"].get(cycle, [])
3775hunk ./src/allmydata/storage/crawler.py 486
3776         if len(last_counts) == len(self.prefixes):
3777             # great, we have a whole cycle.
3778-            num_buckets = sum(last_counts.values())
3779-            self.state["last-complete-bucket-count"] = num_buckets
3780+            num_sharesets = sum(last_counts.values())
3781+            self.state["last-complete-bucket-count"] = num_sharesets
3782             # get rid of old counts
3783             for old_cycle in list(self.state["bucket-counts"].keys()):
3784                 if old_cycle != cycle:
3785hunk ./src/allmydata/storage/crawler.py 494
3786                     del self.state["bucket-counts"][old_cycle]
3787         # get rid of old samples too
3788         for prefix in list(self.state["storage-index-samples"].keys()):
3789-            old_cycle,buckets = self.state["storage-index-samples"][prefix]
3790+            old_cycle, storage_indices = self.state["storage-index-samples"][prefix]
3791             if old_cycle != cycle:
3792                 del self.state["storage-index-samples"][prefix]
3793hunk ./src/allmydata/storage/crawler.py 497
3794-
3795hunk ./src/allmydata/storage/expirer.py 1
3796-import time, os, pickle, struct
3797+
3798+import time, pickle, struct
3799+from twisted.python import log as twlog
3800+
3801 from allmydata.storage.crawler import ShareCrawler
3802hunk ./src/allmydata/storage/expirer.py 6
3803-from allmydata.storage.shares import get_share_file
3804-from allmydata.storage.common import UnknownMutableContainerVersionError, \
3805+from allmydata.storage.common import si_b2a, UnknownMutableContainerVersionError, \
3806      UnknownImmutableContainerVersionError
3807hunk ./src/allmydata/storage/expirer.py 8
3808-from twisted.python import log as twlog
3809+
3810 
3811 class LeaseCheckingCrawler(ShareCrawler):
3812     """I examine the leases on all shares, determining which are still valid
3813hunk ./src/allmydata/storage/expirer.py 17
3814     removed.
3815 
3816     I collect statistics on the leases and make these available to a web
3817-    status page, including::
3818+    status page, including:
3819 
3820     Space recovered during this cycle-so-far:
3821      actual (only if expiration_enabled=True):
3822hunk ./src/allmydata/storage/expirer.py 21
3823-      num-buckets, num-shares, sum of share sizes, real disk usage
3824+      num-storage-indices, num-shares, sum of share sizes, real disk usage
3825       ('real disk usage' means we use stat(fn).st_blocks*512 and include any
3826        space used by the directory)
3827      what it would have been with the original lease expiration time
3828hunk ./src/allmydata/storage/expirer.py 32
3829 
3830     Space recovered during the last 10 cycles  <-- saved in separate pickle
3831 
3832-    Shares/buckets examined:
3833+    Shares/storage-indices examined:
3834      this cycle-so-far
3835      prediction of rest of cycle
3836      during last 10 cycles <-- separate pickle
3837hunk ./src/allmydata/storage/expirer.py 42
3838     Histogram of leases-per-share:
3839      this-cycle-to-date
3840      last 10 cycles <-- separate pickle
3841-    Histogram of lease ages, buckets = 1day
3842+    Histogram of lease ages, storage-indices over 1 day
3843      cycle-to-date
3844      last 10 cycles <-- separate pickle
3845 
3846hunk ./src/allmydata/storage/expirer.py 53
3847     slow_start = 360 # wait 6 minutes after startup
3848     minimum_cycle_time = 12*60*60 # not more than twice per day
3849 
3850-    def __init__(self, server, statefile, historyfile,
3851-                 expiration_enabled, mode,
3852-                 override_lease_duration, # used if expiration_mode=="age"
3853-                 cutoff_date, # used if expiration_mode=="cutoff-date"
3854-                 sharetypes):
3855-        self.historyfile = historyfile
3856-        self.expiration_enabled = expiration_enabled
3857-        self.mode = mode
3858+    def __init__(self, backend, statefp, historyfp, expiration_policy):
3859+        # ShareCrawler.__init__ will call add_initial_state, so self.historyfp has to be set first.
3860+        self.historyfp = historyfp
3861+        ShareCrawler.__init__(self, backend, statefp)
3862+
3863+        self.expiration_enabled = expiration_policy['enabled']
3864+        self.mode = expiration_policy['mode']
3865         self.override_lease_duration = None
3866         self.cutoff_date = None
3867         if self.mode == "age":
3868hunk ./src/allmydata/storage/expirer.py 63
3869-            assert isinstance(override_lease_duration, (int, type(None)))
3870-            self.override_lease_duration = override_lease_duration # seconds
3871+            assert isinstance(expiration_policy['override_lease_duration'], (int, type(None)))
3872+            self.override_lease_duration = expiration_policy['override_lease_duration'] # seconds
3873         elif self.mode == "cutoff-date":
3874hunk ./src/allmydata/storage/expirer.py 66
3875-            assert isinstance(cutoff_date, int) # seconds-since-epoch
3876-            assert cutoff_date is not None
3877-            self.cutoff_date = cutoff_date
3878+            assert isinstance(expiration_policy['cutoff_date'], int) # seconds-since-epoch
3879+            self.cutoff_date = expiration_policy['cutoff_date']
3880         else:
3881hunk ./src/allmydata/storage/expirer.py 69
3882-            raise ValueError("GC mode '%s' must be 'age' or 'cutoff-date'" % mode)
3883-        self.sharetypes_to_expire = sharetypes
3884-        ShareCrawler.__init__(self, server, statefile)
3885+            raise ValueError("GC mode '%s' must be 'age' or 'cutoff-date'" % expiration_policy['mode'])
3886+        self.sharetypes_to_expire = expiration_policy['sharetypes']
3887 
3888     def add_initial_state(self):
3889         # we fill ["cycle-to-date"] here (even though they will be reset in
3890hunk ./src/allmydata/storage/expirer.py 84
3891             self.state["cycle-to-date"].setdefault(k, so_far[k])
3892 
3893         # initialize history
3894-        if not os.path.exists(self.historyfile):
3895+        if not self.historyfp.exists():
3896             history = {} # cyclenum -> dict
3897hunk ./src/allmydata/storage/expirer.py 86
3898-            f = open(self.historyfile, "wb")
3899-            pickle.dump(history, f)
3900-            f.close()
3901+            self.historyfp.setContent(pickle.dumps(history))
3902 
3903     def create_empty_cycle_dict(self):
3904         recovered = self.create_empty_recovered_dict()
3905hunk ./src/allmydata/storage/expirer.py 99
3906 
3907     def create_empty_recovered_dict(self):
3908         recovered = {}
3909+        # "buckets" is ambiguous; here it means the number of sharesets (one per storage index per server)
3910         for a in ("actual", "original", "configured", "examined"):
3911             for b in ("buckets", "shares", "sharebytes", "diskbytes"):
3912                 recovered[a+"-"+b] = 0
3913hunk ./src/allmydata/storage/expirer.py 110
3914     def started_cycle(self, cycle):
3915         self.state["cycle-to-date"] = self.create_empty_cycle_dict()
3916 
3917-    def stat(self, fn):
3918-        return os.stat(fn)
3919-
3920-    def process_bucket(self, cycle, prefix, prefixdir, storage_index_b32):
3921-        bucketdir = os.path.join(prefixdir, storage_index_b32)
3922-        s = self.stat(bucketdir)
3923+    def process_storage_index(self, cycle, prefix, container):
3924         would_keep_shares = []
3925         wks = None
3926hunk ./src/allmydata/storage/expirer.py 113
3927+        sharetype = None
3928 
3929hunk ./src/allmydata/storage/expirer.py 115
3930-        for fn in os.listdir(bucketdir):
3931-            try:
3932-                shnum = int(fn)
3933-            except ValueError:
3934-                continue # non-numeric means not a sharefile
3935-            sharefile = os.path.join(bucketdir, fn)
3936+        for share in container.get_shares():
3937+            sharetype = share.sharetype
3938             try:
3939hunk ./src/allmydata/storage/expirer.py 118
3940-                wks = self.process_share(sharefile)
3941+                wks = self.process_share(share)
3942             except (UnknownMutableContainerVersionError,
3943                     UnknownImmutableContainerVersionError,
3944                     struct.error):
3945hunk ./src/allmydata/storage/expirer.py 122
3946-                twlog.msg("lease-checker error processing %s" % sharefile)
3947+                twlog.msg("lease-checker error processing %r" % (share,))
3948                 twlog.err()
3949hunk ./src/allmydata/storage/expirer.py 124
3950-                which = (storage_index_b32, shnum)
3951+                which = (si_b2a(share.storageindex), share.get_shnum())
3952                 self.state["cycle-to-date"]["corrupt-shares"].append(which)
3953                 wks = (1, 1, 1, "unknown")
3954             would_keep_shares.append(wks)
3955hunk ./src/allmydata/storage/expirer.py 129
3956 
3957-        sharetype = None
3958+        container_type = None
3959         if wks:
3960hunk ./src/allmydata/storage/expirer.py 131
3961-            # use the last share's sharetype as the buckettype
3962-            sharetype = wks[3]
3963+            # use the last share's sharetype as the container type
3964+            container_type = wks[3]
3965         rec = self.state["cycle-to-date"]["space-recovered"]
3966         self.increment(rec, "examined-buckets", 1)
3967         if sharetype:
3968hunk ./src/allmydata/storage/expirer.py 136
3969-            self.increment(rec, "examined-buckets-"+sharetype, 1)
3970+            self.increment(rec, "examined-buckets-"+container_type, 1)
3971+
3972+        container_diskbytes = container.get_overhead()
3973 
3974hunk ./src/allmydata/storage/expirer.py 140
3975-        try:
3976-            bucket_diskbytes = s.st_blocks * 512
3977-        except AttributeError:
3978-            bucket_diskbytes = 0 # no stat().st_blocks on windows
3979         if sum([wks[0] for wks in would_keep_shares]) == 0:
3980hunk ./src/allmydata/storage/expirer.py 141
3981-            self.increment_bucketspace("original", bucket_diskbytes, sharetype)
3982+            self.increment_container_space("original", container_diskbytes, sharetype)
3983         if sum([wks[1] for wks in would_keep_shares]) == 0:
3984hunk ./src/allmydata/storage/expirer.py 143
3985-            self.increment_bucketspace("configured", bucket_diskbytes, sharetype)
3986+            self.increment_container_space("configured", container_diskbytes, sharetype)
3987         if sum([wks[2] for wks in would_keep_shares]) == 0:
3988hunk ./src/allmydata/storage/expirer.py 145
3989-            self.increment_bucketspace("actual", bucket_diskbytes, sharetype)
3990+            self.increment_container_space("actual", container_diskbytes, sharetype)
3991 
3992hunk ./src/allmydata/storage/expirer.py 147
3993-    def process_share(self, sharefilename):
3994-        # first, find out what kind of a share it is
3995-        sf = get_share_file(sharefilename)
3996-        sharetype = sf.sharetype
3997+    def process_share(self, share):
3998+        sharetype = share.sharetype
3999         now = time.time()
4000hunk ./src/allmydata/storage/expirer.py 150
4001-        s = self.stat(sharefilename)
4002+        sharebytes = share.get_size()
4003+        diskbytes = share.get_used_space()
4004 
4005         num_leases = 0
4006         num_valid_leases_original = 0
4007hunk ./src/allmydata/storage/expirer.py 158
4008         num_valid_leases_configured = 0
4009         expired_leases_configured = []
4010 
4011-        for li in sf.get_leases():
4012+        for li in share.get_leases():
4013             num_leases += 1
4014             original_expiration_time = li.get_expiration_time()
4015             grant_renew_time = li.get_grant_renew_time_time()
4016hunk ./src/allmydata/storage/expirer.py 171
4017 
4018             #  expired-or-not according to our configured age limit
4019             expired = False
4020-            if self.mode == "age":
4021-                age_limit = original_expiration_time
4022-                if self.override_lease_duration is not None:
4023-                    age_limit = self.override_lease_duration
4024-                if age > age_limit:
4025-                    expired = True
4026-            else:
4027-                assert self.mode == "cutoff-date"
4028-                if grant_renew_time < self.cutoff_date:
4029-                    expired = True
4030-            if sharetype not in self.sharetypes_to_expire:
4031-                expired = False
4032+            if sharetype in self.sharetypes_to_expire:
4033+                if self.mode == "age":
4034+                    age_limit = original_expiration_time
4035+                    if self.override_lease_duration is not None:
4036+                        age_limit = self.override_lease_duration
4037+                    if age > age_limit:
4038+                        expired = True
4039+                else:
4040+                    assert self.mode == "cutoff-date"
4041+                    if grant_renew_time < self.cutoff_date:
4042+                        expired = True
4043 
4044             if expired:
4045                 expired_leases_configured.append(li)
4046hunk ./src/allmydata/storage/expirer.py 190
4047 
4048         so_far = self.state["cycle-to-date"]
4049         self.increment(so_far["leases-per-share-histogram"], num_leases, 1)
4050-        self.increment_space("examined", s, sharetype)
4051+        self.increment_space("examined", diskbytes, sharetype)
4052 
4053         would_keep_share = [1, 1, 1, sharetype]
4054 
4055hunk ./src/allmydata/storage/expirer.py 196
4056         if self.expiration_enabled:
4057             for li in expired_leases_configured:
4058-                sf.cancel_lease(li.cancel_secret)
4059+                share.cancel_lease(li.cancel_secret)
4060 
4061         if num_valid_leases_original == 0:
4062             would_keep_share[0] = 0
4063hunk ./src/allmydata/storage/expirer.py 200
4064-            self.increment_space("original", s, sharetype)
4065+            self.increment_space("original", sharebytes, diskbytes, sharetype)
4066 
4067         if num_valid_leases_configured == 0:
4068             would_keep_share[1] = 0
4069hunk ./src/allmydata/storage/expirer.py 204
4070-            self.increment_space("configured", s, sharetype)
4071+            self.increment_space("configured", sharebytes, diskbytes, sharetype)
4072             if self.expiration_enabled:
4073                 would_keep_share[2] = 0
4074hunk ./src/allmydata/storage/expirer.py 207
4075-                self.increment_space("actual", s, sharetype)
4076+                self.increment_space("actual", sharebytes, diskbytes, sharetype)
4077 
4078         return would_keep_share
4079 
4080hunk ./src/allmydata/storage/expirer.py 211
4081-    def increment_space(self, a, s, sharetype):
4082-        sharebytes = s.st_size
4083-        try:
4084-            # note that stat(2) says that st_blocks is 512 bytes, and that
4085-            # st_blksize is "optimal file sys I/O ops blocksize", which is
4086-            # independent of the block-size that st_blocks uses.
4087-            diskbytes = s.st_blocks * 512
4088-        except AttributeError:
4089-            # the docs say that st_blocks is only on linux. I also see it on
4090-            # MacOS. But it isn't available on windows.
4091-            diskbytes = sharebytes
4092+    def increment_space(self, a, sharebytes, diskbytes, sharetype):
4093         so_far_sr = self.state["cycle-to-date"]["space-recovered"]
4094         self.increment(so_far_sr, a+"-shares", 1)
4095         self.increment(so_far_sr, a+"-sharebytes", sharebytes)
4096hunk ./src/allmydata/storage/expirer.py 221
4097             self.increment(so_far_sr, a+"-sharebytes-"+sharetype, sharebytes)
4098             self.increment(so_far_sr, a+"-diskbytes-"+sharetype, diskbytes)
4099 
4100-    def increment_bucketspace(self, a, bucket_diskbytes, sharetype):
4101+    def increment_container_space(self, a, container_diskbytes, container_type):
4102         rec = self.state["cycle-to-date"]["space-recovered"]
4103hunk ./src/allmydata/storage/expirer.py 223
4104-        self.increment(rec, a+"-diskbytes", bucket_diskbytes)
4105+        self.increment(rec, a+"-diskbytes", container_diskbytes)
4106         self.increment(rec, a+"-buckets", 1)
4107hunk ./src/allmydata/storage/expirer.py 225
4108-        if sharetype:
4109-            self.increment(rec, a+"-diskbytes-"+sharetype, bucket_diskbytes)
4110-            self.increment(rec, a+"-buckets-"+sharetype, 1)
4111+        if container_type:
4112+            self.increment(rec, a+"-diskbytes-"+container_type, container_diskbytes)
4113+            self.increment(rec, a+"-buckets-"+container_type, 1)
4114 
4115     def increment(self, d, k, delta=1):
4116         if k not in d:
4117hunk ./src/allmydata/storage/expirer.py 281
4118         # copy() needs to become a deepcopy
4119         h["space-recovered"] = s["space-recovered"].copy()
4120 
4121-        history = pickle.load(open(self.historyfile, "rb"))
4122+        history = pickle.load(self.historyfp.getContent())
4123         history[cycle] = h
4124         while len(history) > 10:
4125             oldcycles = sorted(history.keys())
4126hunk ./src/allmydata/storage/expirer.py 286
4127             del history[oldcycles[0]]
4128-        f = open(self.historyfile, "wb")
4129-        pickle.dump(history, f)
4130-        f.close()
4131+        self.historyfp.setContent(pickle.dumps(history))
4132 
4133     def get_state(self):
4134         """In addition to the crawler state described in
4135hunk ./src/allmydata/storage/expirer.py 355
4136         progress = self.get_progress()
4137 
4138         state = ShareCrawler.get_state(self) # does a shallow copy
4139-        history = pickle.load(open(self.historyfile, "rb"))
4140+        history = pickle.load(self.historyfp.getContent())
4141         state["history"] = history
4142 
4143         if not progress["cycle-in-progress"]:
4144hunk ./src/allmydata/storage/lease.py 3
4145 import struct, time
4146 
4147+
4148+class NonExistentLeaseError(Exception):
4149+    pass
4150+
4151 class LeaseInfo:
4152     def __init__(self, owner_num=None, renew_secret=None, cancel_secret=None,
4153                  expiration_time=None, nodeid=None):
4154hunk ./src/allmydata/storage/lease.py 21
4155 
4156     def get_expiration_time(self):
4157         return self.expiration_time
4158+
4159     def get_grant_renew_time_time(self):
4160         # hack, based upon fixed 31day expiration period
4161         return self.expiration_time - 31*24*60*60
4162hunk ./src/allmydata/storage/lease.py 25
4163+
4164     def get_age(self):
4165         return time.time() - self.get_grant_renew_time_time()
4166 
4167hunk ./src/allmydata/storage/lease.py 36
4168          self.expiration_time) = struct.unpack(">L32s32sL", data)
4169         self.nodeid = None
4170         return self
4171+
4172     def to_immutable_data(self):
4173         return struct.pack(">L32s32sL",
4174                            self.owner_num,
4175hunk ./src/allmydata/storage/lease.py 49
4176                            int(self.expiration_time),
4177                            self.renew_secret, self.cancel_secret,
4178                            self.nodeid)
4179+
4180     def from_mutable_data(self, data):
4181         (self.owner_num,
4182          self.expiration_time,
4183hunk ./src/allmydata/storage/server.py 1
4184-import os, re, weakref, struct, time
4185+import weakref, time
4186 
4187 from foolscap.api import Referenceable
4188 from twisted.application import service
4189hunk ./src/allmydata/storage/server.py 7
4190 
4191 from zope.interface import implements
4192-from allmydata.interfaces import RIStorageServer, IStatsProducer
4193-from allmydata.util import fileutil, idlib, log, time_format
4194+from allmydata.interfaces import RIStorageServer, IStatsProducer, IStorageBackend
4195+from allmydata.util.assertutil import precondition
4196+from allmydata.util import idlib, log
4197 import allmydata # for __full_version__
4198 
4199hunk ./src/allmydata/storage/server.py 12
4200-from allmydata.storage.common import si_b2a, si_a2b, storage_index_to_dir
4201-_pyflakes_hush = [si_b2a, si_a2b, storage_index_to_dir] # re-exported
4202+from allmydata.storage.common import si_a2b, si_b2a
4203+[si_a2b]  # hush pyflakes
4204 from allmydata.storage.lease import LeaseInfo
4205hunk ./src/allmydata/storage/server.py 15
4206-from allmydata.storage.mutable import MutableShareFile, EmptyShare, \
4207-     create_mutable_sharefile
4208-from allmydata.storage.immutable import ShareFile, BucketWriter, BucketReader
4209-from allmydata.storage.crawler import BucketCountingCrawler
4210 from allmydata.storage.expirer import LeaseCheckingCrawler
4211hunk ./src/allmydata/storage/server.py 16
4212-
4213-# storage/
4214-# storage/shares/incoming
4215-#   incoming/ holds temp dirs named $START/$STORAGEINDEX/$SHARENUM which will
4216-#   be moved to storage/shares/$START/$STORAGEINDEX/$SHARENUM upon success
4217-# storage/shares/$START/$STORAGEINDEX
4218-# storage/shares/$START/$STORAGEINDEX/$SHARENUM
4219-
4220-# Where "$START" denotes the first 10 bits worth of $STORAGEINDEX (that's 2
4221-# base-32 chars).
4222-
4223-# $SHARENUM matches this regex:
4224-NUM_RE=re.compile("^[0-9]+$")
4225-
4226+from allmydata.storage.crawler import BucketCountingCrawler
4227 
4228 
4229 class StorageServer(service.MultiService, Referenceable):
4230hunk ./src/allmydata/storage/server.py 21
4231     implements(RIStorageServer, IStatsProducer)
4232+
4233     name = 'storage'
4234     LeaseCheckerClass = LeaseCheckingCrawler
4235hunk ./src/allmydata/storage/server.py 24
4236+    DEFAULT_EXPIRATION_POLICY = {
4237+        'enabled': False,
4238+        'mode': 'age',
4239+        'override_lease_duration': None,
4240+        'cutoff_date': None,
4241+        'sharetypes': ('mutable', 'immutable'),
4242+    }
4243 
4244hunk ./src/allmydata/storage/server.py 32
4245-    def __init__(self, storedir, nodeid, reserved_space=0,
4246-                 discard_storage=False, readonly_storage=False,
4247+    def __init__(self, serverid, backend, statedir,
4248                  stats_provider=None,
4249hunk ./src/allmydata/storage/server.py 34
4250-                 expiration_enabled=False,
4251-                 expiration_mode="age",
4252-                 expiration_override_lease_duration=None,
4253-                 expiration_cutoff_date=None,
4254-                 expiration_sharetypes=("mutable", "immutable")):
4255+                 expiration_policy=None):
4256         service.MultiService.__init__(self)
4257hunk ./src/allmydata/storage/server.py 36
4258-        assert isinstance(nodeid, str)
4259-        assert len(nodeid) == 20
4260-        self.my_nodeid = nodeid
4261-        self.storedir = storedir
4262-        sharedir = os.path.join(storedir, "shares")
4263-        fileutil.make_dirs(sharedir)
4264-        self.sharedir = sharedir
4265-        # we don't actually create the corruption-advisory dir until necessary
4266-        self.corruption_advisory_dir = os.path.join(storedir,
4267-                                                    "corruption-advisories")
4268-        self.reserved_space = int(reserved_space)
4269-        self.no_storage = discard_storage
4270-        self.readonly_storage = readonly_storage
4271+        precondition(IStorageBackend.providedBy(backend), backend)
4272+        precondition(isinstance(serverid, str), serverid)
4273+        precondition(len(serverid) == 20, serverid)
4274+
4275+        self._serverid = serverid
4276         self.stats_provider = stats_provider
4277         if self.stats_provider:
4278             self.stats_provider.register_producer(self)
4279hunk ./src/allmydata/storage/server.py 44
4280-        self.incomingdir = os.path.join(sharedir, 'incoming')
4281-        self._clean_incomplete()
4282-        fileutil.make_dirs(self.incomingdir)
4283         self._active_writers = weakref.WeakKeyDictionary()
4284hunk ./src/allmydata/storage/server.py 45
4285+        self.backend = backend
4286+        self.backend.setServiceParent(self)
4287+        self._statedir = statedir
4288         log.msg("StorageServer created", facility="tahoe.storage")
4289 
4290hunk ./src/allmydata/storage/server.py 50
4291-        if reserved_space:
4292-            if self.get_available_space() is None:
4293-                log.msg("warning: [storage]reserved_space= is set, but this platform does not support an API to get disk statistics (statvfs(2) or GetDiskFreeSpaceEx), so this reservation cannot be honored",
4294-                        umin="0wZ27w", level=log.UNUSUAL)
4295-
4296         self.latencies = {"allocate": [], # immutable
4297                           "write": [],
4298                           "close": [],
4299hunk ./src/allmydata/storage/server.py 61
4300                           "renew": [],
4301                           "cancel": [],
4302                           }
4303-        self.add_bucket_counter()
4304-
4305-        statefile = os.path.join(self.storedir, "lease_checker.state")
4306-        historyfile = os.path.join(self.storedir, "lease_checker.history")
4307-        klass = self.LeaseCheckerClass
4308-        self.lease_checker = klass(self, statefile, historyfile,
4309-                                   expiration_enabled, expiration_mode,
4310-                                   expiration_override_lease_duration,
4311-                                   expiration_cutoff_date,
4312-                                   expiration_sharetypes)
4313-        self.lease_checker.setServiceParent(self)
4314+        self._setup_bucket_counter()
4315+        self._setup_lease_checker(expiration_policy or self.DEFAULT_EXPIRATION_POLICY)
4316 
4317     def __repr__(self):
4318hunk ./src/allmydata/storage/server.py 65
4319-        return "<StorageServer %s>" % (idlib.shortnodeid_b2a(self.my_nodeid),)
4320+        return "<StorageServer %s>" % (idlib.shortnodeid_b2a(self._serverid),)
4321 
4322hunk ./src/allmydata/storage/server.py 67
4323-    def add_bucket_counter(self):
4324-        statefile = os.path.join(self.storedir, "bucket_counter.state")
4325-        self.bucket_counter = BucketCountingCrawler(self, statefile)
4326+    def _setup_bucket_counter(self):
4327+        statefp = self._statedir.child("bucket_counter.state")
4328+        self.bucket_counter = BucketCountingCrawler(self.backend, statefp)
4329         self.bucket_counter.setServiceParent(self)
4330 
4331hunk ./src/allmydata/storage/server.py 72
4332+    def _setup_lease_checker(self, expiration_policy):
4333+        statefp = self._statedir.child("lease_checker.state")
4334+        historyfp = self._statedir.child("lease_checker.history")
4335+        self.lease_checker = self.LeaseCheckerClass(self.backend, statefp, historyfp, expiration_policy)
4336+        self.lease_checker.setServiceParent(self)
4337+
4338     def count(self, name, delta=1):
4339         if self.stats_provider:
4340             self.stats_provider.count("storage_server." + name, delta)
4341hunk ./src/allmydata/storage/server.py 92
4342         """Return a dict, indexed by category, that contains a dict of
4343         latency numbers for each category. If there are sufficient samples
4344         for unambiguous interpretation, each dict will contain the
4345-        following keys: mean, 01_0_percentile, 10_0_percentile,
4346+        following keys: samplesize, mean, 01_0_percentile, 10_0_percentile,
4347         50_0_percentile (median), 90_0_percentile, 95_0_percentile,
4348         99_0_percentile, 99_9_percentile.  If there are insufficient
4349         samples for a given percentile to be interpreted unambiguously
4350hunk ./src/allmydata/storage/server.py 114
4351             else:
4352                 stats["mean"] = None
4353 
4354-            orderstatlist = [(0.01, "01_0_percentile", 100), (0.1, "10_0_percentile", 10),\
4355-                             (0.50, "50_0_percentile", 10), (0.90, "90_0_percentile", 10),\
4356-                             (0.95, "95_0_percentile", 20), (0.99, "99_0_percentile", 100),\
4357+            orderstatlist = [(0.1, "10_0_percentile", 10), (0.5, "50_0_percentile", 10), \
4358+                             (0.9, "90_0_percentile", 10), (0.95, "95_0_percentile", 20), \
4359+                             (0.01, "01_0_percentile", 100),  (0.99, "99_0_percentile", 100),\
4360                              (0.999, "99_9_percentile", 1000)]
4361 
4362             for percentile, percentilestring, minnumtoobserve in orderstatlist:
4363hunk ./src/allmydata/storage/server.py 133
4364             kwargs["facility"] = "tahoe.storage"
4365         return log.msg(*args, **kwargs)
4366 
4367-    def _clean_incomplete(self):
4368-        fileutil.rm_dir(self.incomingdir)
4369+    def get_serverid(self):
4370+        return self._serverid
4371 
4372     def get_stats(self):
4373         # remember: RIStatsProvider requires that our return dict
4374hunk ./src/allmydata/storage/server.py 138
4375-        # contains numeric values.
4376+        # contains numeric, or None values.
4377         stats = { 'storage_server.allocated': self.allocated_size(), }
4378hunk ./src/allmydata/storage/server.py 140
4379-        stats['storage_server.reserved_space'] = self.reserved_space
4380         for category,ld in self.get_latencies().items():
4381             for name,v in ld.items():
4382                 stats['storage_server.latencies.%s.%s' % (category, name)] = v
4383hunk ./src/allmydata/storage/server.py 144
4384 
4385-        try:
4386-            disk = fileutil.get_disk_stats(self.sharedir, self.reserved_space)
4387-            writeable = disk['avail'] > 0
4388-
4389-            # spacetime predictors should use disk_avail / (d(disk_used)/dt)
4390-            stats['storage_server.disk_total'] = disk['total']
4391-            stats['storage_server.disk_used'] = disk['used']
4392-            stats['storage_server.disk_free_for_root'] = disk['free_for_root']
4393-            stats['storage_server.disk_free_for_nonroot'] = disk['free_for_nonroot']
4394-            stats['storage_server.disk_avail'] = disk['avail']
4395-        except AttributeError:
4396-            writeable = True
4397-        except EnvironmentError:
4398-            log.msg("OS call to get disk statistics failed", level=log.UNUSUAL)
4399-            writeable = False
4400-
4401-        if self.readonly_storage:
4402-            stats['storage_server.disk_avail'] = 0
4403-            writeable = False
4404+        self.backend.fill_in_space_stats(stats)
4405 
4406hunk ./src/allmydata/storage/server.py 146
4407-        stats['storage_server.accepting_immutable_shares'] = int(writeable)
4408         s = self.bucket_counter.get_state()
4409         bucket_count = s.get("last-complete-bucket-count")
4410         if bucket_count:
4411hunk ./src/allmydata/storage/server.py 153
4412         return stats
4413 
4414     def get_available_space(self):
4415-        """Returns available space for share storage in bytes, or None if no
4416-        API to get this information is available."""
4417-
4418-        if self.readonly_storage:
4419-            return 0
4420-        return fileutil.get_available_space(self.sharedir, self.reserved_space)
4421+        return self.backend.get_available_space()
4422 
4423     def allocated_size(self):
4424         space = 0
4425hunk ./src/allmydata/storage/server.py 162
4426         return space
4427 
4428     def remote_get_version(self):
4429-        remaining_space = self.get_available_space()
4430+        remaining_space = self.backend.get_available_space()
4431         if remaining_space is None:
4432             # We're on a platform that has no API to get disk stats.
4433             remaining_space = 2**64
4434hunk ./src/allmydata/storage/server.py 178
4435                     }
4436         return version
4437 
4438-    def remote_allocate_buckets(self, storage_index,
4439+    def remote_allocate_buckets(self, storageindex,
4440                                 renew_secret, cancel_secret,
4441                                 sharenums, allocated_size,
4442                                 canary, owner_num=0):
4443hunk ./src/allmydata/storage/server.py 182
4444+        # cancel_secret is no longer used.
4445         # owner_num is not for clients to set, but rather it should be
4446hunk ./src/allmydata/storage/server.py 184
4447-        # curried into the PersonalStorageServer instance that is dedicated
4448-        # to a particular owner.
4449+        # curried into a StorageServer instance dedicated to a particular
4450+        # owner.
4451         start = time.time()
4452         self.count("allocate")
4453hunk ./src/allmydata/storage/server.py 188
4454-        alreadygot = set()
4455         bucketwriters = {} # k: shnum, v: BucketWriter
4456hunk ./src/allmydata/storage/server.py 189
4457-        si_dir = storage_index_to_dir(storage_index)
4458-        si_s = si_b2a(storage_index)
4459 
4460hunk ./src/allmydata/storage/server.py 190
4461+        si_s = si_b2a(storageindex)
4462         log.msg("storage: allocate_buckets %s" % si_s)
4463 
4464hunk ./src/allmydata/storage/server.py 193
4465-        # in this implementation, the lease information (including secrets)
4466-        # goes into the share files themselves. It could also be put into a
4467-        # separate database. Note that the lease should not be added until
4468-        # the BucketWriter has been closed.
4469+        # Note that the lease should not be added until the BucketWriter
4470+        # has been closed.
4471         expire_time = time.time() + 31*24*60*60
4472hunk ./src/allmydata/storage/server.py 196
4473-        lease_info = LeaseInfo(owner_num,
4474-                               renew_secret, cancel_secret,
4475-                               expire_time, self.my_nodeid)
4476+        lease_info = LeaseInfo(owner_num, renew_secret,
4477+                               expire_time, self._serverid)
4478 
4479         max_space_per_bucket = allocated_size
4480 
4481hunk ./src/allmydata/storage/server.py 201
4482-        remaining_space = self.get_available_space()
4483+        remaining_space = self.backend.get_available_space()
4484         limited = remaining_space is not None
4485         if limited:
4486hunk ./src/allmydata/storage/server.py 204
4487-            # this is a bit conservative, since some of this allocated_size()
4488-            # has already been written to disk, where it will show up in
4489+            # This is a bit conservative, since some of this allocated_size()
4490+            # has already been written to the backend, where it will show up in
4491             # get_available_space.
4492             remaining_space -= self.allocated_size()
4493hunk ./src/allmydata/storage/server.py 208
4494-        # self.readonly_storage causes remaining_space <= 0
4495+            # If the backend is read-only, remaining_space will be <= 0.
4496+
4497+        shareset = self.backend.get_shareset(storageindex)
4498 
4499hunk ./src/allmydata/storage/server.py 212
4500-        # fill alreadygot with all shares that we have, not just the ones
4501+        # Fill alreadygot with all shares that we have, not just the ones
4502         # they asked about: this will save them a lot of work. Add or update
4503         # leases for all of them: if they want us to hold shares for this
4504hunk ./src/allmydata/storage/server.py 215
4505-        # file, they'll want us to hold leases for this file.
4506-        for (shnum, fn) in self._get_bucket_shares(storage_index):
4507-            alreadygot.add(shnum)
4508-            sf = ShareFile(fn)
4509-            sf.add_or_renew_lease(lease_info)
4510+        # file, they'll want us to hold leases for all the shares of it.
4511+        #
4512+        # XXX should we be making the assumption here that lease info is
4513+        # duplicated in all shares?
4514+        alreadygot = set()
4515+        for share in shareset.get_shares():
4516+            share.add_or_renew_lease(lease_info)
4517+            alreadygot.add(share.shnum)
4518 
4519hunk ./src/allmydata/storage/server.py 224
4520-        for shnum in sharenums:
4521-            incominghome = os.path.join(self.incomingdir, si_dir, "%d" % shnum)
4522-            finalhome = os.path.join(self.sharedir, si_dir, "%d" % shnum)
4523-            if os.path.exists(finalhome):
4524-                # great! we already have it. easy.
4525-                pass
4526-            elif os.path.exists(incominghome):
4527+        for shnum in sharenums - alreadygot:
4528+            if shareset.has_incoming(shnum):
4529                 # Note that we don't create BucketWriters for shnums that
4530                 # have a partial share (in incoming/), so if a second upload
4531                 # occurs while the first is still in progress, the second
4532hunk ./src/allmydata/storage/server.py 232
4533                 # uploader will use different storage servers.
4534                 pass
4535             elif (not limited) or (remaining_space >= max_space_per_bucket):
4536-                # ok! we need to create the new share file.
4537-                bw = BucketWriter(self, incominghome, finalhome,
4538-                                  max_space_per_bucket, lease_info, canary)
4539-                if self.no_storage:
4540-                    bw.throw_out_all_data = True
4541+                bw = shareset.make_bucket_writer(self, shnum, max_space_per_bucket,
4542+                                                 lease_info, canary)
4543                 bucketwriters[shnum] = bw
4544                 self._active_writers[bw] = 1
4545                 if limited:
4546hunk ./src/allmydata/storage/server.py 239
4547                     remaining_space -= max_space_per_bucket
4548             else:
4549-                # bummer! not enough space to accept this bucket
4550+                # Bummer not enough space to accept this share.
4551                 pass
4552 
4553hunk ./src/allmydata/storage/server.py 242
4554-        if bucketwriters:
4555-            fileutil.make_dirs(os.path.join(self.sharedir, si_dir))
4556-
4557         self.add_latency("allocate", time.time() - start)
4558         return alreadygot, bucketwriters
4559 
4560hunk ./src/allmydata/storage/server.py 245
4561-    def _iter_share_files(self, storage_index):
4562-        for shnum, filename in self._get_bucket_shares(storage_index):
4563-            f = open(filename, 'rb')
4564-            header = f.read(32)
4565-            f.close()
4566-            if header[:32] == MutableShareFile.MAGIC:
4567-                sf = MutableShareFile(filename, self)
4568-                # note: if the share has been migrated, the renew_lease()
4569-                # call will throw an exception, with information to help the
4570-                # client update the lease.
4571-            elif header[:4] == struct.pack(">L", 1):
4572-                sf = ShareFile(filename)
4573-            else:
4574-                continue # non-sharefile
4575-            yield sf
4576-
4577-    def remote_add_lease(self, storage_index, renew_secret, cancel_secret,
4578+    def remote_add_lease(self, storageindex, renew_secret, cancel_secret,
4579                          owner_num=1):
4580hunk ./src/allmydata/storage/server.py 247
4581+        # cancel_secret is no longer used.
4582         start = time.time()
4583         self.count("add-lease")
4584         new_expire_time = time.time() + 31*24*60*60
4585hunk ./src/allmydata/storage/server.py 251
4586-        lease_info = LeaseInfo(owner_num,
4587-                               renew_secret, cancel_secret,
4588-                               new_expire_time, self.my_nodeid)
4589-        for sf in self._iter_share_files(storage_index):
4590-            sf.add_or_renew_lease(lease_info)
4591-        self.add_latency("add-lease", time.time() - start)
4592-        return None
4593+        lease_info = LeaseInfo(owner_num, renew_secret,
4594+                               new_expire_time, self._serverid)
4595 
4596hunk ./src/allmydata/storage/server.py 254
4597-    def remote_renew_lease(self, storage_index, renew_secret):
4598+        try:
4599+            self.backend.add_or_renew_lease(lease_info)
4600+        finally:
4601+            self.add_latency("add-lease", time.time() - start)
4602+
4603+    def remote_renew_lease(self, storageindex, renew_secret):
4604         start = time.time()
4605         self.count("renew")
4606hunk ./src/allmydata/storage/server.py 262
4607-        new_expire_time = time.time() + 31*24*60*60
4608-        found_buckets = False
4609-        for sf in self._iter_share_files(storage_index):
4610-            found_buckets = True
4611-            sf.renew_lease(renew_secret, new_expire_time)
4612-        self.add_latency("renew", time.time() - start)
4613-        if not found_buckets:
4614-            raise IndexError("no such lease to renew")
4615+
4616+        try:
4617+            shareset = self.backend.get_shareset(storageindex)
4618+            new_expiration_time = start + 31*24*60*60   # one month from now
4619+            shareset.renew_lease(renew_secret, new_expiration_time)
4620+        finally:
4621+            self.add_latency("renew", time.time() - start)
4622 
4623     def bucket_writer_closed(self, bw, consumed_size):
4624         if self.stats_provider:
4625hunk ./src/allmydata/storage/server.py 275
4626             self.stats_provider.count('storage_server.bytes_added', consumed_size)
4627         del self._active_writers[bw]
4628 
4629-    def _get_bucket_shares(self, storage_index):
4630-        """Return a list of (shnum, pathname) tuples for files that hold
4631-        shares for this storage_index. In each tuple, 'shnum' will always be
4632-        the integer form of the last component of 'pathname'."""
4633-        storagedir = os.path.join(self.sharedir, storage_index_to_dir(storage_index))
4634-        try:
4635-            for f in os.listdir(storagedir):
4636-                if NUM_RE.match(f):
4637-                    filename = os.path.join(storagedir, f)
4638-                    yield (int(f), filename)
4639-        except OSError:
4640-            # Commonly caused by there being no buckets at all.
4641-            pass
4642-
4643-    def remote_get_buckets(self, storage_index):
4644+    def remote_get_buckets(self, storageindex):
4645         start = time.time()
4646         self.count("get")
4647hunk ./src/allmydata/storage/server.py 278
4648-        si_s = si_b2a(storage_index)
4649+        si_s = si_b2a(storageindex)
4650         log.msg("storage: get_buckets %s" % si_s)
4651         bucketreaders = {} # k: sharenum, v: BucketReader
4652hunk ./src/allmydata/storage/server.py 281
4653-        for shnum, filename in self._get_bucket_shares(storage_index):
4654-            bucketreaders[shnum] = BucketReader(self, filename,
4655-                                                storage_index, shnum)
4656-        self.add_latency("get", time.time() - start)
4657-        return bucketreaders
4658 
4659hunk ./src/allmydata/storage/server.py 282
4660-    def get_leases(self, storage_index):
4661-        """Provide an iterator that yields all of the leases attached to this
4662-        bucket. Each lease is returned as a LeaseInfo instance.
4663+        try:
4664+            shareset = self.backend.get_shareset(storageindex)
4665+            for share in shareset.get_shares():
4666+                bucketreaders[share.get_shnum()] = shareset.make_bucket_reader(self, share)
4667+            return bucketreaders
4668+        finally:
4669+            self.add_latency("get", time.time() - start)
4670 
4671hunk ./src/allmydata/storage/server.py 290
4672-        This method is not for client use.
4673+    def get_leases(self, storageindex):
4674         """
4675hunk ./src/allmydata/storage/server.py 292
4676+        Provide an iterator that yields all of the leases attached to this
4677+        bucket. Each lease is returned as a LeaseInfo instance.
4678 
4679hunk ./src/allmydata/storage/server.py 295
4680-        # since all shares get the same lease data, we just grab the leases
4681-        # from the first share
4682-        try:
4683-            shnum, filename = self._get_bucket_shares(storage_index).next()
4684-            sf = ShareFile(filename)
4685-            return sf.get_leases()
4686-        except StopIteration:
4687-            return iter([])
4688+        This method is not for client use. XXX do we need it at all?
4689+        """
4690+        return self.backend.get_shareset(storageindex).get_leases()
4691 
4692hunk ./src/allmydata/storage/server.py 299
4693-    def remote_slot_testv_and_readv_and_writev(self, storage_index,
4694+    def remote_slot_testv_and_readv_and_writev(self, storageindex,
4695                                                secrets,
4696                                                test_and_write_vectors,
4697                                                read_vector):
4698hunk ./src/allmydata/storage/server.py 305
4699         start = time.time()
4700         self.count("writev")
4701-        si_s = si_b2a(storage_index)
4702+        si_s = si_b2a(storageindex)
4703         log.msg("storage: slot_writev %s" % si_s)
4704hunk ./src/allmydata/storage/server.py 307
4705-        si_dir = storage_index_to_dir(storage_index)
4706-        (write_enabler, renew_secret, cancel_secret) = secrets
4707-        # shares exist if there is a file for them
4708-        bucketdir = os.path.join(self.sharedir, si_dir)
4709-        shares = {}
4710-        if os.path.isdir(bucketdir):
4711-            for sharenum_s in os.listdir(bucketdir):
4712-                try:
4713-                    sharenum = int(sharenum_s)
4714-                except ValueError:
4715-                    continue
4716-                filename = os.path.join(bucketdir, sharenum_s)
4717-                msf = MutableShareFile(filename, self)
4718-                msf.check_write_enabler(write_enabler, si_s)
4719-                shares[sharenum] = msf
4720-        # write_enabler is good for all existing shares.
4721-
4722-        # Now evaluate test vectors.
4723-        testv_is_good = True
4724-        for sharenum in test_and_write_vectors:
4725-            (testv, datav, new_length) = test_and_write_vectors[sharenum]
4726-            if sharenum in shares:
4727-                if not shares[sharenum].check_testv(testv):
4728-                    self.log("testv failed: [%d]: %r" % (sharenum, testv))
4729-                    testv_is_good = False
4730-                    break
4731-            else:
4732-                # compare the vectors against an empty share, in which all
4733-                # reads return empty strings.
4734-                if not EmptyShare().check_testv(testv):
4735-                    self.log("testv failed (empty): [%d] %r" % (sharenum,
4736-                                                                testv))
4737-                    testv_is_good = False
4738-                    break
4739-
4740-        # now gather the read vectors, before we do any writes
4741-        read_data = {}
4742-        for sharenum, share in shares.items():
4743-            read_data[sharenum] = share.readv(read_vector)
4744-
4745-        ownerid = 1 # TODO
4746-        expire_time = time.time() + 31*24*60*60   # one month
4747-        lease_info = LeaseInfo(ownerid,
4748-                               renew_secret, cancel_secret,
4749-                               expire_time, self.my_nodeid)
4750-
4751-        if testv_is_good:
4752-            # now apply the write vectors
4753-            for sharenum in test_and_write_vectors:
4754-                (testv, datav, new_length) = test_and_write_vectors[sharenum]
4755-                if new_length == 0:
4756-                    if sharenum in shares:
4757-                        shares[sharenum].unlink()
4758-                else:
4759-                    if sharenum not in shares:
4760-                        # allocate a new share
4761-                        allocated_size = 2000 # arbitrary, really
4762-                        share = self._allocate_slot_share(bucketdir, secrets,
4763-                                                          sharenum,
4764-                                                          allocated_size,
4765-                                                          owner_num=0)
4766-                        shares[sharenum] = share
4767-                    shares[sharenum].writev(datav, new_length)
4768-                    # and update the lease
4769-                    shares[sharenum].add_or_renew_lease(lease_info)
4770-
4771-            if new_length == 0:
4772-                # delete empty bucket directories
4773-                if not os.listdir(bucketdir):
4774-                    os.rmdir(bucketdir)
4775 
4776hunk ./src/allmydata/storage/server.py 308
4777+        try:
4778+            shareset = self.backend.get_shareset(storageindex)
4779+            expiration_time = start + 31*24*60*60   # one month from now
4780+            return shareset.testv_and_readv_and_writev(self, secrets, test_and_write_vectors,
4781+                                                       read_vector, expiration_time)
4782+        finally:
4783+            self.add_latency("writev", time.time() - start)
4784 
4785hunk ./src/allmydata/storage/server.py 316
4786-        # all done
4787-        self.add_latency("writev", time.time() - start)
4788-        return (testv_is_good, read_data)
4789-
4790-    def _allocate_slot_share(self, bucketdir, secrets, sharenum,
4791-                             allocated_size, owner_num=0):
4792-        (write_enabler, renew_secret, cancel_secret) = secrets
4793-        my_nodeid = self.my_nodeid
4794-        fileutil.make_dirs(bucketdir)
4795-        filename = os.path.join(bucketdir, "%d" % sharenum)
4796-        share = create_mutable_sharefile(filename, my_nodeid, write_enabler,
4797-                                         self)
4798-        return share
4799-
4800-    def remote_slot_readv(self, storage_index, shares, readv):
4801+    def remote_slot_readv(self, storageindex, shares, readv):
4802         start = time.time()
4803         self.count("readv")
4804hunk ./src/allmydata/storage/server.py 319
4805-        si_s = si_b2a(storage_index)
4806-        lp = log.msg("storage: slot_readv %s %s" % (si_s, shares),
4807-                     facility="tahoe.storage", level=log.OPERATIONAL)
4808-        si_dir = storage_index_to_dir(storage_index)
4809-        # shares exist if there is a file for them
4810-        bucketdir = os.path.join(self.sharedir, si_dir)
4811-        if not os.path.isdir(bucketdir):
4812+        si_s = si_b2a(storageindex)
4813+        log.msg("storage: slot_readv %s %s" % (si_s, shares),
4814+                facility="tahoe.storage", level=log.OPERATIONAL)
4815+
4816+        try:
4817+            shareset = self.backend.get_shareset(storageindex)
4818+            return shareset.readv(self, shares, readv)
4819+        finally:
4820             self.add_latency("readv", time.time() - start)
4821hunk ./src/allmydata/storage/server.py 328
4822-            return {}
4823-        datavs = {}
4824-        for sharenum_s in os.listdir(bucketdir):
4825-            try:
4826-                sharenum = int(sharenum_s)
4827-            except ValueError:
4828-                continue
4829-            if sharenum in shares or not shares:
4830-                filename = os.path.join(bucketdir, sharenum_s)
4831-                msf = MutableShareFile(filename, self)
4832-                datavs[sharenum] = msf.readv(readv)
4833-        log.msg("returning shares %s" % (datavs.keys(),),
4834-                facility="tahoe.storage", level=log.NOISY, parent=lp)
4835-        self.add_latency("readv", time.time() - start)
4836-        return datavs
4837 
4838hunk ./src/allmydata/storage/server.py 329
4839-    def remote_advise_corrupt_share(self, share_type, storage_index, shnum,
4840-                                    reason):
4841-        fileutil.make_dirs(self.corruption_advisory_dir)
4842-        now = time_format.iso_utc(sep="T")
4843-        si_s = si_b2a(storage_index)
4844-        # windows can't handle colons in the filename
4845-        fn = os.path.join(self.corruption_advisory_dir,
4846-                          "%s--%s-%d" % (now, si_s, shnum)).replace(":","")
4847-        f = open(fn, "w")
4848-        f.write("report: Share Corruption\n")
4849-        f.write("type: %s\n" % share_type)
4850-        f.write("storage_index: %s\n" % si_s)
4851-        f.write("share_number: %d\n" % shnum)
4852-        f.write("\n")
4853-        f.write(reason)
4854-        f.write("\n")
4855-        f.close()
4856-        log.msg(format=("client claims corruption in (%(share_type)s) " +
4857-                        "%(si)s-%(shnum)d: %(reason)s"),
4858-                share_type=share_type, si=si_s, shnum=shnum, reason=reason,
4859-                level=log.SCARY, umid="SGx2fA")
4860-        return None
4861+    def remote_advise_corrupt_share(self, share_type, storage_index, shnum, reason):
4862+        self.backend.advise_corrupt_share(share_type, storage_index, shnum, reason)
4863hunk ./src/allmydata/test/common.py 20
4864 from allmydata.mutable.common import CorruptShareError
4865 from allmydata.mutable.layout import unpack_header
4866 from allmydata.mutable.publish import MutableData
4867-from allmydata.storage.mutable import MutableShareFile
4868+from allmydata.storage.backends.disk.mutable import MutableDiskShare
4869 from allmydata.util import hashutil, log, fileutil, pollmixin
4870 from allmydata.util.assertutil import precondition
4871 from allmydata.util.consumer import download_to_data
4872hunk ./src/allmydata/test/common.py 1297
4873 
4874 def _corrupt_mutable_share_data(data, debug=False):
4875     prefix = data[:32]
4876-    assert prefix == MutableShareFile.MAGIC, "This function is designed to corrupt mutable shares of v1, and the magic number doesn't look right: %r vs %r" % (prefix, MutableShareFile.MAGIC)
4877-    data_offset = MutableShareFile.DATA_OFFSET
4878+    assert prefix == MutableDiskShare.MAGIC, "This function is designed to corrupt mutable shares of v1, and the magic number doesn't look right: %r vs %r" % (prefix, MutableDiskShare.MAGIC)
4879+    data_offset = MutableDiskShare.DATA_OFFSET
4880     sharetype = data[data_offset:data_offset+1]
4881     assert sharetype == "\x00", "non-SDMF mutable shares not supported"
4882     (version, ig_seqnum, ig_roothash, ig_IV, ig_k, ig_N, ig_segsize,
4883hunk ./src/allmydata/test/no_network.py 21
4884 from twisted.application import service
4885 from twisted.internet import defer, reactor
4886 from twisted.python.failure import Failure
4887+from twisted.python.filepath import FilePath
4888 from foolscap.api import Referenceable, fireEventually, RemoteException
4889 from base64 import b32encode
4890hunk ./src/allmydata/test/no_network.py 24
4891+
4892 from allmydata import uri as tahoe_uri
4893 from allmydata.client import Client
4894hunk ./src/allmydata/test/no_network.py 27
4895-from allmydata.storage.server import StorageServer, storage_index_to_dir
4896+from allmydata.storage.server import StorageServer
4897+from allmydata.storage.backends.disk.disk_backend import DiskBackend
4898 from allmydata.util import fileutil, idlib, hashutil
4899 from allmydata.util.hashutil import sha1
4900 from allmydata.test.common_web import HTTPClientGETFactory
4901hunk ./src/allmydata/test/no_network.py 155
4902             seed = server.get_permutation_seed()
4903             return sha1(peer_selection_index + seed).digest()
4904         return sorted(self.get_connected_servers(), key=_permuted)
4905+
4906     def get_connected_servers(self):
4907         return self.client._servers
4908hunk ./src/allmydata/test/no_network.py 158
4909+
4910     def get_nickname_for_serverid(self, serverid):
4911         return None
4912 
4913hunk ./src/allmydata/test/no_network.py 162
4914+    def get_known_servers(self):
4915+        return self.get_connected_servers()
4916+
4917+    def get_all_serverids(self):
4918+        return self.client.get_all_serverids()
4919+
4920+
4921 class NoNetworkClient(Client):
4922     def create_tub(self):
4923         pass
4924hunk ./src/allmydata/test/no_network.py 262
4925 
4926     def make_server(self, i, readonly=False):
4927         serverid = hashutil.tagged_hash("serverid", str(i))[:20]
4928-        serverdir = os.path.join(self.basedir, "servers",
4929-                                 idlib.shortnodeid_b2a(serverid), "storage")
4930-        fileutil.make_dirs(serverdir)
4931-        ss = StorageServer(serverdir, serverid, stats_provider=SimpleStats(),
4932-                           readonly_storage=readonly)
4933+        storagedir = FilePath(self.basedir).child("servers").child(idlib.shortnodeid_b2a(serverid)).child("storage")
4934+
4935+        # The backend will make the storage directory and any necessary parents.
4936+        backend = DiskBackend(storagedir, readonly=readonly)
4937+        ss = StorageServer(serverid, backend, storagedir, stats_provider=SimpleStats())
4938         ss._no_network_server_number = i
4939         return ss
4940 
4941hunk ./src/allmydata/test/no_network.py 276
4942         middleman = service.MultiService()
4943         middleman.setServiceParent(self)
4944         ss.setServiceParent(middleman)
4945-        serverid = ss.my_nodeid
4946+        serverid = ss.get_serverid()
4947         self.servers_by_number[i] = ss
4948         wrapper = wrap_storage_server(ss)
4949         self.wrappers_by_id[serverid] = wrapper
4950hunk ./src/allmydata/test/no_network.py 295
4951         # it's enough to remove the server from c._servers (we don't actually
4952         # have to detach and stopService it)
4953         for i,ss in self.servers_by_number.items():
4954-            if ss.my_nodeid == serverid:
4955+            if ss.get_serverid() == serverid:
4956                 del self.servers_by_number[i]
4957                 break
4958         del self.wrappers_by_id[serverid]
4959hunk ./src/allmydata/test/no_network.py 345
4960     def get_clientdir(self, i=0):
4961         return self.g.clients[i].basedir
4962 
4963+    def get_server(self, i):
4964+        return self.g.servers_by_number[i]
4965+
4966     def get_serverdir(self, i):
4967hunk ./src/allmydata/test/no_network.py 349
4968-        return self.g.servers_by_number[i].storedir
4969+        return self.g.servers_by_number[i].backend.storedir
4970+
4971+    def remove_server(self, i):
4972+        self.g.remove_server(self.g.servers_by_number[i].get_serverid())
4973 
4974     def iterate_servers(self):
4975         for i in sorted(self.g.servers_by_number.keys()):
4976hunk ./src/allmydata/test/no_network.py 357
4977             ss = self.g.servers_by_number[i]
4978-            yield (i, ss, ss.storedir)
4979+            yield (i, ss, ss.backend.storedir)
4980 
4981     def find_uri_shares(self, uri):
4982         si = tahoe_uri.from_string(uri).get_storage_index()
4983hunk ./src/allmydata/test/no_network.py 361
4984-        prefixdir = storage_index_to_dir(si)
4985         shares = []
4986         for i,ss in self.g.servers_by_number.items():
4987hunk ./src/allmydata/test/no_network.py 363
4988-            serverid = ss.my_nodeid
4989-            basedir = os.path.join(ss.sharedir, prefixdir)
4990-            if not os.path.exists(basedir):
4991-                continue
4992-            for f in os.listdir(basedir):
4993-                try:
4994-                    shnum = int(f)
4995-                    shares.append((shnum, serverid, os.path.join(basedir, f)))
4996-                except ValueError:
4997-                    pass
4998+            for share in ss.backend.get_shareset(si).get_shares():
4999+                shares.append((share.get_shnum(), ss.get_serverid(), share._home))
5000         return sorted(shares)
5001 
5002hunk ./src/allmydata/test/no_network.py 367
5003+    def count_leases(self, uri):
5004+        """Return (filename, leasecount) pairs in arbitrary order."""
5005+        si = tahoe_uri.from_string(uri).get_storage_index()
5006+        lease_counts = []
5007+        for i,ss in self.g.servers_by_number.items():
5008+            for share in ss.backend.get_shareset(si).get_shares():
5009+                num_leases = len(list(share.get_leases()))
5010+                lease_counts.append( (share._home.path, num_leases) )
5011+        return lease_counts
5012+
5013     def copy_shares(self, uri):
5014         shares = {}
5015hunk ./src/allmydata/test/no_network.py 379
5016-        for (shnum, serverid, sharefile) in self.find_uri_shares(uri):
5017-            shares[sharefile] = open(sharefile, "rb").read()
5018+        for (shnum, serverid, sharefp) in self.find_uri_shares(uri):
5019+            shares[sharefp.path] = sharefp.getContent()
5020         return shares
5021 
5022hunk ./src/allmydata/test/no_network.py 383
5023+    def copy_share(self, from_share, uri, to_server):
5024+        si = uri.from_string(self.uri).get_storage_index()
5025+        (i_shnum, i_serverid, i_sharefp) = from_share
5026+        shares_dir = to_server.backend.get_shareset(si)._sharehomedir
5027+        i_sharefp.copyTo(shares_dir.child(str(i_shnum)))
5028+
5029     def restore_all_shares(self, shares):
5030hunk ./src/allmydata/test/no_network.py 390
5031-        for sharefile, data in shares.items():
5032-            open(sharefile, "wb").write(data)
5033+        for share, data in shares.items():
5034+            share.home.setContent(data)
5035 
5036hunk ./src/allmydata/test/no_network.py 393
5037-    def delete_share(self, (shnum, serverid, sharefile)):
5038-        os.unlink(sharefile)
5039+    def delete_share(self, (shnum, serverid, sharefp)):
5040+        sharefp.remove()
5041 
5042     def delete_shares_numbered(self, uri, shnums):
5043hunk ./src/allmydata/test/no_network.py 397
5044-        for (i_shnum, i_serverid, i_sharefile) in self.find_uri_shares(uri):
5045+        for (i_shnum, i_serverid, i_sharefp) in self.find_uri_shares(uri):
5046             if i_shnum in shnums:
5047hunk ./src/allmydata/test/no_network.py 399
5048-                os.unlink(i_sharefile)
5049+                i_sharefp.remove()
5050 
5051hunk ./src/allmydata/test/no_network.py 401
5052-    def corrupt_share(self, (shnum, serverid, sharefile), corruptor_function):
5053-        sharedata = open(sharefile, "rb").read()
5054-        corruptdata = corruptor_function(sharedata)
5055-        open(sharefile, "wb").write(corruptdata)
5056+    def corrupt_share(self, (shnum, serverid, sharefp), corruptor_function, debug=False):
5057+        sharedata = sharefp.getContent()
5058+        corruptdata = corruptor_function(sharedata, debug=debug)
5059+        sharefp.setContent(corruptdata)
5060 
5061     def corrupt_shares_numbered(self, uri, shnums, corruptor, debug=False):
5062hunk ./src/allmydata/test/no_network.py 407
5063-        for (i_shnum, i_serverid, i_sharefile) in self.find_uri_shares(uri):
5064+        for (i_shnum, i_serverid, i_sharefp) in self.find_uri_shares(uri):
5065             if i_shnum in shnums:
5066hunk ./src/allmydata/test/no_network.py 409
5067-                sharedata = open(i_sharefile, "rb").read()
5068-                corruptdata = corruptor(sharedata, debug=debug)
5069-                open(i_sharefile, "wb").write(corruptdata)
5070+                self.corrupt_share((i_shnum, i_serverid, i_sharefp), corruptor, debug=debug)
5071 
5072     def corrupt_all_shares(self, uri, corruptor, debug=False):
5073hunk ./src/allmydata/test/no_network.py 412
5074-        for (i_shnum, i_serverid, i_sharefile) in self.find_uri_shares(uri):
5075-            sharedata = open(i_sharefile, "rb").read()
5076-            corruptdata = corruptor(sharedata, debug=debug)
5077-            open(i_sharefile, "wb").write(corruptdata)
5078+        for (i_shnum, i_serverid, i_sharefp) in self.find_uri_shares(uri):
5079+            self.corrupt_share((i_shnum, i_serverid, i_sharefp), corruptor, debug=debug)
5080 
5081     def GET(self, urlpath, followRedirect=False, return_response=False,
5082             method="GET", clientnum=0, **kwargs):
5083hunk ./src/allmydata/test/test_download.py 6
5084 # a previous run. This asserts that the current code is capable of decoding
5085 # shares from a previous version.
5086 
5087-import os
5088 from twisted.trial import unittest
5089 from twisted.internet import defer, reactor
5090 from allmydata import uri
5091hunk ./src/allmydata/test/test_download.py 9
5092-from allmydata.storage.server import storage_index_to_dir
5093 from allmydata.util import base32, fileutil, spans, log, hashutil
5094 from allmydata.util.consumer import download_to_data, MemoryConsumer
5095 from allmydata.immutable import upload, layout
5096hunk ./src/allmydata/test/test_download.py 85
5097         u = upload.Data(plaintext, None)
5098         d = self.c0.upload(u)
5099         f = open("stored_shares.py", "w")
5100-        def _created_immutable(ur):
5101-            # write the generated shares and URI to a file, which can then be
5102-            # incorporated into this one next time.
5103-            f.write('immutable_uri = "%s"\n' % ur.uri)
5104-            f.write('immutable_shares = {\n')
5105-            si = uri.from_string(ur.uri).get_storage_index()
5106-            si_dir = storage_index_to_dir(si)
5107+
5108+        def _write_py(uri):
5109+            si = uri.from_string(uri).get_storage_index()
5110             for (i,ss,ssdir) in self.iterate_servers():
5111hunk ./src/allmydata/test/test_download.py 89
5112-                sharedir = os.path.join(ssdir, "shares", si_dir)
5113                 shares = {}
5114hunk ./src/allmydata/test/test_download.py 90
5115-                for fn in os.listdir(sharedir):
5116-                    shnum = int(fn)
5117-                    sharedata = open(os.path.join(sharedir, fn), "rb").read()
5118-                    shares[shnum] = sharedata
5119-                fileutil.rm_dir(sharedir)
5120+                shareset = ss.backend.get_shareset(si)
5121+                for share in shareset.get_shares():
5122+                    sharedata = share._home.getContent()
5123+                    shares[share.get_shnum()] = sharedata
5124+
5125+                fileutil.fp_remove(shareset._sharehomedir)
5126                 if shares:
5127                     f.write(' %d: { # client[%d]\n' % (i, i))
5128                     for shnum in sorted(shares.keys()):
5129hunk ./src/allmydata/test/test_download.py 103
5130                                 (shnum, base32.b2a(shares[shnum])))
5131                     f.write('    },\n')
5132             f.write('}\n')
5133-            f.write('\n')
5134 
5135hunk ./src/allmydata/test/test_download.py 104
5136+        def _created_immutable(ur):
5137+            # write the generated shares and URI to a file, which can then be
5138+            # incorporated into this one next time.
5139+            f.write('immutable_uri = "%s"\n' % ur.uri)
5140+            f.write('immutable_shares = {\n')
5141+            _write_py(ur.uri)
5142+            f.write('\n')
5143         d.addCallback(_created_immutable)
5144 
5145         d.addCallback(lambda ignored:
5146hunk ./src/allmydata/test/test_download.py 118
5147         def _created_mutable(n):
5148             f.write('mutable_uri = "%s"\n' % n.get_uri())
5149             f.write('mutable_shares = {\n')
5150-            si = uri.from_string(n.get_uri()).get_storage_index()
5151-            si_dir = storage_index_to_dir(si)
5152-            for (i,ss,ssdir) in self.iterate_servers():
5153-                sharedir = os.path.join(ssdir, "shares", si_dir)
5154-                shares = {}
5155-                for fn in os.listdir(sharedir):
5156-                    shnum = int(fn)
5157-                    sharedata = open(os.path.join(sharedir, fn), "rb").read()
5158-                    shares[shnum] = sharedata
5159-                fileutil.rm_dir(sharedir)
5160-                if shares:
5161-                    f.write(' %d: { # client[%d]\n' % (i, i))
5162-                    for shnum in sorted(shares.keys()):
5163-                        f.write('  %d: base32.a2b("%s"),\n' %
5164-                                (shnum, base32.b2a(shares[shnum])))
5165-                    f.write('    },\n')
5166-            f.write('}\n')
5167-
5168-            f.close()
5169+            _write_py(n.get_uri())
5170         d.addCallback(_created_mutable)
5171 
5172         def _done(ignored):
5173hunk ./src/allmydata/test/test_download.py 123
5174             f.close()
5175-        d.addCallback(_done)
5176+        d.addBoth(_done)
5177 
5178         return d
5179 
5180hunk ./src/allmydata/test/test_download.py 127
5181+    def _write_shares(self, uri, shares):
5182+        si = uri.from_string(uri).get_storage_index()
5183+        for i in shares:
5184+            shares_for_server = shares[i]
5185+            for shnum in shares_for_server:
5186+                share_dir = self.get_server(i).backend.get_shareset(si)._sharehomedir
5187+                fileutil.fp_make_dirs(share_dir)
5188+                share_dir.child(str(shnum)).setContent(shares[shnum])
5189+
5190     def load_shares(self, ignored=None):
5191         # this uses the data generated by create_shares() to populate the
5192         # storage servers with pre-generated shares
5193hunk ./src/allmydata/test/test_download.py 139
5194-        si = uri.from_string(immutable_uri).get_storage_index()
5195-        si_dir = storage_index_to_dir(si)
5196-        for i in immutable_shares:
5197-            shares = immutable_shares[i]
5198-            for shnum in shares:
5199-                dn = os.path.join(self.get_serverdir(i), "shares", si_dir)
5200-                fileutil.make_dirs(dn)
5201-                fn = os.path.join(dn, str(shnum))
5202-                f = open(fn, "wb")
5203-                f.write(shares[shnum])
5204-                f.close()
5205-
5206-        si = uri.from_string(mutable_uri).get_storage_index()
5207-        si_dir = storage_index_to_dir(si)
5208-        for i in mutable_shares:
5209-            shares = mutable_shares[i]
5210-            for shnum in shares:
5211-                dn = os.path.join(self.get_serverdir(i), "shares", si_dir)
5212-                fileutil.make_dirs(dn)
5213-                fn = os.path.join(dn, str(shnum))
5214-                f = open(fn, "wb")
5215-                f.write(shares[shnum])
5216-                f.close()
5217+        self._write_shares(immutable_uri, immutable_shares)
5218+        self._write_shares(mutable_uri, mutable_shares)
5219 
5220     def download_immutable(self, ignored=None):
5221         n = self.c0.create_node_from_uri(immutable_uri)
5222hunk ./src/allmydata/test/test_download.py 183
5223 
5224         self.load_shares()
5225         si = uri.from_string(immutable_uri).get_storage_index()
5226-        si_dir = storage_index_to_dir(si)
5227 
5228         n = self.c0.create_node_from_uri(immutable_uri)
5229         d = download_to_data(n)
5230hunk ./src/allmydata/test/test_download.py 198
5231                 for clientnum in immutable_shares:
5232                     for shnum in immutable_shares[clientnum]:
5233                         if s._shnum == shnum:
5234-                            fn = os.path.join(self.get_serverdir(clientnum),
5235-                                              "shares", si_dir, str(shnum))
5236-                            os.unlink(fn)
5237+                            share_dir = self.get_server(clientnum).backend.get_shareset(si)._sharehomedir
5238+                            share_dir.child(str(shnum)).remove()
5239         d.addCallback(_clobber_some_shares)
5240         d.addCallback(lambda ign: download_to_data(n))
5241         d.addCallback(_got_data)
5242hunk ./src/allmydata/test/test_download.py 212
5243                 for shnum in immutable_shares[clientnum]:
5244                     if shnum == save_me:
5245                         continue
5246-                    fn = os.path.join(self.get_serverdir(clientnum),
5247-                                      "shares", si_dir, str(shnum))
5248-                    if os.path.exists(fn):
5249-                        os.unlink(fn)
5250+                    share_dir = self.get_server(clientnum).backend.get_shareset(si)._sharehomedir
5251+                    fileutil.fp_remove(share_dir.child(str(shnum)))
5252             # now the download should fail with NotEnoughSharesError
5253             return self.shouldFail(NotEnoughSharesError, "1shares", None,
5254                                    download_to_data, n)
5255hunk ./src/allmydata/test/test_download.py 223
5256             # delete the last remaining share
5257             for clientnum in immutable_shares:
5258                 for shnum in immutable_shares[clientnum]:
5259-                    fn = os.path.join(self.get_serverdir(clientnum),
5260-                                      "shares", si_dir, str(shnum))
5261-                    if os.path.exists(fn):
5262-                        os.unlink(fn)
5263+                    share_dir = self.get_server(clientnum).backend.get_shareset(si)._sharehomedir
5264+                    share_dir.child(str(shnum)).remove()
5265             # now a new download should fail with NoSharesError. We want a
5266             # new ImmutableFileNode so it will forget about the old shares.
5267             # If we merely called create_node_from_uri() without first
5268hunk ./src/allmydata/test/test_download.py 801
5269         # will report two shares, and the ShareFinder will handle the
5270         # duplicate by attaching both to the same CommonShare instance.
5271         si = uri.from_string(immutable_uri).get_storage_index()
5272-        si_dir = storage_index_to_dir(si)
5273-        sh0_file = [sharefile
5274-                    for (shnum, serverid, sharefile)
5275-                    in self.find_uri_shares(immutable_uri)
5276-                    if shnum == 0][0]
5277-        sh0_data = open(sh0_file, "rb").read()
5278+        sh0_fp = [sharefp for (shnum, serverid, sharefp)
5279+                          in self.find_uri_shares(immutable_uri)
5280+                          if shnum == 0][0]
5281+        sh0_data = sh0_fp.getContent()
5282         for clientnum in immutable_shares:
5283             if 0 in immutable_shares[clientnum]:
5284                 continue
5285hunk ./src/allmydata/test/test_download.py 808
5286-            cdir = self.get_serverdir(clientnum)
5287-            target = os.path.join(cdir, "shares", si_dir, "0")
5288-            outf = open(target, "wb")
5289-            outf.write(sh0_data)
5290-            outf.close()
5291+            cdir = self.get_server(clientnum).backend.get_shareset(si)._sharehomedir
5292+            fileutil.fp_make_dirs(cdir)
5293+            cdir.child(str(shnum)).setContent(sh0_data)
5294 
5295         d = self.download_immutable()
5296         return d
5297hunk ./src/allmydata/test/test_encode.py 134
5298         d.addCallback(_try)
5299         return d
5300 
5301-    def get_share_hashes(self, at_least_these=()):
5302+    def get_share_hashes(self):
5303         d = self._start()
5304         def _try(unused=None):
5305             if self.mode == "bad sharehash":
5306hunk ./src/allmydata/test/test_hung_server.py 3
5307 # -*- coding: utf-8 -*-
5308 
5309-import os, shutil
5310 from twisted.trial import unittest
5311 from twisted.internet import defer
5312hunk ./src/allmydata/test/test_hung_server.py 5
5313-from allmydata import uri
5314+
5315 from allmydata.util.consumer import download_to_data
5316 from allmydata.immutable import upload
5317 from allmydata.mutable.common import UnrecoverableFileError
5318hunk ./src/allmydata/test/test_hung_server.py 10
5319 from allmydata.mutable.publish import MutableData
5320-from allmydata.storage.common import storage_index_to_dir
5321 from allmydata.test.no_network import GridTestMixin
5322 from allmydata.test.common import ShouldFailMixin
5323 from allmydata.util.pollmixin import PollMixin
5324hunk ./src/allmydata/test/test_hung_server.py 18
5325 immutable_plaintext = "data" * 10000
5326 mutable_plaintext = "muta" * 10000
5327 
5328+
5329 class HungServerDownloadTest(GridTestMixin, ShouldFailMixin, PollMixin,
5330                              unittest.TestCase):
5331     # Many of these tests take around 60 seconds on François's ARM buildslave:
5332hunk ./src/allmydata/test/test_hung_server.py 31
5333     timeout = 240
5334 
5335     def _break(self, servers):
5336-        for (id, ss) in servers:
5337-            self.g.break_server(id)
5338+        for ss in servers:
5339+            self.g.break_server(ss.get_serverid())
5340 
5341     def _hang(self, servers, **kwargs):
5342hunk ./src/allmydata/test/test_hung_server.py 35
5343-        for (id, ss) in servers:
5344-            self.g.hang_server(id, **kwargs)
5345+        for ss in servers:
5346+            self.g.hang_server(ss.get_serverid(), **kwargs)
5347 
5348     def _unhang(self, servers, **kwargs):
5349hunk ./src/allmydata/test/test_hung_server.py 39
5350-        for (id, ss) in servers:
5351-            self.g.unhang_server(id, **kwargs)
5352+        for ss in servers:
5353+            self.g.unhang_server(ss.get_serverid(), **kwargs)
5354 
5355     def _hang_shares(self, shnums, **kwargs):
5356         # hang all servers who are holding the given shares
5357hunk ./src/allmydata/test/test_hung_server.py 52
5358                     hung_serverids.add(i_serverid)
5359 
5360     def _delete_all_shares_from(self, servers):
5361-        serverids = [id for (id, ss) in servers]
5362-        for (i_shnum, i_serverid, i_sharefile) in self.shares:
5363+        serverids = [ss.get_serverid() for ss in servers]
5364+        for (i_shnum, i_serverid, i_sharefp) in self.shares:
5365             if i_serverid in serverids:
5366hunk ./src/allmydata/test/test_hung_server.py 55
5367-                os.unlink(i_sharefile)
5368+                i_sharefp.remove()
5369 
5370     def _corrupt_all_shares_in(self, servers, corruptor_func):
5371hunk ./src/allmydata/test/test_hung_server.py 58
5372-        serverids = [id for (id, ss) in servers]
5373-        for (i_shnum, i_serverid, i_sharefile) in self.shares:
5374+        serverids = [ss.get_serverid() for ss in servers]
5375+        for (i_shnum, i_serverid, i_sharefp) in self.shares:
5376             if i_serverid in serverids:
5377hunk ./src/allmydata/test/test_hung_server.py 61
5378-                self._corrupt_share((i_shnum, i_sharefile), corruptor_func)
5379+                self.corrupt_share((i_shnum, i_serverid, i_sharefp), corruptor_func)
5380 
5381     def _copy_all_shares_from(self, from_servers, to_server):
5382hunk ./src/allmydata/test/test_hung_server.py 64
5383-        serverids = [id for (id, ss) in from_servers]
5384-        for (i_shnum, i_serverid, i_sharefile) in self.shares:
5385+        serverids = [ss.get_serverid() for ss in from_servers]
5386+        for (i_shnum, i_serverid, i_sharefp) in self.shares:
5387             if i_serverid in serverids:
5388hunk ./src/allmydata/test/test_hung_server.py 67
5389-                self._copy_share((i_shnum, i_sharefile), to_server)
5390+                self.copy_share((i_shnum, i_serverid, i_sharefp), self.uri, to_server)
5391 
5392hunk ./src/allmydata/test/test_hung_server.py 69
5393-    def _copy_share(self, share, to_server):
5394-        (sharenum, sharefile) = share
5395-        (id, ss) = to_server
5396-        shares_dir = os.path.join(ss.original.storedir, "shares")
5397-        si = uri.from_string(self.uri).get_storage_index()
5398-        si_dir = os.path.join(shares_dir, storage_index_to_dir(si))
5399-        if not os.path.exists(si_dir):
5400-            os.makedirs(si_dir)
5401-        new_sharefile = os.path.join(si_dir, str(sharenum))
5402-        shutil.copy(sharefile, new_sharefile)
5403         self.shares = self.find_uri_shares(self.uri)
5404hunk ./src/allmydata/test/test_hung_server.py 70
5405-        # Make sure that the storage server has the share.
5406-        self.failUnless((sharenum, ss.original.my_nodeid, new_sharefile)
5407-                        in self.shares)
5408-
5409-    def _corrupt_share(self, share, corruptor_func):
5410-        (sharenum, sharefile) = share
5411-        data = open(sharefile, "rb").read()
5412-        newdata = corruptor_func(data)
5413-        os.unlink(sharefile)
5414-        wf = open(sharefile, "wb")
5415-        wf.write(newdata)
5416-        wf.close()
5417 
5418     def _set_up(self, mutable, testdir, num_clients=1, num_servers=10):
5419         self.mutable = mutable
5420hunk ./src/allmydata/test/test_hung_server.py 82
5421 
5422         self.c0 = self.g.clients[0]
5423         nm = self.c0.nodemaker
5424-        self.servers = sorted([(s.get_serverid(), s.get_rref())
5425-                               for s in nm.storage_broker.get_connected_servers()])
5426+        unsorted = [(s.get_serverid(), s.get_rref()) for s in nm.storage_broker.get_connected_servers()]
5427+        self.servers = [ss for (id, ss) in sorted(unsorted)]
5428         self.servers = self.servers[5:] + self.servers[:5]
5429 
5430         if mutable:
5431hunk ./src/allmydata/test/test_hung_server.py 244
5432             # stuck-but-not-overdue, and 4 live requests. All 4 live requests
5433             # will retire before the download is complete and the ShareFinder
5434             # is shut off. That will leave 4 OVERDUE and 1
5435-            # stuck-but-not-overdue, for a total of 5 requests in in
5436+            # stuck-but-not-overdue, for a total of 5 requests in
5437             # _sf.pending_requests
5438             for t in self._sf.overdue_timers.values()[:4]:
5439                 t.reset(-1.0)
5440hunk ./src/allmydata/test/test_mutable.py 21
5441 from foolscap.api import eventually, fireEventually
5442 from foolscap.logging import log
5443 from allmydata.storage_client import StorageFarmBroker
5444-from allmydata.storage.common import storage_index_to_dir
5445 from allmydata.scripts import debug
5446 
5447 from allmydata.mutable.filenode import MutableFileNode, BackoffAgent
5448hunk ./src/allmydata/test/test_mutable.py 3662
5449         # Now execute each assignment by writing the storage.
5450         for (share, servernum) in assignments:
5451             sharedata = base64.b64decode(self.sdmf_old_shares[share])
5452-            storedir = self.get_serverdir(servernum)
5453-            storage_path = os.path.join(storedir, "shares",
5454-                                        storage_index_to_dir(si))
5455-            fileutil.make_dirs(storage_path)
5456-            fileutil.write(os.path.join(storage_path, "%d" % share),
5457-                           sharedata)
5458+            storage_dir = self.get_server(servernum).backend.get_shareset(si).sharehomedir
5459+            fileutil.fp_make_dirs(storage_dir)
5460+            storage_dir.child("%d" % share).setContent(sharedata)
5461         # ...and verify that the shares are there.
5462         shares = self.find_uri_shares(self.sdmf_old_cap)
5463         assert len(shares) == 10
5464hunk ./src/allmydata/test/test_provisioning.py 13
5465 from nevow import inevow
5466 from zope.interface import implements
5467 
5468-class MyRequest:
5469+class MockRequest:
5470     implements(inevow.IRequest)
5471     pass
5472 
5473hunk ./src/allmydata/test/test_provisioning.py 26
5474     def test_load(self):
5475         pt = provisioning.ProvisioningTool()
5476         self.fields = {}
5477-        #r = MyRequest()
5478+        #r = MockRequest()
5479         #r.fields = self.fields
5480         #ctx = RequestContext()
5481         #unfilled = pt.renderSynchronously(ctx)
5482hunk ./src/allmydata/test/test_repairer.py 537
5483         # happiness setting.
5484         def _delete_some_servers(ignored):
5485             for i in xrange(7):
5486-                self.g.remove_server(self.g.servers_by_number[i].my_nodeid)
5487+                self.remove_server(i)
5488 
5489             assert len(self.g.servers_by_number) == 3
5490 
5491hunk ./src/allmydata/test/test_storage.py 14
5492 from allmydata import interfaces
5493 from allmydata.util import fileutil, hashutil, base32, pollmixin, time_format
5494 from allmydata.storage.server import StorageServer
5495-from allmydata.storage.mutable import MutableShareFile
5496-from allmydata.storage.immutable import BucketWriter, BucketReader
5497-from allmydata.storage.common import DataTooLargeError, storage_index_to_dir, \
5498+from allmydata.storage.backends.disk.mutable import MutableDiskShare
5499+from allmydata.storage.bucket import BucketWriter, BucketReader
5500+from allmydata.storage.common import DataTooLargeError, \
5501      UnknownMutableContainerVersionError, UnknownImmutableContainerVersionError
5502 from allmydata.storage.lease import LeaseInfo
5503 from allmydata.storage.crawler import BucketCountingCrawler
5504hunk ./src/allmydata/test/test_storage.py 474
5505         w[0].remote_write(0, "\xff"*10)
5506         w[0].remote_close()
5507 
5508-        fn = os.path.join(ss.sharedir, storage_index_to_dir("si1"), "0")
5509-        f = open(fn, "rb+")
5510+        fp = ss.backend.get_shareset("si1").sharehomedir.child("0")
5511+        f = fp.open("rb+")
5512         f.seek(0)
5513         f.write(struct.pack(">L", 0)) # this is invalid: minimum used is v1
5514         f.close()
5515hunk ./src/allmydata/test/test_storage.py 814
5516     def test_bad_magic(self):
5517         ss = self.create("test_bad_magic")
5518         self.allocate(ss, "si1", "we1", self._lease_secret.next(), set([0]), 10)
5519-        fn = os.path.join(ss.sharedir, storage_index_to_dir("si1"), "0")
5520-        f = open(fn, "rb+")
5521+        fp = ss.backend.get_shareset("si1").sharehomedir.child("0")
5522+        f = fp.open("rb+")
5523         f.seek(0)
5524         f.write("BAD MAGIC")
5525         f.close()
5526hunk ./src/allmydata/test/test_storage.py 842
5527 
5528         # Trying to make the container too large (by sending a write vector
5529         # whose offset is too high) will raise an exception.
5530-        TOOBIG = MutableShareFile.MAX_SIZE + 10
5531+        TOOBIG = MutableDiskShare.MAX_SIZE + 10
5532         self.failUnlessRaises(DataTooLargeError,
5533                               rstaraw, "si1", secrets,
5534                               {0: ([], [(TOOBIG,data)], None)},
5535hunk ./src/allmydata/test/test_storage.py 1229
5536 
5537         # create a random non-numeric file in the bucket directory, to
5538         # exercise the code that's supposed to ignore those.
5539-        bucket_dir = os.path.join(self.workdir("test_leases"),
5540-                                  "shares", storage_index_to_dir("si1"))
5541-        f = open(os.path.join(bucket_dir, "ignore_me.txt"), "w")
5542-        f.write("you ought to be ignoring me\n")
5543-        f.close()
5544+        bucket_dir = ss.backend.get_shareset("si1").sharehomedir
5545+        bucket_dir.child("ignore_me.txt").setContent("you ought to be ignoring me\n")
5546 
5547hunk ./src/allmydata/test/test_storage.py 1232
5548-        s0 = MutableShareFile(os.path.join(bucket_dir, "0"))
5549+        s0 = MutableDiskShare(os.path.join(bucket_dir, "0"))
5550         self.failUnlessEqual(len(list(s0.get_leases())), 1)
5551 
5552         # add-lease on a missing storage index is silently ignored
5553hunk ./src/allmydata/test/test_storage.py 3118
5554         [immutable_si_0, immutable_si_1, mutable_si_2, mutable_si_3] = self.sis
5555 
5556         # add a non-sharefile to exercise another code path
5557-        fn = os.path.join(ss.sharedir,
5558-                          storage_index_to_dir(immutable_si_0),
5559-                          "not-a-share")
5560-        f = open(fn, "wb")
5561-        f.write("I am not a share.\n")
5562-        f.close()
5563+        fp = ss.backend.get_shareset(immutable_si_0).sharehomedir.child("not-a-share")
5564+        fp.setContent("I am not a share.\n")
5565 
5566         # this is before the crawl has started, so we're not in a cycle yet
5567         initial_state = lc.get_state()
5568hunk ./src/allmydata/test/test_storage.py 3282
5569     def test_expire_age(self):
5570         basedir = "storage/LeaseCrawler/expire_age"
5571         fileutil.make_dirs(basedir)
5572-        # setting expiration_time to 2000 means that any lease which is more
5573-        # than 2000s old will be expired.
5574-        ss = InstrumentedStorageServer(basedir, "\x00" * 20,
5575-                                       expiration_enabled=True,
5576-                                       expiration_mode="age",
5577-                                       expiration_override_lease_duration=2000)
5578+        # setting 'override_lease_duration' to 2000 means that any lease that
5579+        # is more than 2000 seconds old will be expired.
5580+        expiration_policy = {
5581+            'enabled': True,
5582+            'mode': 'age',
5583+            'override_lease_duration': 2000,
5584+            'sharetypes': ('mutable', 'immutable'),
5585+        }
5586+        ss = InstrumentedStorageServer(basedir, "\x00" * 20, expiration_policy)
5587         # make it start sooner than usual.
5588         lc = ss.lease_checker
5589         lc.slow_start = 0
5590hunk ./src/allmydata/test/test_storage.py 3423
5591     def test_expire_cutoff_date(self):
5592         basedir = "storage/LeaseCrawler/expire_cutoff_date"
5593         fileutil.make_dirs(basedir)
5594-        # setting cutoff-date to 2000 seconds ago means that any lease which
5595-        # is more than 2000s old will be expired.
5596+        # setting 'cutoff_date' to 2000 seconds ago means that any lease that
5597+        # is more than 2000 seconds old will be expired.
5598         now = time.time()
5599         then = int(now - 2000)
5600hunk ./src/allmydata/test/test_storage.py 3427
5601-        ss = InstrumentedStorageServer(basedir, "\x00" * 20,
5602-                                       expiration_enabled=True,
5603-                                       expiration_mode="cutoff-date",
5604-                                       expiration_cutoff_date=then)
5605+        expiration_policy = {
5606+            'enabled': True,
5607+            'mode': 'cutoff-date',
5608+            'cutoff_date': then,
5609+            'sharetypes': ('mutable', 'immutable'),
5610+        }
5611+        ss = InstrumentedStorageServer(basedir, "\x00" * 20, expiration_policy)
5612         # make it start sooner than usual.
5613         lc = ss.lease_checker
5614         lc.slow_start = 0
5615hunk ./src/allmydata/test/test_storage.py 3575
5616     def test_only_immutable(self):
5617         basedir = "storage/LeaseCrawler/only_immutable"
5618         fileutil.make_dirs(basedir)
5619+        # setting 'cutoff_date' to 2000 seconds ago means that any lease that
5620+        # is more than 2000 seconds old will be expired.
5621         now = time.time()
5622         then = int(now - 2000)
5623hunk ./src/allmydata/test/test_storage.py 3579
5624-        ss = StorageServer(basedir, "\x00" * 20,
5625-                           expiration_enabled=True,
5626-                           expiration_mode="cutoff-date",
5627-                           expiration_cutoff_date=then,
5628-                           expiration_sharetypes=("immutable",))
5629+        expiration_policy = {
5630+            'enabled': True,
5631+            'mode': 'cutoff-date',
5632+            'cutoff_date': then,
5633+            'sharetypes': ('immutable',),
5634+        }
5635+        ss = StorageServer(basedir, "\x00" * 20, expiration_policy)
5636         lc = ss.lease_checker
5637         lc.slow_start = 0
5638         webstatus = StorageStatus(ss)
5639hunk ./src/allmydata/test/test_storage.py 3636
5640     def test_only_mutable(self):
5641         basedir = "storage/LeaseCrawler/only_mutable"
5642         fileutil.make_dirs(basedir)
5643+        # setting 'cutoff_date' to 2000 seconds ago means that any lease that
5644+        # is more than 2000 seconds old will be expired.
5645         now = time.time()
5646         then = int(now - 2000)
5647hunk ./src/allmydata/test/test_storage.py 3640
5648-        ss = StorageServer(basedir, "\x00" * 20,
5649-                           expiration_enabled=True,
5650-                           expiration_mode="cutoff-date",
5651-                           expiration_cutoff_date=then,
5652-                           expiration_sharetypes=("mutable",))
5653+        expiration_policy = {
5654+            'enabled': True,
5655+            'mode': 'cutoff-date',
5656+            'cutoff_date': then,
5657+            'sharetypes': ('mutable',),
5658+        }
5659+        ss = StorageServer(basedir, "\x00" * 20, expiration_policy)
5660         lc = ss.lease_checker
5661         lc.slow_start = 0
5662         webstatus = StorageStatus(ss)
5663hunk ./src/allmydata/test/test_storage.py 3819
5664     def test_no_st_blocks(self):
5665         basedir = "storage/LeaseCrawler/no_st_blocks"
5666         fileutil.make_dirs(basedir)
5667-        ss = No_ST_BLOCKS_StorageServer(basedir, "\x00" * 20,
5668-                                        expiration_mode="age",
5669-                                        expiration_override_lease_duration=-1000)
5670-        # a negative expiration_time= means the "configured-"
5671+        # A negative 'override_lease_duration' means that the "configured-"
5672         # space-recovered counts will be non-zero, since all shares will have
5673hunk ./src/allmydata/test/test_storage.py 3821
5674-        # expired by then
5675+        # expired by then.
5676+        expiration_policy = {
5677+            'enabled': True,
5678+            'mode': 'age',
5679+            'override_lease_duration': -1000,
5680+            'sharetypes': ('mutable', 'immutable'),
5681+        }
5682+        ss = No_ST_BLOCKS_StorageServer(basedir, "\x00" * 20, expiration_policy)
5683 
5684         # make it start sooner than usual.
5685         lc = ss.lease_checker
5686hunk ./src/allmydata/test/test_storage.py 3877
5687         [immutable_si_0, immutable_si_1, mutable_si_2, mutable_si_3] = self.sis
5688         first = min(self.sis)
5689         first_b32 = base32.b2a(first)
5690-        fn = os.path.join(ss.sharedir, storage_index_to_dir(first), "0")
5691-        f = open(fn, "rb+")
5692+        fp = ss.backend.get_shareset(first).sharehomedir.child("0")
5693+        f = fp.open("rb+")
5694         f.seek(0)
5695         f.write("BAD MAGIC")
5696         f.close()
5697hunk ./src/allmydata/test/test_storage.py 3890
5698 
5699         # also create an empty bucket
5700         empty_si = base32.b2a("\x04"*16)
5701-        empty_bucket_dir = os.path.join(ss.sharedir,
5702-                                        storage_index_to_dir(empty_si))
5703-        fileutil.make_dirs(empty_bucket_dir)
5704+        empty_bucket_dir = ss.backend.get_shareset(empty_si).sharehomedir
5705+        fileutil.fp_make_dirs(empty_bucket_dir)
5706 
5707         ss.setServiceParent(self.s)
5708 
5709hunk ./src/allmydata/test/test_system.py 10
5710 
5711 import allmydata
5712 from allmydata import uri
5713-from allmydata.storage.mutable import MutableShareFile
5714+from allmydata.storage.backends.disk.mutable import MutableDiskShare
5715 from allmydata.storage.server import si_a2b
5716 from allmydata.immutable import offloaded, upload
5717 from allmydata.immutable.literal import LiteralFileNode
5718hunk ./src/allmydata/test/test_system.py 421
5719         return shares
5720 
5721     def _corrupt_mutable_share(self, filename, which):
5722-        msf = MutableShareFile(filename)
5723+        msf = MutableDiskShare(filename)
5724         datav = msf.readv([ (0, 1000000) ])
5725         final_share = datav[0]
5726         assert len(final_share) < 1000000 # ought to be truncated
5727hunk ./src/allmydata/test/test_upload.py 22
5728 from allmydata.util.happinessutil import servers_of_happiness, \
5729                                          shares_by_server, merge_servers
5730 from allmydata.storage_client import StorageFarmBroker
5731-from allmydata.storage.server import storage_index_to_dir
5732 
5733 MiB = 1024*1024
5734 
5735hunk ./src/allmydata/test/test_upload.py 821
5736 
5737     def _copy_share_to_server(self, share_number, server_number):
5738         ss = self.g.servers_by_number[server_number]
5739-        # Copy share i from the directory associated with the first
5740-        # storage server to the directory associated with this one.
5741-        assert self.g, "I tried to find a grid at self.g, but failed"
5742-        assert self.shares, "I tried to find shares at self.shares, but failed"
5743-        old_share_location = self.shares[share_number][2]
5744-        new_share_location = os.path.join(ss.storedir, "shares")
5745-        si = uri.from_string(self.uri).get_storage_index()
5746-        new_share_location = os.path.join(new_share_location,
5747-                                          storage_index_to_dir(si))
5748-        if not os.path.exists(new_share_location):
5749-            os.makedirs(new_share_location)
5750-        new_share_location = os.path.join(new_share_location,
5751-                                          str(share_number))
5752-        if old_share_location != new_share_location:
5753-            shutil.copy(old_share_location, new_share_location)
5754-        shares = self.find_uri_shares(self.uri)
5755-        # Make sure that the storage server has the share.
5756-        self.failUnless((share_number, ss.my_nodeid, new_share_location)
5757-                        in shares)
5758+        self.copy_share(self.shares[share_number], ss)
5759 
5760     def _setup_grid(self):
5761         """
5762hunk ./src/allmydata/test/test_upload.py 1103
5763                 self._copy_share_to_server(i, 2)
5764         d.addCallback(_copy_shares)
5765         # Remove the first server, and add a placeholder with share 0
5766-        d.addCallback(lambda ign:
5767-            self.g.remove_server(self.g.servers_by_number[0].my_nodeid))
5768+        d.addCallback(lambda ign: self.remove_server(0))
5769         d.addCallback(lambda ign:
5770             self._add_server_with_share(server_number=4, share_number=0))
5771         # Now try uploading.
5772hunk ./src/allmydata/test/test_upload.py 1134
5773         d.addCallback(lambda ign:
5774             self._add_server(server_number=4))
5775         d.addCallback(_copy_shares)
5776-        d.addCallback(lambda ign:
5777-            self.g.remove_server(self.g.servers_by_number[0].my_nodeid))
5778+        d.addCallback(lambda ign: self.remove_server(0))
5779         d.addCallback(_reset_encoding_parameters)
5780         d.addCallback(lambda client:
5781             client.upload(upload.Data("data" * 10000, convergence="")))
5782hunk ./src/allmydata/test/test_upload.py 1196
5783                 self._copy_share_to_server(i, 2)
5784         d.addCallback(_copy_shares)
5785         # Remove server 0, and add another in its place
5786-        d.addCallback(lambda ign:
5787-            self.g.remove_server(self.g.servers_by_number[0].my_nodeid))
5788+        d.addCallback(lambda ign: self.remove_server(0))
5789         d.addCallback(lambda ign:
5790             self._add_server_with_share(server_number=4, share_number=0,
5791                                         readonly=True))
5792hunk ./src/allmydata/test/test_upload.py 1237
5793             for i in xrange(1, 10):
5794                 self._copy_share_to_server(i, 2)
5795         d.addCallback(_copy_shares)
5796-        d.addCallback(lambda ign:
5797-            self.g.remove_server(self.g.servers_by_number[0].my_nodeid))
5798+        d.addCallback(lambda ign: self.remove_server(0))
5799         def _reset_encoding_parameters(ign, happy=4):
5800             client = self.g.clients[0]
5801             client.DEFAULT_ENCODING_PARAMETERS['happy'] = happy
5802hunk ./src/allmydata/test/test_upload.py 1273
5803         # remove the original server
5804         # (necessary to ensure that the Tahoe2ServerSelector will distribute
5805         #  all the shares)
5806-        def _remove_server(ign):
5807-            server = self.g.servers_by_number[0]
5808-            self.g.remove_server(server.my_nodeid)
5809-        d.addCallback(_remove_server)
5810+        d.addCallback(lambda ign: self.remove_server(0))
5811         # This should succeed; we still have 4 servers, and the
5812         # happiness of the upload is 4.
5813         d.addCallback(lambda ign:
5814hunk ./src/allmydata/test/test_upload.py 1285
5815         d.addCallback(lambda ign:
5816             self._setup_and_upload())
5817         d.addCallback(_do_server_setup)
5818-        d.addCallback(_remove_server)
5819+        d.addCallback(lambda ign: self.remove_server(0))
5820         d.addCallback(lambda ign:
5821             self.shouldFail(UploadUnhappinessError,
5822                             "test_dropped_servers_in_encoder",
5823hunk ./src/allmydata/test/test_upload.py 1307
5824             self._add_server_with_share(4, 7, readonly=True)
5825             self._add_server_with_share(5, 8, readonly=True)
5826         d.addCallback(_do_server_setup_2)
5827-        d.addCallback(_remove_server)
5828+        d.addCallback(lambda ign: self.remove_server(0))
5829         d.addCallback(lambda ign:
5830             self._do_upload_with_broken_servers(1))
5831         d.addCallback(_set_basedir)
5832hunk ./src/allmydata/test/test_upload.py 1314
5833         d.addCallback(lambda ign:
5834             self._setup_and_upload())
5835         d.addCallback(_do_server_setup_2)
5836-        d.addCallback(_remove_server)
5837+        d.addCallback(lambda ign: self.remove_server(0))
5838         d.addCallback(lambda ign:
5839             self.shouldFail(UploadUnhappinessError,
5840                             "test_dropped_servers_in_encoder",
5841hunk ./src/allmydata/test/test_upload.py 1528
5842             for i in xrange(1, 10):
5843                 self._copy_share_to_server(i, 1)
5844         d.addCallback(_copy_shares)
5845-        d.addCallback(lambda ign:
5846-            self.g.remove_server(self.g.servers_by_number[0].my_nodeid))
5847+        d.addCallback(lambda ign: self.remove_server(0))
5848         def _prepare_client(ign):
5849             client = self.g.clients[0]
5850             client.DEFAULT_ENCODING_PARAMETERS['happy'] = 4
5851hunk ./src/allmydata/test/test_upload.py 1550
5852         def _setup(ign):
5853             for i in xrange(1, 11):
5854                 self._add_server(server_number=i)
5855-            self.g.remove_server(self.g.servers_by_number[0].my_nodeid)
5856+            self.remove_server(0)
5857             c = self.g.clients[0]
5858             # We set happy to an unsatisfiable value so that we can check the
5859             # counting in the exception message. The same progress message
5860hunk ./src/allmydata/test/test_upload.py 1577
5861                 self._add_server(server_number=i)
5862             self._add_server(server_number=11, readonly=True)
5863             self._add_server(server_number=12, readonly=True)
5864-            self.g.remove_server(self.g.servers_by_number[0].my_nodeid)
5865+            self.remove_server(0)
5866             c = self.g.clients[0]
5867             c.DEFAULT_ENCODING_PARAMETERS['happy'] = 45
5868             return c
5869hunk ./src/allmydata/test/test_upload.py 1605
5870             # the first one that the selector sees.
5871             for i in xrange(10):
5872                 self._copy_share_to_server(i, 9)
5873-            # Remove server 0, and its contents
5874-            self.g.remove_server(self.g.servers_by_number[0].my_nodeid)
5875+            self.remove_server(0)
5876             # Make happiness unsatisfiable
5877             c = self.g.clients[0]
5878             c.DEFAULT_ENCODING_PARAMETERS['happy'] = 45
5879hunk ./src/allmydata/test/test_upload.py 1625
5880         def _then(ign):
5881             for i in xrange(1, 11):
5882                 self._add_server(server_number=i, readonly=True)
5883-            self.g.remove_server(self.g.servers_by_number[0].my_nodeid)
5884+            self.remove_server(0)
5885             c = self.g.clients[0]
5886             c.DEFAULT_ENCODING_PARAMETERS['k'] = 2
5887             c.DEFAULT_ENCODING_PARAMETERS['happy'] = 4
5888hunk ./src/allmydata/test/test_upload.py 1661
5889             self._add_server(server_number=4, readonly=True))
5890         d.addCallback(lambda ign:
5891             self._add_server(server_number=5, readonly=True))
5892-        d.addCallback(lambda ign:
5893-            self.g.remove_server(self.g.servers_by_number[0].my_nodeid))
5894+        d.addCallback(lambda ign: self.remove_server(0))
5895         def _reset_encoding_parameters(ign, happy=4):
5896             client = self.g.clients[0]
5897             client.DEFAULT_ENCODING_PARAMETERS['happy'] = happy
5898hunk ./src/allmydata/test/test_upload.py 1696
5899         d.addCallback(lambda ign:
5900             self._add_server(server_number=2))
5901         def _break_server_2(ign):
5902-            serverid = self.g.servers_by_number[2].my_nodeid
5903+            serverid = self.get_server(2).get_serverid()
5904             self.g.break_server(serverid)
5905         d.addCallback(_break_server_2)
5906         d.addCallback(lambda ign:
5907hunk ./src/allmydata/test/test_upload.py 1705
5908             self._add_server(server_number=4, readonly=True))
5909         d.addCallback(lambda ign:
5910             self._add_server(server_number=5, readonly=True))
5911-        d.addCallback(lambda ign:
5912-            self.g.remove_server(self.g.servers_by_number[0].my_nodeid))
5913+        d.addCallback(lambda ign: self.remove_server(0))
5914         d.addCallback(_reset_encoding_parameters)
5915         d.addCallback(lambda client:
5916             self.shouldFail(UploadUnhappinessError, "test_selection_exceptions",
5917hunk ./src/allmydata/test/test_upload.py 1816
5918             # Copy shares
5919             self._copy_share_to_server(1, 1)
5920             self._copy_share_to_server(2, 1)
5921-            # Remove server 0
5922-            self.g.remove_server(self.g.servers_by_number[0].my_nodeid)
5923+            self.remove_server(0)
5924             client = self.g.clients[0]
5925             client.DEFAULT_ENCODING_PARAMETERS['happy'] = 3
5926             return client
5927hunk ./src/allmydata/test/test_upload.py 1930
5928                                         readonly=True)
5929             self._add_server_with_share(server_number=4, share_number=3,
5930                                         readonly=True)
5931-            # Remove server 0.
5932-            self.g.remove_server(self.g.servers_by_number[0].my_nodeid)
5933+            self.remove_server(0)
5934             # Set the client appropriately
5935             c = self.g.clients[0]
5936             c.DEFAULT_ENCODING_PARAMETERS['happy'] = 4
5937hunk ./src/allmydata/test/test_util.py 9
5938 from twisted.trial import unittest
5939 from twisted.internet import defer, reactor
5940 from twisted.python.failure import Failure
5941+from twisted.python.filepath import FilePath
5942 from twisted.python import log
5943 from pycryptopp.hash.sha256 import SHA256 as _hash
5944 
5945hunk ./src/allmydata/test/test_util.py 508
5946                 os.chdir(saved_cwd)
5947 
5948     def test_disk_stats(self):
5949-        avail = fileutil.get_available_space('.', 2**14)
5950+        avail = fileutil.get_available_space(FilePath('.'), 2**14)
5951         if avail == 0:
5952             raise unittest.SkipTest("This test will spuriously fail there is no disk space left.")
5953 
5954hunk ./src/allmydata/test/test_util.py 512
5955-        disk = fileutil.get_disk_stats('.', 2**13)
5956+        disk = fileutil.get_disk_stats(FilePath('.'), 2**13)
5957         self.failUnless(disk['total'] > 0, disk['total'])
5958         self.failUnless(disk['used'] > 0, disk['used'])
5959         self.failUnless(disk['free_for_root'] > 0, disk['free_for_root'])
5960hunk ./src/allmydata/test/test_util.py 521
5961 
5962     def test_disk_stats_avail_nonnegative(self):
5963         # This test will spuriously fail if you have more than 2^128
5964-        # bytes of available space on your filesystem.
5965-        disk = fileutil.get_disk_stats('.', 2**128)
5966+        # bytes of available space on your filesystem (lucky you).
5967+        disk = fileutil.get_disk_stats(FilePath('.'), 2**128)
5968         self.failUnlessEqual(disk['avail'], 0)
5969 
5970 class PollMixinTests(unittest.TestCase):
5971hunk ./src/allmydata/test/test_web.py 12
5972 from twisted.python import failure, log
5973 from nevow import rend
5974 from allmydata import interfaces, uri, webish, dirnode
5975-from allmydata.storage.shares import get_share_file
5976 from allmydata.storage_client import StorageFarmBroker
5977 from allmydata.immutable import upload
5978 from allmydata.immutable.downloader.status import DownloadStatus
5979hunk ./src/allmydata/test/test_web.py 4111
5980             good_shares = self.find_uri_shares(self.uris["good"])
5981             self.failUnlessReallyEqual(len(good_shares), 10)
5982             sick_shares = self.find_uri_shares(self.uris["sick"])
5983-            os.unlink(sick_shares[0][2])
5984+            sick_shares[0][2].remove()
5985             dead_shares = self.find_uri_shares(self.uris["dead"])
5986             for i in range(1, 10):
5987hunk ./src/allmydata/test/test_web.py 4114
5988-                os.unlink(dead_shares[i][2])
5989+                dead_shares[i][2].remove()
5990             c_shares = self.find_uri_shares(self.uris["corrupt"])
5991             cso = CorruptShareOptions()
5992             cso.stdout = StringIO()
5993hunk ./src/allmydata/test/test_web.py 4118
5994-            cso.parseOptions([c_shares[0][2]])
5995+            cso.parseOptions([c_shares[0][2].path])
5996             corrupt_share(cso)
5997         d.addCallback(_clobber_shares)
5998 
5999hunk ./src/allmydata/test/test_web.py 4253
6000             good_shares = self.find_uri_shares(self.uris["good"])
6001             self.failUnlessReallyEqual(len(good_shares), 10)
6002             sick_shares = self.find_uri_shares(self.uris["sick"])
6003-            os.unlink(sick_shares[0][2])
6004+            sick_shares[0][2].remove()
6005             dead_shares = self.find_uri_shares(self.uris["dead"])
6006             for i in range(1, 10):
6007hunk ./src/allmydata/test/test_web.py 4256
6008-                os.unlink(dead_shares[i][2])
6009+                dead_shares[i][2].remove()
6010             c_shares = self.find_uri_shares(self.uris["corrupt"])
6011             cso = CorruptShareOptions()
6012             cso.stdout = StringIO()
6013hunk ./src/allmydata/test/test_web.py 4260
6014-            cso.parseOptions([c_shares[0][2]])
6015+            cso.parseOptions([c_shares[0][2].path])
6016             corrupt_share(cso)
6017         d.addCallback(_clobber_shares)
6018 
6019hunk ./src/allmydata/test/test_web.py 4319
6020 
6021         def _clobber_shares(ignored):
6022             sick_shares = self.find_uri_shares(self.uris["sick"])
6023-            os.unlink(sick_shares[0][2])
6024+            sick_shares[0][2].remove()
6025         d.addCallback(_clobber_shares)
6026 
6027         d.addCallback(self.CHECK, "sick", "t=check&repair=true&output=json")
6028hunk ./src/allmydata/test/test_web.py 4811
6029             good_shares = self.find_uri_shares(self.uris["good"])
6030             self.failUnlessReallyEqual(len(good_shares), 10)
6031             sick_shares = self.find_uri_shares(self.uris["sick"])
6032-            os.unlink(sick_shares[0][2])
6033+            sick_shares[0][2].remove()
6034             #dead_shares = self.find_uri_shares(self.uris["dead"])
6035             #for i in range(1, 10):
6036hunk ./src/allmydata/test/test_web.py 4814
6037-            #    os.unlink(dead_shares[i][2])
6038+            #    dead_shares[i][2].remove()
6039 
6040             #c_shares = self.find_uri_shares(self.uris["corrupt"])
6041             #cso = CorruptShareOptions()
6042hunk ./src/allmydata/test/test_web.py 4819
6043             #cso.stdout = StringIO()
6044-            #cso.parseOptions([c_shares[0][2]])
6045+            #cso.parseOptions([c_shares[0][2].path])
6046             #corrupt_share(cso)
6047         d.addCallback(_clobber_shares)
6048 
6049hunk ./src/allmydata/test/test_web.py 4870
6050         d.addErrback(self.explain_web_error)
6051         return d
6052 
6053-    def _count_leases(self, ignored, which):
6054-        u = self.uris[which]
6055-        shares = self.find_uri_shares(u)
6056-        lease_counts = []
6057-        for shnum, serverid, fn in shares:
6058-            sf = get_share_file(fn)
6059-            num_leases = len(list(sf.get_leases()))
6060-            lease_counts.append( (fn, num_leases) )
6061-        return lease_counts
6062-
6063-    def _assert_leasecount(self, lease_counts, expected):
6064+    def _assert_leasecount(self, ignored, which, expected):
6065+        lease_counts = self.count_leases(self.uris[which])
6066         for (fn, num_leases) in lease_counts:
6067             if num_leases != expected:
6068                 self.fail("expected %d leases, have %d, on %s" %
6069hunk ./src/allmydata/test/test_web.py 4903
6070                 self.fileurls[which] = "uri/" + urllib.quote(self.uris[which])
6071         d.addCallback(_compute_fileurls)
6072 
6073-        d.addCallback(self._count_leases, "one")
6074-        d.addCallback(self._assert_leasecount, 1)
6075-        d.addCallback(self._count_leases, "two")
6076-        d.addCallback(self._assert_leasecount, 1)
6077-        d.addCallback(self._count_leases, "mutable")
6078-        d.addCallback(self._assert_leasecount, 1)
6079+        d.addCallback(self._assert_leasecount, "one", 1)
6080+        d.addCallback(self._assert_leasecount, "two", 1)
6081+        d.addCallback(self._assert_leasecount, "mutable", 1)
6082 
6083         d.addCallback(self.CHECK, "one", "t=check") # no add-lease
6084         def _got_html_good(res):
6085hunk ./src/allmydata/test/test_web.py 4913
6086             self.failIf("Not Healthy" in res, res)
6087         d.addCallback(_got_html_good)
6088 
6089-        d.addCallback(self._count_leases, "one")
6090-        d.addCallback(self._assert_leasecount, 1)
6091-        d.addCallback(self._count_leases, "two")
6092-        d.addCallback(self._assert_leasecount, 1)
6093-        d.addCallback(self._count_leases, "mutable")
6094-        d.addCallback(self._assert_leasecount, 1)
6095+        d.addCallback(self._assert_leasecount, "one", 1)
6096+        d.addCallback(self._assert_leasecount, "two", 1)
6097+        d.addCallback(self._assert_leasecount, "mutable", 1)
6098 
6099         # this CHECK uses the original client, which uses the same
6100         # lease-secrets, so it will just renew the original lease
6101hunk ./src/allmydata/test/test_web.py 4922
6102         d.addCallback(self.CHECK, "one", "t=check&add-lease=true")
6103         d.addCallback(_got_html_good)
6104 
6105-        d.addCallback(self._count_leases, "one")
6106-        d.addCallback(self._assert_leasecount, 1)
6107-        d.addCallback(self._count_leases, "two")
6108-        d.addCallback(self._assert_leasecount, 1)
6109-        d.addCallback(self._count_leases, "mutable")
6110-        d.addCallback(self._assert_leasecount, 1)
6111+        d.addCallback(self._assert_leasecount, "one", 1)
6112+        d.addCallback(self._assert_leasecount, "two", 1)
6113+        d.addCallback(self._assert_leasecount, "mutable", 1)
6114 
6115         # this CHECK uses an alternate client, which adds a second lease
6116         d.addCallback(self.CHECK, "one", "t=check&add-lease=true", clientnum=1)
6117hunk ./src/allmydata/test/test_web.py 4930
6118         d.addCallback(_got_html_good)
6119 
6120-        d.addCallback(self._count_leases, "one")
6121-        d.addCallback(self._assert_leasecount, 2)
6122-        d.addCallback(self._count_leases, "two")
6123-        d.addCallback(self._assert_leasecount, 1)
6124-        d.addCallback(self._count_leases, "mutable")
6125-        d.addCallback(self._assert_leasecount, 1)
6126+        d.addCallback(self._assert_leasecount, "one", 2)
6127+        d.addCallback(self._assert_leasecount, "two", 1)
6128+        d.addCallback(self._assert_leasecount, "mutable", 1)
6129 
6130         d.addCallback(self.CHECK, "mutable", "t=check&add-lease=true")
6131         d.addCallback(_got_html_good)
6132hunk ./src/allmydata/test/test_web.py 4937
6133 
6134-        d.addCallback(self._count_leases, "one")
6135-        d.addCallback(self._assert_leasecount, 2)
6136-        d.addCallback(self._count_leases, "two")
6137-        d.addCallback(self._assert_leasecount, 1)
6138-        d.addCallback(self._count_leases, "mutable")
6139-        d.addCallback(self._assert_leasecount, 1)
6140+        d.addCallback(self._assert_leasecount, "one", 2)
6141+        d.addCallback(self._assert_leasecount, "two", 1)
6142+        d.addCallback(self._assert_leasecount, "mutable", 1)
6143 
6144         d.addCallback(self.CHECK, "mutable", "t=check&add-lease=true",
6145                       clientnum=1)
6146hunk ./src/allmydata/test/test_web.py 4945
6147         d.addCallback(_got_html_good)
6148 
6149-        d.addCallback(self._count_leases, "one")
6150-        d.addCallback(self._assert_leasecount, 2)
6151-        d.addCallback(self._count_leases, "two")
6152-        d.addCallback(self._assert_leasecount, 1)
6153-        d.addCallback(self._count_leases, "mutable")
6154-        d.addCallback(self._assert_leasecount, 2)
6155+        d.addCallback(self._assert_leasecount, "one", 2)
6156+        d.addCallback(self._assert_leasecount, "two", 1)
6157+        d.addCallback(self._assert_leasecount, "mutable", 2)
6158 
6159         d.addErrback(self.explain_web_error)
6160         return d
6161hunk ./src/allmydata/test/test_web.py 4989
6162             self.failUnlessReallyEqual(len(units), 4+1)
6163         d.addCallback(_done)
6164 
6165-        d.addCallback(self._count_leases, "root")
6166-        d.addCallback(self._assert_leasecount, 1)
6167-        d.addCallback(self._count_leases, "one")
6168-        d.addCallback(self._assert_leasecount, 1)
6169-        d.addCallback(self._count_leases, "mutable")
6170-        d.addCallback(self._assert_leasecount, 1)
6171+        d.addCallback(self._assert_leasecount, "root", 1)
6172+        d.addCallback(self._assert_leasecount, "one", 1)
6173+        d.addCallback(self._assert_leasecount, "mutable", 1)
6174 
6175         d.addCallback(self.CHECK, "root", "t=stream-deep-check&add-lease=true")
6176         d.addCallback(_done)
6177hunk ./src/allmydata/test/test_web.py 4996
6178 
6179-        d.addCallback(self._count_leases, "root")
6180-        d.addCallback(self._assert_leasecount, 1)
6181-        d.addCallback(self._count_leases, "one")
6182-        d.addCallback(self._assert_leasecount, 1)
6183-        d.addCallback(self._count_leases, "mutable")
6184-        d.addCallback(self._assert_leasecount, 1)
6185+        d.addCallback(self._assert_leasecount, "root", 1)
6186+        d.addCallback(self._assert_leasecount, "one", 1)
6187+        d.addCallback(self._assert_leasecount, "mutable", 1)
6188 
6189         d.addCallback(self.CHECK, "root", "t=stream-deep-check&add-lease=true",
6190                       clientnum=1)
6191hunk ./src/allmydata/test/test_web.py 5004
6192         d.addCallback(_done)
6193 
6194-        d.addCallback(self._count_leases, "root")
6195-        d.addCallback(self._assert_leasecount, 2)
6196-        d.addCallback(self._count_leases, "one")
6197-        d.addCallback(self._assert_leasecount, 2)
6198-        d.addCallback(self._count_leases, "mutable")
6199-        d.addCallback(self._assert_leasecount, 2)
6200+        d.addCallback(self._assert_leasecount, "root", 2)
6201+        d.addCallback(self._assert_leasecount, "one", 2)
6202+        d.addCallback(self._assert_leasecount, "mutable", 2)
6203 
6204         d.addErrback(self.explain_web_error)
6205         return d
6206hunk ./src/allmydata/uri.py 829
6207     def is_mutable(self):
6208         return False
6209 
6210+    def is_readonly(self):
6211+        return True
6212+
6213+    def get_readonly(self):
6214+        return self
6215+
6216+
6217 class DirectoryURIVerifier(_DirectoryBaseURI):
6218     implements(IVerifierURI)
6219 
6220hunk ./src/allmydata/uri.py 855
6221     def is_mutable(self):
6222         return False
6223 
6224+    def is_readonly(self):
6225+        return True
6226+
6227+    def get_readonly(self):
6228+        return self
6229+
6230 
6231 class ImmutableDirectoryURIVerifier(DirectoryURIVerifier):
6232     implements(IVerifierURI)
6233hunk ./src/allmydata/util/encodingutil.py 221
6234 def quote_path(path, quotemarks=True):
6235     return quote_output("/".join(map(to_str, path)), quotemarks=quotemarks)
6236 
6237+def quote_filepath(fp, quotemarks=True, encoding=None):
6238+    path = fp.path
6239+    if isinstance(path, str):
6240+        try:
6241+            path = path.decode(filesystem_encoding)
6242+        except UnicodeDecodeError:
6243+            return 'b"%s"' % (ESCAPABLE_8BIT.sub(_str_escape, path),)
6244+
6245+    return quote_output(path, quotemarks=quotemarks, encoding=encoding)
6246+
6247 
6248 def unicode_platform():
6249     """
6250hunk ./src/allmydata/util/fileutil.py 5
6251 Futz with files like a pro.
6252 """
6253 
6254-import sys, exceptions, os, stat, tempfile, time, binascii
6255+import errno, sys, exceptions, os, stat, tempfile, time, binascii
6256+
6257+from allmydata.util.assertutil import precondition
6258 
6259 from twisted.python import log
6260hunk ./src/allmydata/util/fileutil.py 10
6261+from twisted.python.filepath import FilePath, UnlistableError
6262 
6263 from pycryptopp.cipher.aes import AES
6264 
6265hunk ./src/allmydata/util/fileutil.py 189
6266             raise tx
6267         raise exceptions.IOError, "unknown error prevented creation of directory, or deleted the directory immediately after creation: %s" % dirname # careful not to construct an IOError with a 2-tuple, as that has a special meaning...
6268 
6269-def rm_dir(dirname):
6270+def fp_make_dirs(dirfp):
6271+    """
6272+    An idempotent version of FilePath.makedirs().  If the dir already
6273+    exists, do nothing and return without raising an exception.  If this
6274+    call creates the dir, return without raising an exception.  If there is
6275+    an error that prevents creation or if the directory gets deleted after
6276+    fp_make_dirs() creates it and before fp_make_dirs() checks that it
6277+    exists, raise an exception.
6278+    """
6279+    log.msg( "xxx 0 %s" % (dirfp,))
6280+    tx = None
6281+    try:
6282+        dirfp.makedirs()
6283+    except OSError, x:
6284+        tx = x
6285+
6286+    if not dirfp.isdir():
6287+        if tx:
6288+            raise tx
6289+        raise exceptions.IOError, "unknown error prevented creation of directory, or deleted the directory immediately after creation: %s" % dirfp # careful not to construct an IOError with a 2-tuple, as that has a special meaning...
6290+
6291+def fp_rmdir_if_empty(dirfp):
6292+    """ Remove the directory if it is empty. """
6293+    try:
6294+        os.rmdir(dirfp.path)
6295+    except OSError, e:
6296+        if e.errno != errno.ENOTEMPTY:
6297+            raise
6298+    else:
6299+        dirfp.changed()
6300+
6301+def rmtree(dirname):
6302     """
6303     A threadsafe and idempotent version of shutil.rmtree().  If the dir is
6304     already gone, do nothing and return without raising an exception.  If this
6305hunk ./src/allmydata/util/fileutil.py 239
6306             else:
6307                 remove(fullname)
6308         os.rmdir(dirname)
6309-    except Exception, le:
6310-        # Ignore "No such file or directory"
6311-        if (not isinstance(le, OSError)) or le.args[0] != 2:
6312+    except EnvironmentError, le:
6313+        # Ignore "No such file or directory", collect any other exception.
6314+        if (le.args[0] != 2 and le.args[0] != 3) or (le.args[0] != errno.ENOENT):
6315             excs.append(le)
6316hunk ./src/allmydata/util/fileutil.py 243
6317+    except Exception, le:
6318+        excs.append(le)
6319 
6320     # Okay, now we've recursively removed everything, ignoring any "No
6321     # such file or directory" errors, and collecting any other errors.
6322hunk ./src/allmydata/util/fileutil.py 256
6323             raise OSError, "Failed to remove dir for unknown reason."
6324         raise OSError, excs
6325 
6326+def fp_remove(fp):
6327+    """
6328+    An idempotent version of shutil.rmtree().  If the file/dir is already
6329+    gone, do nothing and return without raising an exception.  If this call
6330+    removes the file/dir, return without raising an exception.  If there is
6331+    an error that prevents removal, or if a file or directory at the same
6332+    path gets created again by someone else after this deletes it and before
6333+    this checks that it is gone, raise an exception.
6334+    """
6335+    try:
6336+        fp.remove()
6337+    except UnlistableError, e:
6338+        if e.originalException.errno != errno.ENOENT:
6339+            raise
6340+    except OSError, e:
6341+        if e.errno != errno.ENOENT:
6342+            raise
6343+
6344+def rm_dir(dirname):
6345+    # Renamed to be like shutil.rmtree and unlike rmdir.
6346+    return rmtree(dirname)
6347 
6348 def remove_if_possible(f):
6349     try:
6350hunk ./src/allmydata/util/fileutil.py 387
6351         import traceback
6352         traceback.print_exc()
6353 
6354-def get_disk_stats(whichdir, reserved_space=0):
6355+def get_disk_stats(whichdirfp, reserved_space=0):
6356     """Return disk statistics for the storage disk, in the form of a dict
6357     with the following fields.
6358       total:            total bytes on disk
6359hunk ./src/allmydata/util/fileutil.py 408
6360     you can pass how many bytes you would like to leave unused on this
6361     filesystem as reserved_space.
6362     """
6363+    precondition(isinstance(whichdirfp, FilePath), whichdirfp)
6364 
6365     if have_GetDiskFreeSpaceExW:
6366         # If this is a Windows system and GetDiskFreeSpaceExW is available, use it.
6367hunk ./src/allmydata/util/fileutil.py 419
6368         n_free_for_nonroot = c_ulonglong(0)
6369         n_total            = c_ulonglong(0)
6370         n_free_for_root    = c_ulonglong(0)
6371-        retval = GetDiskFreeSpaceExW(whichdir, byref(n_free_for_nonroot),
6372-                                               byref(n_total),
6373-                                               byref(n_free_for_root))
6374+        retval = GetDiskFreeSpaceExW(whichdirfp.path, byref(n_free_for_nonroot),
6375+                                                      byref(n_total),
6376+                                                      byref(n_free_for_root))
6377         if retval == 0:
6378             raise OSError("Windows error %d attempting to get disk statistics for %r"
6379hunk ./src/allmydata/util/fileutil.py 424
6380-                          % (GetLastError(), whichdir))
6381+                          % (GetLastError(), whichdirfp.path))
6382         free_for_nonroot = n_free_for_nonroot.value
6383         total            = n_total.value
6384         free_for_root    = n_free_for_root.value
6385hunk ./src/allmydata/util/fileutil.py 433
6386         # <http://docs.python.org/library/os.html#os.statvfs>
6387         # <http://opengroup.org/onlinepubs/7990989799/xsh/fstatvfs.html>
6388         # <http://opengroup.org/onlinepubs/7990989799/xsh/sysstatvfs.h.html>
6389-        s = os.statvfs(whichdir)
6390+        s = os.statvfs(whichdirfp.path)
6391 
6392         # on my mac laptop:
6393         #  statvfs(2) is a wrapper around statfs(2).
6394hunk ./src/allmydata/util/fileutil.py 460
6395              'avail': avail,
6396            }
6397 
6398-def get_available_space(whichdir, reserved_space):
6399+def get_available_space(whichdirfp, reserved_space):
6400     """Returns available space for share storage in bytes, or None if no
6401     API to get this information is available.
6402 
6403hunk ./src/allmydata/util/fileutil.py 472
6404     you can pass how many bytes you would like to leave unused on this
6405     filesystem as reserved_space.
6406     """
6407+    precondition(isinstance(whichdirfp, FilePath), whichdirfp)
6408     try:
6409hunk ./src/allmydata/util/fileutil.py 474
6410-        return get_disk_stats(whichdir, reserved_space)['avail']
6411+        return get_disk_stats(whichdirfp, reserved_space)['avail']
6412     except AttributeError:
6413         return None
6414hunk ./src/allmydata/util/fileutil.py 477
6415-    except EnvironmentError:
6416-        log.msg("OS call to get disk statistics failed")
6417+
6418+
6419+def get_used_space(fp):
6420+    if fp is None:
6421         return 0
6422hunk ./src/allmydata/util/fileutil.py 482
6423+    try:
6424+        s = os.stat(fp.path)
6425+    except EnvironmentError:
6426+        if not fp.exists():
6427+            return 0
6428+        raise
6429+    else:
6430+        # POSIX defines st_blocks (originally a BSDism):
6431+        #   <http://pubs.opengroup.org/onlinepubs/009695399/basedefs/sys/stat.h.html>
6432+        # but does not require stat() to give it a "meaningful value"
6433+        #   <http://pubs.opengroup.org/onlinepubs/009695399/functions/stat.html>
6434+        # and says:
6435+        #   "The unit for the st_blocks member of the stat structure is not defined
6436+        #    within IEEE Std 1003.1-2001. In some implementations it is 512 bytes.
6437+        #    It may differ on a file system basis. There is no correlation between
6438+        #    values of the st_blocks and st_blksize, and the f_bsize (from <sys/statvfs.h>)
6439+        #    structure members."
6440+        #
6441+        # The Linux docs define it as "the number of blocks allocated to the file,
6442+        # [in] 512-byte units." It is also defined that way on MacOS X. Python does
6443+        # not set the attribute on Windows.
6444+        #
6445+        # We consider platforms that define st_blocks but give it a wrong value, or
6446+        # measure it in a unit other than 512 bytes, to be broken. See also
6447+        # <http://bugs.python.org/issue12350>.
6448+
6449+        if hasattr(s, 'st_blocks'):
6450+            return s.st_blocks * 512
6451+        else:
6452+            return s.st_size
6453}
6454[Work-in-progress, includes fix to bug involving BucketWriter. refs #999
6455david-sarah@jacaranda.org**20110920033803
6456 Ignore-this: 64e9e019421454e4d08141d10b6e4eed
6457] {
6458hunk ./src/allmydata/client.py 9
6459 from twisted.internet import reactor, defer
6460 from twisted.application import service
6461 from twisted.application.internet import TimerService
6462+from twisted.python.filepath import FilePath
6463 from foolscap.api import Referenceable
6464 from pycryptopp.publickey import rsa
6465 
6466hunk ./src/allmydata/client.py 15
6467 import allmydata
6468 from allmydata.storage.server import StorageServer
6469+from allmydata.storage.backends.disk.disk_backend import DiskBackend
6470 from allmydata import storage_client
6471 from allmydata.immutable.upload import Uploader
6472 from allmydata.immutable.offloaded import Helper
6473hunk ./src/allmydata/client.py 213
6474             return
6475         readonly = self.get_config("storage", "readonly", False, boolean=True)
6476 
6477-        storedir = os.path.join(self.basedir, self.STOREDIR)
6478+        storedir = FilePath(self.basedir).child(self.STOREDIR)
6479 
6480         data = self.get_config("storage", "reserved_space", None)
6481         reserved = None
6482hunk ./src/allmydata/client.py 255
6483             'cutoff_date': cutoff_date,
6484             'sharetypes': tuple(sharetypes),
6485         }
6486-        ss = StorageServer(storedir, self.nodeid,
6487-                           reserved_space=reserved,
6488-                           discard_storage=discard,
6489-                           readonly_storage=readonly,
6490+
6491+        backend = DiskBackend(storedir, readonly=readonly, reserved_space=reserved,
6492+                              discard_storage=discard)
6493+        ss = StorageServer(nodeid, backend, storedir,
6494                            stats_provider=self.stats_provider,
6495                            expiration_policy=expiration_policy)
6496         self.add_service(ss)
6497hunk ./src/allmydata/interfaces.py 348
6498 
6499     def get_shares():
6500         """
6501-        Generates the IStoredShare objects held in this shareset.
6502+        Generates IStoredShare objects for all completed shares in this shareset.
6503         """
6504 
6505     def has_incoming(shnum):
6506hunk ./src/allmydata/storage/backends/base.py 69
6507         # def _create_mutable_share(self, storageserver, shnum, write_enabler):
6508         #     """create a mutable share with the given shnum and write_enabler"""
6509 
6510-        # secrets might be a triple with cancel_secret in secrets[2], but if
6511-        # so we ignore the cancel_secret.
6512         write_enabler = secrets[0]
6513         renew_secret = secrets[1]
6514hunk ./src/allmydata/storage/backends/base.py 71
6515+        cancel_secret = '\x00'*32
6516+        if len(secrets) > 2:
6517+            cancel_secret = secrets[2]
6518 
6519         si_s = self.get_storage_index_string()
6520         shares = {}
6521hunk ./src/allmydata/storage/backends/base.py 110
6522             read_data[shnum] = share.readv(read_vector)
6523 
6524         ownerid = 1 # TODO
6525-        lease_info = LeaseInfo(ownerid, renew_secret,
6526+        lease_info = LeaseInfo(ownerid, renew_secret, cancel_secret,
6527                                expiration_time, storageserver.get_serverid())
6528 
6529         if testv_is_good:
6530hunk ./src/allmydata/storage/backends/disk/disk_backend.py 34
6531     return newfp.child(sia)
6532 
6533 
6534-def get_share(fp):
6535+def get_share(storageindex, shnum, fp):
6536     f = fp.open('rb')
6537     try:
6538         prefix = f.read(32)
6539hunk ./src/allmydata/storage/backends/disk/disk_backend.py 42
6540         f.close()
6541 
6542     if prefix == MutableDiskShare.MAGIC:
6543-        return MutableDiskShare(fp)
6544+        return MutableDiskShare(storageindex, shnum, fp)
6545     else:
6546         # assume it's immutable
6547hunk ./src/allmydata/storage/backends/disk/disk_backend.py 45
6548-        return ImmutableDiskShare(fp)
6549+        return ImmutableDiskShare(storageindex, shnum, fp)
6550 
6551 
6552 class DiskBackend(Backend):
6553hunk ./src/allmydata/storage/backends/disk/disk_backend.py 174
6554                 if not NUM_RE.match(shnumstr):
6555                     continue
6556                 sharehome = self._sharehomedir.child(shnumstr)
6557-                yield self.get_share(sharehome)
6558+                yield get_share(self.get_storage_index(), int(shnumstr), sharehome)
6559         except UnlistableError:
6560             # There is no shares directory at all.
6561             pass
6562hunk ./src/allmydata/storage/backends/disk/disk_backend.py 185
6563         return self._incominghomedir.child(str(shnum)).exists()
6564 
6565     def make_bucket_writer(self, storageserver, shnum, max_space_per_bucket, lease_info, canary):
6566-        sharehome = self._sharehomedir.child(str(shnum))
6567+        finalhome = self._sharehomedir.child(str(shnum))
6568         incominghome = self._incominghomedir.child(str(shnum))
6569hunk ./src/allmydata/storage/backends/disk/disk_backend.py 187
6570-        immsh = ImmutableDiskShare(self.get_storage_index(), shnum, sharehome, incominghome,
6571-                                   max_size=max_space_per_bucket, create=True)
6572+        immsh = ImmutableDiskShare(self.get_storage_index(), shnum, incominghome, finalhome,
6573+                                   max_size=max_space_per_bucket)
6574         bw = BucketWriter(storageserver, immsh, max_space_per_bucket, lease_info, canary)
6575         if self._discard_storage:
6576             bw.throw_out_all_data = True
6577hunk ./src/allmydata/storage/backends/disk/disk_backend.py 198
6578         fileutil.fp_make_dirs(self._sharehomedir)
6579         sharehome = self._sharehomedir.child(str(shnum))
6580         serverid = storageserver.get_serverid()
6581-        return create_mutable_disk_share(sharehome, serverid, write_enabler, storageserver)
6582+        return create_mutable_disk_share(self.get_storage_index(), shnum, sharehome, serverid, write_enabler, storageserver)
6583 
6584     def _clean_up_after_unlink(self):
6585         fileutil.fp_rmdir_if_empty(self._sharehomedir)
6586hunk ./src/allmydata/storage/backends/disk/immutable.py 48
6587     LEASE_SIZE = struct.calcsize(">L32s32sL")
6588 
6589 
6590-    def __init__(self, storageindex, shnum, finalhome=None, incominghome=None, max_size=None, create=False):
6591-        """ If max_size is not None then I won't allow more than
6592-        max_size to be written to me. If create=True then max_size
6593-        must not be None. """
6594-        precondition((max_size is not None) or (not create), max_size, create)
6595+    def __init__(self, storageindex, shnum, home, finalhome=None, max_size=None):
6596+        """
6597+        If max_size is not None then I won't allow more than max_size to be written to me.
6598+        If finalhome is not None (meaning that we are creating the share) then max_size
6599+        must not be None.
6600+        """
6601+        precondition((max_size is not None) or (finalhome is None), max_size, finalhome)
6602         self._storageindex = storageindex
6603         self._max_size = max_size
6604hunk ./src/allmydata/storage/backends/disk/immutable.py 57
6605-        self._incominghome = incominghome
6606-        self._home = finalhome
6607+
6608+        # If we are creating the share, _finalhome refers to the final path and
6609+        # _home to the incoming path. Otherwise, _finalhome is None.
6610+        self._finalhome = finalhome
6611+        self._home = home
6612         self._shnum = shnum
6613hunk ./src/allmydata/storage/backends/disk/immutable.py 63
6614-        if create:
6615-            # touch the file, so later callers will see that we're working on
6616+
6617+        if self._finalhome is not None:
6618+            # Touch the file, so later callers will see that we're working on
6619             # it. Also construct the metadata.
6620hunk ./src/allmydata/storage/backends/disk/immutable.py 67
6621-            assert not finalhome.exists()
6622-            fp_make_dirs(self._incominghome.parent())
6623+            assert not self._finalhome.exists()
6624+            fp_make_dirs(self._home.parent())
6625             # The second field -- the four-byte share data length -- is no
6626             # longer used as of Tahoe v1.3.0, but we continue to write it in
6627             # there in case someone downgrades a storage server from >=
6628hunk ./src/allmydata/storage/backends/disk/immutable.py 78
6629             # the largest length that can fit into the field. That way, even
6630             # if this does happen, the old < v1.3.0 server will still allow
6631             # clients to read the first part of the share.
6632-            self._incominghome.setContent(struct.pack(">LLL", 1, min(2**32-1, max_size), 0) )
6633+            self._home.setContent(struct.pack(">LLL", 1, min(2**32-1, max_size), 0) )
6634             self._lease_offset = max_size + 0x0c
6635             self._num_leases = 0
6636         else:
6637hunk ./src/allmydata/storage/backends/disk/immutable.py 101
6638                 % (si_b2a(self._storageindex), self._shnum, quote_filepath(self._home)))
6639 
6640     def close(self):
6641-        fileutil.fp_make_dirs(self._home.parent())
6642-        self._incominghome.moveTo(self._home)
6643-        try:
6644-            # self._incominghome is like storage/shares/incoming/ab/abcde/4 .
6645-            # We try to delete the parent (.../ab/abcde) to avoid leaving
6646-            # these directories lying around forever, but the delete might
6647-            # fail if we're working on another share for the same storage
6648-            # index (like ab/abcde/5). The alternative approach would be to
6649-            # use a hierarchy of objects (PrefixHolder, BucketHolder,
6650-            # ShareWriter), each of which is responsible for a single
6651-            # directory on disk, and have them use reference counting of
6652-            # their children to know when they should do the rmdir. This
6653-            # approach is simpler, but relies on os.rmdir refusing to delete
6654-            # a non-empty directory. Do *not* use fileutil.fp_remove() here!
6655-            fileutil.fp_rmdir_if_empty(self._incominghome.parent())
6656-            # we also delete the grandparent (prefix) directory, .../ab ,
6657-            # again to avoid leaving directories lying around. This might
6658-            # fail if there is another bucket open that shares a prefix (like
6659-            # ab/abfff).
6660-            fileutil.fp_rmdir_if_empty(self._incominghome.parent().parent())
6661-            # we leave the great-grandparent (incoming/) directory in place.
6662-        except EnvironmentError:
6663-            # ignore the "can't rmdir because the directory is not empty"
6664-            # exceptions, those are normal consequences of the
6665-            # above-mentioned conditions.
6666-            pass
6667-        pass
6668+        fileutil.fp_make_dirs(self._finalhome.parent())
6669+        self._home.moveTo(self._finalhome)
6670+
6671+        # self._home is like storage/shares/incoming/ab/abcde/4 .
6672+        # We try to delete the parent (.../ab/abcde) to avoid leaving
6673+        # these directories lying around forever, but the delete might
6674+        # fail if we're working on another share for the same storage
6675+        # index (like ab/abcde/5). The alternative approach would be to
6676+        # use a hierarchy of objects (PrefixHolder, BucketHolder,
6677+        # ShareWriter), each of which is responsible for a single
6678+        # directory on disk, and have them use reference counting of
6679+        # their children to know when they should do the rmdir. This
6680+        # approach is simpler, but relies on os.rmdir (used by
6681+        # fp_rmdir_if_empty) refusing to delete a non-empty directory.
6682+        # Do *not* use fileutil.fp_remove() here!
6683+        parent = self._home.parent()
6684+        fileutil.fp_rmdir_if_empty(parent)
6685+
6686+        # we also delete the grandparent (prefix) directory, .../ab ,
6687+        # again to avoid leaving directories lying around. This might
6688+        # fail if there is another bucket open that shares a prefix (like
6689+        # ab/abfff).
6690+        fileutil.fp_rmdir_if_empty(parent.parent())
6691+
6692+        # we leave the great-grandparent (incoming/) directory in place.
6693+
6694+        # allow lease changes after closing.
6695+        self._home = self._finalhome
6696+        self._finalhome = None
6697 
6698     def get_used_space(self):
6699hunk ./src/allmydata/storage/backends/disk/immutable.py 132
6700-        return (fileutil.get_used_space(self._home) +
6701-                fileutil.get_used_space(self._incominghome))
6702+        return (fileutil.get_used_space(self._finalhome) +
6703+                fileutil.get_used_space(self._home))
6704 
6705     def get_storage_index(self):
6706         return self._storageindex
6707hunk ./src/allmydata/storage/backends/disk/immutable.py 175
6708         precondition(offset >= 0, offset)
6709         if self._max_size is not None and offset+length > self._max_size:
6710             raise DataTooLargeError(self._max_size, offset, length)
6711-        f = self._incominghome.open(mode='rb+')
6712+        f = self._home.open(mode='rb+')
6713         try:
6714             real_offset = self._data_offset+offset
6715             f.seek(real_offset)
6716hunk ./src/allmydata/storage/backends/disk/immutable.py 205
6717 
6718     # These lease operations are intended for use by disk_backend.py.
6719     # Other clients should not depend on the fact that the disk backend
6720-    # stores leases in share files.
6721+    # stores leases in share files. XXX bucket.py also relies on this.
6722 
6723     def get_leases(self):
6724         """Yields a LeaseInfo instance for all leases."""
6725hunk ./src/allmydata/storage/backends/disk/immutable.py 221
6726             f.close()
6727 
6728     def add_lease(self, lease_info):
6729-        f = self._incominghome.open(mode='rb')
6730+        f = self._home.open(mode='rb+')
6731         try:
6732             num_leases = self._read_num_leases(f)
6733hunk ./src/allmydata/storage/backends/disk/immutable.py 224
6734-        finally:
6735-            f.close()
6736-        f = self._home.open(mode='wb+')
6737-        try:
6738             self._write_lease_record(f, num_leases, lease_info)
6739             self._write_num_leases(f, num_leases+1)
6740         finally:
6741hunk ./src/allmydata/storage/backends/disk/mutable.py 440
6742         pass
6743 
6744 
6745-def create_mutable_disk_share(fp, serverid, write_enabler, parent):
6746-    ms = MutableDiskShare(fp, parent)
6747+def create_mutable_disk_share(storageindex, shnum, fp, serverid, write_enabler, parent):
6748+    ms = MutableDiskShare(storageindex, shnum, fp, parent)
6749     ms.create(serverid, write_enabler)
6750     del ms
6751hunk ./src/allmydata/storage/backends/disk/mutable.py 444
6752-    return MutableDiskShare(fp, parent)
6753+    return MutableDiskShare(storageindex, shnum, fp, parent)
6754hunk ./src/allmydata/storage/bucket.py 44
6755         start = time.time()
6756 
6757         self._share.close()
6758-        filelen = self._share.stat()
6759+        # XXX should this be self._share.get_used_space() ?
6760+        consumed_size = self._share.get_size()
6761         self._share = None
6762 
6763         self.closed = True
6764hunk ./src/allmydata/storage/bucket.py 51
6765         self._canary.dontNotifyOnDisconnect(self._disconnect_marker)
6766 
6767-        self.ss.bucket_writer_closed(self, filelen)
6768+        self.ss.bucket_writer_closed(self, consumed_size)
6769         self.ss.add_latency("close", time.time() - start)
6770         self.ss.count("close")
6771 
6772hunk ./src/allmydata/storage/server.py 182
6773                                 renew_secret, cancel_secret,
6774                                 sharenums, allocated_size,
6775                                 canary, owner_num=0):
6776-        # cancel_secret is no longer used.
6777         # owner_num is not for clients to set, but rather it should be
6778         # curried into a StorageServer instance dedicated to a particular
6779         # owner.
6780hunk ./src/allmydata/storage/server.py 195
6781         # Note that the lease should not be added until the BucketWriter
6782         # has been closed.
6783         expire_time = time.time() + 31*24*60*60
6784-        lease_info = LeaseInfo(owner_num, renew_secret,
6785+        lease_info = LeaseInfo(owner_num, renew_secret, cancel_secret,
6786                                expire_time, self._serverid)
6787 
6788         max_space_per_bucket = allocated_size
6789hunk ./src/allmydata/test/no_network.py 349
6790         return self.g.servers_by_number[i]
6791 
6792     def get_serverdir(self, i):
6793-        return self.g.servers_by_number[i].backend.storedir
6794+        return self.g.servers_by_number[i].backend._storedir
6795 
6796     def remove_server(self, i):
6797         self.g.remove_server(self.g.servers_by_number[i].get_serverid())
6798hunk ./src/allmydata/test/no_network.py 357
6799     def iterate_servers(self):
6800         for i in sorted(self.g.servers_by_number.keys()):
6801             ss = self.g.servers_by_number[i]
6802-            yield (i, ss, ss.backend.storedir)
6803+            yield (i, ss, ss.backend._storedir)
6804 
6805     def find_uri_shares(self, uri):
6806         si = tahoe_uri.from_string(uri).get_storage_index()
6807hunk ./src/allmydata/test/no_network.py 384
6808         return shares
6809 
6810     def copy_share(self, from_share, uri, to_server):
6811-        si = uri.from_string(self.uri).get_storage_index()
6812+        si = tahoe_uri.from_string(uri).get_storage_index()
6813         (i_shnum, i_serverid, i_sharefp) = from_share
6814         shares_dir = to_server.backend.get_shareset(si)._sharehomedir
6815         i_sharefp.copyTo(shares_dir.child(str(i_shnum)))
6816hunk ./src/allmydata/test/test_download.py 127
6817 
6818         return d
6819 
6820-    def _write_shares(self, uri, shares):
6821-        si = uri.from_string(uri).get_storage_index()
6822+    def _write_shares(self, fileuri, shares):
6823+        si = uri.from_string(fileuri).get_storage_index()
6824         for i in shares:
6825             shares_for_server = shares[i]
6826             for shnum in shares_for_server:
6827hunk ./src/allmydata/test/test_hung_server.py 36
6828 
6829     def _hang(self, servers, **kwargs):
6830         for ss in servers:
6831-            self.g.hang_server(ss.get_serverid(), **kwargs)
6832+            self.g.hang_server(ss.original.get_serverid(), **kwargs)
6833 
6834     def _unhang(self, servers, **kwargs):
6835         for ss in servers:
6836hunk ./src/allmydata/test/test_hung_server.py 40
6837-            self.g.unhang_server(ss.get_serverid(), **kwargs)
6838+            self.g.unhang_server(ss.original.get_serverid(), **kwargs)
6839 
6840     def _hang_shares(self, shnums, **kwargs):
6841         # hang all servers who are holding the given shares
6842hunk ./src/allmydata/test/test_hung_server.py 52
6843                     hung_serverids.add(i_serverid)
6844 
6845     def _delete_all_shares_from(self, servers):
6846-        serverids = [ss.get_serverid() for ss in servers]
6847+        serverids = [ss.original.get_serverid() for ss in servers]
6848         for (i_shnum, i_serverid, i_sharefp) in self.shares:
6849             if i_serverid in serverids:
6850                 i_sharefp.remove()
6851hunk ./src/allmydata/test/test_hung_server.py 58
6852 
6853     def _corrupt_all_shares_in(self, servers, corruptor_func):
6854-        serverids = [ss.get_serverid() for ss in servers]
6855+        serverids = [ss.original.get_serverid() for ss in servers]
6856         for (i_shnum, i_serverid, i_sharefp) in self.shares:
6857             if i_serverid in serverids:
6858                 self.corrupt_share((i_shnum, i_serverid, i_sharefp), corruptor_func)
6859hunk ./src/allmydata/test/test_hung_server.py 64
6860 
6861     def _copy_all_shares_from(self, from_servers, to_server):
6862-        serverids = [ss.get_serverid() for ss in from_servers]
6863+        serverids = [ss.original.get_serverid() for ss in from_servers]
6864         for (i_shnum, i_serverid, i_sharefp) in self.shares:
6865             if i_serverid in serverids:
6866                 self.copy_share((i_shnum, i_serverid, i_sharefp), self.uri, to_server)
6867hunk ./src/allmydata/test/test_mutable.py 2983
6868             fso = debug.FindSharesOptions()
6869             storage_index = base32.b2a(n.get_storage_index())
6870             fso.si_s = storage_index
6871-            fso.nodedirs = [unicode(os.path.dirname(os.path.abspath(storedir)))
6872+            fso.nodedirs = [unicode(storedir.parent().path)
6873                             for (i,ss,storedir)
6874                             in self.iterate_servers()]
6875             fso.stdout = StringIO()
6876hunk ./src/allmydata/test/test_upload.py 818
6877         if share_number is not None:
6878             self._copy_share_to_server(share_number, server_number)
6879 
6880-
6881     def _copy_share_to_server(self, share_number, server_number):
6882         ss = self.g.servers_by_number[server_number]
6883hunk ./src/allmydata/test/test_upload.py 820
6884-        self.copy_share(self.shares[share_number], ss)
6885+        self.copy_share(self.shares[share_number], self.uri, ss)
6886 
6887     def _setup_grid(self):
6888         """
6889}
6890[docs/backends: document the configuration options for the pluggable backends scheme. refs #999
6891david-sarah@jacaranda.org**20110920171737
6892 Ignore-this: 5947e864682a43cb04e557334cda7c19
6893] {
6894adddir ./docs/backends
6895addfile ./docs/backends/S3.rst
6896hunk ./docs/backends/S3.rst 1
6897+====================================================
6898+Storing Shares in Amazon Simple Storage Service (S3)
6899+====================================================
6900+
6901+S3 is a commercial storage service provided by Amazon, described at
6902+`<https://aws.amazon.com/s3/>`_.
6903+
6904+The Tahoe-LAFS storage server can be configured to store its shares in
6905+an S3 bucket, rather than on local filesystem. To enable this, add the
6906+following keys to the server's ``tahoe.cfg`` file:
6907+
6908+``[storage]``
6909+
6910+``backend = s3``
6911+
6912+    This turns off the local filesystem backend and enables use of S3.
6913+
6914+``s3.access_key_id = (string, required)``
6915+``s3.secret_access_key = (string, required)``
6916+
6917+    These two give the storage server permission to access your Amazon
6918+    Web Services account, allowing them to upload and download shares
6919+    from S3.
6920+
6921+``s3.bucket = (string, required)``
6922+
6923+    This controls which bucket will be used to hold shares. The Tahoe-LAFS
6924+    storage server will only modify and access objects in the configured S3
6925+    bucket.
6926+
6927+``s3.url = (URL string, optional)``
6928+
6929+    This URL tells the storage server how to access the S3 service. It
6930+    defaults to ``http://s3.amazonaws.com``, but by setting it to something
6931+    else, you may be able to use some other S3-like service if it is
6932+    sufficiently compatible.
6933+
6934+``s3.max_space = (str, optional)``
6935+
6936+    This tells the server to limit how much space can be used in the S3
6937+    bucket. Before each share is uploaded, the server will ask S3 for the
6938+    current bucket usage, and will only accept the share if it does not cause
6939+    the usage to grow above this limit.
6940+
6941+    The string contains a number, with an optional case-insensitive scale
6942+    suffix like "K" or "M" or "G", and an optional "B" or "iB" suffix. So
6943+    "100MB", "100M", "100000000B", "100000000", and "100000kb" all mean the
6944+    same thing. Likewise, "1MiB", "1024KiB", and "1048576B" all mean the same
6945+    thing.
6946+
6947+    If ``s3.max_space`` is omitted, the default behavior is to allow
6948+    unlimited usage.
6949+
6950+
6951+Once configured, the WUI "storage server" page will provide information about
6952+how much space is being used and how many shares are being stored.
6953+
6954+
6955+Issues
6956+------
6957+
6958+Objects in an S3 bucket cannot be read for free. As a result, when Tahoe-LAFS
6959+is configured to store shares in S3 rather than on local disk, some common
6960+operations may behave differently:
6961+
6962+* Lease crawling/expiration is not yet implemented. As a result, shares will
6963+  be retained forever, and the Storage Server status web page will not show
6964+  information about the number of mutable/immutable shares present.
6965+
6966+* Enabling ``s3.max_space`` causes an extra S3 usage query to be sent for
6967+  each share upload, causing the upload process to run slightly slower and
6968+  incur more S3 request charges.
6969addfile ./docs/backends/disk.rst
6970hunk ./docs/backends/disk.rst 1
6971+====================================
6972+Storing Shares on a Local Filesystem
6973+====================================
6974+
6975+The "disk" backend stores shares on the local filesystem. Versions of
6976+Tahoe-LAFS <= 1.9.0 always stored shares in this way.
6977+
6978+``[storage]``
6979+
6980+``backend = disk``
6981+
6982+    This enables use of the disk backend, and is the default.
6983+
6984+``reserved_space = (str, optional)``
6985+
6986+    If provided, this value defines how much disk space is reserved: the
6987+    storage server will not accept any share that causes the amount of free
6988+    disk space to drop below this value. (The free space is measured by a
6989+    call to statvfs(2) on Unix, or GetDiskFreeSpaceEx on Windows, and is the
6990+    space available to the user account under which the storage server runs.)
6991+
6992+    This string contains a number, with an optional case-insensitive scale
6993+    suffix like "K" or "M" or "G", and an optional "B" or "iB" suffix. So
6994+    "100MB", "100M", "100000000B", "100000000", and "100000kb" all mean the
6995+    same thing. Likewise, "1MiB", "1024KiB", and "1048576B" all mean the same
6996+    thing.
6997+
6998+    "``tahoe create-node``" generates a tahoe.cfg with
6999+    "``reserved_space=1G``", but you may wish to raise, lower, or remove the
7000+    reservation to suit your needs.
7001+
7002+``expire.enabled =``
7003+
7004+``expire.mode =``
7005+
7006+``expire.override_lease_duration =``
7007+
7008+``expire.cutoff_date =``
7009+
7010+``expire.immutable =``
7011+
7012+``expire.mutable =``
7013+
7014+    These settings control garbage collection, causing the server to
7015+    delete shares that no longer have an up-to-date lease on them. Please
7016+    see `<garbage-collection.rst>`_ for full details.
7017hunk ./docs/configuration.rst 412
7018     <http://tahoe-lafs.org/trac/tahoe-lafs/ticket/390>`_ for the current
7019     status of this bug. The default value is ``False``.
7020 
7021-``reserved_space = (str, optional)``
7022+``backend = (string, optional)``
7023 
7024hunk ./docs/configuration.rst 414
7025-    If provided, this value defines how much disk space is reserved: the
7026-    storage server will not accept any share that causes the amount of free
7027-    disk space to drop below this value. (The free space is measured by a
7028-    call to statvfs(2) on Unix, or GetDiskFreeSpaceEx on Windows, and is the
7029-    space available to the user account under which the storage server runs.)
7030+    Storage servers can store the data into different "backends". Clients
7031+    need not be aware of which backend is used by a server. The default
7032+    value is ``disk``.
7033 
7034hunk ./docs/configuration.rst 418
7035-    This string contains a number, with an optional case-insensitive scale
7036-    suffix like "K" or "M" or "G", and an optional "B" or "iB" suffix. So
7037-    "100MB", "100M", "100000000B", "100000000", and "100000kb" all mean the
7038-    same thing. Likewise, "1MiB", "1024KiB", and "1048576B" all mean the same
7039-    thing.
7040+``backend = disk``
7041 
7042hunk ./docs/configuration.rst 420
7043-    "``tahoe create-node``" generates a tahoe.cfg with
7044-    "``reserved_space=1G``", but you may wish to raise, lower, or remove the
7045-    reservation to suit your needs.
7046+    The default is to store shares on the local filesystem (in
7047+    BASEDIR/storage/shares/). For configuration details (including how to
7048+    reserve a minimum amount of free space), see `<backends/disk.rst>`_.
7049 
7050hunk ./docs/configuration.rst 424
7051-``expire.enabled =``
7052+``backend = S3``
7053 
7054hunk ./docs/configuration.rst 426
7055-``expire.mode =``
7056-
7057-``expire.override_lease_duration =``
7058-
7059-``expire.cutoff_date =``
7060-
7061-``expire.immutable =``
7062-
7063-``expire.mutable =``
7064-
7065-    These settings control garbage collection, in which the server will
7066-    delete shares that no longer have an up-to-date lease on them. Please see
7067-    `<garbage-collection.rst>`_ for full details.
7068+    The storage server can store all shares to an Amazon Simple Storage
7069+    Service (S3) bucket. For configuration details, see `<backends/S3.rst>`_.
7070 
7071 
7072 Running A Helper
7073}
7074[Fix some incorrect attribute accesses. refs #999
7075david-sarah@jacaranda.org**20110921031207
7076 Ignore-this: f1ea4c3ea191f6d4b719afaebd2b2bcd
7077] {
7078hunk ./src/allmydata/client.py 258
7079 
7080         backend = DiskBackend(storedir, readonly=readonly, reserved_space=reserved,
7081                               discard_storage=discard)
7082-        ss = StorageServer(nodeid, backend, storedir,
7083+        ss = StorageServer(self.nodeid, backend, storedir,
7084                            stats_provider=self.stats_provider,
7085                            expiration_policy=expiration_policy)
7086         self.add_service(ss)
7087hunk ./src/allmydata/interfaces.py 449
7088         Returns the storage index.
7089         """
7090 
7091+    def get_storage_index_string():
7092+        """
7093+        Returns the base32-encoded storage index.
7094+        """
7095+
7096     def get_shnum():
7097         """
7098         Returns the share number.
7099hunk ./src/allmydata/storage/backends/disk/immutable.py 138
7100     def get_storage_index(self):
7101         return self._storageindex
7102 
7103+    def get_storage_index_string(self):
7104+        return si_b2a(self._storageindex)
7105+
7106     def get_shnum(self):
7107         return self._shnum
7108 
7109hunk ./src/allmydata/storage/backends/disk/mutable.py 119
7110     def get_storage_index(self):
7111         return self._storageindex
7112 
7113+    def get_storage_index_string(self):
7114+        return si_b2a(self._storageindex)
7115+
7116     def get_shnum(self):
7117         return self._shnum
7118 
7119hunk ./src/allmydata/storage/bucket.py 86
7120     def __init__(self, ss, share):
7121         self.ss = ss
7122         self._share = share
7123-        self.storageindex = share.storageindex
7124-        self.shnum = share.shnum
7125+        self.storageindex = share.get_storage_index()
7126+        self.shnum = share.get_shnum()
7127 
7128     def __repr__(self):
7129         return "<%s %s %s>" % (self.__class__.__name__,
7130hunk ./src/allmydata/storage/expirer.py 6
7131 from twisted.python import log as twlog
7132 
7133 from allmydata.storage.crawler import ShareCrawler
7134-from allmydata.storage.common import si_b2a, UnknownMutableContainerVersionError, \
7135+from allmydata.storage.common import UnknownMutableContainerVersionError, \
7136      UnknownImmutableContainerVersionError
7137 
7138 
7139hunk ./src/allmydata/storage/expirer.py 124
7140                     struct.error):
7141                 twlog.msg("lease-checker error processing %r" % (share,))
7142                 twlog.err()
7143-                which = (si_b2a(share.storageindex), share.get_shnum())
7144+                which = (share.get_storage_index_string(), share.get_shnum())
7145                 self.state["cycle-to-date"]["corrupt-shares"].append(which)
7146                 wks = (1, 1, 1, "unknown")
7147             would_keep_shares.append(wks)
7148hunk ./src/allmydata/storage/server.py 221
7149         alreadygot = set()
7150         for share in shareset.get_shares():
7151             share.add_or_renew_lease(lease_info)
7152-            alreadygot.add(share.shnum)
7153+            alreadygot.add(share.get_shnum())
7154 
7155         for shnum in sharenums - alreadygot:
7156             if shareset.has_incoming(shnum):
7157hunk ./src/allmydata/storage/server.py 324
7158 
7159         try:
7160             shareset = self.backend.get_shareset(storageindex)
7161-            return shareset.readv(self, shares, readv)
7162+            return shareset.readv(shares, readv)
7163         finally:
7164             self.add_latency("readv", time.time() - start)
7165 
7166hunk ./src/allmydata/storage/shares.py 1
7167-#! /usr/bin/python
7168-
7169-from allmydata.storage.mutable import MutableShareFile
7170-from allmydata.storage.immutable import ShareFile
7171-
7172-def get_share_file(filename):
7173-    f = open(filename, "rb")
7174-    prefix = f.read(32)
7175-    f.close()
7176-    if prefix == MutableShareFile.MAGIC:
7177-        return MutableShareFile(filename)
7178-    # otherwise assume it's immutable
7179-    return ShareFile(filename)
7180-
7181rmfile ./src/allmydata/storage/shares.py
7182hunk ./src/allmydata/test/no_network.py 387
7183         si = tahoe_uri.from_string(uri).get_storage_index()
7184         (i_shnum, i_serverid, i_sharefp) = from_share
7185         shares_dir = to_server.backend.get_shareset(si)._sharehomedir
7186+        fileutil.fp_make_dirs(shares_dir)
7187         i_sharefp.copyTo(shares_dir.child(str(i_shnum)))
7188 
7189     def restore_all_shares(self, shares):
7190hunk ./src/allmydata/test/no_network.py 391
7191-        for share, data in shares.items():
7192-            share.home.setContent(data)
7193+        for sharepath, data in shares.items():
7194+            FilePath(sharepath).setContent(data)
7195 
7196     def delete_share(self, (shnum, serverid, sharefp)):
7197         sharefp.remove()
7198hunk ./src/allmydata/test/test_upload.py 744
7199         servertoshnums = {} # k: server, v: set(shnum)
7200 
7201         for i, c in self.g.servers_by_number.iteritems():
7202-            for (dirp, dirns, fns) in os.walk(c.sharedir):
7203+            for (dirp, dirns, fns) in os.walk(c.backend._sharedir.path):
7204                 for fn in fns:
7205                     try:
7206                         sharenum = int(fn)
7207}
7208[docs/backends/S3.rst: remove Issues section. refs #999
7209david-sarah@jacaranda.org**20110921031625
7210 Ignore-this: c83d8f52b790bc32488869e6ee1df8c2
7211] hunk ./docs/backends/S3.rst 57
7212 
7213 Once configured, the WUI "storage server" page will provide information about
7214 how much space is being used and how many shares are being stored.
7215-
7216-
7217-Issues
7218-------
7219-
7220-Objects in an S3 bucket cannot be read for free. As a result, when Tahoe-LAFS
7221-is configured to store shares in S3 rather than on local disk, some common
7222-operations may behave differently:
7223-
7224-* Lease crawling/expiration is not yet implemented. As a result, shares will
7225-  be retained forever, and the Storage Server status web page will not show
7226-  information about the number of mutable/immutable shares present.
7227-
7228-* Enabling ``s3.max_space`` causes an extra S3 usage query to be sent for
7229-  each share upload, causing the upload process to run slightly slower and
7230-  incur more S3 request charges.
7231[docs/backends/S3.rst, disk.rst: describe type of space settings as 'quantity of space', not 'str'. refs #999
7232david-sarah@jacaranda.org**20110921031705
7233 Ignore-this: a74ed8e01b0a1ab5f07a1487d7bf138
7234] {
7235hunk ./docs/backends/S3.rst 38
7236     else, you may be able to use some other S3-like service if it is
7237     sufficiently compatible.
7238 
7239-``s3.max_space = (str, optional)``
7240+``s3.max_space = (quantity of space, optional)``
7241 
7242     This tells the server to limit how much space can be used in the S3
7243     bucket. Before each share is uploaded, the server will ask S3 for the
7244hunk ./docs/backends/disk.rst 14
7245 
7246     This enables use of the disk backend, and is the default.
7247 
7248-``reserved_space = (str, optional)``
7249+``reserved_space = (quantity of space, optional)``
7250 
7251     If provided, this value defines how much disk space is reserved: the
7252     storage server will not accept any share that causes the amount of free
7253}
7254[More fixes to tests needed for pluggable backends. refs #999
7255david-sarah@jacaranda.org**20110921184649
7256 Ignore-this: 9be0d3a98e350fd4e17a07d2c00bb4ca
7257] {
7258hunk ./src/allmydata/scripts/debug.py 8
7259 from twisted.python import usage, failure
7260 from twisted.internet import defer
7261 from twisted.scripts import trial as twisted_trial
7262+from twisted.python.filepath import FilePath
7263 
7264 
7265 class DumpOptions(usage.Options):
7266hunk ./src/allmydata/scripts/debug.py 38
7267         self['filename'] = argv_to_abspath(filename)
7268 
7269 def dump_share(options):
7270-    from allmydata.storage.mutable import MutableShareFile
7271+    from allmydata.storage.backends.disk.disk_backend import get_share
7272     from allmydata.util.encodingutil import quote_output
7273 
7274     out = options.stdout
7275hunk ./src/allmydata/scripts/debug.py 46
7276     # check the version, to see if we have a mutable or immutable share
7277     print >>out, "share filename: %s" % quote_output(options['filename'])
7278 
7279-    f = open(options['filename'], "rb")
7280-    prefix = f.read(32)
7281-    f.close()
7282-    if prefix == MutableShareFile.MAGIC:
7283-        return dump_mutable_share(options)
7284-    # otherwise assume it's immutable
7285-    return dump_immutable_share(options)
7286-
7287-def dump_immutable_share(options):
7288-    from allmydata.storage.immutable import ShareFile
7289+    share = get_share("", 0, fp)
7290+    if share.sharetype == "mutable":
7291+        return dump_mutable_share(options, share)
7292+    else:
7293+        assert share.sharetype == "immutable", share.sharetype
7294+        return dump_immutable_share(options)
7295 
7296hunk ./src/allmydata/scripts/debug.py 53
7297+def dump_immutable_share(options, share):
7298     out = options.stdout
7299hunk ./src/allmydata/scripts/debug.py 55
7300-    f = ShareFile(options['filename'])
7301     if not options["leases-only"]:
7302hunk ./src/allmydata/scripts/debug.py 56
7303-        dump_immutable_chk_share(f, out, options)
7304-    dump_immutable_lease_info(f, out)
7305+        dump_immutable_chk_share(share, out, options)
7306+    dump_immutable_lease_info(share, out)
7307     print >>out
7308     return 0
7309 
7310hunk ./src/allmydata/scripts/debug.py 166
7311     return when
7312 
7313 
7314-def dump_mutable_share(options):
7315-    from allmydata.storage.mutable import MutableShareFile
7316+def dump_mutable_share(options, m):
7317     from allmydata.util import base32, idlib
7318     out = options.stdout
7319hunk ./src/allmydata/scripts/debug.py 169
7320-    m = MutableShareFile(options['filename'])
7321     f = open(options['filename'], "rb")
7322     WE, nodeid = m._read_write_enabler_and_nodeid(f)
7323     num_extra_leases = m._read_num_extra_leases(f)
7324hunk ./src/allmydata/scripts/debug.py 641
7325     /home/warner/testnet/node-1/storage/shares/44k/44kai1tui348689nrw8fjegc8c/9
7326     /home/warner/testnet/node-2/storage/shares/44k/44kai1tui348689nrw8fjegc8c/2
7327     """
7328-    from allmydata.storage.server import si_a2b, storage_index_to_dir
7329-    from allmydata.util.encodingutil import listdir_unicode
7330+    from allmydata.storage.server import si_a2b
7331+    from allmydata.storage.backends.disk_backend import si_si2dir
7332+    from allmydata.util.encodingutil import quote_filepath
7333 
7334     out = options.stdout
7335hunk ./src/allmydata/scripts/debug.py 646
7336-    sharedir = storage_index_to_dir(si_a2b(options.si_s))
7337-    for d in options.nodedirs:
7338-        d = os.path.join(d, "storage/shares", sharedir)
7339-        if os.path.exists(d):
7340-            for shnum in listdir_unicode(d):
7341-                print >>out, os.path.join(d, shnum)
7342+    si = si_a2b(options.si_s)
7343+    for nodedir in options.nodedirs:
7344+        sharedir = si_si2dir(nodedir.child("storage").child("shares"), si)
7345+        if sharedir.exists():
7346+            for sharefp in sharedir.children():
7347+                print >>out, quote_filepath(sharefp, quotemarks=False)
7348 
7349     return 0
7350 
7351hunk ./src/allmydata/scripts/debug.py 878
7352         print >>err, "Error processing %s" % quote_output(si_dir)
7353         failure.Failure().printTraceback(err)
7354 
7355+
7356 class CorruptShareOptions(usage.Options):
7357     def getSynopsis(self):
7358         return "Usage: tahoe debug corrupt-share SHARE_FILENAME"
7359hunk ./src/allmydata/scripts/debug.py 902
7360 Obviously, this command should not be used in normal operation.
7361 """
7362         return t
7363+
7364     def parseArgs(self, filename):
7365         self['filename'] = filename
7366 
7367hunk ./src/allmydata/scripts/debug.py 907
7368 def corrupt_share(options):
7369+    do_corrupt_share(options.stdout, FilePath(options['filename']), options['offset'])
7370+
7371+def do_corrupt_share(out, fp, offset="block-random"):
7372     import random
7373hunk ./src/allmydata/scripts/debug.py 911
7374-    from allmydata.storage.mutable import MutableShareFile
7375-    from allmydata.storage.immutable import ShareFile
7376+    from allmydata.storage.backends.disk.mutable import MutableDiskShare
7377+    from allmydata.storage.backends.disk.immutable import ImmutableDiskShare
7378     from allmydata.mutable.layout import unpack_header
7379     from allmydata.immutable.layout import ReadBucketProxy
7380hunk ./src/allmydata/scripts/debug.py 915
7381-    out = options.stdout
7382-    fn = options['filename']
7383-    assert options["offset"] == "block-random", "other offsets not implemented"
7384+
7385+    assert offset == "block-random", "other offsets not implemented"
7386+
7387     # first, what kind of share is it?
7388 
7389     def flip_bit(start, end):
7390hunk ./src/allmydata/scripts/debug.py 924
7391         offset = random.randrange(start, end)
7392         bit = random.randrange(0, 8)
7393         print >>out, "[%d..%d):  %d.b%d" % (start, end, offset, bit)
7394-        f = open(fn, "rb+")
7395-        f.seek(offset)
7396-        d = f.read(1)
7397-        d = chr(ord(d) ^ 0x01)
7398-        f.seek(offset)
7399-        f.write(d)
7400-        f.close()
7401+        f = fp.open("rb+")
7402+        try:
7403+            f.seek(offset)
7404+            d = f.read(1)
7405+            d = chr(ord(d) ^ 0x01)
7406+            f.seek(offset)
7407+            f.write(d)
7408+        finally:
7409+            f.close()
7410 
7411hunk ./src/allmydata/scripts/debug.py 934
7412-    f = open(fn, "rb")
7413-    prefix = f.read(32)
7414-    f.close()
7415-    if prefix == MutableShareFile.MAGIC:
7416-        # mutable
7417-        m = MutableShareFile(fn)
7418-        f = open(fn, "rb")
7419-        f.seek(m.DATA_OFFSET)
7420-        data = f.read(2000)
7421-        # make sure this slot contains an SMDF share
7422-        assert data[0] == "\x00", "non-SDMF mutable shares not supported"
7423+    f = fp.open("rb")
7424+    try:
7425+        prefix = f.read(32)
7426+    finally:
7427         f.close()
7428hunk ./src/allmydata/scripts/debug.py 939
7429+    if prefix == MutableDiskShare.MAGIC:
7430+        # mutable
7431+        m = MutableDiskShare("", 0, fp)
7432+        f = fp.open("rb")
7433+        try:
7434+            f.seek(m.DATA_OFFSET)
7435+            data = f.read(2000)
7436+            # make sure this slot contains an SMDF share
7437+            assert data[0] == "\x00", "non-SDMF mutable shares not supported"
7438+        finally:
7439+            f.close()
7440 
7441         (version, ig_seqnum, ig_roothash, ig_IV, ig_k, ig_N, ig_segsize,
7442          ig_datalen, offsets) = unpack_header(data)
7443hunk ./src/allmydata/scripts/debug.py 960
7444         flip_bit(start, end)
7445     else:
7446         # otherwise assume it's immutable
7447-        f = ShareFile(fn)
7448+        f = ImmutableDiskShare("", 0, fp)
7449         bp = ReadBucketProxy(None, None, '')
7450         offsets = bp._parse_offsets(f.read_share_data(0, 0x24))
7451         start = f._data_offset + offsets["data"]
7452hunk ./src/allmydata/storage/backends/base.py 92
7453             (testv, datav, new_length) = test_and_write_vectors[sharenum]
7454             if sharenum in shares:
7455                 if not shares[sharenum].check_testv(testv):
7456-                    self.log("testv failed: [%d]: %r" % (sharenum, testv))
7457+                    storageserver.log("testv failed: [%d]: %r" % (sharenum, testv))
7458                     testv_is_good = False
7459                     break
7460             else:
7461hunk ./src/allmydata/storage/backends/base.py 99
7462                 # compare the vectors against an empty share, in which all
7463                 # reads return empty strings
7464                 if not EmptyShare().check_testv(testv):
7465-                    self.log("testv failed (empty): [%d] %r" % (sharenum,
7466-                                                                testv))
7467+                    storageserver.log("testv failed (empty): [%d] %r" % (sharenum, testv))
7468                     testv_is_good = False
7469                     break
7470 
7471hunk ./src/allmydata/test/test_cli.py 2892
7472             # delete one, corrupt a second
7473             shares = self.find_uri_shares(self.uri)
7474             self.failUnlessReallyEqual(len(shares), 10)
7475-            os.unlink(shares[0][2])
7476-            cso = debug.CorruptShareOptions()
7477-            cso.stdout = StringIO()
7478-            cso.parseOptions([shares[1][2]])
7479+            shares[0][2].remove()
7480+            stdout = StringIO()
7481+            sharefile = shares[1][2]
7482             storage_index = uri.from_string(self.uri).get_storage_index()
7483             self._corrupt_share_line = "  server %s, SI %s, shnum %d" % \
7484                                        (base32.b2a(shares[1][1]),
7485hunk ./src/allmydata/test/test_cli.py 2900
7486                                         base32.b2a(storage_index),
7487                                         shares[1][0])
7488-            debug.corrupt_share(cso)
7489+            debug.do_corrupt_share(stdout, sharefile)
7490         d.addCallback(_clobber_shares)
7491 
7492         d.addCallback(lambda ign: self.do_cli("check", "--verify", self.uri))
7493hunk ./src/allmydata/test/test_cli.py 3017
7494         def _clobber_shares(ignored):
7495             shares = self.find_uri_shares(self.uris[u"g\u00F6\u00F6d"])
7496             self.failUnlessReallyEqual(len(shares), 10)
7497-            os.unlink(shares[0][2])
7498+            shares[0][2].remove()
7499 
7500             shares = self.find_uri_shares(self.uris["mutable"])
7501hunk ./src/allmydata/test/test_cli.py 3020
7502-            cso = debug.CorruptShareOptions()
7503-            cso.stdout = StringIO()
7504-            cso.parseOptions([shares[1][2]])
7505+            stdout = StringIO()
7506+            sharefile = shares[1][2]
7507             storage_index = uri.from_string(self.uris["mutable"]).get_storage_index()
7508             self._corrupt_share_line = " corrupt: server %s, SI %s, shnum %d" % \
7509                                        (base32.b2a(shares[1][1]),
7510hunk ./src/allmydata/test/test_cli.py 3027
7511                                         base32.b2a(storage_index),
7512                                         shares[1][0])
7513-            debug.corrupt_share(cso)
7514+            debug.do_corrupt_share(stdout, sharefile)
7515         d.addCallback(_clobber_shares)
7516 
7517         # root
7518hunk ./src/allmydata/test/test_client.py 90
7519                            "enabled = true\n" + \
7520                            "reserved_space = 1000\n")
7521         c = client.Client(basedir)
7522-        self.failUnlessEqual(c.getServiceNamed("storage").reserved_space, 1000)
7523+        self.failUnlessEqual(c.getServiceNamed("storage").backend._reserved_space, 1000)
7524 
7525     def test_reserved_2(self):
7526         basedir = "client.Basic.test_reserved_2"
7527hunk ./src/allmydata/test/test_client.py 101
7528                            "enabled = true\n" + \
7529                            "reserved_space = 10K\n")
7530         c = client.Client(basedir)
7531-        self.failUnlessEqual(c.getServiceNamed("storage").reserved_space, 10*1000)
7532+        self.failUnlessEqual(c.getServiceNamed("storage").backend._reserved_space, 10*1000)
7533 
7534     def test_reserved_3(self):
7535         basedir = "client.Basic.test_reserved_3"
7536hunk ./src/allmydata/test/test_client.py 112
7537                            "enabled = true\n" + \
7538                            "reserved_space = 5mB\n")
7539         c = client.Client(basedir)
7540-        self.failUnlessEqual(c.getServiceNamed("storage").reserved_space,
7541+        self.failUnlessEqual(c.getServiceNamed("storage").backend._reserved_space,
7542                              5*1000*1000)
7543 
7544     def test_reserved_4(self):
7545hunk ./src/allmydata/test/test_client.py 124
7546                            "enabled = true\n" + \
7547                            "reserved_space = 78Gb\n")
7548         c = client.Client(basedir)
7549-        self.failUnlessEqual(c.getServiceNamed("storage").reserved_space,
7550+        self.failUnlessEqual(c.getServiceNamed("storage").backend._reserved_space,
7551                              78*1000*1000*1000)
7552 
7553     def test_reserved_bad(self):
7554hunk ./src/allmydata/test/test_client.py 136
7555                            "enabled = true\n" + \
7556                            "reserved_space = bogus\n")
7557         c = client.Client(basedir)
7558-        self.failUnlessEqual(c.getServiceNamed("storage").reserved_space, 0)
7559+        self.failUnlessEqual(c.getServiceNamed("storage").backend._reserved_space, 0)
7560 
7561     def _permute(self, sb, key):
7562         return [ s.get_serverid() for s in sb.get_servers_for_psi(key) ]
7563hunk ./src/allmydata/test/test_crawler.py 7
7564 from twisted.trial import unittest
7565 from twisted.application import service
7566 from twisted.internet import defer
7567+from twisted.python.filepath import FilePath
7568 from foolscap.api import eventually, fireEventually
7569 
7570 from allmydata.util import fileutil, hashutil, pollmixin
7571hunk ./src/allmydata/test/test_crawler.py 13
7572 from allmydata.storage.server import StorageServer, si_b2a
7573 from allmydata.storage.crawler import ShareCrawler, TimeSliceExceeded
7574+from allmydata.storage.backends.disk.disk_backend import DiskBackend
7575 
7576 from allmydata.test.test_storage import FakeCanary
7577 from allmydata.test.common_util import StallMixin
7578hunk ./src/allmydata/test/test_crawler.py 115
7579 
7580     def test_immediate(self):
7581         self.basedir = "crawler/Basic/immediate"
7582-        fileutil.make_dirs(self.basedir)
7583         serverid = "\x00" * 20
7584hunk ./src/allmydata/test/test_crawler.py 116
7585-        ss = StorageServer(self.basedir, serverid)
7586+        fp = FilePath(self.basedir)
7587+        backend = DiskBackend(fp)
7588+        ss = StorageServer(serverid, backend, fp)
7589         ss.setServiceParent(self.s)
7590 
7591         sis = [self.write(i, ss, serverid) for i in range(10)]
7592hunk ./src/allmydata/test/test_crawler.py 122
7593-        statefile = os.path.join(self.basedir, "statefile")
7594+        statefp = fp.child("statefile")
7595 
7596hunk ./src/allmydata/test/test_crawler.py 124
7597-        c = BucketEnumeratingCrawler(ss, statefile, allowed_cpu_percentage=.1)
7598+        c = BucketEnumeratingCrawler(backend, statefp, allowed_cpu_percentage=.1)
7599         c.load_state()
7600 
7601         c.start_current_prefix(time.time())
7602hunk ./src/allmydata/test/test_crawler.py 137
7603         self.failUnlessEqual(sorted(sis), sorted(c.all_buckets))
7604 
7605         # check that a new crawler picks up on the state file properly
7606-        c2 = BucketEnumeratingCrawler(ss, statefile)
7607+        c2 = BucketEnumeratingCrawler(backend, statefp)
7608         c2.load_state()
7609 
7610         c2.start_current_prefix(time.time())
7611hunk ./src/allmydata/test/test_crawler.py 145
7612 
7613     def test_service(self):
7614         self.basedir = "crawler/Basic/service"
7615-        fileutil.make_dirs(self.basedir)
7616         serverid = "\x00" * 20
7617hunk ./src/allmydata/test/test_crawler.py 146
7618-        ss = StorageServer(self.basedir, serverid)
7619+        fp = FilePath(self.basedir)
7620+        backend = DiskBackend(fp)
7621+        ss = StorageServer(serverid, backend, fp)
7622         ss.setServiceParent(self.s)
7623 
7624         sis = [self.write(i, ss, serverid) for i in range(10)]
7625hunk ./src/allmydata/test/test_crawler.py 153
7626 
7627-        statefile = os.path.join(self.basedir, "statefile")
7628-        c = BucketEnumeratingCrawler(ss, statefile)
7629+        statefp = fp.child("statefile")
7630+        c = BucketEnumeratingCrawler(backend, statefp)
7631         c.setServiceParent(self.s)
7632 
7633         # it should be legal to call get_state() and get_progress() right
7634hunk ./src/allmydata/test/test_crawler.py 174
7635 
7636     def test_paced(self):
7637         self.basedir = "crawler/Basic/paced"
7638-        fileutil.make_dirs(self.basedir)
7639         serverid = "\x00" * 20
7640hunk ./src/allmydata/test/test_crawler.py 175
7641-        ss = StorageServer(self.basedir, serverid)
7642+        fp = FilePath(self.basedir)
7643+        backend = DiskBackend(fp)
7644+        ss = StorageServer(serverid, backend, fp)
7645         ss.setServiceParent(self.s)
7646 
7647         # put four buckets in each prefixdir
7648hunk ./src/allmydata/test/test_crawler.py 186
7649             for tail in range(4):
7650                 sis.append(self.write(i, ss, serverid, tail))
7651 
7652-        statefile = os.path.join(self.basedir, "statefile")
7653+        statefp = fp.child("statefile")
7654 
7655hunk ./src/allmydata/test/test_crawler.py 188
7656-        c = PacedCrawler(ss, statefile)
7657+        c = PacedCrawler(backend, statefp)
7658         c.load_state()
7659         try:
7660             c.start_current_prefix(time.time())
7661hunk ./src/allmydata/test/test_crawler.py 213
7662         del c
7663 
7664         # start a new crawler, it should start from the beginning
7665-        c = PacedCrawler(ss, statefile)
7666+        c = PacedCrawler(backend, statefp)
7667         c.load_state()
7668         try:
7669             c.start_current_prefix(time.time())
7670hunk ./src/allmydata/test/test_crawler.py 226
7671         c.cpu_slice = PacedCrawler.cpu_slice
7672 
7673         # a third crawler should pick up from where it left off
7674-        c2 = PacedCrawler(ss, statefile)
7675+        c2 = PacedCrawler(backend, statefp)
7676         c2.all_buckets = c.all_buckets[:]
7677         c2.load_state()
7678         c2.countdown = -1
7679hunk ./src/allmydata/test/test_crawler.py 237
7680 
7681         # now stop it at the end of a bucket (countdown=4), to exercise a
7682         # different place that checks the time
7683-        c = PacedCrawler(ss, statefile)
7684+        c = PacedCrawler(backend, statefp)
7685         c.load_state()
7686         c.countdown = 4
7687         try:
7688hunk ./src/allmydata/test/test_crawler.py 256
7689 
7690         # stop it again at the end of the bucket, check that a new checker
7691         # picks up correctly
7692-        c = PacedCrawler(ss, statefile)
7693+        c = PacedCrawler(backend, statefp)
7694         c.load_state()
7695         c.countdown = 4
7696         try:
7697hunk ./src/allmydata/test/test_crawler.py 266
7698         # that should stop at the end of one of the buckets.
7699         c.save_state()
7700 
7701-        c2 = PacedCrawler(ss, statefile)
7702+        c2 = PacedCrawler(backend, statefp)
7703         c2.all_buckets = c.all_buckets[:]
7704         c2.load_state()
7705         c2.countdown = -1
7706hunk ./src/allmydata/test/test_crawler.py 277
7707 
7708     def test_paced_service(self):
7709         self.basedir = "crawler/Basic/paced_service"
7710-        fileutil.make_dirs(self.basedir)
7711         serverid = "\x00" * 20
7712hunk ./src/allmydata/test/test_crawler.py 278
7713-        ss = StorageServer(self.basedir, serverid)
7714+        fp = FilePath(self.basedir)
7715+        backend = DiskBackend(fp)
7716+        ss = StorageServer(serverid, backend, fp)
7717         ss.setServiceParent(self.s)
7718 
7719         sis = [self.write(i, ss, serverid) for i in range(10)]
7720hunk ./src/allmydata/test/test_crawler.py 285
7721 
7722-        statefile = os.path.join(self.basedir, "statefile")
7723-        c = PacedCrawler(ss, statefile)
7724+        statefp = fp.child("statefile")
7725+        c = PacedCrawler(backend, statefp)
7726 
7727         did_check_progress = [False]
7728         def check_progress():
7729hunk ./src/allmydata/test/test_crawler.py 345
7730         # and read the stdout when it runs.
7731 
7732         self.basedir = "crawler/Basic/cpu_usage"
7733-        fileutil.make_dirs(self.basedir)
7734         serverid = "\x00" * 20
7735hunk ./src/allmydata/test/test_crawler.py 346
7736-        ss = StorageServer(self.basedir, serverid)
7737+        fp = FilePath(self.basedir)
7738+        backend = DiskBackend(fp)
7739+        ss = StorageServer(serverid, backend, fp)
7740         ss.setServiceParent(self.s)
7741 
7742         for i in range(10):
7743hunk ./src/allmydata/test/test_crawler.py 354
7744             self.write(i, ss, serverid)
7745 
7746-        statefile = os.path.join(self.basedir, "statefile")
7747-        c = ConsumingCrawler(ss, statefile)
7748+        statefp = fp.child("statefile")
7749+        c = ConsumingCrawler(backend, statefp)
7750         c.setServiceParent(self.s)
7751 
7752         # this will run as fast as it can, consuming about 50ms per call to
7753hunk ./src/allmydata/test/test_crawler.py 391
7754 
7755     def test_empty_subclass(self):
7756         self.basedir = "crawler/Basic/empty_subclass"
7757-        fileutil.make_dirs(self.basedir)
7758         serverid = "\x00" * 20
7759hunk ./src/allmydata/test/test_crawler.py 392
7760-        ss = StorageServer(self.basedir, serverid)
7761+        fp = FilePath(self.basedir)
7762+        backend = DiskBackend(fp)
7763+        ss = StorageServer(serverid, backend, fp)
7764         ss.setServiceParent(self.s)
7765 
7766         for i in range(10):
7767hunk ./src/allmydata/test/test_crawler.py 400
7768             self.write(i, ss, serverid)
7769 
7770-        statefile = os.path.join(self.basedir, "statefile")
7771-        c = ShareCrawler(ss, statefile)
7772+        statefp = fp.child("statefile")
7773+        c = ShareCrawler(backend, statefp)
7774         c.slow_start = 0
7775         c.setServiceParent(self.s)
7776 
7777hunk ./src/allmydata/test/test_crawler.py 417
7778         d.addCallback(_done)
7779         return d
7780 
7781-
7782     def test_oneshot(self):
7783         self.basedir = "crawler/Basic/oneshot"
7784hunk ./src/allmydata/test/test_crawler.py 419
7785-        fileutil.make_dirs(self.basedir)
7786         serverid = "\x00" * 20
7787hunk ./src/allmydata/test/test_crawler.py 420
7788-        ss = StorageServer(self.basedir, serverid)
7789+        fp = FilePath(self.basedir)
7790+        backend = DiskBackend(fp)
7791+        ss = StorageServer(serverid, backend, fp)
7792         ss.setServiceParent(self.s)
7793 
7794         for i in range(30):
7795hunk ./src/allmydata/test/test_crawler.py 428
7796             self.write(i, ss, serverid)
7797 
7798-        statefile = os.path.join(self.basedir, "statefile")
7799-        c = OneShotCrawler(ss, statefile)
7800+        statefp = fp.child("statefile")
7801+        c = OneShotCrawler(backend, statefp)
7802         c.setServiceParent(self.s)
7803 
7804         d = c.finished_d
7805hunk ./src/allmydata/test/test_crawler.py 447
7806             self.failUnlessEqual(s["current-cycle"], None)
7807         d.addCallback(_check)
7808         return d
7809-
7810hunk ./src/allmydata/test/test_deepcheck.py 23
7811      ShouldFailMixin
7812 from allmydata.test.common_util import StallMixin
7813 from allmydata.test.no_network import GridTestMixin
7814+from allmydata.scripts import debug
7815+
7816 
7817 timeout = 2400 # One of these took 1046.091s on Zandr's ARM box.
7818 
7819hunk ./src/allmydata/test/test_deepcheck.py 905
7820         d.addErrback(self.explain_error)
7821         return d
7822 
7823-
7824-
7825     def set_up_damaged_tree(self):
7826         # 6.4s
7827 
7828hunk ./src/allmydata/test/test_deepcheck.py 989
7829 
7830         return d
7831 
7832-    def _run_cli(self, argv):
7833-        stdout, stderr = StringIO(), StringIO()
7834-        # this can only do synchronous operations
7835-        assert argv[0] == "debug"
7836-        runner.runner(argv, run_by_human=False, stdout=stdout, stderr=stderr)
7837-        return stdout.getvalue()
7838-
7839     def _delete_some_shares(self, node):
7840         self.delete_shares_numbered(node.get_uri(), [0,1])
7841 
7842hunk ./src/allmydata/test/test_deepcheck.py 995
7843     def _corrupt_some_shares(self, node):
7844         for (shnum, serverid, sharefile) in self.find_uri_shares(node.get_uri()):
7845             if shnum in (0,1):
7846-                self._run_cli(["debug", "corrupt-share", sharefile])
7847+                debug.do_corrupt_share(StringIO(), sharefile)
7848 
7849     def _delete_most_shares(self, node):
7850         self.delete_shares_numbered(node.get_uri(), range(1,10))
7851hunk ./src/allmydata/test/test_deepcheck.py 1000
7852 
7853-
7854     def check_is_healthy(self, cr, where):
7855         try:
7856             self.failUnless(ICheckResults.providedBy(cr), (cr, type(cr), where))
7857hunk ./src/allmydata/test/test_download.py 134
7858             for shnum in shares_for_server:
7859                 share_dir = self.get_server(i).backend.get_shareset(si)._sharehomedir
7860                 fileutil.fp_make_dirs(share_dir)
7861-                share_dir.child(str(shnum)).setContent(shares[shnum])
7862+                share_dir.child(str(shnum)).setContent(shares_for_server[shnum])
7863 
7864     def load_shares(self, ignored=None):
7865         # this uses the data generated by create_shares() to populate the
7866hunk ./src/allmydata/test/test_hung_server.py 32
7867 
7868     def _break(self, servers):
7869         for ss in servers:
7870-            self.g.break_server(ss.get_serverid())
7871+            self.g.break_server(ss.original.get_serverid())
7872 
7873     def _hang(self, servers, **kwargs):
7874         for ss in servers:
7875hunk ./src/allmydata/test/test_hung_server.py 67
7876         serverids = [ss.original.get_serverid() for ss in from_servers]
7877         for (i_shnum, i_serverid, i_sharefp) in self.shares:
7878             if i_serverid in serverids:
7879-                self.copy_share((i_shnum, i_serverid, i_sharefp), self.uri, to_server)
7880+                self.copy_share((i_shnum, i_serverid, i_sharefp), self.uri, to_server.original)
7881 
7882         self.shares = self.find_uri_shares(self.uri)
7883 
7884hunk ./src/allmydata/test/test_mutable.py 3662
7885         # Now execute each assignment by writing the storage.
7886         for (share, servernum) in assignments:
7887             sharedata = base64.b64decode(self.sdmf_old_shares[share])
7888-            storage_dir = self.get_server(servernum).backend.get_shareset(si).sharehomedir
7889+            storage_dir = self.get_server(servernum).backend.get_shareset(si)._sharehomedir
7890             fileutil.fp_make_dirs(storage_dir)
7891             storage_dir.child("%d" % share).setContent(sharedata)
7892         # ...and verify that the shares are there.
7893hunk ./src/allmydata/test/test_no_network.py 10
7894 from allmydata.immutable.upload import Data
7895 from allmydata.util.consumer import download_to_data
7896 
7897+
7898 class Harness(unittest.TestCase):
7899     def setUp(self):
7900         self.s = service.MultiService()
7901hunk ./src/allmydata/test/test_storage.py 1
7902-import time, os.path, platform, stat, re, simplejson, struct, shutil
7903+import time, os.path, platform, stat, re, simplejson, struct, shutil, itertools
7904 
7905 import mock
7906 
7907hunk ./src/allmydata/test/test_storage.py 6
7908 from twisted.trial import unittest
7909-
7910 from twisted.internet import defer
7911 from twisted.application import service
7912hunk ./src/allmydata/test/test_storage.py 8
7913+from twisted.python.filepath import FilePath
7914 from foolscap.api import fireEventually
7915hunk ./src/allmydata/test/test_storage.py 10
7916-import itertools
7917+
7918 from allmydata import interfaces
7919 from allmydata.util import fileutil, hashutil, base32, pollmixin, time_format
7920 from allmydata.storage.server import StorageServer
7921hunk ./src/allmydata/test/test_storage.py 14
7922+from allmydata.storage.backends.disk.disk_backend import DiskBackend
7923 from allmydata.storage.backends.disk.mutable import MutableDiskShare
7924 from allmydata.storage.bucket import BucketWriter, BucketReader
7925 from allmydata.storage.common import DataTooLargeError, \
7926hunk ./src/allmydata/test/test_storage.py 310
7927         return self.sparent.stopService()
7928 
7929     def workdir(self, name):
7930-        basedir = os.path.join("storage", "Server", name)
7931-        return basedir
7932+        return FilePath("storage").child("Server").child(name)
7933 
7934     def create(self, name, reserved_space=0, klass=StorageServer):
7935         workdir = self.workdir(name)
7936hunk ./src/allmydata/test/test_storage.py 314
7937-        ss = klass(workdir, "\x00" * 20, reserved_space=reserved_space,
7938+        backend = DiskBackend(workdir, readonly=False, reserved_space=reserved_space)
7939+        ss = klass("\x00" * 20, backend, workdir,
7940                    stats_provider=FakeStatsProvider())
7941         ss.setServiceParent(self.sparent)
7942         return ss
7943hunk ./src/allmydata/test/test_storage.py 1386
7944 
7945     def tearDown(self):
7946         self.sparent.stopService()
7947-        shutil.rmtree(self.workdir("MDMFProxies storage test server"))
7948+        fileutil.fp_remove(self.workdir("MDMFProxies storage test server"))
7949 
7950 
7951     def write_enabler(self, we_tag):
7952hunk ./src/allmydata/test/test_storage.py 2781
7953         return self.sparent.stopService()
7954 
7955     def workdir(self, name):
7956-        basedir = os.path.join("storage", "Server", name)
7957-        return basedir
7958+        return FilePath("storage").child("Server").child(name)
7959 
7960     def create(self, name):
7961         workdir = self.workdir(name)
7962hunk ./src/allmydata/test/test_storage.py 2785
7963-        ss = StorageServer(workdir, "\x00" * 20)
7964+        backend = DiskBackend(workdir)
7965+        ss = StorageServer("\x00" * 20, backend, workdir)
7966         ss.setServiceParent(self.sparent)
7967         return ss
7968 
7969hunk ./src/allmydata/test/test_storage.py 4061
7970         }
7971 
7972         basedir = "storage/WebStatus/status_right_disk_stats"
7973-        fileutil.make_dirs(basedir)
7974-        ss = StorageServer(basedir, "\x00" * 20, reserved_space=reserved_space)
7975-        expecteddir = ss.sharedir
7976+        fp = FilePath(basedir)
7977+        backend = DiskBackend(fp, readonly=False, reserved_space=reserved_space)
7978+        ss = StorageServer("\x00" * 20, backend, fp)
7979+        expecteddir = backend._sharedir
7980         ss.setServiceParent(self.s)
7981         w = StorageStatus(ss)
7982         html = w.renderSynchronously()
7983hunk ./src/allmydata/test/test_storage.py 4084
7984 
7985     def test_readonly(self):
7986         basedir = "storage/WebStatus/readonly"
7987-        fileutil.make_dirs(basedir)
7988-        ss = StorageServer(basedir, "\x00" * 20, readonly_storage=True)
7989+        fp = FilePath(basedir)
7990+        backend = DiskBackend(fp, readonly=True)
7991+        ss = StorageServer("\x00" * 20, backend, fp)
7992         ss.setServiceParent(self.s)
7993         w = StorageStatus(ss)
7994         html = w.renderSynchronously()
7995hunk ./src/allmydata/test/test_storage.py 4096
7996 
7997     def test_reserved(self):
7998         basedir = "storage/WebStatus/reserved"
7999-        fileutil.make_dirs(basedir)
8000-        ss = StorageServer(basedir, "\x00" * 20, reserved_space=10e6)
8001-        ss.setServiceParent(self.s)
8002-        w = StorageStatus(ss)
8003-        html = w.renderSynchronously()
8004-        self.failUnlessIn("<h1>Storage Server Status</h1>", html)
8005-        s = remove_tags(html)
8006-        self.failUnlessIn("Reserved space: - 10.00 MB (10000000)", s)
8007-
8008-    def test_huge_reserved(self):
8009-        basedir = "storage/WebStatus/reserved"
8010-        fileutil.make_dirs(basedir)
8011-        ss = StorageServer(basedir, "\x00" * 20, reserved_space=10e6)
8012+        fp = FilePath(basedir)
8013+        backend = DiskBackend(fp, readonly=False, reserved_space=10e6)
8014+        ss = StorageServer("\x00" * 20, backend, fp)
8015         ss.setServiceParent(self.s)
8016         w = StorageStatus(ss)
8017         html = w.renderSynchronously()
8018hunk ./src/allmydata/test/test_upload.py 3
8019 # -*- coding: utf-8 -*-
8020 
8021-import os, shutil
8022+import os
8023 from cStringIO import StringIO
8024 from twisted.trial import unittest
8025 from twisted.python.failure import Failure
8026hunk ./src/allmydata/test/test_upload.py 14
8027 from allmydata import uri, monitor, client
8028 from allmydata.immutable import upload, encode
8029 from allmydata.interfaces import FileTooLargeError, UploadUnhappinessError
8030-from allmydata.util import log
8031+from allmydata.util import log, fileutil
8032 from allmydata.util.assertutil import precondition
8033 from allmydata.util.deferredutil import DeferredListShouldSucceed
8034 from allmydata.test.no_network import GridTestMixin
8035hunk ./src/allmydata/test/test_upload.py 972
8036                                         readonly=True))
8037         # Remove the first share from server 0.
8038         def _remove_share_0_from_server_0():
8039-            share_location = self.shares[0][2]
8040-            os.remove(share_location)
8041+            self.shares[0][2].remove()
8042         d.addCallback(lambda ign:
8043             _remove_share_0_from_server_0())
8044         # Set happy = 4 in the client.
8045hunk ./src/allmydata/test/test_upload.py 1847
8046             self._copy_share_to_server(3, 1)
8047             storedir = self.get_serverdir(0)
8048             # remove the storedir, wiping out any existing shares
8049-            shutil.rmtree(storedir)
8050+            fileutil.fp_remove(storedir)
8051             # create an empty storedir to replace the one we just removed
8052hunk ./src/allmydata/test/test_upload.py 1849
8053-            os.mkdir(storedir)
8054+            storedir.mkdir()
8055             client = self.g.clients[0]
8056             client.DEFAULT_ENCODING_PARAMETERS['happy'] = 4
8057             return client
8058hunk ./src/allmydata/test/test_upload.py 1888
8059             self._copy_share_to_server(3, 1)
8060             storedir = self.get_serverdir(0)
8061             # remove the storedir, wiping out any existing shares
8062-            shutil.rmtree(storedir)
8063+            fileutil.fp_remove(storedir)
8064             # create an empty storedir to replace the one we just removed
8065hunk ./src/allmydata/test/test_upload.py 1890
8066-            os.mkdir(storedir)
8067+            storedir.mkdir()
8068             client = self.g.clients[0]
8069             client.DEFAULT_ENCODING_PARAMETERS['happy'] = 4
8070             return client
8071hunk ./src/allmydata/test/test_web.py 4870
8072         d.addErrback(self.explain_web_error)
8073         return d
8074 
8075-    def _assert_leasecount(self, ignored, which, expected):
8076+    def _assert_leasecount(self, which, expected):
8077         lease_counts = self.count_leases(self.uris[which])
8078         for (fn, num_leases) in lease_counts:
8079             if num_leases != expected:
8080hunk ./src/allmydata/test/test_web.py 4903
8081                 self.fileurls[which] = "uri/" + urllib.quote(self.uris[which])
8082         d.addCallback(_compute_fileurls)
8083 
8084-        d.addCallback(self._assert_leasecount, "one", 1)
8085-        d.addCallback(self._assert_leasecount, "two", 1)
8086-        d.addCallback(self._assert_leasecount, "mutable", 1)
8087+        d.addCallback(lambda ign: self._assert_leasecount("one", 1))
8088+        d.addCallback(lambda ign: self._assert_leasecount("two", 1))
8089+        d.addCallback(lambda ign: self._assert_leasecount("mutable", 1))
8090 
8091         d.addCallback(self.CHECK, "one", "t=check") # no add-lease
8092         def _got_html_good(res):
8093hunk ./src/allmydata/test/test_web.py 4913
8094             self.failIf("Not Healthy" in res, res)
8095         d.addCallback(_got_html_good)
8096 
8097-        d.addCallback(self._assert_leasecount, "one", 1)
8098-        d.addCallback(self._assert_leasecount, "two", 1)
8099-        d.addCallback(self._assert_leasecount, "mutable", 1)
8100+        d.addCallback(lambda ign: self._assert_leasecount("one", 1))
8101+        d.addCallback(lambda ign: self._assert_leasecount("two", 1))
8102+        d.addCallback(lambda ign: self._assert_leasecount("mutable", 1))
8103 
8104         # this CHECK uses the original client, which uses the same
8105         # lease-secrets, so it will just renew the original lease
8106hunk ./src/allmydata/test/test_web.py 4922
8107         d.addCallback(self.CHECK, "one", "t=check&add-lease=true")
8108         d.addCallback(_got_html_good)
8109 
8110-        d.addCallback(self._assert_leasecount, "one", 1)
8111-        d.addCallback(self._assert_leasecount, "two", 1)
8112-        d.addCallback(self._assert_leasecount, "mutable", 1)
8113+        d.addCallback(lambda ign: self._assert_leasecount("one", 1))
8114+        d.addCallback(lambda ign: self._assert_leasecount("two", 1))
8115+        d.addCallback(lambda ign: self._assert_leasecount("mutable", 1))
8116 
8117         # this CHECK uses an alternate client, which adds a second lease
8118         d.addCallback(self.CHECK, "one", "t=check&add-lease=true", clientnum=1)
8119hunk ./src/allmydata/test/test_web.py 4930
8120         d.addCallback(_got_html_good)
8121 
8122-        d.addCallback(self._assert_leasecount, "one", 2)
8123-        d.addCallback(self._assert_leasecount, "two", 1)
8124-        d.addCallback(self._assert_leasecount, "mutable", 1)
8125+        d.addCallback(lambda ign: self._assert_leasecount("one", 2))
8126+        d.addCallback(lambda ign: self._assert_leasecount("two", 1))
8127+        d.addCallback(lambda ign: self._assert_leasecount("mutable", 1))
8128 
8129         d.addCallback(self.CHECK, "mutable", "t=check&add-lease=true")
8130         d.addCallback(_got_html_good)
8131hunk ./src/allmydata/test/test_web.py 4937
8132 
8133-        d.addCallback(self._assert_leasecount, "one", 2)
8134-        d.addCallback(self._assert_leasecount, "two", 1)
8135-        d.addCallback(self._assert_leasecount, "mutable", 1)
8136+        d.addCallback(lambda ign: self._assert_leasecount("one", 2))
8137+        d.addCallback(lambda ign: self._assert_leasecount("two", 1))
8138+        d.addCallback(lambda ign: self._assert_leasecount("mutable", 1))
8139 
8140         d.addCallback(self.CHECK, "mutable", "t=check&add-lease=true",
8141                       clientnum=1)
8142hunk ./src/allmydata/test/test_web.py 4945
8143         d.addCallback(_got_html_good)
8144 
8145-        d.addCallback(self._assert_leasecount, "one", 2)
8146-        d.addCallback(self._assert_leasecount, "two", 1)
8147-        d.addCallback(self._assert_leasecount, "mutable", 2)
8148+        d.addCallback(lambda ign: self._assert_leasecount("one", 2))
8149+        d.addCallback(lambda ign: self._assert_leasecount("two", 1))
8150+        d.addCallback(lambda ign: self._assert_leasecount("mutable", 2))
8151 
8152         d.addErrback(self.explain_web_error)
8153         return d
8154hunk ./src/allmydata/test/test_web.py 4989
8155             self.failUnlessReallyEqual(len(units), 4+1)
8156         d.addCallback(_done)
8157 
8158-        d.addCallback(self._assert_leasecount, "root", 1)
8159-        d.addCallback(self._assert_leasecount, "one", 1)
8160-        d.addCallback(self._assert_leasecount, "mutable", 1)
8161+        d.addCallback(lambda ign: self._assert_leasecount("root", 1))
8162+        d.addCallback(lambda ign: self._assert_leasecount("one", 1))
8163+        d.addCallback(lambda ign: self._assert_leasecount("mutable", 1))
8164 
8165         d.addCallback(self.CHECK, "root", "t=stream-deep-check&add-lease=true")
8166         d.addCallback(_done)
8167hunk ./src/allmydata/test/test_web.py 4996
8168 
8169-        d.addCallback(self._assert_leasecount, "root", 1)
8170-        d.addCallback(self._assert_leasecount, "one", 1)
8171-        d.addCallback(self._assert_leasecount, "mutable", 1)
8172+        d.addCallback(lambda ign: self._assert_leasecount("root", 1))
8173+        d.addCallback(lambda ign: self._assert_leasecount("one", 1))
8174+        d.addCallback(lambda ign: self._assert_leasecount("mutable", 1))
8175 
8176         d.addCallback(self.CHECK, "root", "t=stream-deep-check&add-lease=true",
8177                       clientnum=1)
8178hunk ./src/allmydata/test/test_web.py 5004
8179         d.addCallback(_done)
8180 
8181-        d.addCallback(self._assert_leasecount, "root", 2)
8182-        d.addCallback(self._assert_leasecount, "one", 2)
8183-        d.addCallback(self._assert_leasecount, "mutable", 2)
8184+        d.addCallback(lambda ign: self._assert_leasecount("root", 2))
8185+        d.addCallback(lambda ign: self._assert_leasecount("one", 2))
8186+        d.addCallback(lambda ign: self._assert_leasecount("mutable", 2))
8187 
8188         d.addErrback(self.explain_web_error)
8189         return d
8190}
8191
8192Context:
8193
8194[Make platform-detection code tolerate linux-3.0, patch by zooko.
8195Brian Warner <warner@lothar.com>**20110915202620
8196 Ignore-this: af63cf9177ae531984dea7a1cad03762
8197 
8198 Otherwise address-autodetection can't find ifconfig. refs #1536
8199]
8200[test_web.py: fix a bug in _count_leases that was causing us to check only the lease count of one share file, not of all share files as intended.
8201david-sarah@jacaranda.org**20110915185126
8202 Ignore-this: d96632bc48d770b9b577cda1bbd8ff94
8203]
8204[docs: insert a newline at the beginning of known_issues.rst to see if this makes it render more nicely in trac
8205zooko@zooko.com**20110914064728
8206 Ignore-this: aca15190fa22083c5d4114d3965f5d65
8207]
8208[docs: remove the coding: utf-8 declaration at the to of known_issues.rst, since the trac rendering doesn't hide it
8209zooko@zooko.com**20110914055713
8210 Ignore-this: 941ed32f83ead377171aa7a6bd198fcf
8211]
8212[docs: more cleanup of known_issues.rst -- now it passes "rst2html --verbose" without comment
8213zooko@zooko.com**20110914055419
8214 Ignore-this: 5505b3d76934bd97d0312cc59ed53879
8215]
8216[docs: more formatting improvements to known_issues.rst
8217zooko@zooko.com**20110914051639
8218 Ignore-this: 9ae9230ec9a38a312cbacaf370826691
8219]
8220[docs: reformatting of known_issues.rst
8221zooko@zooko.com**20110914050240
8222 Ignore-this: b8be0375079fb478be9d07500f9aaa87
8223]
8224[docs: fix formatting error in docs/known_issues.rst
8225zooko@zooko.com**20110914045909
8226 Ignore-this: f73fe74ad2b9e655aa0c6075acced15a
8227]
8228[merge Tahoe-LAFS v1.8.3 release announcement with trunk
8229zooko@zooko.com**20110913210544
8230 Ignore-this: 163f2c3ddacca387d7308e4b9332516e
8231]
8232[docs: release notes for Tahoe-LAFS v1.8.3
8233zooko@zooko.com**20110913165826
8234 Ignore-this: 84223604985b14733a956d2fbaeb4e9f
8235]
8236[tests: bump up the timeout in this test that fails on FreeStorm's CentOS in order to see if it is just very slow
8237zooko@zooko.com**20110913024255
8238 Ignore-this: 6a86d691e878cec583722faad06fb8e4
8239]
8240[interfaces: document that the 'fills-holes-with-zero-bytes' key should be used to detect whether a storage server has that behavior. refs #1528
8241david-sarah@jacaranda.org**20110913002843
8242 Ignore-this: 1a00a6029d40f6792af48c5578c1fd69
8243]
8244[CREDITS: more CREDITS for Kevan and David-Sarah
8245zooko@zooko.com**20110912223357
8246 Ignore-this: 4ea8f0d6f2918171d2f5359c25ad1ada
8247]
8248[merge NEWS about the mutable file bounds fixes with NEWS about work-in-progress
8249zooko@zooko.com**20110913205521
8250 Ignore-this: 4289a4225f848d6ae6860dd39bc92fa8
8251]
8252[doc: add NEWS item about fixes to potential palimpsest issues in mutable files
8253zooko@zooko.com**20110912223329
8254 Ignore-this: 9d63c95ddf95c7d5453c94a1ba4d406a
8255 ref. #1528
8256]
8257[merge the NEWS about the security fix (#1528) with the work-in-progress NEWS
8258zooko@zooko.com**20110913205153
8259 Ignore-this: 88e88a2ad140238c62010cf7c66953fc
8260]
8261[doc: add NEWS entry about the issue which allows unauthorized deletion of shares
8262zooko@zooko.com**20110912223246
8263 Ignore-this: 77e06d09103d2ef6bb51ea3e5d6e80b0
8264 ref. #1528
8265]
8266[doc: add entry in known_issues.rst about the issue which allows unauthorized deletion of shares
8267zooko@zooko.com**20110912223135
8268 Ignore-this: b26c6ea96b6c8740b93da1f602b5a4cd
8269 ref. #1528
8270]
8271[storage: more paranoid handling of bounds and palimpsests in mutable share files
8272zooko@zooko.com**20110912222655
8273 Ignore-this: a20782fa423779ee851ea086901e1507
8274 * storage server ignores requests to extend shares by sending a new_length
8275 * storage server fills exposed holes (created by sending a write vector whose offset begins after the end of the current data) with 0 to avoid "palimpsest" exposure of previous contents
8276 * storage server zeroes out lease info at the old location when moving it to a new location
8277 ref. #1528
8278]
8279[storage: test that the storage server ignores requests to extend shares by sending a new_length, and that the storage server fills exposed holes with 0 to avoid "palimpsest" exposure of previous contents
8280zooko@zooko.com**20110912222554
8281 Ignore-this: 61ebd7b11250963efdf5b1734a35271
8282 ref. #1528
8283]
8284[immutable: prevent clients from reading past the end of share data, which would allow them to learn the cancellation secret
8285zooko@zooko.com**20110912222458
8286 Ignore-this: da1ebd31433ea052087b75b2e3480c25
8287 Declare explicitly that we prevent this problem in the server's version dict.
8288 fixes #1528 (there are two patches that are each a sufficient fix to #1528 and this is one of them)
8289]
8290[storage: remove the storage server's "remote_cancel_lease" function
8291zooko@zooko.com**20110912222331
8292 Ignore-this: 1c32dee50e0981408576daffad648c50
8293 We're removing this function because it is currently unused, because it is dangerous, and because the bug described in #1528 leaks the cancellation secret, which allows anyone who knows a file's storage index to abuse this function to delete shares of that file.
8294 fixes #1528 (there are two patches that are each a sufficient fix to #1528 and this is one of them)
8295]
8296[storage: test that the storage server does *not* have a "remote_cancel_lease" function
8297zooko@zooko.com**20110912222324
8298 Ignore-this: 21c652009704652d35f34651f98dd403
8299 We're removing this function because it is currently unused, because it is dangerous, and because the bug described in #1528 leaks the cancellation secret, which allows anyone who knows a file's storage index to abuse this function to delete shares of that file.
8300 ref. #1528
8301]
8302[immutable: test whether the server allows clients to read past the end of share data, which would allow them to learn the cancellation secret
8303zooko@zooko.com**20110912221201
8304 Ignore-this: 376e47b346c713d37096531491176349
8305 Also test whether the server explicitly declares that it prevents this problem.
8306 ref #1528
8307]
8308[Retrieve._activate_enough_peers: rewrite Verify logic
8309Brian Warner <warner@lothar.com>**20110909181150
8310 Ignore-this: 9367c11e1eacbf025f75ce034030d717
8311]
8312[Retrieve: implement/test stopProducing
8313Brian Warner <warner@lothar.com>**20110909181150
8314 Ignore-this: 47b2c3df7dc69835e0a066ca12e3c178
8315]
8316[move DownloadStopped from download.common to interfaces
8317Brian Warner <warner@lothar.com>**20110909181150
8318 Ignore-this: 8572acd3bb16e50341dbed8eb1d90a50
8319]
8320[retrieve.py: remove vestigal self._validated_readers
8321Brian Warner <warner@lothar.com>**20110909181150
8322 Ignore-this: faab2ec14e314a53a2ffb714de626e2d
8323]
8324[Retrieve: rewrite flow-control: use a top-level loop() to catch all errors
8325Brian Warner <warner@lothar.com>**20110909181150
8326 Ignore-this: e162d2cd53b3d3144fc6bc757e2c7714
8327 
8328 This ought to close the potential for dropped errors and hanging downloads.
8329 Verify needs to be examined, I may have broken it, although all tests pass.
8330]
8331[Retrieve: merge _validate_active_prefixes into _add_active_peers
8332Brian Warner <warner@lothar.com>**20110909181150
8333 Ignore-this: d3ead31e17e69394ae7058eeb5beaf4c