Ticket #999: passtest_status_bad_disk_stats.darcs.patch

File passtest_status_bad_disk_stats.darcs.patch, 500.1 KB (added by zancas, at 2011-09-27T06:37:30Z)

contains changes in v12

Line 
1Wed Aug 24 17:32:17 PDT 2011  david-sarah@jacaranda.org
2  * interfaces.py: 'which -> that' grammar cleanup.
3
4Mon Sep 19 16:29:26 PDT 2011  david-sarah@jacaranda.org
5  * Pluggable backends -- new and moved files, changes to moved files. refs #999
6
7Mon Sep 19 16:32:56 PDT 2011  david-sarah@jacaranda.org
8  * Pluggable backends -- all other changes. refs #999
9
10Mon Sep 19 20:38:03 PDT 2011  david-sarah@jacaranda.org
11  * Work-in-progress, includes fix to bug involving BucketWriter. refs #999
12
13Tue Sep 20 10:17:37 PDT 2011  david-sarah@jacaranda.org
14  * docs/backends: document the configuration options for the pluggable backends scheme. refs #999
15
16Tue Sep 20 20:12:07 PDT 2011  david-sarah@jacaranda.org
17  * Fix some incorrect attribute accesses. refs #999
18
19Tue Sep 20 20:16:25 PDT 2011  david-sarah@jacaranda.org
20  * docs/backends/S3.rst: remove Issues section. refs #999
21
22Tue Sep 20 20:17:05 PDT 2011  david-sarah@jacaranda.org
23  * docs/backends/S3.rst, disk.rst: describe type of space settings as 'quantity of space', not 'str'. refs #999
24
25Wed Sep 21 11:46:49 PDT 2011  david-sarah@jacaranda.org
26  * More fixes to tests needed for pluggable backends. refs #999
27
28Wed Sep 21 15:14:21 PDT 2011  david-sarah@jacaranda.org
29  * Fix more shallow bugs, mainly FilePathification. Also, remove the max_space_per_bucket parameter from BucketWriter since it can be obtained from the _max_size attribute of the share (via a new get_allocated_size() accessor). refs #999
30
31Wed Sep 21 15:20:38 PDT 2011  david-sarah@jacaranda.org
32  * uri.py: resolve a conflict between trunk and the pluggable-backends patches. refs #999
33
34Wed Sep 21 21:54:51 PDT 2011  david-sarah@jacaranda.org
35  * Fix some more test failures. refs #999
36
37Thu Sep 22 11:30:08 PDT 2011  david-sarah@jacaranda.org
38  * Fix most of the crawler tests. refs #999
39
40Thu Sep 22 11:33:23 PDT 2011  david-sarah@jacaranda.org
41  * Reinstate the cancel_lease methods of ImmutableDiskShare and MutableDiskShare, since they are needed for lease expiry. refs #999
42
43Thu Sep 22 18:20:44 PDT 2011  david-sarah@jacaranda.org
44  * Blank line cleanups.
45
46Thu Sep 22 21:08:25 PDT 2011  david-sarah@jacaranda.org
47  * mutable/publish.py: elements should not be removed from a dictionary while it is being iterated over. refs #393
48
49Thu Sep 22 21:10:03 PDT 2011  david-sarah@jacaranda.org
50  * A few comment cleanups. refs #999
51
52Thu Sep 22 21:11:15 PDT 2011  david-sarah@jacaranda.org
53  * Move advise_corrupt_share to allmydata/storage/backends/base.py, since it will be common to the disk and S3 backends. refs #999
54
55Thu Sep 22 21:13:14 PDT 2011  david-sarah@jacaranda.org
56  * Add incomplete S3 backend. refs #999
57
58Fri Sep 23 13:37:23 PDT 2011  david-sarah@jacaranda.org
59  * interfaces.py: add fill_in_space_stats method to IStorageBackend. refs #999
60
61Fri Sep 23 13:44:25 PDT 2011  david-sarah@jacaranda.org
62  * Remove redundant si_s argument from check_write_enabler. refs #999
63
64Fri Sep 23 13:46:11 PDT 2011  david-sarah@jacaranda.org
65  * Implement readv for immutable shares. refs #999
66
67Fri Sep 23 13:49:14 PDT 2011  david-sarah@jacaranda.org
68  * The cancel secret needs to be unique, even if it isn't explicitly provided. refs #999
69
70Fri Sep 23 13:49:45 PDT 2011  david-sarah@jacaranda.org
71  * Make EmptyShare.check_testv a simple function. refs #999
72
73Fri Sep 23 13:52:19 PDT 2011  david-sarah@jacaranda.org
74  * Update the null backend to take into account interface changes. Also, it now records which shares are present, but not their contents. refs #999
75
76Fri Sep 23 13:53:45 PDT 2011  david-sarah@jacaranda.org
77  * Update the S3 backend. refs #999
78
79Fri Sep 23 13:55:10 PDT 2011  david-sarah@jacaranda.org
80  * Minor cleanup to disk backend. refs #999
81
82Mon Sep 26 23:31:56 PDT 2011  wilcoxjg@gmail.com
83  * disk/disk_backends.py:  Modify get_available_space to handle EnvironmentErrors with a return value of 0, indicating that there's an error in the OS call.
84
85New patches:
86
87[interfaces.py: 'which -> that' grammar cleanup.
88david-sarah@jacaranda.org**20110825003217
89 Ignore-this: a3e15f3676de1b346ad78aabdfb8cac6
90] {
91hunk ./src/allmydata/interfaces.py 38
92     the StubClient. This object doesn't actually offer any services, but the
93     announcement helps the Introducer keep track of which clients are
94     subscribed (so the grid admin can keep track of things like the size of
95-    the grid and the client versions in use. This is the (empty)
96+    the grid and the client versions in use). This is the (empty)
97     RemoteInterface for the StubClient."""
98 
99 class RIBucketWriter(RemoteInterface):
100hunk ./src/allmydata/interfaces.py 276
101         (binary) storage index string, and 'shnum' is the integer share
102         number. 'reason' is a human-readable explanation of the problem,
103         probably including some expected hash values and the computed ones
104-        which did not match. Corruption advisories for mutable shares should
105+        that did not match. Corruption advisories for mutable shares should
106         include a hash of the public key (the same value that appears in the
107         mutable-file verify-cap), since the current share format does not
108         store that on disk.
109hunk ./src/allmydata/interfaces.py 413
110           remote_host: the IAddress, if connected, otherwise None
111 
112         This method is intended for monitoring interfaces, such as a web page
113-        which describes connecting and connected peers.
114+        that describes connecting and connected peers.
115         """
116 
117     def get_all_peerids():
118hunk ./src/allmydata/interfaces.py 515
119 
120     # TODO: rename to get_read_cap()
121     def get_readonly():
122-        """Return another IURI instance, which represents a read-only form of
123+        """Return another IURI instance that represents a read-only form of
124         this one. If is_readonly() is True, this returns self."""
125 
126     def get_verify_cap():
127hunk ./src/allmydata/interfaces.py 542
128         passing into init_from_string."""
129 
130 class IDirnodeURI(Interface):
131-    """I am a URI which represents a dirnode."""
132+    """I am a URI that represents a dirnode."""
133 
134 class IFileURI(Interface):
135hunk ./src/allmydata/interfaces.py 545
136-    """I am a URI which represents a filenode."""
137+    """I am a URI that represents a filenode."""
138     def get_size():
139         """Return the length (in bytes) of the file that I represent."""
140 
141hunk ./src/allmydata/interfaces.py 553
142     pass
143 
144 class IMutableFileURI(Interface):
145-    """I am a URI which represents a mutable filenode."""
146+    """I am a URI that represents a mutable filenode."""
147     def get_extension_params():
148         """Return the extension parameters in the URI"""
149 
150hunk ./src/allmydata/interfaces.py 856
151         """
152 
153 class IFileNode(IFilesystemNode):
154-    """I am a node which represents a file: a sequence of bytes. I am not a
155+    """I am a node that represents a file: a sequence of bytes. I am not a
156     container, like IDirectoryNode."""
157     def get_best_readable_version():
158         """Return a Deferred that fires with an IReadable for the 'best'
159hunk ./src/allmydata/interfaces.py 905
160     multiple versions of a file present in the grid, some of which might be
161     unrecoverable (i.e. have fewer than 'k' shares). These versions are
162     loosely ordered: each has a sequence number and a hash, and any version
163-    with seqnum=N was uploaded by a node which has seen at least one version
164+    with seqnum=N was uploaded by a node that has seen at least one version
165     with seqnum=N-1.
166 
167     The 'servermap' (an instance of IMutableFileServerMap) is used to
168hunk ./src/allmydata/interfaces.py 1014
169         as a guide to where the shares are located.
170 
171         I return a Deferred that fires with the requested contents, or
172-        errbacks with UnrecoverableFileError. Note that a servermap which was
173+        errbacks with UnrecoverableFileError. Note that a servermap that was
174         updated with MODE_ANYTHING or MODE_READ may not know about shares for
175         all versions (those modes stop querying servers as soon as they can
176         fulfil their goals), so you may want to use MODE_CHECK (which checks
177hunk ./src/allmydata/interfaces.py 1073
178     """Upload was unable to satisfy 'servers_of_happiness'"""
179 
180 class UnableToFetchCriticalDownloadDataError(Exception):
181-    """I was unable to fetch some piece of critical data which is supposed to
182+    """I was unable to fetch some piece of critical data that is supposed to
183     be identically present in all shares."""
184 
185 class NoServersError(Exception):
186hunk ./src/allmydata/interfaces.py 1085
187     exists, and overwrite= was set to False."""
188 
189 class NoSuchChildError(Exception):
190-    """A directory node was asked to fetch a child which does not exist."""
191+    """A directory node was asked to fetch a child that does not exist."""
192 
193 class ChildOfWrongTypeError(Exception):
194     """An operation was attempted on a child of the wrong type (file or directory)."""
195hunk ./src/allmydata/interfaces.py 1403
196         if you initially thought you were going to use 10 peers, started
197         encoding, and then two of the peers dropped out: you could use
198         desired_share_ids= to skip the work (both memory and CPU) of
199-        producing shares for the peers which are no longer available.
200+        producing shares for the peers that are no longer available.
201 
202         """
203 
204hunk ./src/allmydata/interfaces.py 1478
205         if you initially thought you were going to use 10 peers, started
206         encoding, and then two of the peers dropped out: you could use
207         desired_share_ids= to skip the work (both memory and CPU) of
208-        producing shares for the peers which are no longer available.
209+        producing shares for the peers that are no longer available.
210 
211         For each call, encode() will return a Deferred that fires with two
212         lists, one containing shares and the other containing the shareids.
213hunk ./src/allmydata/interfaces.py 1535
214         required to be of the same length.  The i'th element of their_shareids
215         is required to be the shareid of the i'th buffer in some_shares.
216 
217-        This returns a Deferred which fires with a sequence of buffers. This
218+        This returns a Deferred that fires with a sequence of buffers. This
219         sequence will contain all of the segments of the original data, in
220         order. The sum of the lengths of all of the buffers will be the
221         'data_size' value passed into the original ICodecEncode.set_params()
222hunk ./src/allmydata/interfaces.py 1582
223         Encoding parameters can be set in three ways. 1: The Encoder class
224         provides defaults (3/7/10). 2: the Encoder can be constructed with
225         an 'options' dictionary, in which the
226-        needed_and_happy_and_total_shares' key can be a (k,d,n) tuple. 3:
227+        'needed_and_happy_and_total_shares' key can be a (k,d,n) tuple. 3:
228         set_params((k,d,n)) can be called.
229 
230         If you intend to use set_params(), you must call it before
231hunk ./src/allmydata/interfaces.py 1780
232         produced, so that the segment hashes can be generated with only a
233         single pass.
234 
235-        This returns a Deferred which fires with a sequence of hashes, using:
236+        This returns a Deferred that fires with a sequence of hashes, using:
237 
238          tuple(segment_hashes[first:last])
239 
240hunk ./src/allmydata/interfaces.py 1796
241     def get_plaintext_hash():
242         """OBSOLETE; Get the hash of the whole plaintext.
243 
244-        This returns a Deferred which fires with a tagged SHA-256 hash of the
245+        This returns a Deferred that fires with a tagged SHA-256 hash of the
246         whole plaintext, obtained from hashutil.plaintext_hash(data).
247         """
248 
249hunk ./src/allmydata/interfaces.py 1856
250         be used to encrypt the data. The key will also be hashed to derive
251         the StorageIndex.
252 
253-        Uploadables which want to achieve convergence should hash their file
254+        Uploadables that want to achieve convergence should hash their file
255         contents and the serialized_encoding_parameters to form the key
256         (which of course requires a full pass over the data). Uploadables can
257         use the upload.ConvergentUploadMixin class to achieve this
258hunk ./src/allmydata/interfaces.py 1862
259         automatically.
260 
261-        Uploadables which do not care about convergence (or do not wish to
262+        Uploadables that do not care about convergence (or do not wish to
263         make multiple passes over the data) can simply return a
264         strongly-random 16 byte string.
265 
266hunk ./src/allmydata/interfaces.py 1872
267 
268     def read(length):
269         """Return a Deferred that fires with a list of strings (perhaps with
270-        only a single element) which, when concatenated together, contain the
271+        only a single element) that, when concatenated together, contain the
272         next 'length' bytes of data. If EOF is near, this may provide fewer
273         than 'length' bytes. The total number of bytes provided by read()
274         before it signals EOF must equal the size provided by get_size().
275hunk ./src/allmydata/interfaces.py 1919
276 
277     def read(length):
278         """
279-        Returns a list of strings which, when concatenated, are the next
280+        Returns a list of strings that, when concatenated, are the next
281         length bytes of the file, or fewer if there are fewer bytes
282         between the current location and the end of the file.
283         """
284hunk ./src/allmydata/interfaces.py 1932
285 
286 class IUploadResults(Interface):
287     """I am returned by upload() methods. I contain a number of public
288-    attributes which can be read to determine the results of the upload. Some
289+    attributes that can be read to determine the results of the upload. Some
290     of these are functional, some are timing information. All of these may be
291     None.
292 
293hunk ./src/allmydata/interfaces.py 1965
294 
295 class IDownloadResults(Interface):
296     """I am created internally by download() methods. I contain a number of
297-    public attributes which contain details about the download process.::
298+    public attributes that contain details about the download process.::
299 
300      .file_size : the size of the file, in bytes
301      .servers_used : set of server peerids that were used during download
302hunk ./src/allmydata/interfaces.py 1991
303 class IUploader(Interface):
304     def upload(uploadable):
305         """Upload the file. 'uploadable' must impement IUploadable. This
306-        returns a Deferred which fires with an IUploadResults instance, from
307+        returns a Deferred that fires with an IUploadResults instance, from
308         which the URI of the file can be obtained as results.uri ."""
309 
310     def upload_ssk(write_capability, new_version, uploadable):
311hunk ./src/allmydata/interfaces.py 2041
312         kind of lease that is obtained (which account number to claim, etc).
313 
314         TODO: any problems seen during checking will be reported to the
315-        health-manager.furl, a centralized object which is responsible for
316+        health-manager.furl, a centralized object that is responsible for
317         figuring out why files are unhealthy so corrective action can be
318         taken.
319         """
320hunk ./src/allmydata/interfaces.py 2056
321         will be put in the check-and-repair results. The Deferred will not
322         fire until the repair is complete.
323 
324-        This returns a Deferred which fires with an instance of
325+        This returns a Deferred that fires with an instance of
326         ICheckAndRepairResults."""
327 
328 class IDeepCheckable(Interface):
329hunk ./src/allmydata/interfaces.py 2141
330                               that was found to be corrupt. Each share
331                               locator is a list of (serverid, storage_index,
332                               sharenum).
333-         count-incompatible-shares: the number of shares which are of a share
334+         count-incompatible-shares: the number of shares that are of a share
335                                     format unknown to this checker
336          list-incompatible-shares: a list of 'share locators', one for each
337                                    share that was found to be of an unknown
338hunk ./src/allmydata/interfaces.py 2148
339                                    format. Each share locator is a list of
340                                    (serverid, storage_index, sharenum).
341          servers-responding: list of (binary) storage server identifiers,
342-                             one for each server which responded to the share
343+                             one for each server that responded to the share
344                              query (even if they said they didn't have
345                              shares, and even if they said they did have
346                              shares but then didn't send them when asked, or
347hunk ./src/allmydata/interfaces.py 2345
348         will use the data in the checker results to guide the repair process,
349         such as which servers provided bad data and should therefore be
350         avoided. The ICheckResults object is inside the
351-        ICheckAndRepairResults object, which is returned by the
352+        ICheckAndRepairResults object that is returned by the
353         ICheckable.check() method::
354 
355          d = filenode.check(repair=False)
356hunk ./src/allmydata/interfaces.py 2436
357         methods to create new objects. I return synchronously."""
358 
359     def create_mutable_file(contents=None, keysize=None):
360-        """I create a new mutable file, and return a Deferred which will fire
361+        """I create a new mutable file, and return a Deferred that will fire
362         with the IMutableFileNode instance when it is ready. If contents= is
363         provided (a bytestring), it will be used as the initial contents of
364         the new file, otherwise the file will contain zero bytes. keysize= is
365hunk ./src/allmydata/interfaces.py 2444
366         usual."""
367 
368     def create_new_mutable_directory(initial_children={}):
369-        """I create a new mutable directory, and return a Deferred which will
370+        """I create a new mutable directory, and return a Deferred that will
371         fire with the IDirectoryNode instance when it is ready. If
372         initial_children= is provided (a dict mapping unicode child name to
373         (childnode, metadata_dict) tuples), the directory will be populated
374hunk ./src/allmydata/interfaces.py 2452
375 
376 class IClientStatus(Interface):
377     def list_all_uploads():
378-        """Return a list of uploader objects, one for each upload which
379+        """Return a list of uploader objects, one for each upload that
380         currently has an object available (tracked with weakrefs). This is
381         intended for debugging purposes."""
382     def list_active_uploads():
383hunk ./src/allmydata/interfaces.py 2462
384         started uploads."""
385 
386     def list_all_downloads():
387-        """Return a list of downloader objects, one for each download which
388+        """Return a list of downloader objects, one for each download that
389         currently has an object available (tracked with weakrefs). This is
390         intended for debugging purposes."""
391     def list_active_downloads():
392hunk ./src/allmydata/interfaces.py 2689
393 
394     def provide(provider=RIStatsProvider, nickname=str):
395         """
396-        @param provider: a stats collector instance which should be polled
397+        @param provider: a stats collector instance that should be polled
398                          periodically by the gatherer to collect stats.
399         @param nickname: a name useful to identify the provided client
400         """
401hunk ./src/allmydata/interfaces.py 2722
402 
403 class IValidatedThingProxy(Interface):
404     def start():
405-        """ Acquire a thing and validate it. Return a deferred which is
406+        """ Acquire a thing and validate it. Return a deferred that is
407         eventually fired with self if the thing is valid or errbacked if it
408         can't be acquired or validated."""
409 
410}
411[Pluggable backends -- new and moved files, changes to moved files. refs #999
412david-sarah@jacaranda.org**20110919232926
413 Ignore-this: ec5d2d1362a092d919e84327d3092424
414] {
415adddir ./src/allmydata/storage/backends
416adddir ./src/allmydata/storage/backends/disk
417move ./src/allmydata/storage/immutable.py ./src/allmydata/storage/backends/disk/immutable.py
418move ./src/allmydata/storage/mutable.py ./src/allmydata/storage/backends/disk/mutable.py
419adddir ./src/allmydata/storage/backends/null
420addfile ./src/allmydata/storage/backends/__init__.py
421addfile ./src/allmydata/storage/backends/base.py
422hunk ./src/allmydata/storage/backends/base.py 1
423+
424+from twisted.application import service
425+
426+from allmydata.storage.common import si_b2a
427+from allmydata.storage.lease import LeaseInfo
428+from allmydata.storage.bucket import BucketReader
429+
430+
431+class Backend(service.MultiService):
432+    def __init__(self):
433+        service.MultiService.__init__(self)
434+
435+
436+class ShareSet(object):
437+    """
438+    This class implements shareset logic that could work for all backends, but
439+    might be useful to override for efficiency.
440+    """
441+
442+    def __init__(self, storageindex):
443+        self.storageindex = storageindex
444+
445+    def get_storage_index(self):
446+        return self.storageindex
447+
448+    def get_storage_index_string(self):
449+        return si_b2a(self.storageindex)
450+
451+    def renew_lease(self, renew_secret, new_expiration_time):
452+        found_shares = False
453+        for share in self.get_shares():
454+            found_shares = True
455+            share.renew_lease(renew_secret, new_expiration_time)
456+
457+        if not found_shares:
458+            raise IndexError("no such lease to renew")
459+
460+    def get_leases(self):
461+        # Since all shares get the same lease data, we just grab the leases
462+        # from the first share.
463+        try:
464+            sf = self.get_shares().next()
465+            return sf.get_leases()
466+        except StopIteration:
467+            return iter([])
468+
469+    def add_or_renew_lease(self, lease_info):
470+        # This implementation assumes that lease data is duplicated in
471+        # all shares of a shareset, which might not be true for all backends.
472+        for share in self.get_shares():
473+            share.add_or_renew_lease(lease_info)
474+
475+    def make_bucket_reader(self, storageserver, share):
476+        return BucketReader(storageserver, share)
477+
478+    def testv_and_readv_and_writev(self, storageserver, secrets,
479+                                   test_and_write_vectors, read_vector,
480+                                   expiration_time):
481+        # The implementation here depends on the following helper methods,
482+        # which must be provided by subclasses:
483+        #
484+        # def _clean_up_after_unlink(self):
485+        #     """clean up resources associated with the shareset after some
486+        #     shares might have been deleted"""
487+        #
488+        # def _create_mutable_share(self, storageserver, shnum, write_enabler):
489+        #     """create a mutable share with the given shnum and write_enabler"""
490+
491+        # secrets might be a triple with cancel_secret in secrets[2], but if
492+        # so we ignore the cancel_secret.
493+        write_enabler = secrets[0]
494+        renew_secret = secrets[1]
495+
496+        si_s = self.get_storage_index_string()
497+        shares = {}
498+        for share in self.get_shares():
499+            # XXX is it correct to ignore immutable shares? Maybe get_shares should
500+            # have a parameter saying what type it's expecting.
501+            if share.sharetype == "mutable":
502+                share.check_write_enabler(write_enabler, si_s)
503+                shares[share.get_shnum()] = share
504+
505+        # write_enabler is good for all existing shares
506+
507+        # now evaluate test vectors
508+        testv_is_good = True
509+        for sharenum in test_and_write_vectors:
510+            (testv, datav, new_length) = test_and_write_vectors[sharenum]
511+            if sharenum in shares:
512+                if not shares[sharenum].check_testv(testv):
513+                    self.log("testv failed: [%d]: %r" % (sharenum, testv))
514+                    testv_is_good = False
515+                    break
516+            else:
517+                # compare the vectors against an empty share, in which all
518+                # reads return empty strings
519+                if not EmptyShare().check_testv(testv):
520+                    self.log("testv failed (empty): [%d] %r" % (sharenum,
521+                                                                testv))
522+                    testv_is_good = False
523+                    break
524+
525+        # gather the read vectors, before we do any writes
526+        read_data = {}
527+        for shnum, share in shares.items():
528+            read_data[shnum] = share.readv(read_vector)
529+
530+        ownerid = 1 # TODO
531+        lease_info = LeaseInfo(ownerid, renew_secret,
532+                               expiration_time, storageserver.get_serverid())
533+
534+        if testv_is_good:
535+            # now apply the write vectors
536+            for shnum in test_and_write_vectors:
537+                (testv, datav, new_length) = test_and_write_vectors[shnum]
538+                if new_length == 0:
539+                    if shnum in shares:
540+                        shares[shnum].unlink()
541+                else:
542+                    if shnum not in shares:
543+                        # allocate a new share
544+                        share = self._create_mutable_share(storageserver, shnum, write_enabler)
545+                        shares[shnum] = share
546+                    shares[shnum].writev(datav, new_length)
547+                    # and update the lease
548+                    shares[shnum].add_or_renew_lease(lease_info)
549+
550+            if new_length == 0:
551+                self._clean_up_after_unlink()
552+
553+        return (testv_is_good, read_data)
554+
555+    def readv(self, wanted_shnums, read_vector):
556+        """
557+        Read a vector from the numbered shares in this shareset. An empty
558+        shares list means to return data from all known shares.
559+
560+        @param wanted_shnums=ListOf(int)
561+        @param read_vector=ReadVector
562+        @return DictOf(int, ReadData): shnum -> results, with one key per share
563+        """
564+        datavs = {}
565+        for share in self.get_shares():
566+            shnum = share.get_shnum()
567+            if not wanted_shnums or shnum in wanted_shnums:
568+                datavs[shnum] = share.readv(read_vector)
569+
570+        return datavs
571+
572+
573+def testv_compare(a, op, b):
574+    assert op in ("lt", "le", "eq", "ne", "ge", "gt")
575+    if op == "lt":
576+        return a < b
577+    if op == "le":
578+        return a <= b
579+    if op == "eq":
580+        return a == b
581+    if op == "ne":
582+        return a != b
583+    if op == "ge":
584+        return a >= b
585+    if op == "gt":
586+        return a > b
587+    # never reached
588+
589+
590+class EmptyShare:
591+    def check_testv(self, testv):
592+        test_good = True
593+        for (offset, length, operator, specimen) in testv:
594+            data = ""
595+            if not testv_compare(data, operator, specimen):
596+                test_good = False
597+                break
598+        return test_good
599+
600addfile ./src/allmydata/storage/backends/disk/__init__.py
601addfile ./src/allmydata/storage/backends/disk/disk_backend.py
602hunk ./src/allmydata/storage/backends/disk/disk_backend.py 1
603+
604+import re
605+
606+from twisted.python.filepath import UnlistableError
607+
608+from zope.interface import implements
609+from allmydata.interfaces import IStorageBackend, IShareSet
610+from allmydata.util import fileutil, log, time_format
611+from allmydata.storage.common import si_b2a, si_a2b
612+from allmydata.storage.bucket import BucketWriter
613+from allmydata.storage.backends.base import Backend, ShareSet
614+from allmydata.storage.backends.disk.immutable import ImmutableDiskShare
615+from allmydata.storage.backends.disk.mutable import MutableDiskShare, create_mutable_disk_share
616+
617+# storage/
618+# storage/shares/incoming
619+#   incoming/ holds temp dirs named $START/$STORAGEINDEX/$SHARENUM which will
620+#   be moved to storage/shares/$START/$STORAGEINDEX/$SHARENUM upon success
621+# storage/shares/$START/$STORAGEINDEX
622+# storage/shares/$START/$STORAGEINDEX/$SHARENUM
623+
624+# Where "$START" denotes the first 10 bits worth of $STORAGEINDEX (that's 2
625+# base-32 chars).
626+# $SHARENUM matches this regex:
627+NUM_RE=re.compile("^[0-9]+$")
628+
629+
630+def si_si2dir(startfp, storageindex):
631+    sia = si_b2a(storageindex)
632+    newfp = startfp.child(sia[:2])
633+    return newfp.child(sia)
634+
635+
636+def get_share(fp):
637+    f = fp.open('rb')
638+    try:
639+        prefix = f.read(32)
640+    finally:
641+        f.close()
642+
643+    if prefix == MutableDiskShare.MAGIC:
644+        return MutableDiskShare(fp)
645+    else:
646+        # assume it's immutable
647+        return ImmutableDiskShare(fp)
648+
649+
650+class DiskBackend(Backend):
651+    implements(IStorageBackend)
652+
653+    def __init__(self, storedir, readonly=False, reserved_space=0, discard_storage=False):
654+        Backend.__init__(self)
655+        self._setup_storage(storedir, readonly, reserved_space, discard_storage)
656+        self._setup_corruption_advisory()
657+
658+    def _setup_storage(self, storedir, readonly, reserved_space, discard_storage):
659+        self._storedir = storedir
660+        self._readonly = readonly
661+        self._reserved_space = int(reserved_space)
662+        self._discard_storage = discard_storage
663+        self._sharedir = self._storedir.child("shares")
664+        fileutil.fp_make_dirs(self._sharedir)
665+        self._incomingdir = self._sharedir.child('incoming')
666+        self._clean_incomplete()
667+        if self._reserved_space and (self.get_available_space() is None):
668+            log.msg("warning: [storage]reserved_space= is set, but this platform does not support an API to get disk statistics (statvfs(2) or GetDiskFreeSpaceEx), so this reservation cannot be honored",
669+                    umid="0wZ27w", level=log.UNUSUAL)
670+
671+    def _clean_incomplete(self):
672+        fileutil.fp_remove(self._incomingdir)
673+        fileutil.fp_make_dirs(self._incomingdir)
674+
675+    def _setup_corruption_advisory(self):
676+        # we don't actually create the corruption-advisory dir until necessary
677+        self._corruption_advisory_dir = self._storedir.child("corruption-advisories")
678+
679+    def _make_shareset(self, sharehomedir):
680+        return self.get_shareset(si_a2b(sharehomedir.basename()))
681+
682+    def get_sharesets_for_prefix(self, prefix):
683+        prefixfp = self._sharedir.child(prefix)
684+        try:
685+            sharesets = map(self._make_shareset, prefixfp.children())
686+            def _by_base32si(b):
687+                return b.get_storage_index_string()
688+            sharesets.sort(key=_by_base32si)
689+        except EnvironmentError:
690+            sharesets = []
691+        return sharesets
692+
693+    def get_shareset(self, storageindex):
694+        sharehomedir = si_si2dir(self._sharedir, storageindex)
695+        incominghomedir = si_si2dir(self._incomingdir, storageindex)
696+        return DiskShareSet(storageindex, sharehomedir, incominghomedir, discard_storage=self._discard_storage)
697+
698+    def fill_in_space_stats(self, stats):
699+        stats['storage_server.reserved_space'] = self._reserved_space
700+        try:
701+            disk = fileutil.get_disk_stats(self._sharedir, self._reserved_space)
702+            writeable = disk['avail'] > 0
703+
704+            # spacetime predictors should use disk_avail / (d(disk_used)/dt)
705+            stats['storage_server.disk_total'] = disk['total']
706+            stats['storage_server.disk_used'] = disk['used']
707+            stats['storage_server.disk_free_for_root'] = disk['free_for_root']
708+            stats['storage_server.disk_free_for_nonroot'] = disk['free_for_nonroot']
709+            stats['storage_server.disk_avail'] = disk['avail']
710+        except AttributeError:
711+            writeable = True
712+        except EnvironmentError:
713+            log.msg("OS call to get disk statistics failed", level=log.UNUSUAL)
714+            writeable = False
715+
716+        if self._readonly:
717+            stats['storage_server.disk_avail'] = 0
718+            writeable = False
719+
720+        stats['storage_server.accepting_immutable_shares'] = int(writeable)
721+
722+    def get_available_space(self):
723+        if self._readonly:
724+            return 0
725+        return fileutil.get_available_space(self._sharedir, self._reserved_space)
726+
727+    def advise_corrupt_share(self, sharetype, storageindex, shnum, reason):
728+        fileutil.fp_make_dirs(self._corruption_advisory_dir)
729+        now = time_format.iso_utc(sep="T")
730+        si_s = si_b2a(storageindex)
731+
732+        # Windows can't handle colons in the filename.
733+        name = ("%s--%s-%d" % (now, si_s, shnum)).replace(":", "")
734+        f = self._corruption_advisory_dir.child(name).open("w")
735+        try:
736+            f.write("report: Share Corruption\n")
737+            f.write("type: %s\n" % sharetype)
738+            f.write("storage_index: %s\n" % si_s)
739+            f.write("share_number: %d\n" % shnum)
740+            f.write("\n")
741+            f.write(reason)
742+            f.write("\n")
743+        finally:
744+            f.close()
745+
746+        log.msg(format=("client claims corruption in (%(share_type)s) " +
747+                        "%(si)s-%(shnum)d: %(reason)s"),
748+                share_type=sharetype, si=si_s, shnum=shnum, reason=reason,
749+                level=log.SCARY, umid="SGx2fA")
750+
751+
752+class DiskShareSet(ShareSet):
753+    implements(IShareSet)
754+
755+    def __init__(self, storageindex, sharehomedir, incominghomedir=None, discard_storage=False):
756+        ShareSet.__init__(self, storageindex)
757+        self._sharehomedir = sharehomedir
758+        self._incominghomedir = incominghomedir
759+        self._discard_storage = discard_storage
760+
761+    def get_overhead(self):
762+        return (fileutil.get_disk_usage(self._sharehomedir) +
763+                fileutil.get_disk_usage(self._incominghomedir))
764+
765+    def get_shares(self):
766+        """
767+        Generate IStorageBackendShare objects for shares we have for this storage index.
768+        ("Shares we have" means completed ones, excluding incoming ones.)
769+        """
770+        try:
771+            for fp in self._sharehomedir.children():
772+                shnumstr = fp.basename()
773+                if not NUM_RE.match(shnumstr):
774+                    continue
775+                sharehome = self._sharehomedir.child(shnumstr)
776+                yield self.get_share(sharehome)
777+        except UnlistableError:
778+            # There is no shares directory at all.
779+            pass
780+
781+    def has_incoming(self, shnum):
782+        if self._incominghomedir is None:
783+            return False
784+        return self._incominghomedir.child(str(shnum)).exists()
785+
786+    def make_bucket_writer(self, storageserver, shnum, max_space_per_bucket, lease_info, canary):
787+        sharehome = self._sharehomedir.child(str(shnum))
788+        incominghome = self._incominghomedir.child(str(shnum))
789+        immsh = ImmutableDiskShare(self.get_storage_index(), shnum, sharehome, incominghome,
790+                                   max_size=max_space_per_bucket, create=True)
791+        bw = BucketWriter(storageserver, immsh, max_space_per_bucket, lease_info, canary)
792+        if self._discard_storage:
793+            bw.throw_out_all_data = True
794+        return bw
795+
796+    def _create_mutable_share(self, storageserver, shnum, write_enabler):
797+        fileutil.fp_make_dirs(self._sharehomedir)
798+        sharehome = self._sharehomedir.child(str(shnum))
799+        serverid = storageserver.get_serverid()
800+        return create_mutable_disk_share(sharehome, serverid, write_enabler, storageserver)
801+
802+    def _clean_up_after_unlink(self):
803+        fileutil.fp_rmdir_if_empty(self._sharehomedir)
804+
805hunk ./src/allmydata/storage/backends/disk/immutable.py 1
806-import os, stat, struct, time
807 
808hunk ./src/allmydata/storage/backends/disk/immutable.py 2
809-from foolscap.api import Referenceable
810+import struct
811 
812 from zope.interface import implements
813hunk ./src/allmydata/storage/backends/disk/immutable.py 5
814-from allmydata.interfaces import RIBucketWriter, RIBucketReader
815-from allmydata.util import base32, fileutil, log
816+
817+from allmydata.interfaces import IStoredShare
818+from allmydata.util import fileutil
819 from allmydata.util.assertutil import precondition
820hunk ./src/allmydata/storage/backends/disk/immutable.py 9
821+from allmydata.util.fileutil import fp_make_dirs
822 from allmydata.util.hashutil import constant_time_compare
823hunk ./src/allmydata/storage/backends/disk/immutable.py 11
824+from allmydata.util.encodingutil import quote_filepath
825+from allmydata.storage.common import si_b2a, UnknownImmutableContainerVersionError, DataTooLargeError
826 from allmydata.storage.lease import LeaseInfo
827hunk ./src/allmydata/storage/backends/disk/immutable.py 14
828-from allmydata.storage.common import UnknownImmutableContainerVersionError, \
829-     DataTooLargeError
830+
831 
832 # each share file (in storage/shares/$SI/$SHNUM) contains lease information
833 # and share data. The share data is accessed by RIBucketWriter.write and
834hunk ./src/allmydata/storage/backends/disk/immutable.py 41
835 # then the value stored in this field will be the actual share data length
836 # modulo 2**32.
837 
838-class ShareFile:
839-    LEASE_SIZE = struct.calcsize(">L32s32sL")
840+class ImmutableDiskShare(object):
841+    implements(IStoredShare)
842+
843     sharetype = "immutable"
844hunk ./src/allmydata/storage/backends/disk/immutable.py 45
845+    LEASE_SIZE = struct.calcsize(">L32s32sL")
846+
847 
848hunk ./src/allmydata/storage/backends/disk/immutable.py 48
849-    def __init__(self, filename, max_size=None, create=False):
850-        """ If max_size is not None then I won't allow more than max_size to be written to me. If create=True and max_size must not be None. """
851+    def __init__(self, storageindex, shnum, finalhome=None, incominghome=None, max_size=None, create=False):
852+        """ If max_size is not None then I won't allow more than
853+        max_size to be written to me. If create=True then max_size
854+        must not be None. """
855         precondition((max_size is not None) or (not create), max_size, create)
856hunk ./src/allmydata/storage/backends/disk/immutable.py 53
857-        self.home = filename
858+        self._storageindex = storageindex
859         self._max_size = max_size
860hunk ./src/allmydata/storage/backends/disk/immutable.py 55
861+        self._incominghome = incominghome
862+        self._home = finalhome
863+        self._shnum = shnum
864         if create:
865             # touch the file, so later callers will see that we're working on
866             # it. Also construct the metadata.
867hunk ./src/allmydata/storage/backends/disk/immutable.py 61
868-            assert not os.path.exists(self.home)
869-            fileutil.make_dirs(os.path.dirname(self.home))
870-            f = open(self.home, 'wb')
871+            assert not finalhome.exists()
872+            fp_make_dirs(self._incominghome.parent())
873             # The second field -- the four-byte share data length -- is no
874             # longer used as of Tahoe v1.3.0, but we continue to write it in
875             # there in case someone downgrades a storage server from >=
876hunk ./src/allmydata/storage/backends/disk/immutable.py 72
877             # the largest length that can fit into the field. That way, even
878             # if this does happen, the old < v1.3.0 server will still allow
879             # clients to read the first part of the share.
880-            f.write(struct.pack(">LLL", 1, min(2**32-1, max_size), 0))
881-            f.close()
882+            self._incominghome.setContent(struct.pack(">LLL", 1, min(2**32-1, max_size), 0) )
883             self._lease_offset = max_size + 0x0c
884             self._num_leases = 0
885         else:
886hunk ./src/allmydata/storage/backends/disk/immutable.py 76
887-            f = open(self.home, 'rb')
888-            filesize = os.path.getsize(self.home)
889-            (version, unused, num_leases) = struct.unpack(">LLL", f.read(0xc))
890-            f.close()
891+            f = self._home.open(mode='rb')
892+            try:
893+                (version, unused, num_leases) = struct.unpack(">LLL", f.read(0xc))
894+            finally:
895+                f.close()
896+            filesize = self._home.getsize()
897             if version != 1:
898                 msg = "sharefile %s had version %d but we wanted 1" % \
899hunk ./src/allmydata/storage/backends/disk/immutable.py 84
900-                      (filename, version)
901+                      (self._home, version)
902                 raise UnknownImmutableContainerVersionError(msg)
903             self._num_leases = num_leases
904             self._lease_offset = filesize - (num_leases * self.LEASE_SIZE)
905hunk ./src/allmydata/storage/backends/disk/immutable.py 90
906         self._data_offset = 0xc
907 
908+    def __repr__(self):
909+        return ("<ImmutableDiskShare %s:%r at %s>"
910+                % (si_b2a(self._storageindex), self._shnum, quote_filepath(self._home)))
911+
912+    def close(self):
913+        fileutil.fp_make_dirs(self._home.parent())
914+        self._incominghome.moveTo(self._home)
915+        try:
916+            # self._incominghome is like storage/shares/incoming/ab/abcde/4 .
917+            # We try to delete the parent (.../ab/abcde) to avoid leaving
918+            # these directories lying around forever, but the delete might
919+            # fail if we're working on another share for the same storage
920+            # index (like ab/abcde/5). The alternative approach would be to
921+            # use a hierarchy of objects (PrefixHolder, BucketHolder,
922+            # ShareWriter), each of which is responsible for a single
923+            # directory on disk, and have them use reference counting of
924+            # their children to know when they should do the rmdir. This
925+            # approach is simpler, but relies on os.rmdir refusing to delete
926+            # a non-empty directory. Do *not* use fileutil.fp_remove() here!
927+            fileutil.fp_rmdir_if_empty(self._incominghome.parent())
928+            # we also delete the grandparent (prefix) directory, .../ab ,
929+            # again to avoid leaving directories lying around. This might
930+            # fail if there is another bucket open that shares a prefix (like
931+            # ab/abfff).
932+            fileutil.fp_rmdir_if_empty(self._incominghome.parent().parent())
933+            # we leave the great-grandparent (incoming/) directory in place.
934+        except EnvironmentError:
935+            # ignore the "can't rmdir because the directory is not empty"
936+            # exceptions, those are normal consequences of the
937+            # above-mentioned conditions.
938+            pass
939+        pass
940+
941+    def get_used_space(self):
942+        return (fileutil.get_used_space(self._home) +
943+                fileutil.get_used_space(self._incominghome))
944+
945+    def get_storage_index(self):
946+        return self._storageindex
947+
948+    def get_shnum(self):
949+        return self._shnum
950+
951     def unlink(self):
952hunk ./src/allmydata/storage/backends/disk/immutable.py 134
953-        os.unlink(self.home)
954+        self._home.remove()
955+
956+    def get_size(self):
957+        return self._home.getsize()
958+
959+    def get_data_length(self):
960+        return self._lease_offset - self._data_offset
961+
962+    #def readv(self, read_vector):
963+    #    ...
964 
965     def read_share_data(self, offset, length):
966         precondition(offset >= 0)
967hunk ./src/allmydata/storage/backends/disk/immutable.py 147
968-        # reads beyond the end of the data are truncated. Reads that start
969+
970+        # Reads beyond the end of the data are truncated. Reads that start
971         # beyond the end of the data return an empty string.
972         seekpos = self._data_offset+offset
973         actuallength = max(0, min(length, self._lease_offset-seekpos))
974hunk ./src/allmydata/storage/backends/disk/immutable.py 154
975         if actuallength == 0:
976             return ""
977-        f = open(self.home, 'rb')
978-        f.seek(seekpos)
979-        return f.read(actuallength)
980+        f = self._home.open(mode='rb')
981+        try:
982+            f.seek(seekpos)
983+            sharedata = f.read(actuallength)
984+        finally:
985+            f.close()
986+        return sharedata
987 
988     def write_share_data(self, offset, data):
989         length = len(data)
990hunk ./src/allmydata/storage/backends/disk/immutable.py 167
991         precondition(offset >= 0, offset)
992         if self._max_size is not None and offset+length > self._max_size:
993             raise DataTooLargeError(self._max_size, offset, length)
994-        f = open(self.home, 'rb+')
995-        real_offset = self._data_offset+offset
996-        f.seek(real_offset)
997-        assert f.tell() == real_offset
998-        f.write(data)
999-        f.close()
1000+        f = self._incominghome.open(mode='rb+')
1001+        try:
1002+            real_offset = self._data_offset+offset
1003+            f.seek(real_offset)
1004+            assert f.tell() == real_offset
1005+            f.write(data)
1006+        finally:
1007+            f.close()
1008 
1009     def _write_lease_record(self, f, lease_number, lease_info):
1010         offset = self._lease_offset + lease_number * self.LEASE_SIZE
1011hunk ./src/allmydata/storage/backends/disk/immutable.py 184
1012 
1013     def _read_num_leases(self, f):
1014         f.seek(0x08)
1015-        (num_leases,) = struct.unpack(">L", f.read(4))
1016+        ro = f.read(4)
1017+        (num_leases,) = struct.unpack(">L", ro)
1018         return num_leases
1019 
1020     def _write_num_leases(self, f, num_leases):
1021hunk ./src/allmydata/storage/backends/disk/immutable.py 195
1022     def _truncate_leases(self, f, num_leases):
1023         f.truncate(self._lease_offset + num_leases * self.LEASE_SIZE)
1024 
1025+    # These lease operations are intended for use by disk_backend.py.
1026+    # Other clients should not depend on the fact that the disk backend
1027+    # stores leases in share files.
1028+
1029     def get_leases(self):
1030         """Yields a LeaseInfo instance for all leases."""
1031hunk ./src/allmydata/storage/backends/disk/immutable.py 201
1032-        f = open(self.home, 'rb')
1033-        (version, unused, num_leases) = struct.unpack(">LLL", f.read(0xc))
1034-        f.seek(self._lease_offset)
1035-        for i in range(num_leases):
1036-            data = f.read(self.LEASE_SIZE)
1037-            if data:
1038-                yield LeaseInfo().from_immutable_data(data)
1039+        f = self._home.open(mode='rb')
1040+        try:
1041+            (version, unused, num_leases) = struct.unpack(">LLL", f.read(0xc))
1042+            f.seek(self._lease_offset)
1043+            for i in range(num_leases):
1044+                data = f.read(self.LEASE_SIZE)
1045+                if data:
1046+                    yield LeaseInfo().from_immutable_data(data)
1047+        finally:
1048+            f.close()
1049 
1050     def add_lease(self, lease_info):
1051hunk ./src/allmydata/storage/backends/disk/immutable.py 213
1052-        f = open(self.home, 'rb+')
1053-        num_leases = self._read_num_leases(f)
1054-        self._write_lease_record(f, num_leases, lease_info)
1055-        self._write_num_leases(f, num_leases+1)
1056-        f.close()
1057+        f = self._incominghome.open(mode='rb')
1058+        try:
1059+            num_leases = self._read_num_leases(f)
1060+        finally:
1061+            f.close()
1062+        f = self._home.open(mode='wb+')
1063+        try:
1064+            self._write_lease_record(f, num_leases, lease_info)
1065+            self._write_num_leases(f, num_leases+1)
1066+        finally:
1067+            f.close()
1068 
1069     def renew_lease(self, renew_secret, new_expire_time):
1070hunk ./src/allmydata/storage/backends/disk/immutable.py 226
1071-        for i,lease in enumerate(self.get_leases()):
1072-            if constant_time_compare(lease.renew_secret, renew_secret):
1073-                # yup. See if we need to update the owner time.
1074-                if new_expire_time > lease.expiration_time:
1075-                    # yes
1076-                    lease.expiration_time = new_expire_time
1077-                    f = open(self.home, 'rb+')
1078-                    self._write_lease_record(f, i, lease)
1079-                    f.close()
1080-                return
1081+        try:
1082+            for i, lease in enumerate(self.get_leases()):
1083+                if constant_time_compare(lease.renew_secret, renew_secret):
1084+                    # yup. See if we need to update the owner time.
1085+                    if new_expire_time > lease.expiration_time:
1086+                        # yes
1087+                        lease.expiration_time = new_expire_time
1088+                        f = self._home.open('rb+')
1089+                        try:
1090+                            self._write_lease_record(f, i, lease)
1091+                        finally:
1092+                            f.close()
1093+                    return
1094+        except IndexError, e:
1095+            raise Exception("IndexError: %s" % (e,))
1096         raise IndexError("unable to renew non-existent lease")
1097 
1098     def add_or_renew_lease(self, lease_info):
1099hunk ./src/allmydata/storage/backends/disk/immutable.py 249
1100                              lease_info.expiration_time)
1101         except IndexError:
1102             self.add_lease(lease_info)
1103-
1104-
1105-    def cancel_lease(self, cancel_secret):
1106-        """Remove a lease with the given cancel_secret. If the last lease is
1107-        cancelled, the file will be removed. Return the number of bytes that
1108-        were freed (by truncating the list of leases, and possibly by
1109-        deleting the file. Raise IndexError if there was no lease with the
1110-        given cancel_secret.
1111-        """
1112-
1113-        leases = list(self.get_leases())
1114-        num_leases_removed = 0
1115-        for i,lease in enumerate(leases):
1116-            if constant_time_compare(lease.cancel_secret, cancel_secret):
1117-                leases[i] = None
1118-                num_leases_removed += 1
1119-        if not num_leases_removed:
1120-            raise IndexError("unable to find matching lease to cancel")
1121-        if num_leases_removed:
1122-            # pack and write out the remaining leases. We write these out in
1123-            # the same order as they were added, so that if we crash while
1124-            # doing this, we won't lose any non-cancelled leases.
1125-            leases = [l for l in leases if l] # remove the cancelled leases
1126-            f = open(self.home, 'rb+')
1127-            for i,lease in enumerate(leases):
1128-                self._write_lease_record(f, i, lease)
1129-            self._write_num_leases(f, len(leases))
1130-            self._truncate_leases(f, len(leases))
1131-            f.close()
1132-        space_freed = self.LEASE_SIZE * num_leases_removed
1133-        if not len(leases):
1134-            space_freed += os.stat(self.home)[stat.ST_SIZE]
1135-            self.unlink()
1136-        return space_freed
1137-
1138-
1139-class BucketWriter(Referenceable):
1140-    implements(RIBucketWriter)
1141-
1142-    def __init__(self, ss, incominghome, finalhome, max_size, lease_info, canary):
1143-        self.ss = ss
1144-        self.incominghome = incominghome
1145-        self.finalhome = finalhome
1146-        self._max_size = max_size # don't allow the client to write more than this
1147-        self._canary = canary
1148-        self._disconnect_marker = canary.notifyOnDisconnect(self._disconnected)
1149-        self.closed = False
1150-        self.throw_out_all_data = False
1151-        self._sharefile = ShareFile(incominghome, create=True, max_size=max_size)
1152-        # also, add our lease to the file now, so that other ones can be
1153-        # added by simultaneous uploaders
1154-        self._sharefile.add_lease(lease_info)
1155-
1156-    def allocated_size(self):
1157-        return self._max_size
1158-
1159-    def remote_write(self, offset, data):
1160-        start = time.time()
1161-        precondition(not self.closed)
1162-        if self.throw_out_all_data:
1163-            return
1164-        self._sharefile.write_share_data(offset, data)
1165-        self.ss.add_latency("write", time.time() - start)
1166-        self.ss.count("write")
1167-
1168-    def remote_close(self):
1169-        precondition(not self.closed)
1170-        start = time.time()
1171-
1172-        fileutil.make_dirs(os.path.dirname(self.finalhome))
1173-        fileutil.rename(self.incominghome, self.finalhome)
1174-        try:
1175-            # self.incominghome is like storage/shares/incoming/ab/abcde/4 .
1176-            # We try to delete the parent (.../ab/abcde) to avoid leaving
1177-            # these directories lying around forever, but the delete might
1178-            # fail if we're working on another share for the same storage
1179-            # index (like ab/abcde/5). The alternative approach would be to
1180-            # use a hierarchy of objects (PrefixHolder, BucketHolder,
1181-            # ShareWriter), each of which is responsible for a single
1182-            # directory on disk, and have them use reference counting of
1183-            # their children to know when they should do the rmdir. This
1184-            # approach is simpler, but relies on os.rmdir refusing to delete
1185-            # a non-empty directory. Do *not* use fileutil.rm_dir() here!
1186-            os.rmdir(os.path.dirname(self.incominghome))
1187-            # we also delete the grandparent (prefix) directory, .../ab ,
1188-            # again to avoid leaving directories lying around. This might
1189-            # fail if there is another bucket open that shares a prefix (like
1190-            # ab/abfff).
1191-            os.rmdir(os.path.dirname(os.path.dirname(self.incominghome)))
1192-            # we leave the great-grandparent (incoming/) directory in place.
1193-        except EnvironmentError:
1194-            # ignore the "can't rmdir because the directory is not empty"
1195-            # exceptions, those are normal consequences of the
1196-            # above-mentioned conditions.
1197-            pass
1198-        self._sharefile = None
1199-        self.closed = True
1200-        self._canary.dontNotifyOnDisconnect(self._disconnect_marker)
1201-
1202-        filelen = os.stat(self.finalhome)[stat.ST_SIZE]
1203-        self.ss.bucket_writer_closed(self, filelen)
1204-        self.ss.add_latency("close", time.time() - start)
1205-        self.ss.count("close")
1206-
1207-    def _disconnected(self):
1208-        if not self.closed:
1209-            self._abort()
1210-
1211-    def remote_abort(self):
1212-        log.msg("storage: aborting sharefile %s" % self.incominghome,
1213-                facility="tahoe.storage", level=log.UNUSUAL)
1214-        if not self.closed:
1215-            self._canary.dontNotifyOnDisconnect(self._disconnect_marker)
1216-        self._abort()
1217-        self.ss.count("abort")
1218-
1219-    def _abort(self):
1220-        if self.closed:
1221-            return
1222-
1223-        os.remove(self.incominghome)
1224-        # if we were the last share to be moved, remove the incoming/
1225-        # directory that was our parent
1226-        parentdir = os.path.split(self.incominghome)[0]
1227-        if not os.listdir(parentdir):
1228-            os.rmdir(parentdir)
1229-        self._sharefile = None
1230-
1231-        # We are now considered closed for further writing. We must tell
1232-        # the storage server about this so that it stops expecting us to
1233-        # use the space it allocated for us earlier.
1234-        self.closed = True
1235-        self.ss.bucket_writer_closed(self, 0)
1236-
1237-
1238-class BucketReader(Referenceable):
1239-    implements(RIBucketReader)
1240-
1241-    def __init__(self, ss, sharefname, storage_index=None, shnum=None):
1242-        self.ss = ss
1243-        self._share_file = ShareFile(sharefname)
1244-        self.storage_index = storage_index
1245-        self.shnum = shnum
1246-
1247-    def __repr__(self):
1248-        return "<%s %s %s>" % (self.__class__.__name__,
1249-                               base32.b2a_l(self.storage_index[:8], 60),
1250-                               self.shnum)
1251-
1252-    def remote_read(self, offset, length):
1253-        start = time.time()
1254-        data = self._share_file.read_share_data(offset, length)
1255-        self.ss.add_latency("read", time.time() - start)
1256-        self.ss.count("read")
1257-        return data
1258-
1259-    def remote_advise_corrupt_share(self, reason):
1260-        return self.ss.remote_advise_corrupt_share("immutable",
1261-                                                   self.storage_index,
1262-                                                   self.shnum,
1263-                                                   reason)
1264hunk ./src/allmydata/storage/backends/disk/mutable.py 1
1265-import os, stat, struct
1266 
1267hunk ./src/allmydata/storage/backends/disk/mutable.py 2
1268-from allmydata.interfaces import BadWriteEnablerError
1269-from allmydata.util import idlib, log
1270+import struct
1271+
1272+from zope.interface import implements
1273+
1274+from allmydata.interfaces import IStoredMutableShare, BadWriteEnablerError
1275+from allmydata.util import fileutil, idlib, log
1276 from allmydata.util.assertutil import precondition
1277 from allmydata.util.hashutil import constant_time_compare
1278hunk ./src/allmydata/storage/backends/disk/mutable.py 10
1279-from allmydata.storage.lease import LeaseInfo
1280-from allmydata.storage.common import UnknownMutableContainerVersionError, \
1281+from allmydata.util.encodingutil import quote_filepath
1282+from allmydata.storage.common import si_b2a, UnknownMutableContainerVersionError, \
1283      DataTooLargeError
1284hunk ./src/allmydata/storage/backends/disk/mutable.py 13
1285+from allmydata.storage.lease import LeaseInfo
1286+from allmydata.storage.backends.base import testv_compare
1287 
1288hunk ./src/allmydata/storage/backends/disk/mutable.py 16
1289-# the MutableShareFile is like the ShareFile, but used for mutable data. It
1290-# has a different layout. See docs/mutable.txt for more details.
1291+
1292+# The MutableDiskShare is like the ImmutableDiskShare, but used for mutable data.
1293+# It has a different layout. See docs/mutable.rst for more details.
1294 
1295 # #   offset    size    name
1296 # 1   0         32      magic verstr "tahoe mutable container v1" plus binary
1297hunk ./src/allmydata/storage/backends/disk/mutable.py 31
1298 #                        4    4   expiration timestamp
1299 #                        8   32   renewal token
1300 #                        40  32   cancel token
1301-#                        72  20   nodeid which accepted the tokens
1302+#                        72  20   nodeid that accepted the tokens
1303 # 7   468       (a)     data
1304 # 8   ??        4       count of extra leases
1305 # 9   ??        n*92    extra leases
1306hunk ./src/allmydata/storage/backends/disk/mutable.py 37
1307 
1308 
1309-# The struct module doc says that L's are 4 bytes in size., and that Q's are
1310+# The struct module doc says that L's are 4 bytes in size, and that Q's are
1311 # 8 bytes in size. Since compatibility depends upon this, double-check it.
1312 assert struct.calcsize(">L") == 4, struct.calcsize(">L")
1313 assert struct.calcsize(">Q") == 8, struct.calcsize(">Q")
1314hunk ./src/allmydata/storage/backends/disk/mutable.py 42
1315 
1316-class MutableShareFile:
1317+
1318+class MutableDiskShare(object):
1319+    implements(IStoredMutableShare)
1320 
1321     sharetype = "mutable"
1322     DATA_LENGTH_OFFSET = struct.calcsize(">32s20s32s")
1323hunk ./src/allmydata/storage/backends/disk/mutable.py 54
1324     assert LEASE_SIZE == 92
1325     DATA_OFFSET = HEADER_SIZE + 4*LEASE_SIZE
1326     assert DATA_OFFSET == 468, DATA_OFFSET
1327+
1328     # our sharefiles share with a recognizable string, plus some random
1329     # binary data to reduce the chance that a regular text file will look
1330     # like a sharefile.
1331hunk ./src/allmydata/storage/backends/disk/mutable.py 63
1332     MAX_SIZE = 2*1000*1000*1000 # 2GB, kind of arbitrary
1333     # TODO: decide upon a policy for max share size
1334 
1335-    def __init__(self, filename, parent=None):
1336-        self.home = filename
1337-        if os.path.exists(self.home):
1338+    def __init__(self, storageindex, shnum, home, parent=None):
1339+        self._storageindex = storageindex
1340+        self._shnum = shnum
1341+        self._home = home
1342+        if self._home.exists():
1343             # we don't cache anything, just check the magic
1344hunk ./src/allmydata/storage/backends/disk/mutable.py 69
1345-            f = open(self.home, 'rb')
1346-            data = f.read(self.HEADER_SIZE)
1347-            (magic,
1348-             write_enabler_nodeid, write_enabler,
1349-             data_length, extra_least_offset) = \
1350-             struct.unpack(">32s20s32sQQ", data)
1351-            if magic != self.MAGIC:
1352-                msg = "sharefile %s had magic '%r' but we wanted '%r'" % \
1353-                      (filename, magic, self.MAGIC)
1354-                raise UnknownMutableContainerVersionError(msg)
1355+            f = self._home.open('rb')
1356+            try:
1357+                data = f.read(self.HEADER_SIZE)
1358+                (magic,
1359+                 write_enabler_nodeid, write_enabler,
1360+                 data_length, extra_least_offset) = \
1361+                 struct.unpack(">32s20s32sQQ", data)
1362+                if magic != self.MAGIC:
1363+                    msg = "sharefile %s had magic '%r' but we wanted '%r'" % \
1364+                          (quote_filepath(self._home), magic, self.MAGIC)
1365+                    raise UnknownMutableContainerVersionError(msg)
1366+            finally:
1367+                f.close()
1368         self.parent = parent # for logging
1369 
1370     def log(self, *args, **kwargs):
1371hunk ./src/allmydata/storage/backends/disk/mutable.py 87
1372         return self.parent.log(*args, **kwargs)
1373 
1374-    def create(self, my_nodeid, write_enabler):
1375-        assert not os.path.exists(self.home)
1376+    def create(self, serverid, write_enabler):
1377+        assert not self._home.exists()
1378         data_length = 0
1379         extra_lease_offset = (self.HEADER_SIZE
1380                               + 4 * self.LEASE_SIZE
1381hunk ./src/allmydata/storage/backends/disk/mutable.py 95
1382                               + data_length)
1383         assert extra_lease_offset == self.DATA_OFFSET # true at creation
1384         num_extra_leases = 0
1385-        f = open(self.home, 'wb')
1386-        header = struct.pack(">32s20s32sQQ",
1387-                             self.MAGIC, my_nodeid, write_enabler,
1388-                             data_length, extra_lease_offset,
1389-                             )
1390-        leases = ("\x00"*self.LEASE_SIZE) * 4
1391-        f.write(header + leases)
1392-        # data goes here, empty after creation
1393-        f.write(struct.pack(">L", num_extra_leases))
1394-        # extra leases go here, none at creation
1395-        f.close()
1396+        f = self._home.open('wb')
1397+        try:
1398+            header = struct.pack(">32s20s32sQQ",
1399+                                 self.MAGIC, serverid, write_enabler,
1400+                                 data_length, extra_lease_offset,
1401+                                 )
1402+            leases = ("\x00"*self.LEASE_SIZE) * 4
1403+            f.write(header + leases)
1404+            # data goes here, empty after creation
1405+            f.write(struct.pack(">L", num_extra_leases))
1406+            # extra leases go here, none at creation
1407+        finally:
1408+            f.close()
1409+
1410+    def __repr__(self):
1411+        return ("<MutableDiskShare %s:%r at %s>"
1412+                % (si_b2a(self._storageindex), self._shnum, quote_filepath(self._home)))
1413+
1414+    def get_used_space(self):
1415+        return fileutil.get_used_space(self._home)
1416+
1417+    def get_storage_index(self):
1418+        return self._storageindex
1419+
1420+    def get_shnum(self):
1421+        return self._shnum
1422 
1423     def unlink(self):
1424hunk ./src/allmydata/storage/backends/disk/mutable.py 123
1425-        os.unlink(self.home)
1426+        self._home.remove()
1427 
1428     def _read_data_length(self, f):
1429         f.seek(self.DATA_LENGTH_OFFSET)
1430hunk ./src/allmydata/storage/backends/disk/mutable.py 291
1431 
1432     def get_leases(self):
1433         """Yields a LeaseInfo instance for all leases."""
1434-        f = open(self.home, 'rb')
1435-        for i, lease in self._enumerate_leases(f):
1436-            yield lease
1437-        f.close()
1438+        f = self._home.open('rb')
1439+        try:
1440+            for i, lease in self._enumerate_leases(f):
1441+                yield lease
1442+        finally:
1443+            f.close()
1444 
1445     def _enumerate_leases(self, f):
1446         for i in range(self._get_num_lease_slots(f)):
1447hunk ./src/allmydata/storage/backends/disk/mutable.py 303
1448             try:
1449                 data = self._read_lease_record(f, i)
1450                 if data is not None:
1451-                    yield i,data
1452+                    yield i, data
1453             except IndexError:
1454                 return
1455 
1456hunk ./src/allmydata/storage/backends/disk/mutable.py 307
1457+    # These lease operations are intended for use by disk_backend.py.
1458+    # Other non-test clients should not depend on the fact that the disk
1459+    # backend stores leases in share files.
1460+
1461     def add_lease(self, lease_info):
1462         precondition(lease_info.owner_num != 0) # 0 means "no lease here"
1463hunk ./src/allmydata/storage/backends/disk/mutable.py 313
1464-        f = open(self.home, 'rb+')
1465-        num_lease_slots = self._get_num_lease_slots(f)
1466-        empty_slot = self._get_first_empty_lease_slot(f)
1467-        if empty_slot is not None:
1468-            self._write_lease_record(f, empty_slot, lease_info)
1469-        else:
1470-            self._write_lease_record(f, num_lease_slots, lease_info)
1471-        f.close()
1472+        f = self._home.open('rb+')
1473+        try:
1474+            num_lease_slots = self._get_num_lease_slots(f)
1475+            empty_slot = self._get_first_empty_lease_slot(f)
1476+            if empty_slot is not None:
1477+                self._write_lease_record(f, empty_slot, lease_info)
1478+            else:
1479+                self._write_lease_record(f, num_lease_slots, lease_info)
1480+        finally:
1481+            f.close()
1482 
1483     def renew_lease(self, renew_secret, new_expire_time):
1484         accepting_nodeids = set()
1485hunk ./src/allmydata/storage/backends/disk/mutable.py 326
1486-        f = open(self.home, 'rb+')
1487-        for (leasenum,lease) in self._enumerate_leases(f):
1488-            if constant_time_compare(lease.renew_secret, renew_secret):
1489-                # yup. See if we need to update the owner time.
1490-                if new_expire_time > lease.expiration_time:
1491-                    # yes
1492-                    lease.expiration_time = new_expire_time
1493-                    self._write_lease_record(f, leasenum, lease)
1494-                f.close()
1495-                return
1496-            accepting_nodeids.add(lease.nodeid)
1497-        f.close()
1498+        f = self._home.open('rb+')
1499+        try:
1500+            for (leasenum, lease) in self._enumerate_leases(f):
1501+                if constant_time_compare(lease.renew_secret, renew_secret):
1502+                    # yup. See if we need to update the owner time.
1503+                    if new_expire_time > lease.expiration_time:
1504+                        # yes
1505+                        lease.expiration_time = new_expire_time
1506+                        self._write_lease_record(f, leasenum, lease)
1507+                    return
1508+                accepting_nodeids.add(lease.nodeid)
1509+        finally:
1510+            f.close()
1511         # Return the accepting_nodeids set, to give the client a chance to
1512hunk ./src/allmydata/storage/backends/disk/mutable.py 340
1513-        # update the leases on a share which has been migrated from its
1514+        # update the leases on a share that has been migrated from its
1515         # original server to a new one.
1516         msg = ("Unable to renew non-existent lease. I have leases accepted by"
1517                " nodeids: ")
1518hunk ./src/allmydata/storage/backends/disk/mutable.py 357
1519         except IndexError:
1520             self.add_lease(lease_info)
1521 
1522-    def cancel_lease(self, cancel_secret):
1523-        """Remove any leases with the given cancel_secret. If the last lease
1524-        is cancelled, the file will be removed. Return the number of bytes
1525-        that were freed (by truncating the list of leases, and possibly by
1526-        deleting the file. Raise IndexError if there was no lease with the
1527-        given cancel_secret."""
1528-
1529-        accepting_nodeids = set()
1530-        modified = 0
1531-        remaining = 0
1532-        blank_lease = LeaseInfo(owner_num=0,
1533-                                renew_secret="\x00"*32,
1534-                                cancel_secret="\x00"*32,
1535-                                expiration_time=0,
1536-                                nodeid="\x00"*20)
1537-        f = open(self.home, 'rb+')
1538-        for (leasenum,lease) in self._enumerate_leases(f):
1539-            accepting_nodeids.add(lease.nodeid)
1540-            if constant_time_compare(lease.cancel_secret, cancel_secret):
1541-                self._write_lease_record(f, leasenum, blank_lease)
1542-                modified += 1
1543-            else:
1544-                remaining += 1
1545-        if modified:
1546-            freed_space = self._pack_leases(f)
1547-            f.close()
1548-            if not remaining:
1549-                freed_space += os.stat(self.home)[stat.ST_SIZE]
1550-                self.unlink()
1551-            return freed_space
1552-
1553-        msg = ("Unable to cancel non-existent lease. I have leases "
1554-               "accepted by nodeids: ")
1555-        msg += ",".join([("'%s'" % idlib.nodeid_b2a(anid))
1556-                         for anid in accepting_nodeids])
1557-        msg += " ."
1558-        raise IndexError(msg)
1559-
1560-    def _pack_leases(self, f):
1561-        # TODO: reclaim space from cancelled leases
1562-        return 0
1563-
1564     def _read_write_enabler_and_nodeid(self, f):
1565         f.seek(0)
1566         data = f.read(self.HEADER_SIZE)
1567hunk ./src/allmydata/storage/backends/disk/mutable.py 369
1568 
1569     def readv(self, readv):
1570         datav = []
1571-        f = open(self.home, 'rb')
1572-        for (offset, length) in readv:
1573-            datav.append(self._read_share_data(f, offset, length))
1574-        f.close()
1575+        f = self._home.open('rb')
1576+        try:
1577+            for (offset, length) in readv:
1578+                datav.append(self._read_share_data(f, offset, length))
1579+        finally:
1580+            f.close()
1581         return datav
1582 
1583hunk ./src/allmydata/storage/backends/disk/mutable.py 377
1584-#    def remote_get_length(self):
1585-#        f = open(self.home, 'rb')
1586-#        data_length = self._read_data_length(f)
1587-#        f.close()
1588-#        return data_length
1589+    def get_size(self):
1590+        return self._home.getsize()
1591+
1592+    def get_data_length(self):
1593+        f = self._home.open('rb')
1594+        try:
1595+            data_length = self._read_data_length(f)
1596+        finally:
1597+            f.close()
1598+        return data_length
1599 
1600     def check_write_enabler(self, write_enabler, si_s):
1601hunk ./src/allmydata/storage/backends/disk/mutable.py 389
1602-        f = open(self.home, 'rb+')
1603-        (real_write_enabler, write_enabler_nodeid) = \
1604-                             self._read_write_enabler_and_nodeid(f)
1605-        f.close()
1606+        f = self._home.open('rb+')
1607+        try:
1608+            (real_write_enabler, write_enabler_nodeid) = self._read_write_enabler_and_nodeid(f)
1609+        finally:
1610+            f.close()
1611         # avoid a timing attack
1612         #if write_enabler != real_write_enabler:
1613         if not constant_time_compare(write_enabler, real_write_enabler):
1614hunk ./src/allmydata/storage/backends/disk/mutable.py 410
1615 
1616     def check_testv(self, testv):
1617         test_good = True
1618-        f = open(self.home, 'rb+')
1619-        for (offset, length, operator, specimen) in testv:
1620-            data = self._read_share_data(f, offset, length)
1621-            if not testv_compare(data, operator, specimen):
1622-                test_good = False
1623-                break
1624-        f.close()
1625+        f = self._home.open('rb+')
1626+        try:
1627+            for (offset, length, operator, specimen) in testv:
1628+                data = self._read_share_data(f, offset, length)
1629+                if not testv_compare(data, operator, specimen):
1630+                    test_good = False
1631+                    break
1632+        finally:
1633+            f.close()
1634         return test_good
1635 
1636     def writev(self, datav, new_length):
1637hunk ./src/allmydata/storage/backends/disk/mutable.py 422
1638-        f = open(self.home, 'rb+')
1639-        for (offset, data) in datav:
1640-            self._write_share_data(f, offset, data)
1641-        if new_length is not None:
1642-            cur_length = self._read_data_length(f)
1643-            if new_length < cur_length:
1644-                self._write_data_length(f, new_length)
1645-                # TODO: if we're going to shrink the share file when the
1646-                # share data has shrunk, then call
1647-                # self._change_container_size() here.
1648-        f.close()
1649-
1650-def testv_compare(a, op, b):
1651-    assert op in ("lt", "le", "eq", "ne", "ge", "gt")
1652-    if op == "lt":
1653-        return a < b
1654-    if op == "le":
1655-        return a <= b
1656-    if op == "eq":
1657-        return a == b
1658-    if op == "ne":
1659-        return a != b
1660-    if op == "ge":
1661-        return a >= b
1662-    if op == "gt":
1663-        return a > b
1664-    # never reached
1665+        f = self._home.open('rb+')
1666+        try:
1667+            for (offset, data) in datav:
1668+                self._write_share_data(f, offset, data)
1669+            if new_length is not None:
1670+                cur_length = self._read_data_length(f)
1671+                if new_length < cur_length:
1672+                    self._write_data_length(f, new_length)
1673+                    # TODO: if we're going to shrink the share file when the
1674+                    # share data has shrunk, then call
1675+                    # self._change_container_size() here.
1676+        finally:
1677+            f.close()
1678 
1679hunk ./src/allmydata/storage/backends/disk/mutable.py 436
1680-class EmptyShare:
1681+    def close(self):
1682+        pass
1683 
1684hunk ./src/allmydata/storage/backends/disk/mutable.py 439
1685-    def check_testv(self, testv):
1686-        test_good = True
1687-        for (offset, length, operator, specimen) in testv:
1688-            data = ""
1689-            if not testv_compare(data, operator, specimen):
1690-                test_good = False
1691-                break
1692-        return test_good
1693 
1694hunk ./src/allmydata/storage/backends/disk/mutable.py 440
1695-def create_mutable_sharefile(filename, my_nodeid, write_enabler, parent):
1696-    ms = MutableShareFile(filename, parent)
1697-    ms.create(my_nodeid, write_enabler)
1698+def create_mutable_disk_share(fp, serverid, write_enabler, parent):
1699+    ms = MutableDiskShare(fp, parent)
1700+    ms.create(serverid, write_enabler)
1701     del ms
1702hunk ./src/allmydata/storage/backends/disk/mutable.py 444
1703-    return MutableShareFile(filename, parent)
1704-
1705+    return MutableDiskShare(fp, parent)
1706addfile ./src/allmydata/storage/backends/null/__init__.py
1707addfile ./src/allmydata/storage/backends/null/null_backend.py
1708hunk ./src/allmydata/storage/backends/null/null_backend.py 2
1709 
1710+import os, struct
1711+
1712+from zope.interface import implements
1713+
1714+from allmydata.interfaces import IStorageBackend, IShareSet, IStoredShare, IStoredMutableShare
1715+from allmydata.util.assertutil import precondition
1716+from allmydata.util.hashutil import constant_time_compare
1717+from allmydata.storage.backends.base import Backend, ShareSet
1718+from allmydata.storage.bucket import BucketWriter
1719+from allmydata.storage.common import si_b2a
1720+from allmydata.storage.lease import LeaseInfo
1721+
1722+
1723+class NullBackend(Backend):
1724+    implements(IStorageBackend)
1725+
1726+    def __init__(self):
1727+        Backend.__init__(self)
1728+
1729+    def get_available_space(self, reserved_space):
1730+        return None
1731+
1732+    def get_sharesets_for_prefix(self, prefix):
1733+        pass
1734+
1735+    def get_shareset(self, storageindex):
1736+        return NullShareSet(storageindex)
1737+
1738+    def fill_in_space_stats(self, stats):
1739+        pass
1740+
1741+    def set_storage_server(self, ss):
1742+        self.ss = ss
1743+
1744+    def advise_corrupt_share(self, sharetype, storageindex, shnum, reason):
1745+        pass
1746+
1747+
1748+class NullShareSet(ShareSet):
1749+    implements(IShareSet)
1750+
1751+    def __init__(self, storageindex):
1752+        self.storageindex = storageindex
1753+
1754+    def get_overhead(self):
1755+        return 0
1756+
1757+    def get_incoming_shnums(self):
1758+        return frozenset()
1759+
1760+    def get_shares(self):
1761+        pass
1762+
1763+    def get_share(self, shnum):
1764+        return None
1765+
1766+    def get_storage_index(self):
1767+        return self.storageindex
1768+
1769+    def get_storage_index_string(self):
1770+        return si_b2a(self.storageindex)
1771+
1772+    def make_bucket_writer(self, storageserver, shnum, max_space_per_bucket, lease_info, canary):
1773+        immutableshare = ImmutableNullShare()
1774+        return BucketWriter(self.ss, immutableshare, max_space_per_bucket, lease_info, canary)
1775+
1776+    def _create_mutable_share(self, storageserver, shnum, write_enabler):
1777+        return MutableNullShare()
1778+
1779+    def _clean_up_after_unlink(self):
1780+        pass
1781+
1782+
1783+class ImmutableNullShare:
1784+    implements(IStoredShare)
1785+    sharetype = "immutable"
1786+
1787+    def __init__(self):
1788+        """ If max_size is not None then I won't allow more than
1789+        max_size to be written to me. If create=True then max_size
1790+        must not be None. """
1791+        pass
1792+
1793+    def get_shnum(self):
1794+        return self.shnum
1795+
1796+    def unlink(self):
1797+        os.unlink(self.fname)
1798+
1799+    def read_share_data(self, offset, length):
1800+        precondition(offset >= 0)
1801+        # Reads beyond the end of the data are truncated. Reads that start
1802+        # beyond the end of the data return an empty string.
1803+        seekpos = self._data_offset+offset
1804+        fsize = os.path.getsize(self.fname)
1805+        actuallength = max(0, min(length, fsize-seekpos)) # XXX #1528
1806+        if actuallength == 0:
1807+            return ""
1808+        f = open(self.fname, 'rb')
1809+        f.seek(seekpos)
1810+        return f.read(actuallength)
1811+
1812+    def write_share_data(self, offset, data):
1813+        pass
1814+
1815+    def _write_lease_record(self, f, lease_number, lease_info):
1816+        offset = self._lease_offset + lease_number * self.LEASE_SIZE
1817+        f.seek(offset)
1818+        assert f.tell() == offset
1819+        f.write(lease_info.to_immutable_data())
1820+
1821+    def _read_num_leases(self, f):
1822+        f.seek(0x08)
1823+        (num_leases,) = struct.unpack(">L", f.read(4))
1824+        return num_leases
1825+
1826+    def _write_num_leases(self, f, num_leases):
1827+        f.seek(0x08)
1828+        f.write(struct.pack(">L", num_leases))
1829+
1830+    def _truncate_leases(self, f, num_leases):
1831+        f.truncate(self._lease_offset + num_leases * self.LEASE_SIZE)
1832+
1833+    def get_leases(self):
1834+        """Yields a LeaseInfo instance for all leases."""
1835+        f = open(self.fname, 'rb')
1836+        (version, unused, num_leases) = struct.unpack(">LLL", f.read(0xc))
1837+        f.seek(self._lease_offset)
1838+        for i in range(num_leases):
1839+            data = f.read(self.LEASE_SIZE)
1840+            if data:
1841+                yield LeaseInfo().from_immutable_data(data)
1842+
1843+    def add_lease(self, lease):
1844+        pass
1845+
1846+    def renew_lease(self, renew_secret, new_expire_time):
1847+        for i,lease in enumerate(self.get_leases()):
1848+            if constant_time_compare(lease.renew_secret, renew_secret):
1849+                # yup. See if we need to update the owner time.
1850+                if new_expire_time > lease.expiration_time:
1851+                    # yes
1852+                    lease.expiration_time = new_expire_time
1853+                    f = open(self.fname, 'rb+')
1854+                    self._write_lease_record(f, i, lease)
1855+                    f.close()
1856+                return
1857+        raise IndexError("unable to renew non-existent lease")
1858+
1859+    def add_or_renew_lease(self, lease_info):
1860+        try:
1861+            self.renew_lease(lease_info.renew_secret,
1862+                             lease_info.expiration_time)
1863+        except IndexError:
1864+            self.add_lease(lease_info)
1865+
1866+
1867+class MutableNullShare:
1868+    implements(IStoredMutableShare)
1869+    sharetype = "mutable"
1870+
1871+    """ XXX: TODO """
1872addfile ./src/allmydata/storage/bucket.py
1873hunk ./src/allmydata/storage/bucket.py 1
1874+
1875+import time
1876+
1877+from foolscap.api import Referenceable
1878+
1879+from zope.interface import implements
1880+from allmydata.interfaces import RIBucketWriter, RIBucketReader
1881+from allmydata.util import base32, log
1882+from allmydata.util.assertutil import precondition
1883+
1884+
1885+class BucketWriter(Referenceable):
1886+    implements(RIBucketWriter)
1887+
1888+    def __init__(self, ss, immutableshare, max_size, lease_info, canary):
1889+        self.ss = ss
1890+        self._max_size = max_size # don't allow the client to write more than this
1891+        self._canary = canary
1892+        self._disconnect_marker = canary.notifyOnDisconnect(self._disconnected)
1893+        self.closed = False
1894+        self.throw_out_all_data = False
1895+        self._share = immutableshare
1896+        # also, add our lease to the file now, so that other ones can be
1897+        # added by simultaneous uploaders
1898+        self._share.add_lease(lease_info)
1899+
1900+    def allocated_size(self):
1901+        return self._max_size
1902+
1903+    def remote_write(self, offset, data):
1904+        start = time.time()
1905+        precondition(not self.closed)
1906+        if self.throw_out_all_data:
1907+            return
1908+        self._share.write_share_data(offset, data)
1909+        self.ss.add_latency("write", time.time() - start)
1910+        self.ss.count("write")
1911+
1912+    def remote_close(self):
1913+        precondition(not self.closed)
1914+        start = time.time()
1915+
1916+        self._share.close()
1917+        filelen = self._share.stat()
1918+        self._share = None
1919+
1920+        self.closed = True
1921+        self._canary.dontNotifyOnDisconnect(self._disconnect_marker)
1922+
1923+        self.ss.bucket_writer_closed(self, filelen)
1924+        self.ss.add_latency("close", time.time() - start)
1925+        self.ss.count("close")
1926+
1927+    def _disconnected(self):
1928+        if not self.closed:
1929+            self._abort()
1930+
1931+    def remote_abort(self):
1932+        log.msg("storage: aborting write to share %r" % self._share,
1933+                facility="tahoe.storage", level=log.UNUSUAL)
1934+        if not self.closed:
1935+            self._canary.dontNotifyOnDisconnect(self._disconnect_marker)
1936+        self._abort()
1937+        self.ss.count("abort")
1938+
1939+    def _abort(self):
1940+        if self.closed:
1941+            return
1942+        self._share.unlink()
1943+        self._share = None
1944+
1945+        # We are now considered closed for further writing. We must tell
1946+        # the storage server about this so that it stops expecting us to
1947+        # use the space it allocated for us earlier.
1948+        self.closed = True
1949+        self.ss.bucket_writer_closed(self, 0)
1950+
1951+
1952+class BucketReader(Referenceable):
1953+    implements(RIBucketReader)
1954+
1955+    def __init__(self, ss, share):
1956+        self.ss = ss
1957+        self._share = share
1958+        self.storageindex = share.storageindex
1959+        self.shnum = share.shnum
1960+
1961+    def __repr__(self):
1962+        return "<%s %s %s>" % (self.__class__.__name__,
1963+                               base32.b2a_l(self.storageindex[:8], 60),
1964+                               self.shnum)
1965+
1966+    def remote_read(self, offset, length):
1967+        start = time.time()
1968+        data = self._share.read_share_data(offset, length)
1969+        self.ss.add_latency("read", time.time() - start)
1970+        self.ss.count("read")
1971+        return data
1972+
1973+    def remote_advise_corrupt_share(self, reason):
1974+        return self.ss.remote_advise_corrupt_share("immutable",
1975+                                                   self.storageindex,
1976+                                                   self.shnum,
1977+                                                   reason)
1978addfile ./src/allmydata/test/test_backends.py
1979hunk ./src/allmydata/test/test_backends.py 1
1980+import os, stat
1981+from twisted.trial import unittest
1982+from allmydata.util.log import msg
1983+from allmydata.test.common_util import ReallyEqualMixin
1984+import mock
1985+
1986+# This is the code that we're going to be testing.
1987+from allmydata.storage.server import StorageServer
1988+from allmydata.storage.backends.disk.disk_backend import DiskBackend, si_si2dir
1989+from allmydata.storage.backends.null.null_backend import NullBackend
1990+
1991+# The following share file content was generated with
1992+# storage.immutable.ShareFile from Tahoe-LAFS v1.8.2
1993+# with share data == 'a'. The total size of this input
1994+# is 85 bytes.
1995+shareversionnumber = '\x00\x00\x00\x01'
1996+sharedatalength = '\x00\x00\x00\x01'
1997+numberofleases = '\x00\x00\x00\x01'
1998+shareinputdata = 'a'
1999+ownernumber = '\x00\x00\x00\x00'
2000+renewsecret  = 'x'*32
2001+cancelsecret = 'y'*32
2002+expirationtime = '\x00(\xde\x80'
2003+nextlease = ''
2004+containerdata = shareversionnumber + sharedatalength + numberofleases
2005+client_data = shareinputdata + ownernumber + renewsecret + \
2006+    cancelsecret + expirationtime + nextlease
2007+share_data = containerdata + client_data
2008+testnodeid = 'testnodeidxxxxxxxxxx'
2009+
2010+
2011+class MockFileSystem(unittest.TestCase):
2012+    """ I simulate a filesystem that the code under test can use. I simulate
2013+    just the parts of the filesystem that the current implementation of Disk
2014+    backend needs. """
2015+    def setUp(self):
2016+        # Make patcher, patch, and effects for disk-using functions.
2017+        msg( "%s.setUp()" % (self,))
2018+        self.mockedfilepaths = {}
2019+        # keys are pathnames, values are MockFilePath objects. This is necessary because
2020+        # MockFilePath behavior sometimes depends on the filesystem. Where it does,
2021+        # self.mockedfilepaths has the relevant information.
2022+        self.storedir = MockFilePath('teststoredir', self.mockedfilepaths)
2023+        self.basedir = self.storedir.child('shares')
2024+        self.baseincdir = self.basedir.child('incoming')
2025+        self.sharedirfinalname = self.basedir.child('or').child('orsxg5dtorxxeylhmvpws3temv4a')
2026+        self.sharedirincomingname = self.baseincdir.child('or').child('orsxg5dtorxxeylhmvpws3temv4a')
2027+        self.shareincomingname = self.sharedirincomingname.child('0')
2028+        self.sharefinalname = self.sharedirfinalname.child('0')
2029+
2030+        # FIXME: these patches won't work; disk_backend no longer imports FilePath, BucketCountingCrawler,
2031+        # or LeaseCheckingCrawler.
2032+
2033+        self.FilePathFake = mock.patch('allmydata.storage.backends.disk.disk_backend.FilePath', new = MockFilePath)
2034+        self.FilePathFake.__enter__()
2035+
2036+        self.BCountingCrawler = mock.patch('allmydata.storage.backends.disk.disk_backend.BucketCountingCrawler')
2037+        FakeBCC = self.BCountingCrawler.__enter__()
2038+        FakeBCC.side_effect = self.call_FakeBCC
2039+
2040+        self.LeaseCheckingCrawler = mock.patch('allmydata.storage.backends.disk.disk_backend.LeaseCheckingCrawler')
2041+        FakeLCC = self.LeaseCheckingCrawler.__enter__()
2042+        FakeLCC.side_effect = self.call_FakeLCC
2043+
2044+        self.get_available_space = mock.patch('allmydata.util.fileutil.get_available_space')
2045+        GetSpace = self.get_available_space.__enter__()
2046+        GetSpace.side_effect = self.call_get_available_space
2047+
2048+        self.statforsize = mock.patch('allmydata.storage.backends.disk.core.filepath.stat')
2049+        getsize = self.statforsize.__enter__()
2050+        getsize.side_effect = self.call_statforsize
2051+
2052+    def call_FakeBCC(self, StateFile):
2053+        return MockBCC()
2054+
2055+    def call_FakeLCC(self, StateFile, HistoryFile, ExpirationPolicy):
2056+        return MockLCC()
2057+
2058+    def call_get_available_space(self, storedir, reservedspace):
2059+        # The input vector has an input size of 85.
2060+        return 85 - reservedspace
2061+
2062+    def call_statforsize(self, fakefpname):
2063+        return self.mockedfilepaths[fakefpname].fileobject.size()
2064+
2065+    def tearDown(self):
2066+        msg( "%s.tearDown()" % (self,))
2067+        self.FilePathFake.__exit__()
2068+        self.mockedfilepaths = {}
2069+
2070+
2071+class MockFilePath:
2072+    def __init__(self, pathstring, ffpathsenvironment, existence=False):
2073+        #  I can't just make the values MockFileObjects because they may be directories.
2074+        self.mockedfilepaths = ffpathsenvironment
2075+        self.path = pathstring
2076+        self.existence = existence
2077+        if not self.mockedfilepaths.has_key(self.path):
2078+            #  The first MockFilePath object is special
2079+            self.mockedfilepaths[self.path] = self
2080+            self.fileobject = None
2081+        else:
2082+            self.fileobject = self.mockedfilepaths[self.path].fileobject
2083+        self.spawn = {}
2084+        self.antecedent = os.path.dirname(self.path)
2085+
2086+    def setContent(self, contentstring):
2087+        # This method rewrites the data in the file that corresponds to its path
2088+        # name whether it preexisted or not.
2089+        self.fileobject = MockFileObject(contentstring)
2090+        self.existence = True
2091+        self.mockedfilepaths[self.path].fileobject = self.fileobject
2092+        self.mockedfilepaths[self.path].existence = self.existence
2093+        self.setparents()
2094+
2095+    def create(self):
2096+        # This method chokes if there's a pre-existing file!
2097+        if self.mockedfilepaths[self.path].fileobject:
2098+            raise OSError
2099+        else:
2100+            self.existence = True
2101+            self.mockedfilepaths[self.path].fileobject = self.fileobject
2102+            self.mockedfilepaths[self.path].existence = self.existence
2103+            self.setparents()
2104+
2105+    def open(self, mode='r'):
2106+        # XXX Makes no use of mode.
2107+        if not self.mockedfilepaths[self.path].fileobject:
2108+            # If there's no fileobject there already then make one and put it there.
2109+            self.fileobject = MockFileObject()
2110+            self.existence = True
2111+            self.mockedfilepaths[self.path].fileobject = self.fileobject
2112+            self.mockedfilepaths[self.path].existence = self.existence
2113+        else:
2114+            # Otherwise get a ref to it.
2115+            self.fileobject = self.mockedfilepaths[self.path].fileobject
2116+            self.existence = self.mockedfilepaths[self.path].existence
2117+        return self.fileobject.open(mode)
2118+
2119+    def child(self, childstring):
2120+        arg2child = os.path.join(self.path, childstring)
2121+        child = MockFilePath(arg2child, self.mockedfilepaths)
2122+        return child
2123+
2124+    def children(self):
2125+        childrenfromffs = [ffp for ffp in self.mockedfilepaths.values() if ffp.path.startswith(self.path)]
2126+        childrenfromffs = [ffp for ffp in childrenfromffs if not ffp.path.endswith(self.path)]
2127+        childrenfromffs = [ffp for ffp in childrenfromffs if ffp.exists()]
2128+        self.spawn = frozenset(childrenfromffs)
2129+        return self.spawn
2130+
2131+    def parent(self):
2132+        if self.mockedfilepaths.has_key(self.antecedent):
2133+            parent = self.mockedfilepaths[self.antecedent]
2134+        else:
2135+            parent = MockFilePath(self.antecedent, self.mockedfilepaths)
2136+        return parent
2137+
2138+    def parents(self):
2139+        antecedents = []
2140+        def f(fps, antecedents):
2141+            newfps = os.path.split(fps)[0]
2142+            if newfps:
2143+                antecedents.append(newfps)
2144+                f(newfps, antecedents)
2145+        f(self.path, antecedents)
2146+        return antecedents
2147+
2148+    def setparents(self):
2149+        for fps in self.parents():
2150+            if not self.mockedfilepaths.has_key(fps):
2151+                self.mockedfilepaths[fps] = MockFilePath(fps, self.mockedfilepaths, exists=True)
2152+
2153+    def basename(self):
2154+        return os.path.split(self.path)[1]
2155+
2156+    def moveTo(self, newffp):
2157+        #  XXX Makes no distinction between file and directory arguments, this is deviation from filepath.moveTo
2158+        if self.mockedfilepaths[newffp.path].exists():
2159+            raise OSError
2160+        else:
2161+            self.mockedfilepaths[newffp.path] = self
2162+            self.path = newffp.path
2163+
2164+    def getsize(self):
2165+        return self.fileobject.getsize()
2166+
2167+    def exists(self):
2168+        return self.existence
2169+
2170+    def isdir(self):
2171+        return True
2172+
2173+    def makedirs(self):
2174+        # XXX These methods assume that fp_<FOO> functions in fileutil will be tested elsewhere!
2175+        pass
2176+
2177+    def remove(self):
2178+        pass
2179+
2180+
2181+class MockFileObject:
2182+    def __init__(self, contentstring=''):
2183+        self.buffer = contentstring
2184+        self.pos = 0
2185+    def open(self, mode='r'):
2186+        return self
2187+    def write(self, instring):
2188+        begin = self.pos
2189+        padlen = begin - len(self.buffer)
2190+        if padlen > 0:
2191+            self.buffer += '\x00' * padlen
2192+        end = self.pos + len(instring)
2193+        self.buffer = self.buffer[:begin]+instring+self.buffer[end:]
2194+        self.pos = end
2195+    def close(self):
2196+        self.pos = 0
2197+    def seek(self, pos):
2198+        self.pos = pos
2199+    def read(self, numberbytes):
2200+        return self.buffer[self.pos:self.pos+numberbytes]
2201+    def tell(self):
2202+        return self.pos
2203+    def size(self):
2204+        # XXX This method A: Is not to be found in a real file B: Is part of a wild-mung-up of filepath.stat!
2205+        # XXX Finally we shall hopefully use a getsize method soon, must consult first though.
2206+        # Hmmm...  perhaps we need to sometimes stat the address when there's not a mockfileobject present?
2207+        return {stat.ST_SIZE:len(self.buffer)}
2208+    def getsize(self):
2209+        return len(self.buffer)
2210+
2211+class MockBCC:
2212+    def setServiceParent(self, Parent):
2213+        pass
2214+
2215+
2216+class MockLCC:
2217+    def setServiceParent(self, Parent):
2218+        pass
2219+
2220+
2221+class TestServerWithNullBackend(unittest.TestCase, ReallyEqualMixin):
2222+    """ NullBackend is just for testing and executable documentation, so
2223+    this test is actually a test of StorageServer in which we're using
2224+    NullBackend as helper code for the test, rather than a test of
2225+    NullBackend. """
2226+    def setUp(self):
2227+        self.ss = StorageServer(testnodeid, NullBackend())
2228+
2229+    @mock.patch('os.mkdir')
2230+    @mock.patch('__builtin__.open')
2231+    @mock.patch('os.listdir')
2232+    @mock.patch('os.path.isdir')
2233+    def test_write_share(self, mockisdir, mocklistdir, mockopen, mockmkdir):
2234+        """
2235+        Write a new share. This tests that StorageServer's remote_allocate_buckets
2236+        generates the correct return types when given test-vector arguments. That
2237+        bs is of the correct type is verified by attempting to invoke remote_write
2238+        on bs[0].
2239+        """
2240+        alreadygot, bs = self.ss.remote_allocate_buckets('teststorage_index', 'x'*32, 'y'*32, set((0,)), 1, mock.Mock())
2241+        bs[0].remote_write(0, 'a')
2242+        self.failIf(mockisdir.called)
2243+        self.failIf(mocklistdir.called)
2244+        self.failIf(mockopen.called)
2245+        self.failIf(mockmkdir.called)
2246+
2247+
2248+class TestServerConstruction(MockFileSystem, ReallyEqualMixin):
2249+    def test_create_server_disk_backend(self):
2250+        """ This tests whether a server instance can be constructed with a
2251+        filesystem backend. To pass the test, it mustn't use the filesystem
2252+        outside of its configured storedir. """
2253+        StorageServer(testnodeid, DiskBackend(self.storedir))
2254+
2255+
2256+class TestServerAndDiskBackend(MockFileSystem, ReallyEqualMixin):
2257+    """ This tests both the StorageServer and the Disk backend together. """
2258+    def setUp(self):
2259+        MockFileSystem.setUp(self)
2260+        try:
2261+            self.backend = DiskBackend(self.storedir)
2262+            self.ss = StorageServer(testnodeid, self.backend)
2263+
2264+            self.backendwithreserve = DiskBackend(self.storedir, reserved_space = 1)
2265+            self.sswithreserve = StorageServer(testnodeid, self.backendwithreserve)
2266+        except:
2267+            MockFileSystem.tearDown(self)
2268+            raise
2269+
2270+    @mock.patch('time.time')
2271+    @mock.patch('allmydata.util.fileutil.get_available_space')
2272+    def test_out_of_space(self, mockget_available_space, mocktime):
2273+        mocktime.return_value = 0
2274+
2275+        def call_get_available_space(dir, reserve):
2276+            return 0
2277+
2278+        mockget_available_space.side_effect = call_get_available_space
2279+        alreadygotc, bsc = self.sswithreserve.remote_allocate_buckets('teststorage_index', 'x'*32, 'y'*32, set((0,)), 1, mock.Mock())
2280+        self.failUnlessReallyEqual(bsc, {})
2281+
2282+    @mock.patch('time.time')
2283+    def test_write_and_read_share(self, mocktime):
2284+        """
2285+        Write a new share, read it, and test the server's (and disk backend's)
2286+        handling of simultaneous and successive attempts to write the same
2287+        share.
2288+        """
2289+        mocktime.return_value = 0
2290+        # Inspect incoming and fail unless it's empty.
2291+        incomingset = self.ss.backend.get_incoming_shnums('teststorage_index')
2292+
2293+        self.failUnlessReallyEqual(incomingset, frozenset())
2294+
2295+        # Populate incoming with the sharenum: 0.
2296+        alreadygot, bs = self.ss.remote_allocate_buckets('teststorage_index', 'x'*32, 'y'*32, frozenset((0,)), 1, mock.Mock())
2297+
2298+        # This is a transparent-box test: Inspect incoming and fail unless the sharenum: 0 is listed there.
2299+        self.failUnlessReallyEqual(self.ss.backend.get_incoming_shnums('teststorage_index'), frozenset((0,)))
2300+
2301+
2302+
2303+        # Attempt to create a second share writer with the same sharenum.
2304+        alreadygota, bsa = self.ss.remote_allocate_buckets('teststorage_index', 'x'*32, 'y'*32, frozenset((0,)), 1, mock.Mock())
2305+
2306+        # Show that no sharewriter results from a remote_allocate_buckets
2307+        # with the same si and sharenum, until BucketWriter.remote_close()
2308+        # has been called.
2309+        self.failIf(bsa)
2310+
2311+        # Test allocated size.
2312+        spaceint = self.ss.allocated_size()
2313+        self.failUnlessReallyEqual(spaceint, 1)
2314+
2315+        # Write 'a' to shnum 0. Only tested together with close and read.
2316+        bs[0].remote_write(0, 'a')
2317+
2318+        # Preclose: Inspect final, failUnless nothing there.
2319+        self.failUnlessReallyEqual(len(list(self.backend.get_shares('teststorage_index'))), 0)
2320+        bs[0].remote_close()
2321+
2322+        # Postclose: (Omnibus) failUnless written data is in final.
2323+        sharesinfinal = list(self.backend.get_shares('teststorage_index'))
2324+        self.failUnlessReallyEqual(len(sharesinfinal), 1)
2325+        contents = sharesinfinal[0].read_share_data(0, 73)
2326+        self.failUnlessReallyEqual(contents, client_data)
2327+
2328+        # Exercise the case that the share we're asking to allocate is
2329+        # already (completely) uploaded.
2330+        self.ss.remote_allocate_buckets('teststorage_index', 'x'*32, 'y'*32, set((0,)), 1, mock.Mock())
2331+
2332+
2333+    def test_read_old_share(self):
2334+        """ This tests whether the code correctly finds and reads
2335+        shares written out by old (Tahoe-LAFS <= v1.8.2)
2336+        servers. There is a similar test in test_download, but that one
2337+        is from the perspective of the client and exercises a deeper
2338+        stack of code. This one is for exercising just the
2339+        StorageServer object. """
2340+        # Contruct a file with the appropriate contents in the mockfilesystem.
2341+        datalen = len(share_data)
2342+        finalhome = si_si2dir(self.basedir, 'teststorage_index').child(str(0))
2343+        finalhome.setContent(share_data)
2344+
2345+        # Now begin the test.
2346+        bs = self.ss.remote_get_buckets('teststorage_index')
2347+
2348+        self.failUnlessEqual(len(bs), 1)
2349+        b = bs['0']
2350+        # These should match by definition, the next two cases cover cases without (completely) unambiguous behaviors.
2351+        self.failUnlessReallyEqual(b.remote_read(0, datalen), client_data)
2352+        # If you try to read past the end you get the as much data as is there.
2353+        self.failUnlessReallyEqual(b.remote_read(0, datalen+20), client_data)
2354+        # If you start reading past the end of the file you get the empty string.
2355+        self.failUnlessReallyEqual(b.remote_read(datalen+1, 3), '')
2356}
2357[Pluggable backends -- all other changes. refs #999
2358david-sarah@jacaranda.org**20110919233256
2359 Ignore-this: 1a77b6b5d178b32a9b914b699ba7e957
2360] {
2361hunk ./src/allmydata/client.py 245
2362             sharetypes.append("immutable")
2363         if self.get_config("storage", "expire.mutable", True, boolean=True):
2364             sharetypes.append("mutable")
2365-        expiration_sharetypes = tuple(sharetypes)
2366 
2367hunk ./src/allmydata/client.py 246
2368+        expiration_policy = {
2369+            'enabled': expire,
2370+            'mode': mode,
2371+            'override_lease_duration': o_l_d,
2372+            'cutoff_date': cutoff_date,
2373+            'sharetypes': tuple(sharetypes),
2374+        }
2375         ss = StorageServer(storedir, self.nodeid,
2376                            reserved_space=reserved,
2377                            discard_storage=discard,
2378hunk ./src/allmydata/client.py 258
2379                            readonly_storage=readonly,
2380                            stats_provider=self.stats_provider,
2381-                           expiration_enabled=expire,
2382-                           expiration_mode=mode,
2383-                           expiration_override_lease_duration=o_l_d,
2384-                           expiration_cutoff_date=cutoff_date,
2385-                           expiration_sharetypes=expiration_sharetypes)
2386+                           expiration_policy=expiration_policy)
2387         self.add_service(ss)
2388 
2389         d = self.when_tub_ready()
2390hunk ./src/allmydata/immutable/offloaded.py 306
2391         if os.path.exists(self._encoding_file):
2392             self.log("ciphertext already present, bypassing fetch",
2393                      level=log.UNUSUAL)
2394+            # XXX the following comment is probably stale, since
2395+            # LocalCiphertextReader.get_plaintext_hashtree_leaves does not exist.
2396+            #
2397             # we'll still need the plaintext hashes (when
2398             # LocalCiphertextReader.get_plaintext_hashtree_leaves() is
2399             # called), and currently the easiest way to get them is to ask
2400hunk ./src/allmydata/immutable/upload.py 765
2401             self._status.set_progress(1, progress)
2402         return cryptdata
2403 
2404-
2405     def get_plaintext_hashtree_leaves(self, first, last, num_segments):
2406hunk ./src/allmydata/immutable/upload.py 766
2407+        """OBSOLETE; Get the leaf nodes of a merkle hash tree over the
2408+        plaintext segments, i.e. get the tagged hashes of the given segments.
2409+        The segment size is expected to be generated by the
2410+        IEncryptedUploadable before any plaintext is read or ciphertext
2411+        produced, so that the segment hashes can be generated with only a
2412+        single pass.
2413+
2414+        This returns a Deferred that fires with a sequence of hashes, using:
2415+
2416+         tuple(segment_hashes[first:last])
2417+
2418+        'num_segments' is used to assert that the number of segments that the
2419+        IEncryptedUploadable handled matches the number of segments that the
2420+        encoder was expecting.
2421+
2422+        This method must not be called until the final byte has been read
2423+        from read_encrypted(). Once this method is called, read_encrypted()
2424+        can never be called again.
2425+        """
2426         # this is currently unused, but will live again when we fix #453
2427         if len(self._plaintext_segment_hashes) < num_segments:
2428             # close out the last one
2429hunk ./src/allmydata/immutable/upload.py 803
2430         return defer.succeed(tuple(self._plaintext_segment_hashes[first:last]))
2431 
2432     def get_plaintext_hash(self):
2433+        """OBSOLETE; Get the hash of the whole plaintext.
2434+
2435+        This returns a Deferred that fires with a tagged SHA-256 hash of the
2436+        whole plaintext, obtained from hashutil.plaintext_hash(data).
2437+        """
2438+        # this is currently unused, but will live again when we fix #453
2439         h = self._plaintext_hasher.digest()
2440         return defer.succeed(h)
2441 
2442hunk ./src/allmydata/interfaces.py 29
2443 Number = IntegerConstraint(8) # 2**(8*8) == 16EiB ~= 18e18 ~= 18 exabytes
2444 Offset = Number
2445 ReadSize = int # the 'int' constraint is 2**31 == 2Gib -- large files are processed in not-so-large increments
2446-WriteEnablerSecret = Hash # used to protect mutable bucket modifications
2447-LeaseRenewSecret = Hash # used to protect bucket lease renewal requests
2448-LeaseCancelSecret = Hash # used to protect bucket lease cancellation requests
2449+WriteEnablerSecret = Hash # used to protect mutable share modifications
2450+LeaseRenewSecret = Hash # used to protect lease renewal requests
2451+LeaseCancelSecret = Hash # used to protect lease cancellation requests
2452 
2453 class RIStubClient(RemoteInterface):
2454     """Each client publishes a service announcement for a dummy object called
2455hunk ./src/allmydata/interfaces.py 106
2456                          sharenums=SetOf(int, maxLength=MAX_BUCKETS),
2457                          allocated_size=Offset, canary=Referenceable):
2458         """
2459-        @param storage_index: the index of the bucket to be created or
2460+        @param storage_index: the index of the shareset to be created or
2461                               increfed.
2462         @param sharenums: these are the share numbers (probably between 0 and
2463                           99) that the sender is proposing to store on this
2464hunk ./src/allmydata/interfaces.py 111
2465                           server.
2466-        @param renew_secret: This is the secret used to protect bucket refresh
2467+        @param renew_secret: This is the secret used to protect lease renewal.
2468                              This secret is generated by the client and
2469                              stored for later comparison by the server. Each
2470                              server is given a different secret.
2471hunk ./src/allmydata/interfaces.py 115
2472-        @param cancel_secret: Like renew_secret, but protects bucket decref.
2473-        @param canary: If the canary is lost before close(), the bucket is
2474+        @param cancel_secret: ignored
2475+        @param canary: If the canary is lost before close(), the allocation is
2476                        deleted.
2477         @return: tuple of (alreadygot, allocated), where alreadygot is what we
2478                  already have and allocated is what we hereby agree to accept.
2479hunk ./src/allmydata/interfaces.py 129
2480                   renew_secret=LeaseRenewSecret,
2481                   cancel_secret=LeaseCancelSecret):
2482         """
2483-        Add a new lease on the given bucket. If the renew_secret matches an
2484+        Add a new lease on the given shareset. If the renew_secret matches an
2485         existing lease, that lease will be renewed instead. If there is no
2486hunk ./src/allmydata/interfaces.py 131
2487-        bucket for the given storage_index, return silently. (note that in
2488+        shareset for the given storage_index, return silently. (Note that in
2489         tahoe-1.3.0 and earlier, IndexError was raised if there was no
2490hunk ./src/allmydata/interfaces.py 133
2491-        bucket)
2492+        shareset.)
2493         """
2494         return Any() # returns None now, but future versions might change
2495 
2496hunk ./src/allmydata/interfaces.py 139
2497     def renew_lease(storage_index=StorageIndex, renew_secret=LeaseRenewSecret):
2498         """
2499-        Renew the lease on a given bucket, resetting the timer to 31 days.
2500-        Some networks will use this, some will not. If there is no bucket for
2501+        Renew the lease on a given shareset, resetting the timer to 31 days.
2502+        Some networks will use this, some will not. If there is no shareset for
2503         the given storage_index, IndexError will be raised.
2504 
2505         For mutable shares, if the given renew_secret does not match an
2506hunk ./src/allmydata/interfaces.py 146
2507         existing lease, IndexError will be raised with a note listing the
2508         server-nodeids on the existing leases, so leases on migrated shares
2509-        can be renewed or cancelled. For immutable shares, IndexError
2510-        (without the note) will be raised.
2511+        can be renewed. For immutable shares, IndexError (without the note)
2512+        will be raised.
2513         """
2514         return Any()
2515 
2516hunk ./src/allmydata/interfaces.py 154
2517     def get_buckets(storage_index=StorageIndex):
2518         return DictOf(int, RIBucketReader, maxKeys=MAX_BUCKETS)
2519 
2520-
2521-
2522     def slot_readv(storage_index=StorageIndex,
2523                    shares=ListOf(int), readv=ReadVector):
2524         """Read a vector from the numbered shares associated with the given
2525hunk ./src/allmydata/interfaces.py 163
2526 
2527     def slot_testv_and_readv_and_writev(storage_index=StorageIndex,
2528                                         secrets=TupleOf(WriteEnablerSecret,
2529-                                                        LeaseRenewSecret,
2530-                                                        LeaseCancelSecret),
2531+                                                        LeaseRenewSecret),
2532                                         tw_vectors=TestAndWriteVectorsForShares,
2533                                         r_vector=ReadVector,
2534                                         ):
2535hunk ./src/allmydata/interfaces.py 167
2536-        """General-purpose test-and-set operation for mutable slots. Perform
2537-        a bunch of comparisons against the existing shares. If they all pass,
2538-        then apply a bunch of write vectors to those shares. Then use the
2539-        read vectors to extract data from all the shares and return the data.
2540+        """
2541+        General-purpose atomic test-read-and-set operation for mutable slots.
2542+        Perform a bunch of comparisons against the existing shares. If they
2543+        all pass: use the read vectors to extract data from all the shares,
2544+        then apply a bunch of write vectors to those shares. Return the read
2545+        data, which does not include any modifications made by the writes.
2546 
2547         This method is, um, large. The goal is to allow clients to update all
2548         the shares associated with a mutable file in a single round trip.
2549hunk ./src/allmydata/interfaces.py 177
2550 
2551-        @param storage_index: the index of the bucket to be created or
2552+        @param storage_index: the index of the shareset to be created or
2553                               increfed.
2554         @param write_enabler: a secret that is stored along with the slot.
2555                               Writes are accepted from any caller who can
2556hunk ./src/allmydata/interfaces.py 183
2557                               present the matching secret. A different secret
2558                               should be used for each slot*server pair.
2559-        @param renew_secret: This is the secret used to protect bucket refresh
2560+        @param renew_secret: This is the secret used to protect lease renewal.
2561                              This secret is generated by the client and
2562                              stored for later comparison by the server. Each
2563                              server is given a different secret.
2564hunk ./src/allmydata/interfaces.py 187
2565-        @param cancel_secret: Like renew_secret, but protects bucket decref.
2566+        @param cancel_secret: ignored
2567 
2568hunk ./src/allmydata/interfaces.py 189
2569-        The 'secrets' argument is a tuple of (write_enabler, renew_secret,
2570-        cancel_secret). The first is required to perform any write. The
2571-        latter two are used when allocating new shares. To simply acquire a
2572-        new lease on existing shares, use an empty testv and an empty writev.
2573+        The 'secrets' argument is a tuple with (write_enabler, renew_secret).
2574+        The write_enabler is required to perform any write. The renew_secret
2575+        is used when allocating new shares.
2576 
2577         Each share can have a separate test vector (i.e. a list of
2578         comparisons to perform). If all vectors for all shares pass, then all
2579hunk ./src/allmydata/interfaces.py 280
2580         store that on disk.
2581         """
2582 
2583-class IStorageBucketWriter(Interface):
2584+
2585+class IStorageBackend(Interface):
2586     """
2587hunk ./src/allmydata/interfaces.py 283
2588-    Objects of this kind live on the client side.
2589+    Objects of this kind live on the server side and are used by the
2590+    storage server object.
2591     """
2592hunk ./src/allmydata/interfaces.py 286
2593-    def put_block(segmentnum=int, data=ShareData):
2594-        """@param data: For most segments, this data will be 'blocksize'
2595-        bytes in length. The last segment might be shorter.
2596-        @return: a Deferred that fires (with None) when the operation completes
2597+    def get_available_space():
2598+        """
2599+        Returns available space for share storage in bytes, or
2600+        None if this information is not available or if the available
2601+        space is unlimited.
2602+
2603+        If the backend is configured for read-only mode then this will
2604+        return 0.
2605+        """
2606+
2607+    def get_sharesets_for_prefix(prefix):
2608+        """
2609+        Generates IShareSet objects for all storage indices matching the
2610+        given prefix for which this backend holds shares.
2611+        """
2612+
2613+    def get_shareset(storageindex):
2614+        """
2615+        Get an IShareSet object for the given storage index.
2616+        """
2617+
2618+    def advise_corrupt_share(storageindex, sharetype, shnum, reason):
2619+        """
2620+        Clients who discover hash failures in shares that they have
2621+        downloaded from me will use this method to inform me about the
2622+        failures. I will record their concern so that my operator can
2623+        manually inspect the shares in question.
2624+
2625+        'sharetype' is either 'mutable' or 'immutable'. 'shnum' is the integer
2626+        share number. 'reason' is a human-readable explanation of the problem,
2627+        probably including some expected hash values and the computed ones
2628+        that did not match. Corruption advisories for mutable shares should
2629+        include a hash of the public key (the same value that appears in the
2630+        mutable-file verify-cap), since the current share format does not
2631+        store that on disk.
2632+
2633+        @param storageindex=str
2634+        @param sharetype=str
2635+        @param shnum=int
2636+        @param reason=str
2637+        """
2638+
2639+
2640+class IShareSet(Interface):
2641+    def get_storage_index():
2642+        """
2643+        Returns the storage index for this shareset.
2644+        """
2645+
2646+    def get_storage_index_string():
2647+        """
2648+        Returns the base32-encoded storage index for this shareset.
2649+        """
2650+
2651+    def get_overhead():
2652+        """
2653+        Returns the storage overhead, in bytes, of this shareset (exclusive
2654+        of the space used by its shares).
2655+        """
2656+
2657+    def get_shares():
2658+        """
2659+        Generates the IStoredShare objects held in this shareset.
2660+        """
2661+
2662+    def has_incoming(shnum):
2663+        """
2664+        Returns True if this shareset has an incoming (partial) share with this number, otherwise False.
2665+        """
2666+
2667+    def make_bucket_writer(storageserver, shnum, max_space_per_bucket, lease_info, canary):
2668+        """
2669+        Create a bucket writer that can be used to write data to a given share.
2670+
2671+        @param storageserver=RIStorageServer
2672+        @param shnum=int: A share number in this shareset
2673+        @param max_space_per_bucket=int: The maximum space allocated for the
2674+                 share, in bytes
2675+        @param lease_info=LeaseInfo: The initial lease information
2676+        @param canary=Referenceable: If the canary is lost before close(), the
2677+                 bucket is deleted.
2678+        @return an IStorageBucketWriter for the given share
2679+        """
2680+
2681+    def make_bucket_reader(storageserver, share):
2682+        """
2683+        Create a bucket reader that can be used to read data from a given share.
2684+
2685+        @param storageserver=RIStorageServer
2686+        @param share=IStoredShare
2687+        @return an IStorageBucketReader for the given share
2688+        """
2689+
2690+    def readv(wanted_shnums, read_vector):
2691+        """
2692+        Read a vector from the numbered shares in this shareset. An empty
2693+        wanted_shnums list means to return data from all known shares.
2694+
2695+        @param wanted_shnums=ListOf(int)
2696+        @param read_vector=ReadVector
2697+        @return DictOf(int, ReadData): shnum -> results, with one key per share
2698+        """
2699+
2700+    def testv_and_readv_and_writev(storageserver, secrets, test_and_write_vectors, read_vector, expiration_time):
2701+        """
2702+        General-purpose atomic test-read-and-set operation for mutable slots.
2703+        Perform a bunch of comparisons against the existing shares in this
2704+        shareset. If they all pass: use the read vectors to extract data from
2705+        all the shares, then apply a bunch of write vectors to those shares.
2706+        Return the read data, which does not include any modifications made by
2707+        the writes.
2708+
2709+        See the similar method in RIStorageServer for more detail.
2710+
2711+        @param storageserver=RIStorageServer
2712+        @param secrets=TupleOf(WriteEnablerSecret, LeaseRenewSecret[, ...])
2713+        @param test_and_write_vectors=TestAndWriteVectorsForShares
2714+        @param read_vector=ReadVector
2715+        @param expiration_time=int
2716+        @return TupleOf(bool, DictOf(int, ReadData))
2717+        """
2718+
2719+    def add_or_renew_lease(lease_info):
2720+        """
2721+        Add a new lease on the shares in this shareset. If the renew_secret
2722+        matches an existing lease, that lease will be renewed instead. If
2723+        there are no shares in this shareset, return silently.
2724+
2725+        @param lease_info=LeaseInfo
2726+        """
2727+
2728+    def renew_lease(renew_secret, new_expiration_time):
2729+        """
2730+        Renew a lease on the shares in this shareset, resetting the timer
2731+        to 31 days. Some grids will use this, some will not. If there are no
2732+        shares in this shareset, IndexError will be raised.
2733+
2734+        For mutable shares, if the given renew_secret does not match an
2735+        existing lease, IndexError will be raised with a note listing the
2736+        server-nodeids on the existing leases, so leases on migrated shares
2737+        can be renewed. For immutable shares, IndexError (without the note)
2738+        will be raised.
2739+
2740+        @param renew_secret=LeaseRenewSecret
2741+        """
2742+
2743+
2744+class IStoredShare(Interface):
2745+    """
2746+    This object contains as much as all of the share data.  It is intended
2747+    for lazy evaluation, such that in many use cases substantially less than
2748+    all of the share data will be accessed.
2749+    """
2750+    def close():
2751+        """
2752+        Complete writing to this share.
2753+        """
2754+
2755+    def get_storage_index():
2756+        """
2757+        Returns the storage index.
2758+        """
2759+
2760+    def get_shnum():
2761+        """
2762+        Returns the share number.
2763+        """
2764+
2765+    def get_data_length():
2766+        """
2767+        Returns the data length in bytes.
2768+        """
2769+
2770+    def get_size():
2771+        """
2772+        Returns the size of the share in bytes.
2773+        """
2774+
2775+    def get_used_space():
2776+        """
2777+        Returns the amount of backend storage including overhead, in bytes, used
2778+        by this share.
2779+        """
2780+
2781+    def unlink():
2782+        """
2783+        Signal that this share can be removed from the backend storage. This does
2784+        not guarantee that the share data will be immediately inaccessible, or
2785+        that it will be securely erased.
2786+        """
2787+
2788+    def readv(read_vector):
2789+        """
2790+        XXX
2791+        """
2792+
2793+
2794+class IStoredMutableShare(IStoredShare):
2795+    def check_write_enabler(write_enabler, si_s):
2796+        """
2797+        XXX
2798         """
2799 
2800hunk ./src/allmydata/interfaces.py 489
2801-    def put_plaintext_hashes(hashes=ListOf(Hash)):
2802+    def check_testv(test_vector):
2803+        """
2804+        XXX
2805+        """
2806+
2807+    def writev(datav, new_length):
2808+        """
2809+        XXX
2810+        """
2811+
2812+
2813+class IStorageBucketWriter(Interface):
2814+    """
2815+    Objects of this kind live on the client side.
2816+    """
2817+    def put_block(segmentnum, data):
2818         """
2819hunk ./src/allmydata/interfaces.py 506
2820+        @param segmentnum=int
2821+        @param data=ShareData: For most segments, this data will be 'blocksize'
2822+        bytes in length. The last segment might be shorter.
2823         @return: a Deferred that fires (with None) when the operation completes
2824         """
2825 
2826hunk ./src/allmydata/interfaces.py 512
2827-    def put_crypttext_hashes(hashes=ListOf(Hash)):
2828+    def put_crypttext_hashes(hashes):
2829         """
2830hunk ./src/allmydata/interfaces.py 514
2831+        @param hashes=ListOf(Hash)
2832         @return: a Deferred that fires (with None) when the operation completes
2833         """
2834 
2835hunk ./src/allmydata/interfaces.py 518
2836-    def put_block_hashes(blockhashes=ListOf(Hash)):
2837+    def put_block_hashes(blockhashes):
2838         """
2839hunk ./src/allmydata/interfaces.py 520
2840+        @param blockhashes=ListOf(Hash)
2841         @return: a Deferred that fires (with None) when the operation completes
2842         """
2843 
2844hunk ./src/allmydata/interfaces.py 524
2845-    def put_share_hashes(sharehashes=ListOf(TupleOf(int, Hash))):
2846+    def put_share_hashes(sharehashes):
2847         """
2848hunk ./src/allmydata/interfaces.py 526
2849+        @param sharehashes=ListOf(TupleOf(int, Hash))
2850         @return: a Deferred that fires (with None) when the operation completes
2851         """
2852 
2853hunk ./src/allmydata/interfaces.py 530
2854-    def put_uri_extension(data=URIExtensionData):
2855+    def put_uri_extension(data):
2856         """This block of data contains integrity-checking information (hashes
2857         of plaintext, crypttext, and shares), as well as encoding parameters
2858         that are necessary to recover the data. This is a serialized dict
2859hunk ./src/allmydata/interfaces.py 535
2860         mapping strings to other strings. The hash of this data is kept in
2861-        the URI and verified before any of the data is used. All buckets for
2862-        a given file contain identical copies of this data.
2863+        the URI and verified before any of the data is used. All share
2864+        containers for a given file contain identical copies of this data.
2865 
2866         The serialization format is specified with the following pseudocode:
2867         for k in sorted(dict.keys()):
2868hunk ./src/allmydata/interfaces.py 543
2869             assert re.match(r'^[a-zA-Z_\-]+$', k)
2870             write(k + ':' + netstring(dict[k]))
2871 
2872+        @param data=URIExtensionData
2873         @return: a Deferred that fires (with None) when the operation completes
2874         """
2875 
2876hunk ./src/allmydata/interfaces.py 558
2877 
2878 class IStorageBucketReader(Interface):
2879 
2880-    def get_block_data(blocknum=int, blocksize=int, size=int):
2881+    def get_block_data(blocknum, blocksize, size):
2882         """Most blocks will be the same size. The last block might be shorter
2883         than the others.
2884 
2885hunk ./src/allmydata/interfaces.py 562
2886+        @param blocknum=int
2887+        @param blocksize=int
2888+        @param size=int
2889         @return: ShareData
2890         """
2891 
2892hunk ./src/allmydata/interfaces.py 573
2893         @return: ListOf(Hash)
2894         """
2895 
2896-    def get_block_hashes(at_least_these=SetOf(int)):
2897+    def get_block_hashes(at_least_these=()):
2898         """
2899hunk ./src/allmydata/interfaces.py 575
2900+        @param at_least_these=SetOf(int)
2901         @return: ListOf(Hash)
2902         """
2903 
2904hunk ./src/allmydata/interfaces.py 579
2905-    def get_share_hashes(at_least_these=SetOf(int)):
2906+    def get_share_hashes():
2907         """
2908         @return: ListOf(TupleOf(int, Hash))
2909         """
2910hunk ./src/allmydata/interfaces.py 611
2911         @return: unicode nickname, or None
2912         """
2913 
2914-    # methods moved from IntroducerClient, need review
2915-    def get_all_connections():
2916-        """Return a frozenset of (nodeid, service_name, rref) tuples, one for
2917-        each active connection we've established to a remote service. This is
2918-        mostly useful for unit tests that need to wait until a certain number
2919-        of connections have been made."""
2920-
2921-    def get_all_connectors():
2922-        """Return a dict that maps from (nodeid, service_name) to a
2923-        RemoteServiceConnector instance for all services that we are actively
2924-        trying to connect to. Each RemoteServiceConnector has the following
2925-        public attributes::
2926-
2927-          service_name: the type of service provided, like 'storage'
2928-          announcement_time: when we first heard about this service
2929-          last_connect_time: when we last established a connection
2930-          last_loss_time: when we last lost a connection
2931-
2932-          version: the peer's version, from the most recent connection
2933-          oldest_supported: the peer's oldest supported version, same
2934-
2935-          rref: the RemoteReference, if connected, otherwise None
2936-          remote_host: the IAddress, if connected, otherwise None
2937-
2938-        This method is intended for monitoring interfaces, such as a web page
2939-        that describes connecting and connected peers.
2940-        """
2941-
2942-    def get_all_peerids():
2943-        """Return a frozenset of all peerids to whom we have a connection (to
2944-        one or more services) established. Mostly useful for unit tests."""
2945-
2946-    def get_all_connections_for(service_name):
2947-        """Return a frozenset of (nodeid, service_name, rref) tuples, one
2948-        for each active connection that provides the given SERVICE_NAME."""
2949-
2950-    def get_permuted_peers(service_name, key):
2951-        """Returns an ordered list of (peerid, rref) tuples, selecting from
2952-        the connections that provide SERVICE_NAME, using a hash-based
2953-        permutation keyed by KEY. This randomizes the service list in a
2954-        repeatable way, to distribute load over many peers.
2955-        """
2956-
2957 
2958 class IMutableSlotWriter(Interface):
2959     """
2960hunk ./src/allmydata/interfaces.py 616
2961     The interface for a writer around a mutable slot on a remote server.
2962     """
2963-    def set_checkstring(checkstring, *args):
2964+    def set_checkstring(seqnum_or_checkstring, root_hash=None, salt=None):
2965         """
2966         Set the checkstring that I will pass to the remote server when
2967         writing.
2968hunk ./src/allmydata/interfaces.py 640
2969         Add a block and salt to the share.
2970         """
2971 
2972-    def put_encprivey(encprivkey):
2973+    def put_encprivkey(encprivkey):
2974         """
2975         Add the encrypted private key to the share.
2976         """
2977hunk ./src/allmydata/interfaces.py 645
2978 
2979-    def put_blockhashes(blockhashes=list):
2980+    def put_blockhashes(blockhashes):
2981         """
2982hunk ./src/allmydata/interfaces.py 647
2983+        @param blockhashes=list
2984         Add the block hash tree to the share.
2985         """
2986 
2987hunk ./src/allmydata/interfaces.py 651
2988-    def put_sharehashes(sharehashes=dict):
2989+    def put_sharehashes(sharehashes):
2990         """
2991hunk ./src/allmydata/interfaces.py 653
2992+        @param sharehashes=dict
2993         Add the share hash chain to the share.
2994         """
2995 
2996hunk ./src/allmydata/interfaces.py 739
2997     def get_extension_params():
2998         """Return the extension parameters in the URI"""
2999 
3000-    def set_extension_params():
3001+    def set_extension_params(params):
3002         """Set the extension parameters that should be in the URI"""
3003 
3004 class IDirectoryURI(Interface):
3005hunk ./src/allmydata/interfaces.py 879
3006         writer-visible data using this writekey.
3007         """
3008 
3009-    # TODO: Can this be overwrite instead of replace?
3010-    def replace(new_contents):
3011-        """Replace the contents of the mutable file, provided that no other
3012+    def overwrite(new_contents):
3013+        """Overwrite the contents of the mutable file, provided that no other
3014         node has published (or is attempting to publish, concurrently) a
3015         newer version of the file than this one.
3016 
3017hunk ./src/allmydata/interfaces.py 1346
3018         is empty, the metadata will be an empty dictionary.
3019         """
3020 
3021-    def set_uri(name, writecap, readcap=None, metadata=None, overwrite=True):
3022+    def set_uri(name, writecap, readcap, metadata=None, overwrite=True):
3023         """I add a child (by writecap+readcap) at the specific name. I return
3024         a Deferred that fires when the operation finishes. If overwrite= is
3025         True, I will replace any existing child of the same name, otherwise
3026hunk ./src/allmydata/interfaces.py 1745
3027     Block Hash, and the encoding parameters, both of which must be included
3028     in the URI.
3029 
3030-    I do not choose shareholders, that is left to the IUploader. I must be
3031-    given a dict of RemoteReferences to storage buckets that are ready and
3032-    willing to receive data.
3033+    I do not choose shareholders, that is left to the IUploader.
3034     """
3035 
3036     def set_size(size):
3037hunk ./src/allmydata/interfaces.py 1752
3038         """Specify the number of bytes that will be encoded. This must be
3039         peformed before get_serialized_params() can be called.
3040         """
3041+
3042     def set_params(params):
3043         """Override the default encoding parameters. 'params' is a tuple of
3044         (k,d,n), where 'k' is the number of required shares, 'd' is the
3045hunk ./src/allmydata/interfaces.py 1848
3046     download, validate, decode, and decrypt data from them, writing the
3047     results to an output file.
3048 
3049-    I do not locate the shareholders, that is left to the IDownloader. I must
3050-    be given a dict of RemoteReferences to storage buckets that are ready to
3051-    send data.
3052+    I do not locate the shareholders, that is left to the IDownloader.
3053     """
3054 
3055     def setup(outfile):
3056hunk ./src/allmydata/interfaces.py 1950
3057         resuming an interrupted upload (where we need to compute the
3058         plaintext hashes, but don't need the redundant encrypted data)."""
3059 
3060-    def get_plaintext_hashtree_leaves(first, last, num_segments):
3061-        """OBSOLETE; Get the leaf nodes of a merkle hash tree over the
3062-        plaintext segments, i.e. get the tagged hashes of the given segments.
3063-        The segment size is expected to be generated by the
3064-        IEncryptedUploadable before any plaintext is read or ciphertext
3065-        produced, so that the segment hashes can be generated with only a
3066-        single pass.
3067-
3068-        This returns a Deferred that fires with a sequence of hashes, using:
3069-
3070-         tuple(segment_hashes[first:last])
3071-
3072-        'num_segments' is used to assert that the number of segments that the
3073-        IEncryptedUploadable handled matches the number of segments that the
3074-        encoder was expecting.
3075-
3076-        This method must not be called until the final byte has been read
3077-        from read_encrypted(). Once this method is called, read_encrypted()
3078-        can never be called again.
3079-        """
3080-
3081-    def get_plaintext_hash():
3082-        """OBSOLETE; Get the hash of the whole plaintext.
3083-
3084-        This returns a Deferred that fires with a tagged SHA-256 hash of the
3085-        whole plaintext, obtained from hashutil.plaintext_hash(data).
3086-        """
3087-
3088     def close():
3089         """Just like IUploadable.close()."""
3090 
3091hunk ./src/allmydata/interfaces.py 2144
3092         returns a Deferred that fires with an IUploadResults instance, from
3093         which the URI of the file can be obtained as results.uri ."""
3094 
3095-    def upload_ssk(write_capability, new_version, uploadable):
3096-        """TODO: how should this work?"""
3097-
3098 class ICheckable(Interface):
3099     def check(monitor, verify=False, add_lease=False):
3100         """Check up on my health, optionally repairing any problems.
3101hunk ./src/allmydata/interfaces.py 2505
3102 
3103 class IRepairResults(Interface):
3104     """I contain the results of a repair operation."""
3105-    def get_successful(self):
3106+    def get_successful():
3107         """Returns a boolean: True if the repair made the file healthy, False
3108         if not. Repair failure generally indicates a file that has been
3109         damaged beyond repair."""
3110hunk ./src/allmydata/interfaces.py 2577
3111     Tahoe process will typically have a single NodeMaker, but unit tests may
3112     create simplified/mocked forms for testing purposes.
3113     """
3114-    def create_from_cap(writecap, readcap=None, **kwargs):
3115+    def create_from_cap(writecap, readcap=None, deep_immutable=False, name=u"<unknown name>"):
3116         """I create an IFilesystemNode from the given writecap/readcap. I can
3117         only provide nodes for existing file/directory objects: use my other
3118         methods to create new objects. I return synchronously."""
3119hunk ./src/allmydata/monitor.py 30
3120 
3121     # the following methods are provided for the operation code
3122 
3123-    def is_cancelled(self):
3124+    def is_cancelled():
3125         """Returns True if the operation has been cancelled. If True,
3126         operation code should stop creating new work, and attempt to stop any
3127         work already in progress."""
3128hunk ./src/allmydata/monitor.py 35
3129 
3130-    def raise_if_cancelled(self):
3131+    def raise_if_cancelled():
3132         """Raise OperationCancelledError if the operation has been cancelled.
3133         Operation code that has a robust error-handling path can simply call
3134         this periodically."""
3135hunk ./src/allmydata/monitor.py 40
3136 
3137-    def set_status(self, status):
3138+    def set_status(status):
3139         """Sets the Monitor's 'status' object to an arbitrary value.
3140         Different operations will store different sorts of status information
3141         here. Operation code should use get+modify+set sequences to update
3142hunk ./src/allmydata/monitor.py 46
3143         this."""
3144 
3145-    def get_status(self):
3146+    def get_status():
3147         """Return the status object. If the operation failed, this will be a
3148         Failure instance."""
3149 
3150hunk ./src/allmydata/monitor.py 50
3151-    def finish(self, status):
3152+    def finish(status):
3153         """Call this when the operation is done, successful or not. The
3154         Monitor's lifetime is influenced by the completion of the operation
3155         it is monitoring. The Monitor's 'status' value will be set with the
3156hunk ./src/allmydata/monitor.py 63
3157 
3158     # the following methods are provided for the initiator of the operation
3159 
3160-    def is_finished(self):
3161+    def is_finished():
3162         """Return a boolean, True if the operation is done (whether
3163         successful or failed), False if it is still running."""
3164 
3165hunk ./src/allmydata/monitor.py 67
3166-    def when_done(self):
3167+    def when_done():
3168         """Return a Deferred that fires when the operation is complete. It
3169         will fire with the operation status, the same value as returned by
3170         get_status()."""
3171hunk ./src/allmydata/monitor.py 72
3172 
3173-    def cancel(self):
3174+    def cancel():
3175         """Cancel the operation as soon as possible. is_cancelled() will
3176         start returning True after this is called."""
3177 
3178hunk ./src/allmydata/mutable/filenode.py 753
3179         self._writekey = writekey
3180         self._serializer = defer.succeed(None)
3181 
3182-
3183     def get_sequence_number(self):
3184         """
3185         Get the sequence number of the mutable version that I represent.
3186hunk ./src/allmydata/mutable/filenode.py 759
3187         """
3188         return self._version[0] # verinfo[0] == the sequence number
3189 
3190+    def get_servermap(self):
3191+        return self._servermap
3192 
3193hunk ./src/allmydata/mutable/filenode.py 762
3194-    # TODO: Terminology?
3195     def get_writekey(self):
3196         """
3197         I return a writekey or None if I don't have a writekey.
3198hunk ./src/allmydata/mutable/filenode.py 768
3199         """
3200         return self._writekey
3201 
3202-
3203     def set_downloader_hints(self, hints):
3204         """
3205         I set the downloader hints.
3206hunk ./src/allmydata/mutable/filenode.py 776
3207 
3208         self._downloader_hints = hints
3209 
3210-
3211     def get_downloader_hints(self):
3212         """
3213         I return the downloader hints.
3214hunk ./src/allmydata/mutable/filenode.py 782
3215         """
3216         return self._downloader_hints
3217 
3218-
3219     def overwrite(self, new_contents):
3220         """
3221         I overwrite the contents of this mutable file version with the
3222hunk ./src/allmydata/mutable/filenode.py 791
3223 
3224         return self._do_serialized(self._overwrite, new_contents)
3225 
3226-
3227     def _overwrite(self, new_contents):
3228         assert IMutableUploadable.providedBy(new_contents)
3229         assert self._servermap.last_update_mode == MODE_WRITE
3230hunk ./src/allmydata/mutable/filenode.py 797
3231 
3232         return self._upload(new_contents)
3233 
3234-
3235     def modify(self, modifier, backoffer=None):
3236         """I use a modifier callback to apply a change to the mutable file.
3237         I implement the following pseudocode::
3238hunk ./src/allmydata/mutable/filenode.py 841
3239 
3240         return self._do_serialized(self._modify, modifier, backoffer)
3241 
3242-
3243     def _modify(self, modifier, backoffer):
3244         if backoffer is None:
3245             backoffer = BackoffAgent().delay
3246hunk ./src/allmydata/mutable/filenode.py 846
3247         return self._modify_and_retry(modifier, backoffer, True)
3248 
3249-
3250     def _modify_and_retry(self, modifier, backoffer, first_time):
3251         """
3252         I try to apply modifier to the contents of this version of the
3253hunk ./src/allmydata/mutable/filenode.py 878
3254         d.addErrback(_retry)
3255         return d
3256 
3257-
3258     def _modify_once(self, modifier, first_time):
3259         """
3260         I attempt to apply a modifier to the contents of the mutable
3261hunk ./src/allmydata/mutable/filenode.py 913
3262         d.addCallback(_apply)
3263         return d
3264 
3265-
3266     def is_readonly(self):
3267         """
3268         I return True if this MutableFileVersion provides no write
3269hunk ./src/allmydata/mutable/filenode.py 921
3270         """
3271         return self._writekey is None
3272 
3273-
3274     def is_mutable(self):
3275         """
3276         I return True, since mutable files are always mutable by
3277hunk ./src/allmydata/mutable/filenode.py 928
3278         """
3279         return True
3280 
3281-
3282     def get_storage_index(self):
3283         """
3284         I return the storage index of the reference that I encapsulate.
3285hunk ./src/allmydata/mutable/filenode.py 934
3286         """
3287         return self._storage_index
3288 
3289-
3290     def get_size(self):
3291         """
3292         I return the length, in bytes, of this readable object.
3293hunk ./src/allmydata/mutable/filenode.py 940
3294         """
3295         return self._servermap.size_of_version(self._version)
3296 
3297-
3298     def download_to_data(self, fetch_privkey=False):
3299         """
3300         I return a Deferred that fires with the contents of this
3301hunk ./src/allmydata/mutable/filenode.py 951
3302         d.addCallback(lambda mc: "".join(mc.chunks))
3303         return d
3304 
3305-
3306     def _try_to_download_data(self):
3307         """
3308         I am an unserialized cousin of download_to_data; I am called
3309hunk ./src/allmydata/mutable/filenode.py 963
3310         d.addCallback(lambda mc: "".join(mc.chunks))
3311         return d
3312 
3313-
3314     def read(self, consumer, offset=0, size=None, fetch_privkey=False):
3315         """
3316         I read a portion (possibly all) of the mutable file that I
3317hunk ./src/allmydata/mutable/filenode.py 971
3318         return self._do_serialized(self._read, consumer, offset, size,
3319                                    fetch_privkey)
3320 
3321-
3322     def _read(self, consumer, offset=0, size=None, fetch_privkey=False):
3323         """
3324         I am the serialized companion of read.
3325hunk ./src/allmydata/mutable/filenode.py 981
3326         d = r.download(consumer, offset, size)
3327         return d
3328 
3329-
3330     def _do_serialized(self, cb, *args, **kwargs):
3331         # note: to avoid deadlock, this callable is *not* allowed to invoke
3332         # other serialized methods within this (or any other)
3333hunk ./src/allmydata/mutable/filenode.py 999
3334         self._serializer.addErrback(log.err)
3335         return d
3336 
3337-
3338     def _upload(self, new_contents):
3339         #assert self._pubkey, "update_servermap must be called before publish"
3340         p = Publish(self._node, self._storage_broker, self._servermap)
3341hunk ./src/allmydata/mutable/filenode.py 1009
3342         d.addCallback(self._did_upload, new_contents.get_size())
3343         return d
3344 
3345-
3346     def _did_upload(self, res, size):
3347         self._most_recent_size = size
3348         return res
3349hunk ./src/allmydata/mutable/filenode.py 1029
3350         """
3351         return self._do_serialized(self._update, data, offset)
3352 
3353-
3354     def _update(self, data, offset):
3355         """
3356         I update the mutable file version represented by this particular
3357hunk ./src/allmydata/mutable/filenode.py 1058
3358         d.addCallback(self._build_uploadable_and_finish, data, offset)
3359         return d
3360 
3361-
3362     def _do_modify_update(self, data, offset):
3363         """
3364         I perform a file update by modifying the contents of the file
3365hunk ./src/allmydata/mutable/filenode.py 1073
3366             return new
3367         return self._modify(m, None)
3368 
3369-
3370     def _do_update_update(self, data, offset):
3371         """
3372         I start the Servermap update that gets us the data we need to
3373hunk ./src/allmydata/mutable/filenode.py 1108
3374         return self._update_servermap(update_range=(start_segment,
3375                                                     end_segment))
3376 
3377-
3378     def _decode_and_decrypt_segments(self, ignored, data, offset):
3379         """
3380         After the servermap update, I take the encrypted and encoded
3381hunk ./src/allmydata/mutable/filenode.py 1148
3382         d3 = defer.succeed(blockhashes)
3383         return deferredutil.gatherResults([d1, d2, d3])
3384 
3385-
3386     def _build_uploadable_and_finish(self, segments_and_bht, data, offset):
3387         """
3388         After the process has the plaintext segments, I build the
3389hunk ./src/allmydata/mutable/filenode.py 1163
3390         p = Publish(self._node, self._storage_broker, self._servermap)
3391         return p.update(u, offset, segments_and_bht[2], self._version)
3392 
3393-
3394     def _update_servermap(self, mode=MODE_WRITE, update_range=None):
3395         """
3396         I update the servermap. I return a Deferred that fires when the
3397hunk ./src/allmydata/storage/common.py 1
3398-
3399-import os.path
3400 from allmydata.util import base32
3401 
3402 class DataTooLargeError(Exception):
3403hunk ./src/allmydata/storage/common.py 5
3404     pass
3405+
3406 class UnknownMutableContainerVersionError(Exception):
3407     pass
3408hunk ./src/allmydata/storage/common.py 8
3409+
3410 class UnknownImmutableContainerVersionError(Exception):
3411     pass
3412 
3413hunk ./src/allmydata/storage/common.py 18
3414 
3415 def si_a2b(ascii_storageindex):
3416     return base32.a2b(ascii_storageindex)
3417-
3418-def storage_index_to_dir(storageindex):
3419-    sia = si_b2a(storageindex)
3420-    return os.path.join(sia[:2], sia)
3421hunk ./src/allmydata/storage/crawler.py 2
3422 
3423-import os, time, struct
3424+import time, struct
3425 import cPickle as pickle
3426 from twisted.internet import reactor
3427 from twisted.application import service
3428hunk ./src/allmydata/storage/crawler.py 6
3429+
3430+from allmydata.util.assertutil import precondition
3431+from allmydata.interfaces import IStorageBackend
3432 from allmydata.storage.common import si_b2a
3433hunk ./src/allmydata/storage/crawler.py 10
3434-from allmydata.util import fileutil
3435+
3436 
3437 class TimeSliceExceeded(Exception):
3438     pass
3439hunk ./src/allmydata/storage/crawler.py 15
3440 
3441+
3442 class ShareCrawler(service.MultiService):
3443hunk ./src/allmydata/storage/crawler.py 17
3444-    """A ShareCrawler subclass is attached to a StorageServer, and
3445-    periodically walks all of its shares, processing each one in some
3446-    fashion. This crawl is rate-limited, to reduce the IO burden on the host,
3447-    since large servers can easily have a terabyte of shares, in several
3448-    million files, which can take hours or days to read.
3449+    """
3450+    An instance of a subclass of ShareCrawler is attached to a storage
3451+    backend, and periodically walks the backend's shares, processing them
3452+    in some fashion. This crawl is rate-limited to reduce the I/O burden on
3453+    the host, since large servers can easily have a terabyte of shares in
3454+    several million files, which can take hours or days to read.
3455 
3456     Once the crawler starts a cycle, it will proceed at a rate limited by the
3457     allowed_cpu_percentage= and cpu_slice= parameters: yielding the reactor
3458hunk ./src/allmydata/storage/crawler.py 33
3459     long enough to ensure that 'minimum_cycle_time' elapses between the start
3460     of two consecutive cycles.
3461 
3462-    We assume that the normal upload/download/get_buckets traffic of a tahoe
3463+    We assume that the normal upload/download/DYHB traffic of a Tahoe-LAFS
3464     grid will cause the prefixdir contents to be mostly cached in the kernel,
3465hunk ./src/allmydata/storage/crawler.py 35
3466-    or that the number of buckets in each prefixdir will be small enough to
3467-    load quickly. A 1TB allmydata.com server was measured to have 2.56M
3468-    buckets, spread into the 1024 prefixdirs, with about 2500 buckets per
3469+    or that the number of sharesets in each prefixdir will be small enough to
3470+    load quickly. A 1TB allmydata.com server was measured to have 2.56 million
3471+    sharesets, spread into the 1024 prefixdirs, with about 2500 sharesets per
3472     prefix. On this server, each prefixdir took 130ms-200ms to list the first
3473     time, and 17ms to list the second time.
3474 
3475hunk ./src/allmydata/storage/crawler.py 41
3476-    To use a crawler, create a subclass which implements the process_bucket()
3477-    method. It will be called with a prefixdir and a base32 storage index
3478-    string. process_bucket() must run synchronously. Any keys added to
3479-    self.state will be preserved. Override add_initial_state() to set up
3480-    initial state keys. Override finished_cycle() to perform additional
3481-    processing when the cycle is complete. Any status that the crawler
3482-    produces should be put in the self.state dictionary. Status renderers
3483-    (like a web page which describes the accomplishments of your crawler)
3484-    will use crawler.get_state() to retrieve this dictionary; they can
3485-    present the contents as they see fit.
3486+    To implement a crawler, create a subclass that implements the
3487+    process_shareset() method. It will be called with a prefixdir and an
3488+    object providing the IShareSet interface. process_shareset() must run
3489+    synchronously. Any keys added to self.state will be preserved. Override
3490+    add_initial_state() to set up initial state keys. Override
3491+    finished_cycle() to perform additional processing when the cycle is
3492+    complete. Any status that the crawler produces should be put in the
3493+    self.state dictionary. Status renderers (like a web page describing the
3494+    accomplishments of your crawler) will use crawler.get_state() to retrieve
3495+    this dictionary; they can present the contents as they see fit.
3496 
3497hunk ./src/allmydata/storage/crawler.py 52
3498-    Then create an instance, with a reference to a StorageServer and a
3499-    filename where it can store persistent state. The statefile is used to
3500-    keep track of how far around the ring the process has travelled, as well
3501-    as timing history to allow the pace to be predicted and controlled. The
3502-    statefile will be updated and written to disk after each time slice (just
3503-    before the crawler yields to the reactor), and also after each cycle is
3504-    finished, and also when stopService() is called. Note that this means
3505-    that a crawler which is interrupted with SIGKILL while it is in the
3506-    middle of a time slice will lose progress: the next time the node is
3507-    started, the crawler will repeat some unknown amount of work.
3508+    Then create an instance, with a reference to a backend object providing
3509+    the IStorageBackend interface, and a filename where it can store
3510+    persistent state. The statefile is used to keep track of how far around
3511+    the ring the process has travelled, as well as timing history to allow
3512+    the pace to be predicted and controlled. The statefile will be updated
3513+    and written to disk after each time slice (just before the crawler yields
3514+    to the reactor), and also after each cycle is finished, and also when
3515+    stopService() is called. Note that this means that a crawler that is
3516+    interrupted with SIGKILL while it is in the middle of a time slice will
3517+    lose progress: the next time the node is started, the crawler will repeat
3518+    some unknown amount of work.
3519 
3520     The crawler instance must be started with startService() before it will
3521hunk ./src/allmydata/storage/crawler.py 65
3522-    do any work. To make it stop doing work, call stopService().
3523+    do any work. To make it stop doing work, call stopService(). A crawler
3524+    is usually a child service of a StorageServer, although it should not
3525+    depend on that.
3526+
3527+    For historical reasons, some dictionary key names use the term "bucket"
3528+    for what is now preferably called a "shareset" (the set of shares that a
3529+    server holds under a given storage index).
3530     """
3531 
3532     slow_start = 300 # don't start crawling for 5 minutes after startup
3533hunk ./src/allmydata/storage/crawler.py 80
3534     cpu_slice = 1.0 # use up to 1.0 seconds before yielding
3535     minimum_cycle_time = 300 # don't run a cycle faster than this
3536 
3537-    def __init__(self, server, statefile, allowed_cpu_percentage=None):
3538+    def __init__(self, backend, statefp, allowed_cpu_percentage=None):
3539+        precondition(IStorageBackend.providedBy(backend), backend)
3540         service.MultiService.__init__(self)
3541hunk ./src/allmydata/storage/crawler.py 83
3542+        self.backend = backend
3543+        self.statefp = statefp
3544         if allowed_cpu_percentage is not None:
3545             self.allowed_cpu_percentage = allowed_cpu_percentage
3546hunk ./src/allmydata/storage/crawler.py 87
3547-        self.server = server
3548-        self.sharedir = server.sharedir
3549-        self.statefile = statefile
3550         self.prefixes = [si_b2a(struct.pack(">H", i << (16-10)))[:2]
3551                          for i in range(2**10)]
3552         self.prefixes.sort()
3553hunk ./src/allmydata/storage/crawler.py 91
3554         self.timer = None
3555-        self.bucket_cache = (None, [])
3556+        self.shareset_cache = (None, [])
3557         self.current_sleep_time = None
3558         self.next_wake_time = None
3559         self.last_prefix_finished_time = None
3560hunk ./src/allmydata/storage/crawler.py 154
3561                 left = len(self.prefixes) - self.last_complete_prefix_index
3562                 remaining = left * self.last_prefix_elapsed_time
3563                 # TODO: remainder of this prefix: we need to estimate the
3564-                # per-bucket time, probably by measuring the time spent on
3565-                # this prefix so far, divided by the number of buckets we've
3566+                # per-shareset time, probably by measuring the time spent on
3567+                # this prefix so far, divided by the number of sharesets we've
3568                 # processed.
3569             d["estimated-cycle-complete-time-left"] = remaining
3570             # it's possible to call get_progress() from inside a crawler's
3571hunk ./src/allmydata/storage/crawler.py 175
3572         state dictionary.
3573 
3574         If we are not currently sleeping (i.e. get_state() was called from
3575-        inside the process_prefixdir, process_bucket, or finished_cycle()
3576+        inside the process_prefixdir, process_shareset, or finished_cycle()
3577         methods, or if startService has not yet been called on this crawler),
3578         these two keys will be None.
3579 
3580hunk ./src/allmydata/storage/crawler.py 188
3581     def load_state(self):
3582         # we use this to store state for both the crawler's internals and
3583         # anything the subclass-specific code needs. The state is stored
3584-        # after each bucket is processed, after each prefixdir is processed,
3585+        # after each shareset is processed, after each prefixdir is processed,
3586         # and after a cycle is complete. The internal keys we use are:
3587         #  ["version"]: int, always 1
3588         #  ["last-cycle-finished"]: int, or None if we have not yet finished
3589hunk ./src/allmydata/storage/crawler.py 202
3590         #                            are sleeping between cycles, or if we
3591         #                            have not yet finished any prefixdir since
3592         #                            a cycle was started
3593-        #  ["last-complete-bucket"]: str, base32 storage index bucket name
3594-        #                            of the last bucket to be processed, or
3595-        #                            None if we are sleeping between cycles
3596+        #  ["last-complete-bucket"]: str, base32 storage index of the last
3597+        #                            shareset to be processed, or None if we
3598+        #                            are sleeping between cycles
3599         try:
3600hunk ./src/allmydata/storage/crawler.py 206
3601-            f = open(self.statefile, "rb")
3602-            state = pickle.load(f)
3603-            f.close()
3604+            state = pickle.loads(self.statefp.getContent())
3605         except EnvironmentError:
3606             state = {"version": 1,
3607                      "last-cycle-finished": None,
3608hunk ./src/allmydata/storage/crawler.py 242
3609         else:
3610             last_complete_prefix = self.prefixes[lcpi]
3611         self.state["last-complete-prefix"] = last_complete_prefix
3612-        tmpfile = self.statefile + ".tmp"
3613-        f = open(tmpfile, "wb")
3614-        pickle.dump(self.state, f)
3615-        f.close()
3616-        fileutil.move_into_place(tmpfile, self.statefile)
3617+        self.statefp.setContent(pickle.dumps(self.state))
3618 
3619     def startService(self):
3620         # arrange things to look like we were just sleeping, so
3621hunk ./src/allmydata/storage/crawler.py 284
3622         sleep_time = (this_slice / self.allowed_cpu_percentage) - this_slice
3623         # if the math gets weird, or a timequake happens, don't sleep
3624         # forever. Note that this means that, while a cycle is running, we
3625-        # will process at least one bucket every 5 minutes, no matter how
3626-        # long that bucket takes.
3627+        # will process at least one shareset every 5 minutes, no matter how
3628+        # long that shareset takes.
3629         sleep_time = max(0.0, min(sleep_time, 299))
3630         if finished_cycle:
3631             # how long should we sleep between cycles? Don't run faster than
3632hunk ./src/allmydata/storage/crawler.py 315
3633         for i in range(self.last_complete_prefix_index+1, len(self.prefixes)):
3634             # if we want to yield earlier, just raise TimeSliceExceeded()
3635             prefix = self.prefixes[i]
3636-            prefixdir = os.path.join(self.sharedir, prefix)
3637-            if i == self.bucket_cache[0]:
3638-                buckets = self.bucket_cache[1]
3639+            if i == self.shareset_cache[0]:
3640+                sharesets = self.shareset_cache[1]
3641             else:
3642hunk ./src/allmydata/storage/crawler.py 318
3643-                try:
3644-                    buckets = os.listdir(prefixdir)
3645-                    buckets.sort()
3646-                except EnvironmentError:
3647-                    buckets = []
3648-                self.bucket_cache = (i, buckets)
3649-            self.process_prefixdir(cycle, prefix, prefixdir,
3650-                                   buckets, start_slice)
3651+                sharesets = self.backend.get_sharesets_for_prefix(prefix)
3652+                self.shareset_cache = (i, sharesets)
3653+            self.process_prefixdir(cycle, prefix, sharesets, start_slice)
3654             self.last_complete_prefix_index = i
3655 
3656             now = time.time()
3657hunk ./src/allmydata/storage/crawler.py 345
3658         self.finished_cycle(cycle)
3659         self.save_state()
3660 
3661-    def process_prefixdir(self, cycle, prefix, prefixdir, buckets, start_slice):
3662-        """This gets a list of bucket names (i.e. storage index strings,
3663+    def process_prefixdir(self, cycle, prefix, sharesets, start_slice):
3664+        """
3665+        This gets a list of shareset names (i.e. storage index strings,
3666         base32-encoded) in sorted order.
3667 
3668         You can override this if your crawler doesn't care about the actual
3669hunk ./src/allmydata/storage/crawler.py 352
3670         shares, for example a crawler which merely keeps track of how many
3671-        buckets are being managed by this server.
3672+        sharesets are being managed by this server.
3673 
3674hunk ./src/allmydata/storage/crawler.py 354
3675-        Subclasses which *do* care about actual bucket should leave this
3676-        method along, and implement process_bucket() instead.
3677+        Subclasses which *do* care about actual shareset should leave this
3678+        method alone, and implement process_shareset() instead.
3679         """
3680 
3681hunk ./src/allmydata/storage/crawler.py 358
3682-        for bucket in buckets:
3683-            if bucket <= self.state["last-complete-bucket"]:
3684+        for shareset in sharesets:
3685+            base32si = shareset.get_storage_index_string()
3686+            if base32si <= self.state["last-complete-bucket"]:
3687                 continue
3688hunk ./src/allmydata/storage/crawler.py 362
3689-            self.process_bucket(cycle, prefix, prefixdir, bucket)
3690-            self.state["last-complete-bucket"] = bucket
3691+            self.process_shareset(cycle, prefix, shareset)
3692+            self.state["last-complete-bucket"] = base32si
3693             if time.time() >= start_slice + self.cpu_slice:
3694                 raise TimeSliceExceeded()
3695 
3696hunk ./src/allmydata/storage/crawler.py 370
3697     # the remaining methods are explictly for subclasses to implement.
3698 
3699     def started_cycle(self, cycle):
3700-        """Notify a subclass that the crawler is about to start a cycle.
3701+        """
3702+        Notify a subclass that the crawler is about to start a cycle.
3703 
3704         This method is for subclasses to override. No upcall is necessary.
3705         """
3706hunk ./src/allmydata/storage/crawler.py 377
3707         pass
3708 
3709-    def process_bucket(self, cycle, prefix, prefixdir, storage_index_b32):
3710-        """Examine a single bucket. Subclasses should do whatever they want
3711+    def process_shareset(self, cycle, prefix, shareset):
3712+        """
3713+        Examine a single shareset. Subclasses should do whatever they want
3714         to do to the shares therein, then update self.state as necessary.
3715 
3716         If the crawler is never interrupted by SIGKILL, this method will be
3717hunk ./src/allmydata/storage/crawler.py 383
3718-        called exactly once per share (per cycle). If it *is* interrupted,
3719+        called exactly once per shareset (per cycle). If it *is* interrupted,
3720         then the next time the node is started, some amount of work will be
3721         duplicated, according to when self.save_state() was last called. By
3722         default, save_state() is called at the end of each timeslice, and
3723hunk ./src/allmydata/storage/crawler.py 391
3724 
3725         To reduce the chance of duplicate work (i.e. to avoid adding multiple
3726         records to a database), you can call save_state() at the end of your
3727-        process_bucket() method. This will reduce the maximum duplicated work
3728-        to one bucket per SIGKILL. It will also add overhead, probably 1-20ms
3729-        per bucket (and some disk writes), which will count against your
3730-        allowed_cpu_percentage, and which may be considerable if
3731-        process_bucket() runs quickly.
3732+        process_shareset() method. This will reduce the maximum duplicated
3733+        work to one shareset per SIGKILL. It will also add overhead, probably
3734+        1-20ms per shareset (and some disk writes), which will count against
3735+        your allowed_cpu_percentage, and which may be considerable if
3736+        process_shareset() runs quickly.
3737 
3738         This method is for subclasses to override. No upcall is necessary.
3739         """
3740hunk ./src/allmydata/storage/crawler.py 402
3741         pass
3742 
3743     def finished_prefix(self, cycle, prefix):
3744-        """Notify a subclass that the crawler has just finished processing a
3745-        prefix directory (all buckets with the same two-character/10bit
3746+        """
3747+        Notify a subclass that the crawler has just finished processing a
3748+        prefix directory (all sharesets with the same two-character/10-bit
3749         prefix). To impose a limit on how much work might be duplicated by a
3750         SIGKILL that occurs during a timeslice, you can call
3751         self.save_state() here, but be aware that it may represent a
3752hunk ./src/allmydata/storage/crawler.py 415
3753         pass
3754 
3755     def finished_cycle(self, cycle):
3756-        """Notify subclass that a cycle (one complete traversal of all
3757+        """
3758+        Notify subclass that a cycle (one complete traversal of all
3759         prefixdirs) has just finished. 'cycle' is the number of the cycle
3760         that just finished. This method should perform summary work and
3761         update self.state to publish information to status displays.
3762hunk ./src/allmydata/storage/crawler.py 433
3763         pass
3764 
3765     def yielding(self, sleep_time):
3766-        """The crawler is about to sleep for 'sleep_time' seconds. This
3767+        """
3768+        The crawler is about to sleep for 'sleep_time' seconds. This
3769         method is mostly for the convenience of unit tests.
3770 
3771         This method is for subclasses to override. No upcall is necessary.
3772hunk ./src/allmydata/storage/crawler.py 443
3773 
3774 
3775 class BucketCountingCrawler(ShareCrawler):
3776-    """I keep track of how many buckets are being managed by this server.
3777-    This is equivalent to the number of distributed files and directories for
3778-    which I am providing storage. The actual number of files+directories in
3779-    the full grid is probably higher (especially when there are more servers
3780-    than 'N', the number of generated shares), because some files+directories
3781-    will have shares on other servers instead of me. Also note that the
3782-    number of buckets will differ from the number of shares in small grids,
3783-    when more than one share is placed on a single server.
3784+    """
3785+    I keep track of how many sharesets, each corresponding to a storage index,
3786+    are being managed by this server. This is equivalent to the number of
3787+    distributed files and directories for which I am providing storage. The
3788+    actual number of files and directories in the full grid is probably higher
3789+    (especially when there are more servers than 'N', the number of generated
3790+    shares), because some files and directories will have shares on other
3791+    servers instead of me. Also note that the number of sharesets will differ
3792+    from the number of shares in small grids, when more than one share is
3793+    placed on a single server.
3794     """
3795 
3796     minimum_cycle_time = 60*60 # we don't need this more than once an hour
3797hunk ./src/allmydata/storage/crawler.py 457
3798 
3799-    def __init__(self, server, statefile, num_sample_prefixes=1):
3800-        ShareCrawler.__init__(self, server, statefile)
3801+    def __init__(self, backend, statefp, num_sample_prefixes=1):
3802+        ShareCrawler.__init__(self, backend, statefp)
3803         self.num_sample_prefixes = num_sample_prefixes
3804 
3805     def add_initial_state(self):
3806hunk ./src/allmydata/storage/crawler.py 471
3807         self.state.setdefault("last-complete-bucket-count", None)
3808         self.state.setdefault("storage-index-samples", {})
3809 
3810-    def process_prefixdir(self, cycle, prefix, prefixdir, buckets, start_slice):
3811+    def process_prefixdir(self, cycle, prefix, sharesets, start_slice):
3812         # we override process_prefixdir() because we don't want to look at
3813hunk ./src/allmydata/storage/crawler.py 473
3814-        # the individual buckets. We'll save state after each one. On my
3815+        # the individual sharesets. We'll save state after each one. On my
3816         # laptop, a mostly-empty storage server can process about 70
3817         # prefixdirs in a 1.0s slice.
3818         if cycle not in self.state["bucket-counts"]:
3819hunk ./src/allmydata/storage/crawler.py 478
3820             self.state["bucket-counts"][cycle] = {}
3821-        self.state["bucket-counts"][cycle][prefix] = len(buckets)
3822+        self.state["bucket-counts"][cycle][prefix] = len(sharesets)
3823         if prefix in self.prefixes[:self.num_sample_prefixes]:
3824hunk ./src/allmydata/storage/crawler.py 480
3825-            self.state["storage-index-samples"][prefix] = (cycle, buckets)
3826+            self.state["storage-index-samples"][prefix] = (cycle, sharesets)
3827 
3828     def finished_cycle(self, cycle):
3829         last_counts = self.state["bucket-counts"].get(cycle, [])
3830hunk ./src/allmydata/storage/crawler.py 486
3831         if len(last_counts) == len(self.prefixes):
3832             # great, we have a whole cycle.
3833-            num_buckets = sum(last_counts.values())
3834-            self.state["last-complete-bucket-count"] = num_buckets
3835+            num_sharesets = sum(last_counts.values())
3836+            self.state["last-complete-bucket-count"] = num_sharesets
3837             # get rid of old counts
3838             for old_cycle in list(self.state["bucket-counts"].keys()):
3839                 if old_cycle != cycle:
3840hunk ./src/allmydata/storage/crawler.py 494
3841                     del self.state["bucket-counts"][old_cycle]
3842         # get rid of old samples too
3843         for prefix in list(self.state["storage-index-samples"].keys()):
3844-            old_cycle,buckets = self.state["storage-index-samples"][prefix]
3845+            old_cycle, storage_indices = self.state["storage-index-samples"][prefix]
3846             if old_cycle != cycle:
3847                 del self.state["storage-index-samples"][prefix]
3848hunk ./src/allmydata/storage/crawler.py 497
3849-
3850hunk ./src/allmydata/storage/expirer.py 1
3851-import time, os, pickle, struct
3852+
3853+import time, pickle, struct
3854+from twisted.python import log as twlog
3855+
3856 from allmydata.storage.crawler import ShareCrawler
3857hunk ./src/allmydata/storage/expirer.py 6
3858-from allmydata.storage.shares import get_share_file
3859-from allmydata.storage.common import UnknownMutableContainerVersionError, \
3860+from allmydata.storage.common import si_b2a, UnknownMutableContainerVersionError, \
3861      UnknownImmutableContainerVersionError
3862hunk ./src/allmydata/storage/expirer.py 8
3863-from twisted.python import log as twlog
3864+
3865 
3866 class LeaseCheckingCrawler(ShareCrawler):
3867     """I examine the leases on all shares, determining which are still valid
3868hunk ./src/allmydata/storage/expirer.py 17
3869     removed.
3870 
3871     I collect statistics on the leases and make these available to a web
3872-    status page, including::
3873+    status page, including:
3874 
3875     Space recovered during this cycle-so-far:
3876      actual (only if expiration_enabled=True):
3877hunk ./src/allmydata/storage/expirer.py 21
3878-      num-buckets, num-shares, sum of share sizes, real disk usage
3879+      num-storage-indices, num-shares, sum of share sizes, real disk usage
3880       ('real disk usage' means we use stat(fn).st_blocks*512 and include any
3881        space used by the directory)
3882      what it would have been with the original lease expiration time
3883hunk ./src/allmydata/storage/expirer.py 32
3884 
3885     Space recovered during the last 10 cycles  <-- saved in separate pickle
3886 
3887-    Shares/buckets examined:
3888+    Shares/storage-indices examined:
3889      this cycle-so-far
3890      prediction of rest of cycle
3891      during last 10 cycles <-- separate pickle
3892hunk ./src/allmydata/storage/expirer.py 42
3893     Histogram of leases-per-share:
3894      this-cycle-to-date
3895      last 10 cycles <-- separate pickle
3896-    Histogram of lease ages, buckets = 1day
3897+    Histogram of lease ages, storage-indices over 1 day
3898      cycle-to-date
3899      last 10 cycles <-- separate pickle
3900 
3901hunk ./src/allmydata/storage/expirer.py 53
3902     slow_start = 360 # wait 6 minutes after startup
3903     minimum_cycle_time = 12*60*60 # not more than twice per day
3904 
3905-    def __init__(self, server, statefile, historyfile,
3906-                 expiration_enabled, mode,
3907-                 override_lease_duration, # used if expiration_mode=="age"
3908-                 cutoff_date, # used if expiration_mode=="cutoff-date"
3909-                 sharetypes):
3910-        self.historyfile = historyfile
3911-        self.expiration_enabled = expiration_enabled
3912-        self.mode = mode
3913+    def __init__(self, backend, statefp, historyfp, expiration_policy):
3914+        # ShareCrawler.__init__ will call add_initial_state, so self.historyfp has to be set first.
3915+        self.historyfp = historyfp
3916+        ShareCrawler.__init__(self, backend, statefp)
3917+
3918+        self.expiration_enabled = expiration_policy['enabled']
3919+        self.mode = expiration_policy['mode']
3920         self.override_lease_duration = None
3921         self.cutoff_date = None
3922         if self.mode == "age":
3923hunk ./src/allmydata/storage/expirer.py 63
3924-            assert isinstance(override_lease_duration, (int, type(None)))
3925-            self.override_lease_duration = override_lease_duration # seconds
3926+            assert isinstance(expiration_policy['override_lease_duration'], (int, type(None)))
3927+            self.override_lease_duration = expiration_policy['override_lease_duration'] # seconds
3928         elif self.mode == "cutoff-date":
3929hunk ./src/allmydata/storage/expirer.py 66
3930-            assert isinstance(cutoff_date, int) # seconds-since-epoch
3931-            assert cutoff_date is not None
3932-            self.cutoff_date = cutoff_date
3933+            assert isinstance(expiration_policy['cutoff_date'], int) # seconds-since-epoch
3934+            self.cutoff_date = expiration_policy['cutoff_date']
3935         else:
3936hunk ./src/allmydata/storage/expirer.py 69
3937-            raise ValueError("GC mode '%s' must be 'age' or 'cutoff-date'" % mode)
3938-        self.sharetypes_to_expire = sharetypes
3939-        ShareCrawler.__init__(self, server, statefile)
3940+            raise ValueError("GC mode '%s' must be 'age' or 'cutoff-date'" % expiration_policy['mode'])
3941+        self.sharetypes_to_expire = expiration_policy['sharetypes']
3942 
3943     def add_initial_state(self):
3944         # we fill ["cycle-to-date"] here (even though they will be reset in
3945hunk ./src/allmydata/storage/expirer.py 84
3946             self.state["cycle-to-date"].setdefault(k, so_far[k])
3947 
3948         # initialize history
3949-        if not os.path.exists(self.historyfile):
3950+        if not self.historyfp.exists():
3951             history = {} # cyclenum -> dict
3952hunk ./src/allmydata/storage/expirer.py 86
3953-            f = open(self.historyfile, "wb")
3954-            pickle.dump(history, f)
3955-            f.close()
3956+            self.historyfp.setContent(pickle.dumps(history))
3957 
3958     def create_empty_cycle_dict(self):
3959         recovered = self.create_empty_recovered_dict()
3960hunk ./src/allmydata/storage/expirer.py 99
3961 
3962     def create_empty_recovered_dict(self):
3963         recovered = {}
3964+        # "buckets" is ambiguous; here it means the number of sharesets (one per storage index per server)
3965         for a in ("actual", "original", "configured", "examined"):
3966             for b in ("buckets", "shares", "sharebytes", "diskbytes"):
3967                 recovered[a+"-"+b] = 0
3968hunk ./src/allmydata/storage/expirer.py 110
3969     def started_cycle(self, cycle):
3970         self.state["cycle-to-date"] = self.create_empty_cycle_dict()
3971 
3972-    def stat(self, fn):
3973-        return os.stat(fn)
3974-
3975-    def process_bucket(self, cycle, prefix, prefixdir, storage_index_b32):
3976-        bucketdir = os.path.join(prefixdir, storage_index_b32)
3977-        s = self.stat(bucketdir)
3978+    def process_storage_index(self, cycle, prefix, container):
3979         would_keep_shares = []
3980         wks = None
3981hunk ./src/allmydata/storage/expirer.py 113
3982+        sharetype = None
3983 
3984hunk ./src/allmydata/storage/expirer.py 115
3985-        for fn in os.listdir(bucketdir):
3986-            try:
3987-                shnum = int(fn)
3988-            except ValueError:
3989-                continue # non-numeric means not a sharefile
3990-            sharefile = os.path.join(bucketdir, fn)
3991+        for share in container.get_shares():
3992+            sharetype = share.sharetype
3993             try:
3994hunk ./src/allmydata/storage/expirer.py 118
3995-                wks = self.process_share(sharefile)
3996+                wks = self.process_share(share)
3997             except (UnknownMutableContainerVersionError,
3998                     UnknownImmutableContainerVersionError,
3999                     struct.error):
4000hunk ./src/allmydata/storage/expirer.py 122
4001-                twlog.msg("lease-checker error processing %s" % sharefile)
4002+                twlog.msg("lease-checker error processing %r" % (share,))
4003                 twlog.err()
4004hunk ./src/allmydata/storage/expirer.py 124
4005-                which = (storage_index_b32, shnum)
4006+                which = (si_b2a(share.storageindex), share.get_shnum())
4007                 self.state["cycle-to-date"]["corrupt-shares"].append(which)
4008                 wks = (1, 1, 1, "unknown")
4009             would_keep_shares.append(wks)
4010hunk ./src/allmydata/storage/expirer.py 129
4011 
4012-        sharetype = None
4013+        container_type = None
4014         if wks:
4015hunk ./src/allmydata/storage/expirer.py 131
4016-            # use the last share's sharetype as the buckettype
4017-            sharetype = wks[3]
4018+            # use the last share's sharetype as the container type
4019+            container_type = wks[3]
4020         rec = self.state["cycle-to-date"]["space-recovered"]
4021         self.increment(rec, "examined-buckets", 1)
4022         if sharetype:
4023hunk ./src/allmydata/storage/expirer.py 136
4024-            self.increment(rec, "examined-buckets-"+sharetype, 1)
4025+            self.increment(rec, "examined-buckets-"+container_type, 1)
4026+
4027+        container_diskbytes = container.get_overhead()
4028 
4029hunk ./src/allmydata/storage/expirer.py 140
4030-        try:
4031-            bucket_diskbytes = s.st_blocks * 512
4032-        except AttributeError:
4033-            bucket_diskbytes = 0 # no stat().st_blocks on windows
4034         if sum([wks[0] for wks in would_keep_shares]) == 0:
4035hunk ./src/allmydata/storage/expirer.py 141
4036-            self.increment_bucketspace("original", bucket_diskbytes, sharetype)
4037+            self.increment_container_space("original", container_diskbytes, sharetype)
4038         if sum([wks[1] for wks in would_keep_shares]) == 0:
4039hunk ./src/allmydata/storage/expirer.py 143
4040-            self.increment_bucketspace("configured", bucket_diskbytes, sharetype)
4041+            self.increment_container_space("configured", container_diskbytes, sharetype)
4042         if sum([wks[2] for wks in would_keep_shares]) == 0:
4043hunk ./src/allmydata/storage/expirer.py 145
4044-            self.increment_bucketspace("actual", bucket_diskbytes, sharetype)
4045+            self.increment_container_space("actual", container_diskbytes, sharetype)
4046 
4047hunk ./src/allmydata/storage/expirer.py 147
4048-    def process_share(self, sharefilename):
4049-        # first, find out what kind of a share it is
4050-        sf = get_share_file(sharefilename)
4051-        sharetype = sf.sharetype
4052+    def process_share(self, share):
4053+        sharetype = share.sharetype
4054         now = time.time()
4055hunk ./src/allmydata/storage/expirer.py 150
4056-        s = self.stat(sharefilename)
4057+        sharebytes = share.get_size()
4058+        diskbytes = share.get_used_space()
4059 
4060         num_leases = 0
4061         num_valid_leases_original = 0
4062hunk ./src/allmydata/storage/expirer.py 158
4063         num_valid_leases_configured = 0
4064         expired_leases_configured = []
4065 
4066-        for li in sf.get_leases():
4067+        for li in share.get_leases():
4068             num_leases += 1
4069             original_expiration_time = li.get_expiration_time()
4070             grant_renew_time = li.get_grant_renew_time_time()
4071hunk ./src/allmydata/storage/expirer.py 171
4072 
4073             #  expired-or-not according to our configured age limit
4074             expired = False
4075-            if self.mode == "age":
4076-                age_limit = original_expiration_time
4077-                if self.override_lease_duration is not None:
4078-                    age_limit = self.override_lease_duration
4079-                if age > age_limit:
4080-                    expired = True
4081-            else:
4082-                assert self.mode == "cutoff-date"
4083-                if grant_renew_time < self.cutoff_date:
4084-                    expired = True
4085-            if sharetype not in self.sharetypes_to_expire:
4086-                expired = False
4087+            if sharetype in self.sharetypes_to_expire:
4088+                if self.mode == "age":
4089+                    age_limit = original_expiration_time
4090+                    if self.override_lease_duration is not None:
4091+                        age_limit = self.override_lease_duration
4092+                    if age > age_limit:
4093+                        expired = True
4094+                else:
4095+                    assert self.mode == "cutoff-date"
4096+                    if grant_renew_time < self.cutoff_date:
4097+                        expired = True
4098 
4099             if expired:
4100                 expired_leases_configured.append(li)
4101hunk ./src/allmydata/storage/expirer.py 190
4102 
4103         so_far = self.state["cycle-to-date"]
4104         self.increment(so_far["leases-per-share-histogram"], num_leases, 1)
4105-        self.increment_space("examined", s, sharetype)
4106+        self.increment_space("examined", diskbytes, sharetype)
4107 
4108         would_keep_share = [1, 1, 1, sharetype]
4109 
4110hunk ./src/allmydata/storage/expirer.py 196
4111         if self.expiration_enabled:
4112             for li in expired_leases_configured:
4113-                sf.cancel_lease(li.cancel_secret)
4114+                share.cancel_lease(li.cancel_secret)
4115 
4116         if num_valid_leases_original == 0:
4117             would_keep_share[0] = 0
4118hunk ./src/allmydata/storage/expirer.py 200
4119-            self.increment_space("original", s, sharetype)
4120+            self.increment_space("original", sharebytes, diskbytes, sharetype)
4121 
4122         if num_valid_leases_configured == 0:
4123             would_keep_share[1] = 0
4124hunk ./src/allmydata/storage/expirer.py 204
4125-            self.increment_space("configured", s, sharetype)
4126+            self.increment_space("configured", sharebytes, diskbytes, sharetype)
4127             if self.expiration_enabled:
4128                 would_keep_share[2] = 0
4129hunk ./src/allmydata/storage/expirer.py 207
4130-                self.increment_space("actual", s, sharetype)
4131+                self.increment_space("actual", sharebytes, diskbytes, sharetype)
4132 
4133         return would_keep_share
4134 
4135hunk ./src/allmydata/storage/expirer.py 211
4136-    def increment_space(self, a, s, sharetype):
4137-        sharebytes = s.st_size
4138-        try:
4139-            # note that stat(2) says that st_blocks is 512 bytes, and that
4140-            # st_blksize is "optimal file sys I/O ops blocksize", which is
4141-            # independent of the block-size that st_blocks uses.
4142-            diskbytes = s.st_blocks * 512
4143-        except AttributeError:
4144-            # the docs say that st_blocks is only on linux. I also see it on
4145-            # MacOS. But it isn't available on windows.
4146-            diskbytes = sharebytes
4147+    def increment_space(self, a, sharebytes, diskbytes, sharetype):
4148         so_far_sr = self.state["cycle-to-date"]["space-recovered"]
4149         self.increment(so_far_sr, a+"-shares", 1)
4150         self.increment(so_far_sr, a+"-sharebytes", sharebytes)
4151hunk ./src/allmydata/storage/expirer.py 221
4152             self.increment(so_far_sr, a+"-sharebytes-"+sharetype, sharebytes)
4153             self.increment(so_far_sr, a+"-diskbytes-"+sharetype, diskbytes)
4154 
4155-    def increment_bucketspace(self, a, bucket_diskbytes, sharetype):
4156+    def increment_container_space(self, a, container_diskbytes, container_type):
4157         rec = self.state["cycle-to-date"]["space-recovered"]
4158hunk ./src/allmydata/storage/expirer.py 223
4159-        self.increment(rec, a+"-diskbytes", bucket_diskbytes)
4160+        self.increment(rec, a+"-diskbytes", container_diskbytes)
4161         self.increment(rec, a+"-buckets", 1)
4162hunk ./src/allmydata/storage/expirer.py 225
4163-        if sharetype:
4164-            self.increment(rec, a+"-diskbytes-"+sharetype, bucket_diskbytes)
4165-            self.increment(rec, a+"-buckets-"+sharetype, 1)
4166+        if container_type:
4167+            self.increment(rec, a+"-diskbytes-"+container_type, container_diskbytes)
4168+            self.increment(rec, a+"-buckets-"+container_type, 1)
4169 
4170     def increment(self, d, k, delta=1):
4171         if k not in d:
4172hunk ./src/allmydata/storage/expirer.py 281
4173         # copy() needs to become a deepcopy
4174         h["space-recovered"] = s["space-recovered"].copy()
4175 
4176-        history = pickle.load(open(self.historyfile, "rb"))
4177+        history = pickle.load(self.historyfp.getContent())
4178         history[cycle] = h
4179         while len(history) > 10:
4180             oldcycles = sorted(history.keys())
4181hunk ./src/allmydata/storage/expirer.py 286
4182             del history[oldcycles[0]]
4183-        f = open(self.historyfile, "wb")
4184-        pickle.dump(history, f)
4185-        f.close()
4186+        self.historyfp.setContent(pickle.dumps(history))
4187 
4188     def get_state(self):
4189         """In addition to the crawler state described in
4190hunk ./src/allmydata/storage/expirer.py 355
4191         progress = self.get_progress()
4192 
4193         state = ShareCrawler.get_state(self) # does a shallow copy
4194-        history = pickle.load(open(self.historyfile, "rb"))
4195+        history = pickle.load(self.historyfp.getContent())
4196         state["history"] = history
4197 
4198         if not progress["cycle-in-progress"]:
4199hunk ./src/allmydata/storage/lease.py 3
4200 import struct, time
4201 
4202+
4203+class NonExistentLeaseError(Exception):
4204+    pass
4205+
4206 class LeaseInfo:
4207     def __init__(self, owner_num=None, renew_secret=None, cancel_secret=None,
4208                  expiration_time=None, nodeid=None):
4209hunk ./src/allmydata/storage/lease.py 21
4210 
4211     def get_expiration_time(self):
4212         return self.expiration_time
4213+
4214     def get_grant_renew_time_time(self):
4215         # hack, based upon fixed 31day expiration period
4216         return self.expiration_time - 31*24*60*60
4217hunk ./src/allmydata/storage/lease.py 25
4218+
4219     def get_age(self):
4220         return time.time() - self.get_grant_renew_time_time()
4221 
4222hunk ./src/allmydata/storage/lease.py 36
4223          self.expiration_time) = struct.unpack(">L32s32sL", data)
4224         self.nodeid = None
4225         return self
4226+
4227     def to_immutable_data(self):
4228         return struct.pack(">L32s32sL",
4229                            self.owner_num,
4230hunk ./src/allmydata/storage/lease.py 49
4231                            int(self.expiration_time),
4232                            self.renew_secret, self.cancel_secret,
4233                            self.nodeid)
4234+
4235     def from_mutable_data(self, data):
4236         (self.owner_num,
4237          self.expiration_time,
4238hunk ./src/allmydata/storage/server.py 1
4239-import os, re, weakref, struct, time
4240+import weakref, time
4241 
4242 from foolscap.api import Referenceable
4243 from twisted.application import service
4244hunk ./src/allmydata/storage/server.py 7
4245 
4246 from zope.interface import implements
4247-from allmydata.interfaces import RIStorageServer, IStatsProducer
4248-from allmydata.util import fileutil, idlib, log, time_format
4249+from allmydata.interfaces import RIStorageServer, IStatsProducer, IStorageBackend
4250+from allmydata.util.assertutil import precondition
4251+from allmydata.util import idlib, log
4252 import allmydata # for __full_version__
4253 
4254hunk ./src/allmydata/storage/server.py 12
4255-from allmydata.storage.common import si_b2a, si_a2b, storage_index_to_dir
4256-_pyflakes_hush = [si_b2a, si_a2b, storage_index_to_dir] # re-exported
4257+from allmydata.storage.common import si_a2b, si_b2a
4258+[si_a2b]  # hush pyflakes
4259 from allmydata.storage.lease import LeaseInfo
4260hunk ./src/allmydata/storage/server.py 15
4261-from allmydata.storage.mutable import MutableShareFile, EmptyShare, \
4262-     create_mutable_sharefile
4263-from allmydata.storage.immutable import ShareFile, BucketWriter, BucketReader
4264-from allmydata.storage.crawler import BucketCountingCrawler
4265 from allmydata.storage.expirer import LeaseCheckingCrawler
4266hunk ./src/allmydata/storage/server.py 16
4267-
4268-# storage/
4269-# storage/shares/incoming
4270-#   incoming/ holds temp dirs named $START/$STORAGEINDEX/$SHARENUM which will
4271-#   be moved to storage/shares/$START/$STORAGEINDEX/$SHARENUM upon success
4272-# storage/shares/$START/$STORAGEINDEX
4273-# storage/shares/$START/$STORAGEINDEX/$SHARENUM
4274-
4275-# Where "$START" denotes the first 10 bits worth of $STORAGEINDEX (that's 2
4276-# base-32 chars).
4277-
4278-# $SHARENUM matches this regex:
4279-NUM_RE=re.compile("^[0-9]+$")
4280-
4281+from allmydata.storage.crawler import BucketCountingCrawler
4282 
4283 
4284 class StorageServer(service.MultiService, Referenceable):
4285hunk ./src/allmydata/storage/server.py 21
4286     implements(RIStorageServer, IStatsProducer)
4287+
4288     name = 'storage'
4289     LeaseCheckerClass = LeaseCheckingCrawler
4290hunk ./src/allmydata/storage/server.py 24
4291+    DEFAULT_EXPIRATION_POLICY = {
4292+        'enabled': False,
4293+        'mode': 'age',
4294+        'override_lease_duration': None,
4295+        'cutoff_date': None,
4296+        'sharetypes': ('mutable', 'immutable'),
4297+    }
4298 
4299hunk ./src/allmydata/storage/server.py 32
4300-    def __init__(self, storedir, nodeid, reserved_space=0,
4301-                 discard_storage=False, readonly_storage=False,
4302+    def __init__(self, serverid, backend, statedir,
4303                  stats_provider=None,
4304hunk ./src/allmydata/storage/server.py 34
4305-                 expiration_enabled=False,
4306-                 expiration_mode="age",
4307-                 expiration_override_lease_duration=None,
4308-                 expiration_cutoff_date=None,
4309-                 expiration_sharetypes=("mutable", "immutable")):
4310+                 expiration_policy=None):
4311         service.MultiService.__init__(self)
4312hunk ./src/allmydata/storage/server.py 36
4313-        assert isinstance(nodeid, str)
4314-        assert len(nodeid) == 20
4315-        self.my_nodeid = nodeid
4316-        self.storedir = storedir
4317-        sharedir = os.path.join(storedir, "shares")
4318-        fileutil.make_dirs(sharedir)
4319-        self.sharedir = sharedir
4320-        # we don't actually create the corruption-advisory dir until necessary
4321-        self.corruption_advisory_dir = os.path.join(storedir,
4322-                                                    "corruption-advisories")
4323-        self.reserved_space = int(reserved_space)
4324-        self.no_storage = discard_storage
4325-        self.readonly_storage = readonly_storage
4326+        precondition(IStorageBackend.providedBy(backend), backend)
4327+        precondition(isinstance(serverid, str), serverid)
4328+        precondition(len(serverid) == 20, serverid)
4329+
4330+        self._serverid = serverid
4331         self.stats_provider = stats_provider
4332         if self.stats_provider:
4333             self.stats_provider.register_producer(self)
4334hunk ./src/allmydata/storage/server.py 44
4335-        self.incomingdir = os.path.join(sharedir, 'incoming')
4336-        self._clean_incomplete()
4337-        fileutil.make_dirs(self.incomingdir)
4338         self._active_writers = weakref.WeakKeyDictionary()
4339hunk ./src/allmydata/storage/server.py 45
4340+        self.backend = backend
4341+        self.backend.setServiceParent(self)
4342+        self._statedir = statedir
4343         log.msg("StorageServer created", facility="tahoe.storage")
4344 
4345hunk ./src/allmydata/storage/server.py 50
4346-        if reserved_space:
4347-            if self.get_available_space() is None:
4348-                log.msg("warning: [storage]reserved_space= is set, but this platform does not support an API to get disk statistics (statvfs(2) or GetDiskFreeSpaceEx), so this reservation cannot be honored",
4349-                        umin="0wZ27w", level=log.UNUSUAL)
4350-
4351         self.latencies = {"allocate": [], # immutable
4352                           "write": [],
4353                           "close": [],
4354hunk ./src/allmydata/storage/server.py 61
4355                           "renew": [],
4356                           "cancel": [],
4357                           }
4358-        self.add_bucket_counter()
4359-
4360-        statefile = os.path.join(self.storedir, "lease_checker.state")
4361-        historyfile = os.path.join(self.storedir, "lease_checker.history")
4362-        klass = self.LeaseCheckerClass
4363-        self.lease_checker = klass(self, statefile, historyfile,
4364-                                   expiration_enabled, expiration_mode,
4365-                                   expiration_override_lease_duration,
4366-                                   expiration_cutoff_date,
4367-                                   expiration_sharetypes)
4368-        self.lease_checker.setServiceParent(self)
4369+        self._setup_bucket_counter()
4370+        self._setup_lease_checker(expiration_policy or self.DEFAULT_EXPIRATION_POLICY)
4371 
4372     def __repr__(self):
4373hunk ./src/allmydata/storage/server.py 65
4374-        return "<StorageServer %s>" % (idlib.shortnodeid_b2a(self.my_nodeid),)
4375+        return "<StorageServer %s>" % (idlib.shortnodeid_b2a(self._serverid),)
4376 
4377hunk ./src/allmydata/storage/server.py 67
4378-    def add_bucket_counter(self):
4379-        statefile = os.path.join(self.storedir, "bucket_counter.state")
4380-        self.bucket_counter = BucketCountingCrawler(self, statefile)
4381+    def _setup_bucket_counter(self):
4382+        statefp = self._statedir.child("bucket_counter.state")
4383+        self.bucket_counter = BucketCountingCrawler(self.backend, statefp)
4384         self.bucket_counter.setServiceParent(self)
4385 
4386hunk ./src/allmydata/storage/server.py 72
4387+    def _setup_lease_checker(self, expiration_policy):
4388+        statefp = self._statedir.child("lease_checker.state")
4389+        historyfp = self._statedir.child("lease_checker.history")
4390+        self.lease_checker = self.LeaseCheckerClass(self.backend, statefp, historyfp, expiration_policy)
4391+        self.lease_checker.setServiceParent(self)
4392+
4393     def count(self, name, delta=1):
4394         if self.stats_provider:
4395             self.stats_provider.count("storage_server." + name, delta)
4396hunk ./src/allmydata/storage/server.py 92
4397         """Return a dict, indexed by category, that contains a dict of
4398         latency numbers for each category. If there are sufficient samples
4399         for unambiguous interpretation, each dict will contain the
4400-        following keys: mean, 01_0_percentile, 10_0_percentile,
4401+        following keys: samplesize, mean, 01_0_percentile, 10_0_percentile,
4402         50_0_percentile (median), 90_0_percentile, 95_0_percentile,
4403         99_0_percentile, 99_9_percentile.  If there are insufficient
4404         samples for a given percentile to be interpreted unambiguously
4405hunk ./src/allmydata/storage/server.py 114
4406             else:
4407                 stats["mean"] = None
4408 
4409-            orderstatlist = [(0.01, "01_0_percentile", 100), (0.1, "10_0_percentile", 10),\
4410-                             (0.50, "50_0_percentile", 10), (0.90, "90_0_percentile", 10),\
4411-                             (0.95, "95_0_percentile", 20), (0.99, "99_0_percentile", 100),\
4412+            orderstatlist = [(0.1, "10_0_percentile", 10), (0.5, "50_0_percentile", 10), \
4413+                             (0.9, "90_0_percentile", 10), (0.95, "95_0_percentile", 20), \
4414+                             (0.01, "01_0_percentile", 100),  (0.99, "99_0_percentile", 100),\
4415                              (0.999, "99_9_percentile", 1000)]
4416 
4417             for percentile, percentilestring, minnumtoobserve in orderstatlist:
4418hunk ./src/allmydata/storage/server.py 133
4419             kwargs["facility"] = "tahoe.storage"
4420         return log.msg(*args, **kwargs)
4421 
4422-    def _clean_incomplete(self):
4423-        fileutil.rm_dir(self.incomingdir)
4424+    def get_serverid(self):
4425+        return self._serverid
4426 
4427     def get_stats(self):
4428         # remember: RIStatsProvider requires that our return dict
4429hunk ./src/allmydata/storage/server.py 138
4430-        # contains numeric values.
4431+        # contains numeric, or None values.
4432         stats = { 'storage_server.allocated': self.allocated_size(), }
4433hunk ./src/allmydata/storage/server.py 140
4434-        stats['storage_server.reserved_space'] = self.reserved_space
4435         for category,ld in self.get_latencies().items():
4436             for name,v in ld.items():
4437                 stats['storage_server.latencies.%s.%s' % (category, name)] = v
4438hunk ./src/allmydata/storage/server.py 144
4439 
4440-        try:
4441-            disk = fileutil.get_disk_stats(self.sharedir, self.reserved_space)
4442-            writeable = disk['avail'] > 0
4443-
4444-            # spacetime predictors should use disk_avail / (d(disk_used)/dt)
4445-            stats['storage_server.disk_total'] = disk['total']
4446-            stats['storage_server.disk_used'] = disk['used']
4447-            stats['storage_server.disk_free_for_root'] = disk['free_for_root']
4448-            stats['storage_server.disk_free_for_nonroot'] = disk['free_for_nonroot']
4449-            stats['storage_server.disk_avail'] = disk['avail']
4450-        except AttributeError:
4451-            writeable = True
4452-        except EnvironmentError:
4453-            log.msg("OS call to get disk statistics failed", level=log.UNUSUAL)
4454-            writeable = False
4455-
4456-        if self.readonly_storage:
4457-            stats['storage_server.disk_avail'] = 0
4458-            writeable = False
4459+        self.backend.fill_in_space_stats(stats)
4460 
4461hunk ./src/allmydata/storage/server.py 146
4462-        stats['storage_server.accepting_immutable_shares'] = int(writeable)
4463         s = self.bucket_counter.get_state()
4464         bucket_count = s.get("last-complete-bucket-count")
4465         if bucket_count:
4466hunk ./src/allmydata/storage/server.py 153
4467         return stats
4468 
4469     def get_available_space(self):
4470-        """Returns available space for share storage in bytes, or None if no
4471-        API to get this information is available."""
4472-
4473-        if self.readonly_storage:
4474-            return 0
4475-        return fileutil.get_available_space(self.sharedir, self.reserved_space)
4476+        return self.backend.get_available_space()
4477 
4478     def allocated_size(self):
4479         space = 0
4480hunk ./src/allmydata/storage/server.py 162
4481         return space
4482 
4483     def remote_get_version(self):
4484-        remaining_space = self.get_available_space()
4485+        remaining_space = self.backend.get_available_space()
4486         if remaining_space is None:
4487             # We're on a platform that has no API to get disk stats.
4488             remaining_space = 2**64
4489hunk ./src/allmydata/storage/server.py 178
4490                     }
4491         return version
4492 
4493-    def remote_allocate_buckets(self, storage_index,
4494+    def remote_allocate_buckets(self, storageindex,
4495                                 renew_secret, cancel_secret,
4496                                 sharenums, allocated_size,
4497                                 canary, owner_num=0):
4498hunk ./src/allmydata/storage/server.py 182
4499+        # cancel_secret is no longer used.
4500         # owner_num is not for clients to set, but rather it should be
4501hunk ./src/allmydata/storage/server.py 184
4502-        # curried into the PersonalStorageServer instance that is dedicated
4503-        # to a particular owner.
4504+        # curried into a StorageServer instance dedicated to a particular
4505+        # owner.
4506         start = time.time()
4507         self.count("allocate")
4508hunk ./src/allmydata/storage/server.py 188
4509-        alreadygot = set()
4510         bucketwriters = {} # k: shnum, v: BucketWriter
4511hunk ./src/allmydata/storage/server.py 189
4512-        si_dir = storage_index_to_dir(storage_index)
4513-        si_s = si_b2a(storage_index)
4514 
4515hunk ./src/allmydata/storage/server.py 190
4516+        si_s = si_b2a(storageindex)
4517         log.msg("storage: allocate_buckets %s" % si_s)
4518 
4519hunk ./src/allmydata/storage/server.py 193
4520-        # in this implementation, the lease information (including secrets)
4521-        # goes into the share files themselves. It could also be put into a
4522-        # separate database. Note that the lease should not be added until
4523-        # the BucketWriter has been closed.
4524+        # Note that the lease should not be added until the BucketWriter
4525+        # has been closed.
4526         expire_time = time.time() + 31*24*60*60
4527hunk ./src/allmydata/storage/server.py 196
4528-        lease_info = LeaseInfo(owner_num,
4529-                               renew_secret, cancel_secret,
4530-                               expire_time, self.my_nodeid)
4531+        lease_info = LeaseInfo(owner_num, renew_secret,
4532+                               expire_time, self._serverid)
4533 
4534         max_space_per_bucket = allocated_size
4535 
4536hunk ./src/allmydata/storage/server.py 201
4537-        remaining_space = self.get_available_space()
4538+        remaining_space = self.backend.get_available_space()
4539         limited = remaining_space is not None
4540         if limited:
4541hunk ./src/allmydata/storage/server.py 204
4542-            # this is a bit conservative, since some of this allocated_size()
4543-            # has already been written to disk, where it will show up in
4544+            # This is a bit conservative, since some of this allocated_size()
4545+            # has already been written to the backend, where it will show up in
4546             # get_available_space.
4547             remaining_space -= self.allocated_size()
4548hunk ./src/allmydata/storage/server.py 208
4549-        # self.readonly_storage causes remaining_space <= 0
4550+            # If the backend is read-only, remaining_space will be <= 0.
4551+
4552+        shareset = self.backend.get_shareset(storageindex)
4553 
4554hunk ./src/allmydata/storage/server.py 212
4555-        # fill alreadygot with all shares that we have, not just the ones
4556+        # Fill alreadygot with all shares that we have, not just the ones
4557         # they asked about: this will save them a lot of work. Add or update
4558         # leases for all of them: if they want us to hold shares for this
4559hunk ./src/allmydata/storage/server.py 215
4560-        # file, they'll want us to hold leases for this file.
4561-        for (shnum, fn) in self._get_bucket_shares(storage_index):
4562-            alreadygot.add(shnum)
4563-            sf = ShareFile(fn)
4564-            sf.add_or_renew_lease(lease_info)
4565+        # file, they'll want us to hold leases for all the shares of it.
4566+        #
4567+        # XXX should we be making the assumption here that lease info is
4568+        # duplicated in all shares?
4569+        alreadygot = set()
4570+        for share in shareset.get_shares():
4571+            share.add_or_renew_lease(lease_info)
4572+            alreadygot.add(share.shnum)
4573 
4574hunk ./src/allmydata/storage/server.py 224
4575-        for shnum in sharenums:
4576-            incominghome = os.path.join(self.incomingdir, si_dir, "%d" % shnum)
4577-            finalhome = os.path.join(self.sharedir, si_dir, "%d" % shnum)
4578-            if os.path.exists(finalhome):
4579-                # great! we already have it. easy.
4580-                pass
4581-            elif os.path.exists(incominghome):
4582+        for shnum in sharenums - alreadygot:
4583+            if shareset.has_incoming(shnum):
4584                 # Note that we don't create BucketWriters for shnums that
4585                 # have a partial share (in incoming/), so if a second upload
4586                 # occurs while the first is still in progress, the second
4587hunk ./src/allmydata/storage/server.py 232
4588                 # uploader will use different storage servers.
4589                 pass
4590             elif (not limited) or (remaining_space >= max_space_per_bucket):
4591-                # ok! we need to create the new share file.
4592-                bw = BucketWriter(self, incominghome, finalhome,
4593-                                  max_space_per_bucket, lease_info, canary)
4594-                if self.no_storage:
4595-                    bw.throw_out_all_data = True
4596+                bw = shareset.make_bucket_writer(self, shnum, max_space_per_bucket,
4597+                                                 lease_info, canary)
4598                 bucketwriters[shnum] = bw
4599                 self._active_writers[bw] = 1
4600                 if limited:
4601hunk ./src/allmydata/storage/server.py 239
4602                     remaining_space -= max_space_per_bucket
4603             else:
4604-                # bummer! not enough space to accept this bucket
4605+                # Bummer not enough space to accept this share.
4606                 pass
4607 
4608hunk ./src/allmydata/storage/server.py 242
4609-        if bucketwriters:
4610-            fileutil.make_dirs(os.path.join(self.sharedir, si_dir))
4611-
4612         self.add_latency("allocate", time.time() - start)
4613         return alreadygot, bucketwriters
4614 
4615hunk ./src/allmydata/storage/server.py 245
4616-    def _iter_share_files(self, storage_index):
4617-        for shnum, filename in self._get_bucket_shares(storage_index):
4618-            f = open(filename, 'rb')
4619-            header = f.read(32)
4620-            f.close()
4621-            if header[:32] == MutableShareFile.MAGIC:
4622-                sf = MutableShareFile(filename, self)
4623-                # note: if the share has been migrated, the renew_lease()
4624-                # call will throw an exception, with information to help the
4625-                # client update the lease.
4626-            elif header[:4] == struct.pack(">L", 1):
4627-                sf = ShareFile(filename)
4628-            else:
4629-                continue # non-sharefile
4630-            yield sf
4631-
4632-    def remote_add_lease(self, storage_index, renew_secret, cancel_secret,
4633+    def remote_add_lease(self, storageindex, renew_secret, cancel_secret,
4634                          owner_num=1):
4635hunk ./src/allmydata/storage/server.py 247
4636+        # cancel_secret is no longer used.
4637         start = time.time()
4638         self.count("add-lease")
4639         new_expire_time = time.time() + 31*24*60*60
4640hunk ./src/allmydata/storage/server.py 251
4641-        lease_info = LeaseInfo(owner_num,
4642-                               renew_secret, cancel_secret,
4643-                               new_expire_time, self.my_nodeid)
4644-        for sf in self._iter_share_files(storage_index):
4645-            sf.add_or_renew_lease(lease_info)
4646-        self.add_latency("add-lease", time.time() - start)
4647-        return None
4648+        lease_info = LeaseInfo(owner_num, renew_secret,
4649+                               new_expire_time, self._serverid)
4650 
4651hunk ./src/allmydata/storage/server.py 254
4652-    def remote_renew_lease(self, storage_index, renew_secret):
4653+        try:
4654+            self.backend.add_or_renew_lease(lease_info)
4655+        finally:
4656+            self.add_latency("add-lease", time.time() - start)
4657+
4658+    def remote_renew_lease(self, storageindex, renew_secret):
4659         start = time.time()
4660         self.count("renew")
4661hunk ./src/allmydata/storage/server.py 262
4662-        new_expire_time = time.time() + 31*24*60*60
4663-        found_buckets = False
4664-        for sf in self._iter_share_files(storage_index):
4665-            found_buckets = True
4666-            sf.renew_lease(renew_secret, new_expire_time)
4667-        self.add_latency("renew", time.time() - start)
4668-        if not found_buckets:
4669-            raise IndexError("no such lease to renew")
4670+
4671+        try:
4672+            shareset = self.backend.get_shareset(storageindex)
4673+            new_expiration_time = start + 31*24*60*60   # one month from now
4674+            shareset.renew_lease(renew_secret, new_expiration_time)
4675+        finally:
4676+            self.add_latency("renew", time.time() - start)
4677 
4678     def bucket_writer_closed(self, bw, consumed_size):
4679         if self.stats_provider:
4680hunk ./src/allmydata/storage/server.py 275
4681             self.stats_provider.count('storage_server.bytes_added', consumed_size)
4682         del self._active_writers[bw]
4683 
4684-    def _get_bucket_shares(self, storage_index):
4685-        """Return a list of (shnum, pathname) tuples for files that hold
4686-        shares for this storage_index. In each tuple, 'shnum' will always be
4687-        the integer form of the last component of 'pathname'."""
4688-        storagedir = os.path.join(self.sharedir, storage_index_to_dir(storage_index))
4689-        try:
4690-            for f in os.listdir(storagedir):
4691-                if NUM_RE.match(f):
4692-                    filename = os.path.join(storagedir, f)
4693-                    yield (int(f), filename)
4694-        except OSError:
4695-            # Commonly caused by there being no buckets at all.
4696-            pass
4697-
4698-    def remote_get_buckets(self, storage_index):
4699+    def remote_get_buckets(self, storageindex):
4700         start = time.time()
4701         self.count("get")
4702hunk ./src/allmydata/storage/server.py 278
4703-        si_s = si_b2a(storage_index)
4704+        si_s = si_b2a(storageindex)
4705         log.msg("storage: get_buckets %s" % si_s)
4706         bucketreaders = {} # k: sharenum, v: BucketReader
4707hunk ./src/allmydata/storage/server.py 281
4708-        for shnum, filename in self._get_bucket_shares(storage_index):
4709-            bucketreaders[shnum] = BucketReader(self, filename,
4710-                                                storage_index, shnum)
4711-        self.add_latency("get", time.time() - start)
4712-        return bucketreaders
4713 
4714hunk ./src/allmydata/storage/server.py 282
4715-    def get_leases(self, storage_index):
4716-        """Provide an iterator that yields all of the leases attached to this
4717-        bucket. Each lease is returned as a LeaseInfo instance.
4718+        try:
4719+            shareset = self.backend.get_shareset(storageindex)
4720+            for share in shareset.get_shares():
4721+                bucketreaders[share.get_shnum()] = shareset.make_bucket_reader(self, share)
4722+            return bucketreaders
4723+        finally:
4724+            self.add_latency("get", time.time() - start)
4725 
4726hunk ./src/allmydata/storage/server.py 290
4727-        This method is not for client use.
4728+    def get_leases(self, storageindex):
4729         """
4730hunk ./src/allmydata/storage/server.py 292
4731+        Provide an iterator that yields all of the leases attached to this
4732+        bucket. Each lease is returned as a LeaseInfo instance.
4733 
4734hunk ./src/allmydata/storage/server.py 295
4735-        # since all shares get the same lease data, we just grab the leases
4736-        # from the first share
4737-        try:
4738-            shnum, filename = self._get_bucket_shares(storage_index).next()
4739-            sf = ShareFile(filename)
4740-            return sf.get_leases()
4741-        except StopIteration:
4742-            return iter([])
4743+        This method is not for client use. XXX do we need it at all?
4744+        """
4745+        return self.backend.get_shareset(storageindex).get_leases()
4746 
4747hunk ./src/allmydata/storage/server.py 299
4748-    def remote_slot_testv_and_readv_and_writev(self, storage_index,
4749+    def remote_slot_testv_and_readv_and_writev(self, storageindex,
4750                                                secrets,
4751                                                test_and_write_vectors,
4752                                                read_vector):
4753hunk ./src/allmydata/storage/server.py 305
4754         start = time.time()
4755         self.count("writev")
4756-        si_s = si_b2a(storage_index)
4757+        si_s = si_b2a(storageindex)
4758         log.msg("storage: slot_writev %s" % si_s)
4759hunk ./src/allmydata/storage/server.py 307
4760-        si_dir = storage_index_to_dir(storage_index)
4761-        (write_enabler, renew_secret, cancel_secret) = secrets
4762-        # shares exist if there is a file for them
4763-        bucketdir = os.path.join(self.sharedir, si_dir)
4764-        shares = {}
4765-        if os.path.isdir(bucketdir):
4766-            for sharenum_s in os.listdir(bucketdir):
4767-                try:
4768-                    sharenum = int(sharenum_s)
4769-                except ValueError:
4770-                    continue
4771-                filename = os.path.join(bucketdir, sharenum_s)
4772-                msf = MutableShareFile(filename, self)
4773-                msf.check_write_enabler(write_enabler, si_s)
4774-                shares[sharenum] = msf
4775-        # write_enabler is good for all existing shares.
4776-
4777-        # Now evaluate test vectors.
4778-        testv_is_good = True
4779-        for sharenum in test_and_write_vectors:
4780-            (testv, datav, new_length) = test_and_write_vectors[sharenum]
4781-            if sharenum in shares:
4782-                if not shares[sharenum].check_testv(testv):
4783-                    self.log("testv failed: [%d]: %r" % (sharenum, testv))
4784-                    testv_is_good = False
4785-                    break
4786-            else:
4787-                # compare the vectors against an empty share, in which all
4788-                # reads return empty strings.
4789-                if not EmptyShare().check_testv(testv):
4790-                    self.log("testv failed (empty): [%d] %r" % (sharenum,
4791-                                                                testv))
4792-                    testv_is_good = False
4793-                    break
4794-
4795-        # now gather the read vectors, before we do any writes
4796-        read_data = {}
4797-        for sharenum, share in shares.items():
4798-            read_data[sharenum] = share.readv(read_vector)
4799-
4800-        ownerid = 1 # TODO
4801-        expire_time = time.time() + 31*24*60*60   # one month
4802-        lease_info = LeaseInfo(ownerid,
4803-                               renew_secret, cancel_secret,
4804-                               expire_time, self.my_nodeid)
4805-
4806-        if testv_is_good:
4807-            # now apply the write vectors
4808-            for sharenum in test_and_write_vectors:
4809-                (testv, datav, new_length) = test_and_write_vectors[sharenum]
4810-                if new_length == 0:
4811-                    if sharenum in shares:
4812-                        shares[sharenum].unlink()
4813-                else:
4814-                    if sharenum not in shares:
4815-                        # allocate a new share
4816-                        allocated_size = 2000 # arbitrary, really
4817-                        share = self._allocate_slot_share(bucketdir, secrets,
4818-                                                          sharenum,
4819-                                                          allocated_size,
4820-                                                          owner_num=0)
4821-                        shares[sharenum] = share
4822-                    shares[sharenum].writev(datav, new_length)
4823-                    # and update the lease
4824-                    shares[sharenum].add_or_renew_lease(lease_info)
4825-
4826-            if new_length == 0:
4827-                # delete empty bucket directories
4828-                if not os.listdir(bucketdir):
4829-                    os.rmdir(bucketdir)
4830 
4831hunk ./src/allmydata/storage/server.py 308
4832+        try:
4833+            shareset = self.backend.get_shareset(storageindex)
4834+            expiration_time = start + 31*24*60*60   # one month from now
4835+            return shareset.testv_and_readv_and_writev(self, secrets, test_and_write_vectors,
4836+                                                       read_vector, expiration_time)
4837+        finally:
4838+            self.add_latency("writev", time.time() - start)
4839 
4840hunk ./src/allmydata/storage/server.py 316
4841-        # all done
4842-        self.add_latency("writev", time.time() - start)
4843-        return (testv_is_good, read_data)
4844-
4845-    def _allocate_slot_share(self, bucketdir, secrets, sharenum,
4846-                             allocated_size, owner_num=0):
4847-        (write_enabler, renew_secret, cancel_secret) = secrets
4848-        my_nodeid = self.my_nodeid
4849-        fileutil.make_dirs(bucketdir)
4850-        filename = os.path.join(bucketdir, "%d" % sharenum)
4851-        share = create_mutable_sharefile(filename, my_nodeid, write_enabler,
4852-                                         self)
4853-        return share
4854-
4855-    def remote_slot_readv(self, storage_index, shares, readv):
4856+    def remote_slot_readv(self, storageindex, shares, readv):
4857         start = time.time()
4858         self.count("readv")
4859hunk ./src/allmydata/storage/server.py 319
4860-        si_s = si_b2a(storage_index)
4861-        lp = log.msg("storage: slot_readv %s %s" % (si_s, shares),
4862-                     facility="tahoe.storage", level=log.OPERATIONAL)
4863-        si_dir = storage_index_to_dir(storage_index)
4864-        # shares exist if there is a file for them
4865-        bucketdir = os.path.join(self.sharedir, si_dir)
4866-        if not os.path.isdir(bucketdir):
4867+        si_s = si_b2a(storageindex)
4868+        log.msg("storage: slot_readv %s %s" % (si_s, shares),
4869+                facility="tahoe.storage", level=log.OPERATIONAL)
4870+
4871+        try:
4872+            shareset = self.backend.get_shareset(storageindex)
4873+            return shareset.readv(self, shares, readv)
4874+        finally:
4875             self.add_latency("readv", time.time() - start)
4876hunk ./src/allmydata/storage/server.py 328
4877-            return {}
4878-        datavs = {}
4879-        for sharenum_s in os.listdir(bucketdir):
4880-            try:
4881-                sharenum = int(sharenum_s)
4882-            except ValueError:
4883-                continue
4884-            if sharenum in shares or not shares:
4885-                filename = os.path.join(bucketdir, sharenum_s)
4886-                msf = MutableShareFile(filename, self)
4887-                datavs[sharenum] = msf.readv(readv)
4888-        log.msg("returning shares %s" % (datavs.keys(),),
4889-                facility="tahoe.storage", level=log.NOISY, parent=lp)
4890-        self.add_latency("readv", time.time() - start)
4891-        return datavs
4892 
4893hunk ./src/allmydata/storage/server.py 329
4894-    def remote_advise_corrupt_share(self, share_type, storage_index, shnum,
4895-                                    reason):
4896-        fileutil.make_dirs(self.corruption_advisory_dir)
4897-        now = time_format.iso_utc(sep="T")
4898-        si_s = si_b2a(storage_index)
4899-        # windows can't handle colons in the filename
4900-        fn = os.path.join(self.corruption_advisory_dir,
4901-                          "%s--%s-%d" % (now, si_s, shnum)).replace(":","")
4902-        f = open(fn, "w")
4903-        f.write("report: Share Corruption\n")
4904-        f.write("type: %s\n" % share_type)
4905-        f.write("storage_index: %s\n" % si_s)
4906-        f.write("share_number: %d\n" % shnum)
4907-        f.write("\n")
4908-        f.write(reason)
4909-        f.write("\n")
4910-        f.close()
4911-        log.msg(format=("client claims corruption in (%(share_type)s) " +
4912-                        "%(si)s-%(shnum)d: %(reason)s"),
4913-                share_type=share_type, si=si_s, shnum=shnum, reason=reason,
4914-                level=log.SCARY, umid="SGx2fA")
4915-        return None
4916+    def remote_advise_corrupt_share(self, share_type, storage_index, shnum, reason):
4917+        self.backend.advise_corrupt_share(share_type, storage_index, shnum, reason)
4918hunk ./src/allmydata/test/common.py 20
4919 from allmydata.mutable.common import CorruptShareError
4920 from allmydata.mutable.layout import unpack_header
4921 from allmydata.mutable.publish import MutableData
4922-from allmydata.storage.mutable import MutableShareFile
4923+from allmydata.storage.backends.disk.mutable import MutableDiskShare
4924 from allmydata.util import hashutil, log, fileutil, pollmixin
4925 from allmydata.util.assertutil import precondition
4926 from allmydata.util.consumer import download_to_data
4927hunk ./src/allmydata/test/common.py 1297
4928 
4929 def _corrupt_mutable_share_data(data, debug=False):
4930     prefix = data[:32]
4931-    assert prefix == MutableShareFile.MAGIC, "This function is designed to corrupt mutable shares of v1, and the magic number doesn't look right: %r vs %r" % (prefix, MutableShareFile.MAGIC)
4932-    data_offset = MutableShareFile.DATA_OFFSET
4933+    assert prefix == MutableDiskShare.MAGIC, "This function is designed to corrupt mutable shares of v1, and the magic number doesn't look right: %r vs %r" % (prefix, MutableDiskShare.MAGIC)
4934+    data_offset = MutableDiskShare.DATA_OFFSET
4935     sharetype = data[data_offset:data_offset+1]
4936     assert sharetype == "\x00", "non-SDMF mutable shares not supported"
4937     (version, ig_seqnum, ig_roothash, ig_IV, ig_k, ig_N, ig_segsize,
4938hunk ./src/allmydata/test/no_network.py 21
4939 from twisted.application import service
4940 from twisted.internet import defer, reactor
4941 from twisted.python.failure import Failure
4942+from twisted.python.filepath import FilePath
4943 from foolscap.api import Referenceable, fireEventually, RemoteException
4944 from base64 import b32encode
4945hunk ./src/allmydata/test/no_network.py 24
4946+
4947 from allmydata import uri as tahoe_uri
4948 from allmydata.client import Client
4949hunk ./src/allmydata/test/no_network.py 27
4950-from allmydata.storage.server import StorageServer, storage_index_to_dir
4951+from allmydata.storage.server import StorageServer
4952+from allmydata.storage.backends.disk.disk_backend import DiskBackend
4953 from allmydata.util import fileutil, idlib, hashutil
4954 from allmydata.util.hashutil import sha1
4955 from allmydata.test.common_web import HTTPClientGETFactory
4956hunk ./src/allmydata/test/no_network.py 155
4957             seed = server.get_permutation_seed()
4958             return sha1(peer_selection_index + seed).digest()
4959         return sorted(self.get_connected_servers(), key=_permuted)
4960+
4961     def get_connected_servers(self):
4962         return self.client._servers
4963hunk ./src/allmydata/test/no_network.py 158
4964+
4965     def get_nickname_for_serverid(self, serverid):
4966         return None
4967 
4968hunk ./src/allmydata/test/no_network.py 162
4969+    def get_known_servers(self):
4970+        return self.get_connected_servers()
4971+
4972+    def get_all_serverids(self):
4973+        return self.client.get_all_serverids()
4974+
4975+
4976 class NoNetworkClient(Client):
4977     def create_tub(self):
4978         pass
4979hunk ./src/allmydata/test/no_network.py 262
4980 
4981     def make_server(self, i, readonly=False):
4982         serverid = hashutil.tagged_hash("serverid", str(i))[:20]
4983-        serverdir = os.path.join(self.basedir, "servers",
4984-                                 idlib.shortnodeid_b2a(serverid), "storage")
4985-        fileutil.make_dirs(serverdir)
4986-        ss = StorageServer(serverdir, serverid, stats_provider=SimpleStats(),
4987-                           readonly_storage=readonly)
4988+        storagedir = FilePath(self.basedir).child("servers").child(idlib.shortnodeid_b2a(serverid)).child("storage")
4989+
4990+        # The backend will make the storage directory and any necessary parents.
4991+        backend = DiskBackend(storagedir, readonly=readonly)
4992+        ss = StorageServer(serverid, backend, storagedir, stats_provider=SimpleStats())
4993         ss._no_network_server_number = i
4994         return ss
4995 
4996hunk ./src/allmydata/test/no_network.py 276
4997         middleman = service.MultiService()
4998         middleman.setServiceParent(self)
4999         ss.setServiceParent(middleman)
5000-        serverid = ss.my_nodeid
5001+        serverid = ss.get_serverid()
5002         self.servers_by_number[i] = ss
5003         wrapper = wrap_storage_server(ss)
5004         self.wrappers_by_id[serverid] = wrapper
5005hunk ./src/allmydata/test/no_network.py 295
5006         # it's enough to remove the server from c._servers (we don't actually
5007         # have to detach and stopService it)
5008         for i,ss in self.servers_by_number.items():
5009-            if ss.my_nodeid == serverid:
5010+            if ss.get_serverid() == serverid:
5011                 del self.servers_by_number[i]
5012                 break
5013         del self.wrappers_by_id[serverid]
5014hunk ./src/allmydata/test/no_network.py 345
5015     def get_clientdir(self, i=0):
5016         return self.g.clients[i].basedir
5017 
5018+    def get_server(self, i):
5019+        return self.g.servers_by_number[i]
5020+
5021     def get_serverdir(self, i):
5022hunk ./src/allmydata/test/no_network.py 349
5023-        return self.g.servers_by_number[i].storedir
5024+        return self.g.servers_by_number[i].backend.storedir
5025+
5026+    def remove_server(self, i):
5027+        self.g.remove_server(self.g.servers_by_number[i].get_serverid())
5028 
5029     def iterate_servers(self):
5030         for i in sorted(self.g.servers_by_number.keys()):
5031hunk ./src/allmydata/test/no_network.py 357
5032             ss = self.g.servers_by_number[i]
5033-            yield (i, ss, ss.storedir)
5034+            yield (i, ss, ss.backend.storedir)
5035 
5036     def find_uri_shares(self, uri):
5037         si = tahoe_uri.from_string(uri).get_storage_index()
5038hunk ./src/allmydata/test/no_network.py 361
5039-        prefixdir = storage_index_to_dir(si)
5040         shares = []
5041         for i,ss in self.g.servers_by_number.items():
5042hunk ./src/allmydata/test/no_network.py 363
5043-            serverid = ss.my_nodeid
5044-            basedir = os.path.join(ss.sharedir, prefixdir)
5045-            if not os.path.exists(basedir):
5046-                continue
5047-            for f in os.listdir(basedir):
5048-                try:
5049-                    shnum = int(f)
5050-                    shares.append((shnum, serverid, os.path.join(basedir, f)))
5051-                except ValueError:
5052-                    pass
5053+            for share in ss.backend.get_shareset(si).get_shares():
5054+                shares.append((share.get_shnum(), ss.get_serverid(), share._home))
5055         return sorted(shares)
5056 
5057hunk ./src/allmydata/test/no_network.py 367
5058+    def count_leases(self, uri):
5059+        """Return (filename, leasecount) pairs in arbitrary order."""
5060+        si = tahoe_uri.from_string(uri).get_storage_index()
5061+        lease_counts = []
5062+        for i,ss in self.g.servers_by_number.items():
5063+            for share in ss.backend.get_shareset(si).get_shares():
5064+                num_leases = len(list(share.get_leases()))
5065+                lease_counts.append( (share._home.path, num_leases) )
5066+        return lease_counts
5067+
5068     def copy_shares(self, uri):
5069         shares = {}
5070hunk ./src/allmydata/test/no_network.py 379
5071-        for (shnum, serverid, sharefile) in self.find_uri_shares(uri):
5072-            shares[sharefile] = open(sharefile, "rb").read()
5073+        for (shnum, serverid, sharefp) in self.find_uri_shares(uri):
5074+            shares[sharefp.path] = sharefp.getContent()
5075         return shares
5076 
5077hunk ./src/allmydata/test/no_network.py 383
5078+    def copy_share(self, from_share, uri, to_server):
5079+        si = uri.from_string(self.uri).get_storage_index()
5080+        (i_shnum, i_serverid, i_sharefp) = from_share
5081+        shares_dir = to_server.backend.get_shareset(si)._sharehomedir
5082+        i_sharefp.copyTo(shares_dir.child(str(i_shnum)))
5083+
5084     def restore_all_shares(self, shares):
5085hunk ./src/allmydata/test/no_network.py 390
5086-        for sharefile, data in shares.items():
5087-            open(sharefile, "wb").write(data)
5088+        for share, data in shares.items():
5089+            share.home.setContent(data)
5090 
5091hunk ./src/allmydata/test/no_network.py 393
5092-    def delete_share(self, (shnum, serverid, sharefile)):
5093-        os.unlink(sharefile)
5094+    def delete_share(self, (shnum, serverid, sharefp)):
5095+        sharefp.remove()
5096 
5097     def delete_shares_numbered(self, uri, shnums):
5098hunk ./src/allmydata/test/no_network.py 397
5099-        for (i_shnum, i_serverid, i_sharefile) in self.find_uri_shares(uri):
5100+        for (i_shnum, i_serverid, i_sharefp) in self.find_uri_shares(uri):
5101             if i_shnum in shnums:
5102hunk ./src/allmydata/test/no_network.py 399
5103-                os.unlink(i_sharefile)
5104+                i_sharefp.remove()
5105 
5106hunk ./src/allmydata/test/no_network.py 401
5107-    def corrupt_share(self, (shnum, serverid, sharefile), corruptor_function):
5108-        sharedata = open(sharefile, "rb").read()
5109-        corruptdata = corruptor_function(sharedata)
5110-        open(sharefile, "wb").write(corruptdata)
5111+    def corrupt_share(self, (shnum, serverid, sharefp), corruptor_function, debug=False):
5112+        sharedata = sharefp.getContent()
5113+        corruptdata = corruptor_function(sharedata, debug=debug)
5114+        sharefp.setContent(corruptdata)
5115 
5116     def corrupt_shares_numbered(self, uri, shnums, corruptor, debug=False):
5117hunk ./src/allmydata/test/no_network.py 407
5118-        for (i_shnum, i_serverid, i_sharefile) in self.find_uri_shares(uri):
5119+        for (i_shnum, i_serverid, i_sharefp) in self.find_uri_shares(uri):
5120             if i_shnum in shnums:
5121hunk ./src/allmydata/test/no_network.py 409
5122-                sharedata = open(i_sharefile, "rb").read()
5123-                corruptdata = corruptor(sharedata, debug=debug)
5124-                open(i_sharefile, "wb").write(corruptdata)
5125+                self.corrupt_share((i_shnum, i_serverid, i_sharefp), corruptor, debug=debug)
5126 
5127     def corrupt_all_shares(self, uri, corruptor, debug=False):
5128hunk ./src/allmydata/test/no_network.py 412
5129-        for (i_shnum, i_serverid, i_sharefile) in self.find_uri_shares(uri):
5130-            sharedata = open(i_sharefile, "rb").read()
5131-            corruptdata = corruptor(sharedata, debug=debug)
5132-            open(i_sharefile, "wb").write(corruptdata)
5133+        for (i_shnum, i_serverid, i_sharefp) in self.find_uri_shares(uri):
5134+            self.corrupt_share((i_shnum, i_serverid, i_sharefp), corruptor, debug=debug)
5135 
5136     def GET(self, urlpath, followRedirect=False, return_response=False,
5137             method="GET", clientnum=0, **kwargs):
5138hunk ./src/allmydata/test/test_download.py 6
5139 # a previous run. This asserts that the current code is capable of decoding
5140 # shares from a previous version.
5141 
5142-import os
5143 from twisted.trial import unittest
5144 from twisted.internet import defer, reactor
5145 from allmydata import uri
5146hunk ./src/allmydata/test/test_download.py 9
5147-from allmydata.storage.server import storage_index_to_dir
5148 from allmydata.util import base32, fileutil, spans, log, hashutil
5149 from allmydata.util.consumer import download_to_data, MemoryConsumer
5150 from allmydata.immutable import upload, layout
5151hunk ./src/allmydata/test/test_download.py 85
5152         u = upload.Data(plaintext, None)
5153         d = self.c0.upload(u)
5154         f = open("stored_shares.py", "w")
5155-        def _created_immutable(ur):
5156-            # write the generated shares and URI to a file, which can then be
5157-            # incorporated into this one next time.
5158-            f.write('immutable_uri = "%s"\n' % ur.uri)
5159-            f.write('immutable_shares = {\n')
5160-            si = uri.from_string(ur.uri).get_storage_index()
5161-            si_dir = storage_index_to_dir(si)
5162+
5163+        def _write_py(uri):
5164+            si = uri.from_string(uri).get_storage_index()
5165             for (i,ss,ssdir) in self.iterate_servers():
5166hunk ./src/allmydata/test/test_download.py 89
5167-                sharedir = os.path.join(ssdir, "shares", si_dir)
5168                 shares = {}
5169hunk ./src/allmydata/test/test_download.py 90
5170-                for fn in os.listdir(sharedir):
5171-                    shnum = int(fn)
5172-                    sharedata = open(os.path.join(sharedir, fn), "rb").read()
5173-                    shares[shnum] = sharedata
5174-                fileutil.rm_dir(sharedir)
5175+                shareset = ss.backend.get_shareset(si)
5176+                for share in shareset.get_shares():
5177+                    sharedata = share._home.getContent()
5178+                    shares[share.get_shnum()] = sharedata
5179+
5180+                fileutil.fp_remove(shareset._sharehomedir)
5181                 if shares:
5182                     f.write(' %d: { # client[%d]\n' % (i, i))
5183                     for shnum in sorted(shares.keys()):
5184hunk ./src/allmydata/test/test_download.py 103
5185                                 (shnum, base32.b2a(shares[shnum])))
5186                     f.write('    },\n')
5187             f.write('}\n')
5188-            f.write('\n')
5189 
5190hunk ./src/allmydata/test/test_download.py 104
5191+        def _created_immutable(ur):
5192+            # write the generated shares and URI to a file, which can then be
5193+            # incorporated into this one next time.
5194+            f.write('immutable_uri = "%s"\n' % ur.uri)
5195+            f.write('immutable_shares = {\n')
5196+            _write_py(ur.uri)
5197+            f.write('\n')
5198         d.addCallback(_created_immutable)
5199 
5200         d.addCallback(lambda ignored:
5201hunk ./src/allmydata/test/test_download.py 118
5202         def _created_mutable(n):
5203             f.write('mutable_uri = "%s"\n' % n.get_uri())
5204             f.write('mutable_shares = {\n')
5205-            si = uri.from_string(n.get_uri()).get_storage_index()
5206-            si_dir = storage_index_to_dir(si)
5207-            for (i,ss,ssdir) in self.iterate_servers():
5208-                sharedir = os.path.join(ssdir, "shares", si_dir)
5209-                shares = {}
5210-                for fn in os.listdir(sharedir):
5211-                    shnum = int(fn)
5212-                    sharedata = open(os.path.join(sharedir, fn), "rb").read()
5213-                    shares[shnum] = sharedata
5214-                fileutil.rm_dir(sharedir)
5215-                if shares:
5216-                    f.write(' %d: { # client[%d]\n' % (i, i))
5217-                    for shnum in sorted(shares.keys()):
5218-                        f.write('  %d: base32.a2b("%s"),\n' %
5219-                                (shnum, base32.b2a(shares[shnum])))
5220-                    f.write('    },\n')
5221-            f.write('}\n')
5222-
5223-            f.close()
5224+            _write_py(n.get_uri())
5225         d.addCallback(_created_mutable)
5226 
5227         def _done(ignored):
5228hunk ./src/allmydata/test/test_download.py 123
5229             f.close()
5230-        d.addCallback(_done)
5231+        d.addBoth(_done)
5232 
5233         return d
5234 
5235hunk ./src/allmydata/test/test_download.py 127
5236+    def _write_shares(self, uri, shares):
5237+        si = uri.from_string(uri).get_storage_index()
5238+        for i in shares:
5239+            shares_for_server = shares[i]
5240+            for shnum in shares_for_server:
5241+                share_dir = self.get_server(i).backend.get_shareset(si)._sharehomedir
5242+                fileutil.fp_make_dirs(share_dir)
5243+                share_dir.child(str(shnum)).setContent(shares[shnum])
5244+
5245     def load_shares(self, ignored=None):
5246         # this uses the data generated by create_shares() to populate the
5247         # storage servers with pre-generated shares
5248hunk ./src/allmydata/test/test_download.py 139
5249-        si = uri.from_string(immutable_uri).get_storage_index()
5250-        si_dir = storage_index_to_dir(si)
5251-        for i in immutable_shares:
5252-            shares = immutable_shares[i]
5253-            for shnum in shares:
5254-                dn = os.path.join(self.get_serverdir(i), "shares", si_dir)
5255-                fileutil.make_dirs(dn)
5256-                fn = os.path.join(dn, str(shnum))
5257-                f = open(fn, "wb")
5258-                f.write(shares[shnum])
5259-                f.close()
5260-
5261-        si = uri.from_string(mutable_uri).get_storage_index()
5262-        si_dir = storage_index_to_dir(si)
5263-        for i in mutable_shares:
5264-            shares = mutable_shares[i]
5265-            for shnum in shares:
5266-                dn = os.path.join(self.get_serverdir(i), "shares", si_dir)
5267-                fileutil.make_dirs(dn)
5268-                fn = os.path.join(dn, str(shnum))
5269-                f = open(fn, "wb")
5270-                f.write(shares[shnum])
5271-                f.close()
5272+        self._write_shares(immutable_uri, immutable_shares)
5273+        self._write_shares(mutable_uri, mutable_shares)
5274 
5275     def download_immutable(self, ignored=None):
5276         n = self.c0.create_node_from_uri(immutable_uri)
5277hunk ./src/allmydata/test/test_download.py 183
5278 
5279         self.load_shares()
5280         si = uri.from_string(immutable_uri).get_storage_index()
5281-        si_dir = storage_index_to_dir(si)
5282 
5283         n = self.c0.create_node_from_uri(immutable_uri)
5284         d = download_to_data(n)
5285hunk ./src/allmydata/test/test_download.py 198
5286                 for clientnum in immutable_shares:
5287                     for shnum in immutable_shares[clientnum]:
5288                         if s._shnum == shnum:
5289-                            fn = os.path.join(self.get_serverdir(clientnum),
5290-                                              "shares", si_dir, str(shnum))
5291-                            os.unlink(fn)
5292+                            share_dir = self.get_server(clientnum).backend.get_shareset(si)._sharehomedir
5293+                            share_dir.child(str(shnum)).remove()
5294         d.addCallback(_clobber_some_shares)
5295         d.addCallback(lambda ign: download_to_data(n))
5296         d.addCallback(_got_data)
5297hunk ./src/allmydata/test/test_download.py 212
5298                 for shnum in immutable_shares[clientnum]:
5299                     if shnum == save_me:
5300                         continue
5301-                    fn = os.path.join(self.get_serverdir(clientnum),
5302-                                      "shares", si_dir, str(shnum))
5303-                    if os.path.exists(fn):
5304-                        os.unlink(fn)
5305+                    share_dir = self.get_server(clientnum).backend.get_shareset(si)._sharehomedir
5306+                    fileutil.fp_remove(share_dir.child(str(shnum)))
5307             # now the download should fail with NotEnoughSharesError
5308             return self.shouldFail(NotEnoughSharesError, "1shares", None,
5309                                    download_to_data, n)
5310hunk ./src/allmydata/test/test_download.py 223
5311             # delete the last remaining share
5312             for clientnum in immutable_shares:
5313                 for shnum in immutable_shares[clientnum]:
5314-                    fn = os.path.join(self.get_serverdir(clientnum),
5315-                                      "shares", si_dir, str(shnum))
5316-                    if os.path.exists(fn):
5317-                        os.unlink(fn)
5318+                    share_dir = self.get_server(clientnum).backend.get_shareset(si)._sharehomedir
5319+                    share_dir.child(str(shnum)).remove()
5320             # now a new download should fail with NoSharesError. We want a
5321             # new ImmutableFileNode so it will forget about the old shares.
5322             # If we merely called create_node_from_uri() without first
5323hunk ./src/allmydata/test/test_download.py 801
5324         # will report two shares, and the ShareFinder will handle the
5325         # duplicate by attaching both to the same CommonShare instance.
5326         si = uri.from_string(immutable_uri).get_storage_index()
5327-        si_dir = storage_index_to_dir(si)
5328-        sh0_file = [sharefile
5329-                    for (shnum, serverid, sharefile)
5330-                    in self.find_uri_shares(immutable_uri)
5331-                    if shnum == 0][0]
5332-        sh0_data = open(sh0_file, "rb").read()
5333+        sh0_fp = [sharefp for (shnum, serverid, sharefp)
5334+                          in self.find_uri_shares(immutable_uri)
5335+                          if shnum == 0][0]
5336+        sh0_data = sh0_fp.getContent()
5337         for clientnum in immutable_shares:
5338             if 0 in immutable_shares[clientnum]:
5339                 continue
5340hunk ./src/allmydata/test/test_download.py 808
5341-            cdir = self.get_serverdir(clientnum)
5342-            target = os.path.join(cdir, "shares", si_dir, "0")
5343-            outf = open(target, "wb")
5344-            outf.write(sh0_data)
5345-            outf.close()
5346+            cdir = self.get_server(clientnum).backend.get_shareset(si)._sharehomedir
5347+            fileutil.fp_make_dirs(cdir)
5348+            cdir.child(str(shnum)).setContent(sh0_data)
5349 
5350         d = self.download_immutable()
5351         return d
5352hunk ./src/allmydata/test/test_encode.py 134
5353         d.addCallback(_try)
5354         return d
5355 
5356-    def get_share_hashes(self, at_least_these=()):
5357+    def get_share_hashes(self):
5358         d = self._start()
5359         def _try(unused=None):
5360             if self.mode == "bad sharehash":
5361hunk ./src/allmydata/test/test_hung_server.py 3
5362 # -*- coding: utf-8 -*-
5363 
5364-import os, shutil
5365 from twisted.trial import unittest
5366 from twisted.internet import defer
5367hunk ./src/allmydata/test/test_hung_server.py 5
5368-from allmydata import uri
5369+
5370 from allmydata.util.consumer import download_to_data
5371 from allmydata.immutable import upload
5372 from allmydata.mutable.common import UnrecoverableFileError
5373hunk ./src/allmydata/test/test_hung_server.py 10
5374 from allmydata.mutable.publish import MutableData
5375-from allmydata.storage.common import storage_index_to_dir
5376 from allmydata.test.no_network import GridTestMixin
5377 from allmydata.test.common import ShouldFailMixin
5378 from allmydata.util.pollmixin import PollMixin
5379hunk ./src/allmydata/test/test_hung_server.py 18
5380 immutable_plaintext = "data" * 10000
5381 mutable_plaintext = "muta" * 10000
5382 
5383+
5384 class HungServerDownloadTest(GridTestMixin, ShouldFailMixin, PollMixin,
5385                              unittest.TestCase):
5386     # Many of these tests take around 60 seconds on François's ARM buildslave:
5387hunk ./src/allmydata/test/test_hung_server.py 31
5388     timeout = 240
5389 
5390     def _break(self, servers):
5391-        for (id, ss) in servers:
5392-            self.g.break_server(id)
5393+        for ss in servers:
5394+            self.g.break_server(ss.get_serverid())
5395 
5396     def _hang(self, servers, **kwargs):
5397hunk ./src/allmydata/test/test_hung_server.py 35
5398-        for (id, ss) in servers:
5399-            self.g.hang_server(id, **kwargs)
5400+        for ss in servers:
5401+            self.g.hang_server(ss.get_serverid(), **kwargs)
5402 
5403     def _unhang(self, servers, **kwargs):
5404hunk ./src/allmydata/test/test_hung_server.py 39
5405-        for (id, ss) in servers:
5406-            self.g.unhang_server(id, **kwargs)
5407+        for ss in servers:
5408+            self.g.unhang_server(ss.get_serverid(), **kwargs)
5409 
5410     def _hang_shares(self, shnums, **kwargs):
5411         # hang all servers who are holding the given shares
5412hunk ./src/allmydata/test/test_hung_server.py 52
5413                     hung_serverids.add(i_serverid)
5414 
5415     def _delete_all_shares_from(self, servers):
5416-        serverids = [id for (id, ss) in servers]
5417-        for (i_shnum, i_serverid, i_sharefile) in self.shares:
5418+        serverids = [ss.get_serverid() for ss in servers]
5419+        for (i_shnum, i_serverid, i_sharefp) in self.shares:
5420             if i_serverid in serverids:
5421hunk ./src/allmydata/test/test_hung_server.py 55
5422-                os.unlink(i_sharefile)
5423+                i_sharefp.remove()
5424 
5425     def _corrupt_all_shares_in(self, servers, corruptor_func):
5426hunk ./src/allmydata/test/test_hung_server.py 58
5427-        serverids = [id for (id, ss) in servers]
5428-        for (i_shnum, i_serverid, i_sharefile) in self.shares:
5429+        serverids = [ss.get_serverid() for ss in servers]
5430+        for (i_shnum, i_serverid, i_sharefp) in self.shares:
5431             if i_serverid in serverids:
5432hunk ./src/allmydata/test/test_hung_server.py 61
5433-                self._corrupt_share((i_shnum, i_sharefile), corruptor_func)
5434+                self.corrupt_share((i_shnum, i_serverid, i_sharefp), corruptor_func)
5435 
5436     def _copy_all_shares_from(self, from_servers, to_server):
5437hunk ./src/allmydata/test/test_hung_server.py 64
5438-        serverids = [id for (id, ss) in from_servers]
5439-        for (i_shnum, i_serverid, i_sharefile) in self.shares:
5440+        serverids = [ss.get_serverid() for ss in from_servers]
5441+        for (i_shnum, i_serverid, i_sharefp) in self.shares:
5442             if i_serverid in serverids:
5443hunk ./src/allmydata/test/test_hung_server.py 67
5444-                self._copy_share((i_shnum, i_sharefile), to_server)
5445+                self.copy_share((i_shnum, i_serverid, i_sharefp), self.uri, to_server)
5446 
5447hunk ./src/allmydata/test/test_hung_server.py 69
5448-    def _copy_share(self, share, to_server):
5449-        (sharenum, sharefile) = share
5450-        (id, ss) = to_server
5451-        shares_dir = os.path.join(ss.original.storedir, "shares")
5452-        si = uri.from_string(self.uri).get_storage_index()
5453-        si_dir = os.path.join(shares_dir, storage_index_to_dir(si))
5454-        if not os.path.exists(si_dir):
5455-            os.makedirs(si_dir)
5456-        new_sharefile = os.path.join(si_dir, str(sharenum))
5457-        shutil.copy(sharefile, new_sharefile)
5458         self.shares = self.find_uri_shares(self.uri)
5459hunk ./src/allmydata/test/test_hung_server.py 70
5460-        # Make sure that the storage server has the share.
5461-        self.failUnless((sharenum, ss.original.my_nodeid, new_sharefile)
5462-                        in self.shares)
5463-
5464-    def _corrupt_share(self, share, corruptor_func):
5465-        (sharenum, sharefile) = share
5466-        data = open(sharefile, "rb").read()
5467-        newdata = corruptor_func(data)
5468-        os.unlink(sharefile)
5469-        wf = open(sharefile, "wb")
5470-        wf.write(newdata)
5471-        wf.close()
5472 
5473     def _set_up(self, mutable, testdir, num_clients=1, num_servers=10):
5474         self.mutable = mutable
5475hunk ./src/allmydata/test/test_hung_server.py 82
5476 
5477         self.c0 = self.g.clients[0]
5478         nm = self.c0.nodemaker
5479-        self.servers = sorted([(s.get_serverid(), s.get_rref())
5480-                               for s in nm.storage_broker.get_connected_servers()])
5481+        unsorted = [(s.get_serverid(), s.get_rref()) for s in nm.storage_broker.get_connected_servers()]
5482+        self.servers = [ss for (id, ss) in sorted(unsorted)]
5483         self.servers = self.servers[5:] + self.servers[:5]
5484 
5485         if mutable:
5486hunk ./src/allmydata/test/test_hung_server.py 244
5487             # stuck-but-not-overdue, and 4 live requests. All 4 live requests
5488             # will retire before the download is complete and the ShareFinder
5489             # is shut off. That will leave 4 OVERDUE and 1
5490-            # stuck-but-not-overdue, for a total of 5 requests in in
5491+            # stuck-but-not-overdue, for a total of 5 requests in
5492             # _sf.pending_requests
5493             for t in self._sf.overdue_timers.values()[:4]:
5494                 t.reset(-1.0)
5495hunk ./src/allmydata/test/test_mutable.py 21
5496 from foolscap.api import eventually, fireEventually
5497 from foolscap.logging import log
5498 from allmydata.storage_client import StorageFarmBroker
5499-from allmydata.storage.common import storage_index_to_dir
5500 from allmydata.scripts import debug
5501 
5502 from allmydata.mutable.filenode import MutableFileNode, BackoffAgent
5503hunk ./src/allmydata/test/test_mutable.py 3669
5504         # Now execute each assignment by writing the storage.
5505         for (share, servernum) in assignments:
5506             sharedata = base64.b64decode(self.sdmf_old_shares[share])
5507-            storedir = self.get_serverdir(servernum)
5508-            storage_path = os.path.join(storedir, "shares",
5509-                                        storage_index_to_dir(si))
5510-            fileutil.make_dirs(storage_path)
5511-            fileutil.write(os.path.join(storage_path, "%d" % share),
5512-                           sharedata)
5513+            storage_dir = self.get_server(servernum).backend.get_shareset(si).sharehomedir
5514+            fileutil.fp_make_dirs(storage_dir)
5515+            storage_dir.child("%d" % share).setContent(sharedata)
5516         # ...and verify that the shares are there.
5517         shares = self.find_uri_shares(self.sdmf_old_cap)
5518         assert len(shares) == 10
5519hunk ./src/allmydata/test/test_provisioning.py 13
5520 from nevow import inevow
5521 from zope.interface import implements
5522 
5523-class MyRequest:
5524+class MockRequest:
5525     implements(inevow.IRequest)
5526     pass
5527 
5528hunk ./src/allmydata/test/test_provisioning.py 26
5529     def test_load(self):
5530         pt = provisioning.ProvisioningTool()
5531         self.fields = {}
5532-        #r = MyRequest()
5533+        #r = MockRequest()
5534         #r.fields = self.fields
5535         #ctx = RequestContext()
5536         #unfilled = pt.renderSynchronously(ctx)
5537hunk ./src/allmydata/test/test_repairer.py 537
5538         # happiness setting.
5539         def _delete_some_servers(ignored):
5540             for i in xrange(7):
5541-                self.g.remove_server(self.g.servers_by_number[i].my_nodeid)
5542+                self.remove_server(i)
5543 
5544             assert len(self.g.servers_by_number) == 3
5545 
5546hunk ./src/allmydata/test/test_storage.py 14
5547 from allmydata import interfaces
5548 from allmydata.util import fileutil, hashutil, base32, pollmixin, time_format
5549 from allmydata.storage.server import StorageServer
5550-from allmydata.storage.mutable import MutableShareFile
5551-from allmydata.storage.immutable import BucketWriter, BucketReader
5552-from allmydata.storage.common import DataTooLargeError, storage_index_to_dir, \
5553+from allmydata.storage.backends.disk.mutable import MutableDiskShare
5554+from allmydata.storage.bucket import BucketWriter, BucketReader
5555+from allmydata.storage.common import DataTooLargeError, \
5556      UnknownMutableContainerVersionError, UnknownImmutableContainerVersionError
5557 from allmydata.storage.lease import LeaseInfo
5558 from allmydata.storage.crawler import BucketCountingCrawler
5559hunk ./src/allmydata/test/test_storage.py 474
5560         w[0].remote_write(0, "\xff"*10)
5561         w[0].remote_close()
5562 
5563-        fn = os.path.join(ss.sharedir, storage_index_to_dir("si1"), "0")
5564-        f = open(fn, "rb+")
5565+        fp = ss.backend.get_shareset("si1").sharehomedir.child("0")
5566+        f = fp.open("rb+")
5567         f.seek(0)
5568         f.write(struct.pack(">L", 0)) # this is invalid: minimum used is v1
5569         f.close()
5570hunk ./src/allmydata/test/test_storage.py 814
5571     def test_bad_magic(self):
5572         ss = self.create("test_bad_magic")
5573         self.allocate(ss, "si1", "we1", self._lease_secret.next(), set([0]), 10)
5574-        fn = os.path.join(ss.sharedir, storage_index_to_dir("si1"), "0")
5575-        f = open(fn, "rb+")
5576+        fp = ss.backend.get_shareset("si1").sharehomedir.child("0")
5577+        f = fp.open("rb+")
5578         f.seek(0)
5579         f.write("BAD MAGIC")
5580         f.close()
5581hunk ./src/allmydata/test/test_storage.py 842
5582 
5583         # Trying to make the container too large (by sending a write vector
5584         # whose offset is too high) will raise an exception.
5585-        TOOBIG = MutableShareFile.MAX_SIZE + 10
5586+        TOOBIG = MutableDiskShare.MAX_SIZE + 10
5587         self.failUnlessRaises(DataTooLargeError,
5588                               rstaraw, "si1", secrets,
5589                               {0: ([], [(TOOBIG,data)], None)},
5590hunk ./src/allmydata/test/test_storage.py 1229
5591 
5592         # create a random non-numeric file in the bucket directory, to
5593         # exercise the code that's supposed to ignore those.
5594-        bucket_dir = os.path.join(self.workdir("test_leases"),
5595-                                  "shares", storage_index_to_dir("si1"))
5596-        f = open(os.path.join(bucket_dir, "ignore_me.txt"), "w")
5597-        f.write("you ought to be ignoring me\n")
5598-        f.close()
5599+        bucket_dir = ss.backend.get_shareset("si1").sharehomedir
5600+        bucket_dir.child("ignore_me.txt").setContent("you ought to be ignoring me\n")
5601 
5602hunk ./src/allmydata/test/test_storage.py 1232
5603-        s0 = MutableShareFile(os.path.join(bucket_dir, "0"))
5604+        s0 = MutableDiskShare(os.path.join(bucket_dir, "0"))
5605         self.failUnlessEqual(len(list(s0.get_leases())), 1)
5606 
5607         # add-lease on a missing storage index is silently ignored
5608hunk ./src/allmydata/test/test_storage.py 3118
5609         [immutable_si_0, immutable_si_1, mutable_si_2, mutable_si_3] = self.sis
5610 
5611         # add a non-sharefile to exercise another code path
5612-        fn = os.path.join(ss.sharedir,
5613-                          storage_index_to_dir(immutable_si_0),
5614-                          "not-a-share")
5615-        f = open(fn, "wb")
5616-        f.write("I am not a share.\n")
5617-        f.close()
5618+        fp = ss.backend.get_shareset(immutable_si_0).sharehomedir.child("not-a-share")
5619+        fp.setContent("I am not a share.\n")
5620 
5621         # this is before the crawl has started, so we're not in a cycle yet
5622         initial_state = lc.get_state()
5623hunk ./src/allmydata/test/test_storage.py 3282
5624     def test_expire_age(self):
5625         basedir = "storage/LeaseCrawler/expire_age"
5626         fileutil.make_dirs(basedir)
5627-        # setting expiration_time to 2000 means that any lease which is more
5628-        # than 2000s old will be expired.
5629-        ss = InstrumentedStorageServer(basedir, "\x00" * 20,
5630-                                       expiration_enabled=True,
5631-                                       expiration_mode="age",
5632-                                       expiration_override_lease_duration=2000)
5633+        # setting 'override_lease_duration' to 2000 means that any lease that
5634+        # is more than 2000 seconds old will be expired.
5635+        expiration_policy = {
5636+            'enabled': True,
5637+            'mode': 'age',
5638+            'override_lease_duration': 2000,
5639+            'sharetypes': ('mutable', 'immutable'),
5640+        }
5641+        ss = InstrumentedStorageServer(basedir, "\x00" * 20, expiration_policy)
5642         # make it start sooner than usual.
5643         lc = ss.lease_checker
5644         lc.slow_start = 0
5645hunk ./src/allmydata/test/test_storage.py 3423
5646     def test_expire_cutoff_date(self):
5647         basedir = "storage/LeaseCrawler/expire_cutoff_date"
5648         fileutil.make_dirs(basedir)
5649-        # setting cutoff-date to 2000 seconds ago means that any lease which
5650-        # is more than 2000s old will be expired.
5651+        # setting 'cutoff_date' to 2000 seconds ago means that any lease that
5652+        # is more than 2000 seconds old will be expired.
5653         now = time.time()
5654         then = int(now - 2000)
5655hunk ./src/allmydata/test/test_storage.py 3427
5656-        ss = InstrumentedStorageServer(basedir, "\x00" * 20,
5657-                                       expiration_enabled=True,
5658-                                       expiration_mode="cutoff-date",
5659-                                       expiration_cutoff_date=then)
5660+        expiration_policy = {
5661+            'enabled': True,
5662+            'mode': 'cutoff-date',
5663+            'cutoff_date': then,
5664+            'sharetypes': ('mutable', 'immutable'),
5665+        }
5666+        ss = InstrumentedStorageServer(basedir, "\x00" * 20, expiration_policy)
5667         # make it start sooner than usual.
5668         lc = ss.lease_checker
5669         lc.slow_start = 0
5670hunk ./src/allmydata/test/test_storage.py 3575
5671     def test_only_immutable(self):
5672         basedir = "storage/LeaseCrawler/only_immutable"
5673         fileutil.make_dirs(basedir)
5674+        # setting 'cutoff_date' to 2000 seconds ago means that any lease that
5675+        # is more than 2000 seconds old will be expired.
5676         now = time.time()
5677         then = int(now - 2000)
5678hunk ./src/allmydata/test/test_storage.py 3579
5679-        ss = StorageServer(basedir, "\x00" * 20,
5680-                           expiration_enabled=True,
5681-                           expiration_mode="cutoff-date",
5682-                           expiration_cutoff_date=then,
5683-                           expiration_sharetypes=("immutable",))
5684+        expiration_policy = {
5685+            'enabled': True,
5686+            'mode': 'cutoff-date',
5687+            'cutoff_date': then,
5688+            'sharetypes': ('immutable',),
5689+        }
5690+        ss = StorageServer(basedir, "\x00" * 20, expiration_policy)
5691         lc = ss.lease_checker
5692         lc.slow_start = 0
5693         webstatus = StorageStatus(ss)
5694hunk ./src/allmydata/test/test_storage.py 3636
5695     def test_only_mutable(self):
5696         basedir = "storage/LeaseCrawler/only_mutable"
5697         fileutil.make_dirs(basedir)
5698+        # setting 'cutoff_date' to 2000 seconds ago means that any lease that
5699+        # is more than 2000 seconds old will be expired.
5700         now = time.time()
5701         then = int(now - 2000)
5702hunk ./src/allmydata/test/test_storage.py 3640
5703-        ss = StorageServer(basedir, "\x00" * 20,
5704-                           expiration_enabled=True,
5705-                           expiration_mode="cutoff-date",
5706-                           expiration_cutoff_date=then,
5707-                           expiration_sharetypes=("mutable",))
5708+        expiration_policy = {
5709+            'enabled': True,
5710+            'mode': 'cutoff-date',
5711+            'cutoff_date': then,
5712+            'sharetypes': ('mutable',),
5713+        }
5714+        ss = StorageServer(basedir, "\x00" * 20, expiration_policy)
5715         lc = ss.lease_checker
5716         lc.slow_start = 0
5717         webstatus = StorageStatus(ss)
5718hunk ./src/allmydata/test/test_storage.py 3819
5719     def test_no_st_blocks(self):
5720         basedir = "storage/LeaseCrawler/no_st_blocks"
5721         fileutil.make_dirs(basedir)
5722-        ss = No_ST_BLOCKS_StorageServer(basedir, "\x00" * 20,
5723-                                        expiration_mode="age",
5724-                                        expiration_override_lease_duration=-1000)
5725-        # a negative expiration_time= means the "configured-"
5726+        # A negative 'override_lease_duration' means that the "configured-"
5727         # space-recovered counts will be non-zero, since all shares will have
5728hunk ./src/allmydata/test/test_storage.py 3821
5729-        # expired by then
5730+        # expired by then.
5731+        expiration_policy = {
5732+            'enabled': True,
5733+            'mode': 'age',
5734+            'override_lease_duration': -1000,
5735+            'sharetypes': ('mutable', 'immutable'),
5736+        }
5737+        ss = No_ST_BLOCKS_StorageServer(basedir, "\x00" * 20, expiration_policy)
5738 
5739         # make it start sooner than usual.
5740         lc = ss.lease_checker
5741hunk ./src/allmydata/test/test_storage.py 3877
5742         [immutable_si_0, immutable_si_1, mutable_si_2, mutable_si_3] = self.sis
5743         first = min(self.sis)
5744         first_b32 = base32.b2a(first)
5745-        fn = os.path.join(ss.sharedir, storage_index_to_dir(first), "0")
5746-        f = open(fn, "rb+")
5747+        fp = ss.backend.get_shareset(first).sharehomedir.child("0")
5748+        f = fp.open("rb+")
5749         f.seek(0)
5750         f.write("BAD MAGIC")
5751         f.close()
5752hunk ./src/allmydata/test/test_storage.py 3890
5753 
5754         # also create an empty bucket
5755         empty_si = base32.b2a("\x04"*16)
5756-        empty_bucket_dir = os.path.join(ss.sharedir,
5757-                                        storage_index_to_dir(empty_si))
5758-        fileutil.make_dirs(empty_bucket_dir)
5759+        empty_bucket_dir = ss.backend.get_shareset(empty_si).sharehomedir
5760+        fileutil.fp_make_dirs(empty_bucket_dir)
5761 
5762         ss.setServiceParent(self.s)
5763 
5764hunk ./src/allmydata/test/test_system.py 10
5765 
5766 import allmydata
5767 from allmydata import uri
5768-from allmydata.storage.mutable import MutableShareFile
5769+from allmydata.storage.backends.disk.mutable import MutableDiskShare
5770 from allmydata.storage.server import si_a2b
5771 from allmydata.immutable import offloaded, upload
5772 from allmydata.immutable.literal import LiteralFileNode
5773hunk ./src/allmydata/test/test_system.py 421
5774         return shares
5775 
5776     def _corrupt_mutable_share(self, filename, which):
5777-        msf = MutableShareFile(filename)
5778+        msf = MutableDiskShare(filename)
5779         datav = msf.readv([ (0, 1000000) ])
5780         final_share = datav[0]
5781         assert len(final_share) < 1000000 # ought to be truncated
5782hunk ./src/allmydata/test/test_upload.py 22
5783 from allmydata.util.happinessutil import servers_of_happiness, \
5784                                          shares_by_server, merge_servers
5785 from allmydata.storage_client import StorageFarmBroker
5786-from allmydata.storage.server import storage_index_to_dir
5787 
5788 MiB = 1024*1024
5789 
5790hunk ./src/allmydata/test/test_upload.py 821
5791 
5792     def _copy_share_to_server(self, share_number, server_number):
5793         ss = self.g.servers_by_number[server_number]
5794-        # Copy share i from the directory associated with the first
5795-        # storage server to the directory associated with this one.
5796-        assert self.g, "I tried to find a grid at self.g, but failed"
5797-        assert self.shares, "I tried to find shares at self.shares, but failed"
5798-        old_share_location = self.shares[share_number][2]
5799-        new_share_location = os.path.join(ss.storedir, "shares")
5800-        si = uri.from_string(self.uri).get_storage_index()
5801-        new_share_location = os.path.join(new_share_location,
5802-                                          storage_index_to_dir(si))
5803-        if not os.path.exists(new_share_location):
5804-            os.makedirs(new_share_location)
5805-        new_share_location = os.path.join(new_share_location,
5806-                                          str(share_number))
5807-        if old_share_location != new_share_location:
5808-            shutil.copy(old_share_location, new_share_location)
5809-        shares = self.find_uri_shares(self.uri)
5810-        # Make sure that the storage server has the share.
5811-        self.failUnless((share_number, ss.my_nodeid, new_share_location)
5812-                        in shares)
5813+        self.copy_share(self.shares[share_number], ss)
5814 
5815     def _setup_grid(self):
5816         """
5817hunk ./src/allmydata/test/test_upload.py 1103
5818                 self._copy_share_to_server(i, 2)
5819         d.addCallback(_copy_shares)
5820         # Remove the first server, and add a placeholder with share 0
5821-        d.addCallback(lambda ign:
5822-            self.g.remove_server(self.g.servers_by_number[0].my_nodeid))
5823+        d.addCallback(lambda ign: self.remove_server(0))
5824         d.addCallback(lambda ign:
5825             self._add_server_with_share(server_number=4, share_number=0))
5826         # Now try uploading.
5827hunk ./src/allmydata/test/test_upload.py 1134
5828         d.addCallback(lambda ign:
5829             self._add_server(server_number=4))
5830         d.addCallback(_copy_shares)
5831-        d.addCallback(lambda ign:
5832-            self.g.remove_server(self.g.servers_by_number[0].my_nodeid))
5833+        d.addCallback(lambda ign: self.remove_server(0))
5834         d.addCallback(_reset_encoding_parameters)
5835         d.addCallback(lambda client:
5836             client.upload(upload.Data("data" * 10000, convergence="")))
5837hunk ./src/allmydata/test/test_upload.py 1196
5838                 self._copy_share_to_server(i, 2)
5839         d.addCallback(_copy_shares)
5840         # Remove server 0, and add another in its place
5841-        d.addCallback(lambda ign:
5842-            self.g.remove_server(self.g.servers_by_number[0].my_nodeid))
5843+        d.addCallback(lambda ign: self.remove_server(0))
5844         d.addCallback(lambda ign:
5845             self._add_server_with_share(server_number=4, share_number=0,
5846                                         readonly=True))
5847hunk ./src/allmydata/test/test_upload.py 1237
5848             for i in xrange(1, 10):
5849                 self._copy_share_to_server(i, 2)
5850         d.addCallback(_copy_shares)
5851-        d.addCallback(lambda ign:
5852-            self.g.remove_server(self.g.servers_by_number[0].my_nodeid))
5853+        d.addCallback(lambda ign: self.remove_server(0))
5854         def _reset_encoding_parameters(ign, happy=4):
5855             client = self.g.clients[0]
5856             client.DEFAULT_ENCODING_PARAMETERS['happy'] = happy
5857hunk ./src/allmydata/test/test_upload.py 1273
5858         # remove the original server
5859         # (necessary to ensure that the Tahoe2ServerSelector will distribute
5860         #  all the shares)
5861-        def _remove_server(ign):
5862-            server = self.g.servers_by_number[0]
5863-            self.g.remove_server(server.my_nodeid)
5864-        d.addCallback(_remove_server)
5865+        d.addCallback(lambda ign: self.remove_server(0))
5866         # This should succeed; we still have 4 servers, and the
5867         # happiness of the upload is 4.
5868         d.addCallback(lambda ign:
5869hunk ./src/allmydata/test/test_upload.py 1285
5870         d.addCallback(lambda ign:
5871             self._setup_and_upload())
5872         d.addCallback(_do_server_setup)
5873-        d.addCallback(_remove_server)
5874+        d.addCallback(lambda ign: self.remove_server(0))
5875         d.addCallback(lambda ign:
5876             self.shouldFail(UploadUnhappinessError,
5877                             "test_dropped_servers_in_encoder",
5878hunk ./src/allmydata/test/test_upload.py 1307
5879             self._add_server_with_share(4, 7, readonly=True)
5880             self._add_server_with_share(5, 8, readonly=True)
5881         d.addCallback(_do_server_setup_2)
5882-        d.addCallback(_remove_server)
5883+        d.addCallback(lambda ign: self.remove_server(0))
5884         d.addCallback(lambda ign:
5885             self._do_upload_with_broken_servers(1))
5886         d.addCallback(_set_basedir)
5887hunk ./src/allmydata/test/test_upload.py 1314
5888         d.addCallback(lambda ign:
5889             self._setup_and_upload())
5890         d.addCallback(_do_server_setup_2)
5891-        d.addCallback(_remove_server)
5892+        d.addCallback(lambda ign: self.remove_server(0))
5893         d.addCallback(lambda ign:
5894             self.shouldFail(UploadUnhappinessError,
5895                             "test_dropped_servers_in_encoder",
5896hunk ./src/allmydata/test/test_upload.py 1528
5897             for i in xrange(1, 10):
5898                 self._copy_share_to_server(i, 1)
5899         d.addCallback(_copy_shares)
5900-        d.addCallback(lambda ign:
5901-            self.g.remove_server(self.g.servers_by_number[0].my_nodeid))
5902+        d.addCallback(lambda ign: self.remove_server(0))
5903         def _prepare_client(ign):
5904             client = self.g.clients[0]
5905             client.DEFAULT_ENCODING_PARAMETERS['happy'] = 4
5906hunk ./src/allmydata/test/test_upload.py 1550
5907         def _setup(ign):
5908             for i in xrange(1, 11):
5909                 self._add_server(server_number=i)
5910-            self.g.remove_server(self.g.servers_by_number[0].my_nodeid)
5911+            self.remove_server(0)
5912             c = self.g.clients[0]
5913             # We set happy to an unsatisfiable value so that we can check the
5914             # counting in the exception message. The same progress message
5915hunk ./src/allmydata/test/test_upload.py 1577
5916                 self._add_server(server_number=i)
5917             self._add_server(server_number=11, readonly=True)
5918             self._add_server(server_number=12, readonly=True)
5919-            self.g.remove_server(self.g.servers_by_number[0].my_nodeid)
5920+            self.remove_server(0)
5921             c = self.g.clients[0]
5922             c.DEFAULT_ENCODING_PARAMETERS['happy'] = 45
5923             return c
5924hunk ./src/allmydata/test/test_upload.py 1605
5925             # the first one that the selector sees.
5926             for i in xrange(10):
5927                 self._copy_share_to_server(i, 9)
5928-            # Remove server 0, and its contents
5929-            self.g.remove_server(self.g.servers_by_number[0].my_nodeid)
5930+            self.remove_server(0)
5931             # Make happiness unsatisfiable
5932             c = self.g.clients[0]
5933             c.DEFAULT_ENCODING_PARAMETERS['happy'] = 45
5934hunk ./src/allmydata/test/test_upload.py 1625
5935         def _then(ign):
5936             for i in xrange(1, 11):
5937                 self._add_server(server_number=i, readonly=True)
5938-            self.g.remove_server(self.g.servers_by_number[0].my_nodeid)
5939+            self.remove_server(0)
5940             c = self.g.clients[0]
5941             c.DEFAULT_ENCODING_PARAMETERS['k'] = 2
5942             c.DEFAULT_ENCODING_PARAMETERS['happy'] = 4
5943hunk ./src/allmydata/test/test_upload.py 1661
5944             self._add_server(server_number=4, readonly=True))
5945         d.addCallback(lambda ign:
5946             self._add_server(server_number=5, readonly=True))
5947-        d.addCallback(lambda ign:
5948-            self.g.remove_server(self.g.servers_by_number[0].my_nodeid))
5949+        d.addCallback(lambda ign: self.remove_server(0))
5950         def _reset_encoding_parameters(ign, happy=4):
5951             client = self.g.clients[0]
5952             client.DEFAULT_ENCODING_PARAMETERS['happy'] = happy
5953hunk ./src/allmydata/test/test_upload.py 1696
5954         d.addCallback(lambda ign:
5955             self._add_server(server_number=2))
5956         def _break_server_2(ign):
5957-            serverid = self.g.servers_by_number[2].my_nodeid
5958+            serverid = self.get_server(2).get_serverid()
5959             self.g.break_server(serverid)
5960         d.addCallback(_break_server_2)
5961         d.addCallback(lambda ign:
5962hunk ./src/allmydata/test/test_upload.py 1705
5963             self._add_server(server_number=4, readonly=True))
5964         d.addCallback(lambda ign:
5965             self._add_server(server_number=5, readonly=True))
5966-        d.addCallback(lambda ign:
5967-            self.g.remove_server(self.g.servers_by_number[0].my_nodeid))
5968+        d.addCallback(lambda ign: self.remove_server(0))
5969         d.addCallback(_reset_encoding_parameters)
5970         d.addCallback(lambda client:
5971             self.shouldFail(UploadUnhappinessError, "test_selection_exceptions",
5972hunk ./src/allmydata/test/test_upload.py 1816
5973             # Copy shares
5974             self._copy_share_to_server(1, 1)
5975             self._copy_share_to_server(2, 1)
5976-            # Remove server 0
5977-            self.g.remove_server(self.g.servers_by_number[0].my_nodeid)
5978+            self.remove_server(0)
5979             client = self.g.clients[0]
5980             client.DEFAULT_ENCODING_PARAMETERS['happy'] = 3
5981             return client
5982hunk ./src/allmydata/test/test_upload.py 1930
5983                                         readonly=True)
5984             self._add_server_with_share(server_number=4, share_number=3,
5985                                         readonly=True)
5986-            # Remove server 0.
5987-            self.g.remove_server(self.g.servers_by_number[0].my_nodeid)
5988+            self.remove_server(0)
5989             # Set the client appropriately
5990             c = self.g.clients[0]
5991             c.DEFAULT_ENCODING_PARAMETERS['happy'] = 4
5992hunk ./src/allmydata/test/test_util.py 9
5993 from twisted.trial import unittest
5994 from twisted.internet import defer, reactor
5995 from twisted.python.failure import Failure
5996+from twisted.python.filepath import FilePath
5997 from twisted.python import log
5998 from pycryptopp.hash.sha256 import SHA256 as _hash
5999 
6000hunk ./src/allmydata/test/test_util.py 508
6001                 os.chdir(saved_cwd)
6002 
6003     def test_disk_stats(self):
6004-        avail = fileutil.get_available_space('.', 2**14)
6005+        avail = fileutil.get_available_space(FilePath('.'), 2**14)
6006         if avail == 0:
6007             raise unittest.SkipTest("This test will spuriously fail there is no disk space left.")
6008 
6009hunk ./src/allmydata/test/test_util.py 512
6010-        disk = fileutil.get_disk_stats('.', 2**13)
6011+        disk = fileutil.get_disk_stats(FilePath('.'), 2**13)
6012         self.failUnless(disk['total'] > 0, disk['total'])
6013         self.failUnless(disk['used'] > 0, disk['used'])
6014         self.failUnless(disk['free_for_root'] > 0, disk['free_for_root'])
6015hunk ./src/allmydata/test/test_util.py 521
6016 
6017     def test_disk_stats_avail_nonnegative(self):
6018         # This test will spuriously fail if you have more than 2^128
6019-        # bytes of available space on your filesystem.
6020-        disk = fileutil.get_disk_stats('.', 2**128)
6021+        # bytes of available space on your filesystem (lucky you).
6022+        disk = fileutil.get_disk_stats(FilePath('.'), 2**128)
6023         self.failUnlessEqual(disk['avail'], 0)
6024 
6025 class PollMixinTests(unittest.TestCase):
6026hunk ./src/allmydata/test/test_web.py 12
6027 from twisted.python import failure, log
6028 from nevow import rend
6029 from allmydata import interfaces, uri, webish, dirnode
6030-from allmydata.storage.shares import get_share_file
6031 from allmydata.storage_client import StorageFarmBroker
6032 from allmydata.immutable import upload
6033 from allmydata.immutable.downloader.status import DownloadStatus
6034hunk ./src/allmydata/test/test_web.py 4111
6035             good_shares = self.find_uri_shares(self.uris["good"])
6036             self.failUnlessReallyEqual(len(good_shares), 10)
6037             sick_shares = self.find_uri_shares(self.uris["sick"])
6038-            os.unlink(sick_shares[0][2])
6039+            sick_shares[0][2].remove()
6040             dead_shares = self.find_uri_shares(self.uris["dead"])
6041             for i in range(1, 10):
6042hunk ./src/allmydata/test/test_web.py 4114
6043-                os.unlink(dead_shares[i][2])
6044+                dead_shares[i][2].remove()
6045             c_shares = self.find_uri_shares(self.uris["corrupt"])
6046             cso = CorruptShareOptions()
6047             cso.stdout = StringIO()
6048hunk ./src/allmydata/test/test_web.py 4118
6049-            cso.parseOptions([c_shares[0][2]])
6050+            cso.parseOptions([c_shares[0][2].path])
6051             corrupt_share(cso)
6052         d.addCallback(_clobber_shares)
6053 
6054hunk ./src/allmydata/test/test_web.py 4253
6055             good_shares = self.find_uri_shares(self.uris["good"])
6056             self.failUnlessReallyEqual(len(good_shares), 10)
6057             sick_shares = self.find_uri_shares(self.uris["sick"])
6058-            os.unlink(sick_shares[0][2])
6059+            sick_shares[0][2].remove()
6060             dead_shares = self.find_uri_shares(self.uris["dead"])
6061             for i in range(1, 10):
6062hunk ./src/allmydata/test/test_web.py 4256
6063-                os.unlink(dead_shares[i][2])
6064+                dead_shares[i][2].remove()
6065             c_shares = self.find_uri_shares(self.uris["corrupt"])
6066             cso = CorruptShareOptions()
6067             cso.stdout = StringIO()
6068hunk ./src/allmydata/test/test_web.py 4260
6069-            cso.parseOptions([c_shares[0][2]])
6070+            cso.parseOptions([c_shares[0][2].path])
6071             corrupt_share(cso)
6072         d.addCallback(_clobber_shares)
6073 
6074hunk ./src/allmydata/test/test_web.py 4319
6075 
6076         def _clobber_shares(ignored):
6077             sick_shares = self.find_uri_shares(self.uris["sick"])
6078-            os.unlink(sick_shares[0][2])
6079+            sick_shares[0][2].remove()
6080         d.addCallback(_clobber_shares)
6081 
6082         d.addCallback(self.CHECK, "sick", "t=check&repair=true&output=json")
6083hunk ./src/allmydata/test/test_web.py 4811
6084             good_shares = self.find_uri_shares(self.uris["good"])
6085             self.failUnlessReallyEqual(len(good_shares), 10)
6086             sick_shares = self.find_uri_shares(self.uris["sick"])
6087-            os.unlink(sick_shares[0][2])
6088+            sick_shares[0][2].remove()
6089             #dead_shares = self.find_uri_shares(self.uris["dead"])
6090             #for i in range(1, 10):
6091hunk ./src/allmydata/test/test_web.py 4814
6092-            #    os.unlink(dead_shares[i][2])
6093+            #    dead_shares[i][2].remove()
6094 
6095             #c_shares = self.find_uri_shares(self.uris["corrupt"])
6096             #cso = CorruptShareOptions()
6097hunk ./src/allmydata/test/test_web.py 4819
6098             #cso.stdout = StringIO()
6099-            #cso.parseOptions([c_shares[0][2]])
6100+            #cso.parseOptions([c_shares[0][2].path])
6101             #corrupt_share(cso)
6102         d.addCallback(_clobber_shares)
6103 
6104hunk ./src/allmydata/test/test_web.py 4870
6105         d.addErrback(self.explain_web_error)
6106         return d
6107 
6108-    def _count_leases(self, ignored, which):
6109-        u = self.uris[which]
6110-        shares = self.find_uri_shares(u)
6111-        lease_counts = []
6112-        for shnum, serverid, fn in shares:
6113-            sf = get_share_file(fn)
6114-            num_leases = len(list(sf.get_leases()))
6115-            lease_counts.append( (fn, num_leases) )
6116-        return lease_counts
6117-
6118-    def _assert_leasecount(self, lease_counts, expected):
6119+    def _assert_leasecount(self, ignored, which, expected):
6120+        lease_counts = self.count_leases(self.uris[which])
6121         for (fn, num_leases) in lease_counts:
6122             if num_leases != expected:
6123                 self.fail("expected %d leases, have %d, on %s" %
6124hunk ./src/allmydata/test/test_web.py 4903
6125                 self.fileurls[which] = "uri/" + urllib.quote(self.uris[which])
6126         d.addCallback(_compute_fileurls)
6127 
6128-        d.addCallback(self._count_leases, "one")
6129-        d.addCallback(self._assert_leasecount, 1)
6130-        d.addCallback(self._count_leases, "two")
6131-        d.addCallback(self._assert_leasecount, 1)
6132-        d.addCallback(self._count_leases, "mutable")
6133-        d.addCallback(self._assert_leasecount, 1)
6134+        d.addCallback(self._assert_leasecount, "one", 1)
6135+        d.addCallback(self._assert_leasecount, "two", 1)
6136+        d.addCallback(self._assert_leasecount, "mutable", 1)
6137 
6138         d.addCallback(self.CHECK, "one", "t=check") # no add-lease
6139         def _got_html_good(res):
6140hunk ./src/allmydata/test/test_web.py 4913
6141             self.failIf("Not Healthy" in res, res)
6142         d.addCallback(_got_html_good)
6143 
6144-        d.addCallback(self._count_leases, "one")
6145-        d.addCallback(self._assert_leasecount, 1)
6146-        d.addCallback(self._count_leases, "two")
6147-        d.addCallback(self._assert_leasecount, 1)
6148-        d.addCallback(self._count_leases, "mutable")
6149-        d.addCallback(self._assert_leasecount, 1)
6150+        d.addCallback(self._assert_leasecount, "one", 1)
6151+        d.addCallback(self._assert_leasecount, "two", 1)
6152+        d.addCallback(self._assert_leasecount, "mutable", 1)
6153 
6154         # this CHECK uses the original client, which uses the same
6155         # lease-secrets, so it will just renew the original lease
6156hunk ./src/allmydata/test/test_web.py 4922
6157         d.addCallback(self.CHECK, "one", "t=check&add-lease=true")
6158         d.addCallback(_got_html_good)
6159 
6160-        d.addCallback(self._count_leases, "one")
6161-        d.addCallback(self._assert_leasecount, 1)
6162-        d.addCallback(self._count_leases, "two")
6163-        d.addCallback(self._assert_leasecount, 1)
6164-        d.addCallback(self._count_leases, "mutable")
6165-        d.addCallback(self._assert_leasecount, 1)
6166+        d.addCallback(self._assert_leasecount, "one", 1)
6167+        d.addCallback(self._assert_leasecount, "two", 1)
6168+        d.addCallback(self._assert_leasecount, "mutable", 1)
6169 
6170         # this CHECK uses an alternate client, which adds a second lease
6171         d.addCallback(self.CHECK, "one", "t=check&add-lease=true", clientnum=1)
6172hunk ./src/allmydata/test/test_web.py 4930
6173         d.addCallback(_got_html_good)
6174 
6175-        d.addCallback(self._count_leases, "one")
6176-        d.addCallback(self._assert_leasecount, 2)
6177-        d.addCallback(self._count_leases, "two")
6178-        d.addCallback(self._assert_leasecount, 1)
6179-        d.addCallback(self._count_leases, "mutable")
6180-        d.addCallback(self._assert_leasecount, 1)
6181+        d.addCallback(self._assert_leasecount, "one", 2)
6182+        d.addCallback(self._assert_leasecount, "two", 1)
6183+        d.addCallback(self._assert_leasecount, "mutable", 1)
6184 
6185         d.addCallback(self.CHECK, "mutable", "t=check&add-lease=true")
6186         d.addCallback(_got_html_good)
6187hunk ./src/allmydata/test/test_web.py 4937
6188 
6189-        d.addCallback(self._count_leases, "one")
6190-        d.addCallback(self._assert_leasecount, 2)
6191-        d.addCallback(self._count_leases, "two")
6192-        d.addCallback(self._assert_leasecount, 1)
6193-        d.addCallback(self._count_leases, "mutable")
6194-        d.addCallback(self._assert_leasecount, 1)
6195+        d.addCallback(self._assert_leasecount, "one", 2)
6196+        d.addCallback(self._assert_leasecount, "two", 1)
6197+        d.addCallback(self._assert_leasecount, "mutable", 1)
6198 
6199         d.addCallback(self.CHECK, "mutable", "t=check&add-lease=true",
6200                       clientnum=1)
6201hunk ./src/allmydata/test/test_web.py 4945
6202         d.addCallback(_got_html_good)
6203 
6204-        d.addCallback(self._count_leases, "one")
6205-        d.addCallback(self._assert_leasecount, 2)
6206-        d.addCallback(self._count_leases, "two")
6207-        d.addCallback(self._assert_leasecount, 1)
6208-        d.addCallback(self._count_leases, "mutable")
6209-        d.addCallback(self._assert_leasecount, 2)
6210+        d.addCallback(self._assert_leasecount, "one", 2)
6211+        d.addCallback(self._assert_leasecount, "two", 1)
6212+        d.addCallback(self._assert_leasecount, "mutable", 2)
6213 
6214         d.addErrback(self.explain_web_error)
6215         return d
6216hunk ./src/allmydata/test/test_web.py 4989
6217             self.failUnlessReallyEqual(len(units), 4+1)
6218         d.addCallback(_done)
6219 
6220-        d.addCallback(self._count_leases, "root")
6221-        d.addCallback(self._assert_leasecount, 1)
6222-        d.addCallback(self._count_leases, "one")
6223-        d.addCallback(self._assert_leasecount, 1)
6224-        d.addCallback(self._count_leases, "mutable")
6225-        d.addCallback(self._assert_leasecount, 1)
6226+        d.addCallback(self._assert_leasecount, "root", 1)
6227+        d.addCallback(self._assert_leasecount, "one", 1)
6228+        d.addCallback(self._assert_leasecount, "mutable", 1)
6229 
6230         d.addCallback(self.CHECK, "root", "t=stream-deep-check&add-lease=true")
6231         d.addCallback(_done)
6232hunk ./src/allmydata/test/test_web.py 4996
6233 
6234-        d.addCallback(self._count_leases, "root")
6235-        d.addCallback(self._assert_leasecount, 1)
6236-        d.addCallback(self._count_leases, "one")
6237-        d.addCallback(self._assert_leasecount, 1)
6238-        d.addCallback(self._count_leases, "mutable")
6239-        d.addCallback(self._assert_leasecount, 1)
6240+        d.addCallback(self._assert_leasecount, "root", 1)
6241+        d.addCallback(self._assert_leasecount, "one", 1)
6242+        d.addCallback(self._assert_leasecount, "mutable", 1)
6243 
6244         d.addCallback(self.CHECK, "root", "t=stream-deep-check&add-lease=true",
6245                       clientnum=1)
6246hunk ./src/allmydata/test/test_web.py 5004
6247         d.addCallback(_done)
6248 
6249-        d.addCallback(self._count_leases, "root")
6250-        d.addCallback(self._assert_leasecount, 2)
6251-        d.addCallback(self._count_leases, "one")
6252-        d.addCallback(self._assert_leasecount, 2)
6253-        d.addCallback(self._count_leases, "mutable")
6254-        d.addCallback(self._assert_leasecount, 2)
6255+        d.addCallback(self._assert_leasecount, "root", 2)
6256+        d.addCallback(self._assert_leasecount, "one", 2)
6257+        d.addCallback(self._assert_leasecount, "mutable", 2)
6258 
6259         d.addErrback(self.explain_web_error)
6260         return d
6261merger 0.0 (
6262hunk ./src/allmydata/uri.py 829
6263+    def is_readonly(self):
6264+        return True
6265+
6266+    def get_readonly(self):
6267+        return self
6268+
6269+
6270hunk ./src/allmydata/uri.py 829
6271+    def is_readonly(self):
6272+        return True
6273+
6274+    def get_readonly(self):
6275+        return self
6276+
6277+
6278)
6279merger 0.0 (
6280hunk ./src/allmydata/uri.py 848
6281+    def is_readonly(self):
6282+        return True
6283+
6284+    def get_readonly(self):
6285+        return self
6286+
6287hunk ./src/allmydata/uri.py 848
6288+    def is_readonly(self):
6289+        return True
6290+
6291+    def get_readonly(self):
6292+        return self
6293+
6294)
6295hunk ./src/allmydata/util/encodingutil.py 221
6296 def quote_path(path, quotemarks=True):
6297     return quote_output("/".join(map(to_str, path)), quotemarks=quotemarks)
6298 
6299+def quote_filepath(fp, quotemarks=True, encoding=None):
6300+    path = fp.path
6301+    if isinstance(path, str):
6302+        try:
6303+            path = path.decode(filesystem_encoding)
6304+        except UnicodeDecodeError:
6305+            return 'b"%s"' % (ESCAPABLE_8BIT.sub(_str_escape, path),)
6306+
6307+    return quote_output(path, quotemarks=quotemarks, encoding=encoding)
6308+
6309 
6310 def unicode_platform():
6311     """
6312hunk ./src/allmydata/util/fileutil.py 5
6313 Futz with files like a pro.
6314 """
6315 
6316-import sys, exceptions, os, stat, tempfile, time, binascii
6317+import errno, sys, exceptions, os, stat, tempfile, time, binascii
6318+
6319+from allmydata.util.assertutil import precondition
6320 
6321 from twisted.python import log
6322hunk ./src/allmydata/util/fileutil.py 10
6323+from twisted.python.filepath import FilePath, UnlistableError
6324 
6325 from pycryptopp.cipher.aes import AES
6326 
6327hunk ./src/allmydata/util/fileutil.py 189
6328             raise tx
6329         raise exceptions.IOError, "unknown error prevented creation of directory, or deleted the directory immediately after creation: %s" % dirname # careful not to construct an IOError with a 2-tuple, as that has a special meaning...
6330 
6331-def rm_dir(dirname):
6332+def fp_make_dirs(dirfp):
6333+    """
6334+    An idempotent version of FilePath.makedirs().  If the dir already
6335+    exists, do nothing and return without raising an exception.  If this
6336+    call creates the dir, return without raising an exception.  If there is
6337+    an error that prevents creation or if the directory gets deleted after
6338+    fp_make_dirs() creates it and before fp_make_dirs() checks that it
6339+    exists, raise an exception.
6340+    """
6341+    log.msg( "xxx 0 %s" % (dirfp,))
6342+    tx = None
6343+    try:
6344+        dirfp.makedirs()
6345+    except OSError, x:
6346+        tx = x
6347+
6348+    if not dirfp.isdir():
6349+        if tx:
6350+            raise tx
6351+        raise exceptions.IOError, "unknown error prevented creation of directory, or deleted the directory immediately after creation: %s" % dirfp # careful not to construct an IOError with a 2-tuple, as that has a special meaning...
6352+
6353+def fp_rmdir_if_empty(dirfp):
6354+    """ Remove the directory if it is empty. """
6355+    try:
6356+        os.rmdir(dirfp.path)
6357+    except OSError, e:
6358+        if e.errno != errno.ENOTEMPTY:
6359+            raise
6360+    else:
6361+        dirfp.changed()
6362+
6363+def rmtree(dirname):
6364     """
6365     A threadsafe and idempotent version of shutil.rmtree().  If the dir is
6366     already gone, do nothing and return without raising an exception.  If this
6367hunk ./src/allmydata/util/fileutil.py 239
6368             else:
6369                 remove(fullname)
6370         os.rmdir(dirname)
6371-    except Exception, le:
6372-        # Ignore "No such file or directory"
6373-        if (not isinstance(le, OSError)) or le.args[0] != 2:
6374+    except EnvironmentError, le:
6375+        # Ignore "No such file or directory", collect any other exception.
6376+        if (le.args[0] != 2 and le.args[0] != 3) or (le.args[0] != errno.ENOENT):
6377             excs.append(le)
6378hunk ./src/allmydata/util/fileutil.py 243
6379+    except Exception, le:
6380+        excs.append(le)
6381 
6382     # Okay, now we've recursively removed everything, ignoring any "No
6383     # such file or directory" errors, and collecting any other errors.
6384hunk ./src/allmydata/util/fileutil.py 256
6385             raise OSError, "Failed to remove dir for unknown reason."
6386         raise OSError, excs
6387 
6388+def fp_remove(fp):
6389+    """
6390+    An idempotent version of shutil.rmtree().  If the file/dir is already
6391+    gone, do nothing and return without raising an exception.  If this call
6392+    removes the file/dir, return without raising an exception.  If there is
6393+    an error that prevents removal, or if a file or directory at the same
6394+    path gets created again by someone else after this deletes it and before
6395+    this checks that it is gone, raise an exception.
6396+    """
6397+    try:
6398+        fp.remove()
6399+    except UnlistableError, e:
6400+        if e.originalException.errno != errno.ENOENT:
6401+            raise
6402+    except OSError, e:
6403+        if e.errno != errno.ENOENT:
6404+            raise
6405+
6406+def rm_dir(dirname):
6407+    # Renamed to be like shutil.rmtree and unlike rmdir.
6408+    return rmtree(dirname)
6409 
6410 def remove_if_possible(f):
6411     try:
6412hunk ./src/allmydata/util/fileutil.py 387
6413         import traceback
6414         traceback.print_exc()
6415 
6416-def get_disk_stats(whichdir, reserved_space=0):
6417+def get_disk_stats(whichdirfp, reserved_space=0):
6418     """Return disk statistics for the storage disk, in the form of a dict
6419     with the following fields.
6420       total:            total bytes on disk
6421hunk ./src/allmydata/util/fileutil.py 408
6422     you can pass how many bytes you would like to leave unused on this
6423     filesystem as reserved_space.
6424     """
6425+    precondition(isinstance(whichdirfp, FilePath), whichdirfp)
6426 
6427     if have_GetDiskFreeSpaceExW:
6428         # If this is a Windows system and GetDiskFreeSpaceExW is available, use it.
6429hunk ./src/allmydata/util/fileutil.py 419
6430         n_free_for_nonroot = c_ulonglong(0)
6431         n_total            = c_ulonglong(0)
6432         n_free_for_root    = c_ulonglong(0)
6433-        retval = GetDiskFreeSpaceExW(whichdir, byref(n_free_for_nonroot),
6434-                                               byref(n_total),
6435-                                               byref(n_free_for_root))
6436+        retval = GetDiskFreeSpaceExW(whichdirfp.path, byref(n_free_for_nonroot),
6437+                                                      byref(n_total),
6438+                                                      byref(n_free_for_root))
6439         if retval == 0:
6440             raise OSError("Windows error %d attempting to get disk statistics for %r"
6441hunk ./src/allmydata/util/fileutil.py 424
6442-                          % (GetLastError(), whichdir))
6443+                          % (GetLastError(), whichdirfp.path))
6444         free_for_nonroot = n_free_for_nonroot.value
6445         total            = n_total.value
6446         free_for_root    = n_free_for_root.value
6447hunk ./src/allmydata/util/fileutil.py 433
6448         # <http://docs.python.org/library/os.html#os.statvfs>
6449         # <http://opengroup.org/onlinepubs/7990989799/xsh/fstatvfs.html>
6450         # <http://opengroup.org/onlinepubs/7990989799/xsh/sysstatvfs.h.html>
6451-        s = os.statvfs(whichdir)
6452+        s = os.statvfs(whichdirfp.path)
6453 
6454         # on my mac laptop:
6455         #  statvfs(2) is a wrapper around statfs(2).
6456hunk ./src/allmydata/util/fileutil.py 460
6457              'avail': avail,
6458            }
6459 
6460-def get_available_space(whichdir, reserved_space):
6461+def get_available_space(whichdirfp, reserved_space):
6462     """Returns available space for share storage in bytes, or None if no
6463     API to get this information is available.
6464 
6465hunk ./src/allmydata/util/fileutil.py 472
6466     you can pass how many bytes you would like to leave unused on this
6467     filesystem as reserved_space.
6468     """
6469+    precondition(isinstance(whichdirfp, FilePath), whichdirfp)
6470     try:
6471hunk ./src/allmydata/util/fileutil.py 474
6472-        return get_disk_stats(whichdir, reserved_space)['avail']
6473+        return get_disk_stats(whichdirfp, reserved_space)['avail']
6474     except AttributeError:
6475         return None
6476hunk ./src/allmydata/util/fileutil.py 477
6477-    except EnvironmentError:
6478-        log.msg("OS call to get disk statistics failed")
6479+
6480+
6481+def get_used_space(fp):
6482+    if fp is None:
6483         return 0
6484hunk ./src/allmydata/util/fileutil.py 482
6485+    try:
6486+        s = os.stat(fp.path)
6487+    except EnvironmentError:
6488+        if not fp.exists():
6489+            return 0
6490+        raise
6491+    else:
6492+        # POSIX defines st_blocks (originally a BSDism):
6493+        #   <http://pubs.opengroup.org/onlinepubs/009695399/basedefs/sys/stat.h.html>
6494+        # but does not require stat() to give it a "meaningful value"
6495+        #   <http://pubs.opengroup.org/onlinepubs/009695399/functions/stat.html>
6496+        # and says:
6497+        #   "The unit for the st_blocks member of the stat structure is not defined
6498+        #    within IEEE Std 1003.1-2001. In some implementations it is 512 bytes.
6499+        #    It may differ on a file system basis. There is no correlation between
6500+        #    values of the st_blocks and st_blksize, and the f_bsize (from <sys/statvfs.h>)
6501+        #    structure members."
6502+        #
6503+        # The Linux docs define it as "the number of blocks allocated to the file,
6504+        # [in] 512-byte units." It is also defined that way on MacOS X. Python does
6505+        # not set the attribute on Windows.
6506+        #
6507+        # We consider platforms that define st_blocks but give it a wrong value, or
6508+        # measure it in a unit other than 512 bytes, to be broken. See also
6509+        # <http://bugs.python.org/issue12350>.
6510+
6511+        if hasattr(s, 'st_blocks'):
6512+            return s.st_blocks * 512
6513+        else:
6514+            return s.st_size
6515}
6516[Work-in-progress, includes fix to bug involving BucketWriter. refs #999
6517david-sarah@jacaranda.org**20110920033803
6518 Ignore-this: 64e9e019421454e4d08141d10b6e4eed
6519] {
6520hunk ./src/allmydata/client.py 9
6521 from twisted.internet import reactor, defer
6522 from twisted.application import service
6523 from twisted.application.internet import TimerService
6524+from twisted.python.filepath import FilePath
6525 from foolscap.api import Referenceable
6526 from pycryptopp.publickey import rsa
6527 
6528hunk ./src/allmydata/client.py 15
6529 import allmydata
6530 from allmydata.storage.server import StorageServer
6531+from allmydata.storage.backends.disk.disk_backend import DiskBackend
6532 from allmydata import storage_client
6533 from allmydata.immutable.upload import Uploader
6534 from allmydata.immutable.offloaded import Helper
6535hunk ./src/allmydata/client.py 213
6536             return
6537         readonly = self.get_config("storage", "readonly", False, boolean=True)
6538 
6539-        storedir = os.path.join(self.basedir, self.STOREDIR)
6540+        storedir = FilePath(self.basedir).child(self.STOREDIR)
6541 
6542         data = self.get_config("storage", "reserved_space", None)
6543         reserved = None
6544hunk ./src/allmydata/client.py 255
6545             'cutoff_date': cutoff_date,
6546             'sharetypes': tuple(sharetypes),
6547         }
6548-        ss = StorageServer(storedir, self.nodeid,
6549-                           reserved_space=reserved,
6550-                           discard_storage=discard,
6551-                           readonly_storage=readonly,
6552+
6553+        backend = DiskBackend(storedir, readonly=readonly, reserved_space=reserved,
6554+                              discard_storage=discard)
6555+        ss = StorageServer(nodeid, backend, storedir,
6556                            stats_provider=self.stats_provider,
6557                            expiration_policy=expiration_policy)
6558         self.add_service(ss)
6559hunk ./src/allmydata/interfaces.py 348
6560 
6561     def get_shares():
6562         """
6563-        Generates the IStoredShare objects held in this shareset.
6564+        Generates IStoredShare objects for all completed shares in this shareset.
6565         """
6566 
6567     def has_incoming(shnum):
6568hunk ./src/allmydata/storage/backends/base.py 69
6569         # def _create_mutable_share(self, storageserver, shnum, write_enabler):
6570         #     """create a mutable share with the given shnum and write_enabler"""
6571 
6572-        # secrets might be a triple with cancel_secret in secrets[2], but if
6573-        # so we ignore the cancel_secret.
6574         write_enabler = secrets[0]
6575         renew_secret = secrets[1]
6576hunk ./src/allmydata/storage/backends/base.py 71
6577+        cancel_secret = '\x00'*32
6578+        if len(secrets) > 2:
6579+            cancel_secret = secrets[2]
6580 
6581         si_s = self.get_storage_index_string()
6582         shares = {}
6583hunk ./src/allmydata/storage/backends/base.py 110
6584             read_data[shnum] = share.readv(read_vector)
6585 
6586         ownerid = 1 # TODO
6587-        lease_info = LeaseInfo(ownerid, renew_secret,
6588+        lease_info = LeaseInfo(ownerid, renew_secret, cancel_secret,
6589                                expiration_time, storageserver.get_serverid())
6590 
6591         if testv_is_good:
6592hunk ./src/allmydata/storage/backends/disk/disk_backend.py 34
6593     return newfp.child(sia)
6594 
6595 
6596-def get_share(fp):
6597+def get_share(storageindex, shnum, fp):
6598     f = fp.open('rb')
6599     try:
6600         prefix = f.read(32)
6601hunk ./src/allmydata/storage/backends/disk/disk_backend.py 42
6602         f.close()
6603 
6604     if prefix == MutableDiskShare.MAGIC:
6605-        return MutableDiskShare(fp)
6606+        return MutableDiskShare(storageindex, shnum, fp)
6607     else:
6608         # assume it's immutable
6609hunk ./src/allmydata/storage/backends/disk/disk_backend.py 45
6610-        return ImmutableDiskShare(fp)
6611+        return ImmutableDiskShare(storageindex, shnum, fp)
6612 
6613 
6614 class DiskBackend(Backend):
6615hunk ./src/allmydata/storage/backends/disk/disk_backend.py 174
6616                 if not NUM_RE.match(shnumstr):
6617                     continue
6618                 sharehome = self._sharehomedir.child(shnumstr)
6619-                yield self.get_share(sharehome)
6620+                yield get_share(self.get_storage_index(), int(shnumstr), sharehome)
6621         except UnlistableError:
6622             # There is no shares directory at all.
6623             pass
6624hunk ./src/allmydata/storage/backends/disk/disk_backend.py 185
6625         return self._incominghomedir.child(str(shnum)).exists()
6626 
6627     def make_bucket_writer(self, storageserver, shnum, max_space_per_bucket, lease_info, canary):
6628-        sharehome = self._sharehomedir.child(str(shnum))
6629+        finalhome = self._sharehomedir.child(str(shnum))
6630         incominghome = self._incominghomedir.child(str(shnum))
6631hunk ./src/allmydata/storage/backends/disk/disk_backend.py 187
6632-        immsh = ImmutableDiskShare(self.get_storage_index(), shnum, sharehome, incominghome,
6633-                                   max_size=max_space_per_bucket, create=True)
6634+        immsh = ImmutableDiskShare(self.get_storage_index(), shnum, incominghome, finalhome,
6635+                                   max_size=max_space_per_bucket)
6636         bw = BucketWriter(storageserver, immsh, max_space_per_bucket, lease_info, canary)
6637         if self._discard_storage:
6638             bw.throw_out_all_data = True
6639hunk ./src/allmydata/storage/backends/disk/disk_backend.py 198
6640         fileutil.fp_make_dirs(self._sharehomedir)
6641         sharehome = self._sharehomedir.child(str(shnum))
6642         serverid = storageserver.get_serverid()
6643-        return create_mutable_disk_share(sharehome, serverid, write_enabler, storageserver)
6644+        return create_mutable_disk_share(self.get_storage_index(), shnum, sharehome, serverid, write_enabler, storageserver)
6645 
6646     def _clean_up_after_unlink(self):
6647         fileutil.fp_rmdir_if_empty(self._sharehomedir)
6648hunk ./src/allmydata/storage/backends/disk/immutable.py 48
6649     LEASE_SIZE = struct.calcsize(">L32s32sL")
6650 
6651 
6652-    def __init__(self, storageindex, shnum, finalhome=None, incominghome=None, max_size=None, create=False):
6653-        """ If max_size is not None then I won't allow more than
6654-        max_size to be written to me. If create=True then max_size
6655-        must not be None. """
6656-        precondition((max_size is not None) or (not create), max_size, create)
6657+    def __init__(self, storageindex, shnum, home, finalhome=None, max_size=None):
6658+        """
6659+        If max_size is not None then I won't allow more than max_size to be written to me.
6660+        If finalhome is not None (meaning that we are creating the share) then max_size
6661+        must not be None.
6662+        """
6663+        precondition((max_size is not None) or (finalhome is None), max_size, finalhome)
6664         self._storageindex = storageindex
6665         self._max_size = max_size
6666hunk ./src/allmydata/storage/backends/disk/immutable.py 57
6667-        self._incominghome = incominghome
6668-        self._home = finalhome
6669+
6670+        # If we are creating the share, _finalhome refers to the final path and
6671+        # _home to the incoming path. Otherwise, _finalhome is None.
6672+        self._finalhome = finalhome
6673+        self._home = home
6674         self._shnum = shnum
6675hunk ./src/allmydata/storage/backends/disk/immutable.py 63
6676-        if create:
6677-            # touch the file, so later callers will see that we're working on
6678+
6679+        if self._finalhome is not None:
6680+            # Touch the file, so later callers will see that we're working on
6681             # it. Also construct the metadata.
6682hunk ./src/allmydata/storage/backends/disk/immutable.py 67
6683-            assert not finalhome.exists()
6684-            fp_make_dirs(self._incominghome.parent())
6685+            assert not self._finalhome.exists()
6686+            fp_make_dirs(self._home.parent())
6687             # The second field -- the four-byte share data length -- is no
6688             # longer used as of Tahoe v1.3.0, but we continue to write it in
6689             # there in case someone downgrades a storage server from >=
6690hunk ./src/allmydata/storage/backends/disk/immutable.py 78
6691             # the largest length that can fit into the field. That way, even
6692             # if this does happen, the old < v1.3.0 server will still allow
6693             # clients to read the first part of the share.
6694-            self._incominghome.setContent(struct.pack(">LLL", 1, min(2**32-1, max_size), 0) )
6695+            self._home.setContent(struct.pack(">LLL", 1, min(2**32-1, max_size), 0) )
6696             self._lease_offset = max_size + 0x0c
6697             self._num_leases = 0
6698         else:
6699hunk ./src/allmydata/storage/backends/disk/immutable.py 101
6700                 % (si_b2a(self._storageindex), self._shnum, quote_filepath(self._home)))
6701 
6702     def close(self):
6703-        fileutil.fp_make_dirs(self._home.parent())
6704-        self._incominghome.moveTo(self._home)
6705-        try:
6706-            # self._incominghome is like storage/shares/incoming/ab/abcde/4 .
6707-            # We try to delete the parent (.../ab/abcde) to avoid leaving
6708-            # these directories lying around forever, but the delete might
6709-            # fail if we're working on another share for the same storage
6710-            # index (like ab/abcde/5). The alternative approach would be to
6711-            # use a hierarchy of objects (PrefixHolder, BucketHolder,
6712-            # ShareWriter), each of which is responsible for a single
6713-            # directory on disk, and have them use reference counting of
6714-            # their children to know when they should do the rmdir. This
6715-            # approach is simpler, but relies on os.rmdir refusing to delete
6716-            # a non-empty directory. Do *not* use fileutil.fp_remove() here!
6717-            fileutil.fp_rmdir_if_empty(self._incominghome.parent())
6718-            # we also delete the grandparent (prefix) directory, .../ab ,
6719-            # again to avoid leaving directories lying around. This might
6720-            # fail if there is another bucket open that shares a prefix (like
6721-            # ab/abfff).
6722-            fileutil.fp_rmdir_if_empty(self._incominghome.parent().parent())
6723-            # we leave the great-grandparent (incoming/) directory in place.
6724-        except EnvironmentError:
6725-            # ignore the "can't rmdir because the directory is not empty"
6726-            # exceptions, those are normal consequences of the
6727-            # above-mentioned conditions.
6728-            pass
6729-        pass
6730+        fileutil.fp_make_dirs(self._finalhome.parent())
6731+        self._home.moveTo(self._finalhome)
6732+
6733+        # self._home is like storage/shares/incoming/ab/abcde/4 .
6734+        # We try to delete the parent (.../ab/abcde) to avoid leaving
6735+        # these directories lying around forever, but the delete might
6736+        # fail if we're working on another share for the same storage
6737+        # index (like ab/abcde/5). The alternative approach would be to
6738+        # use a hierarchy of objects (PrefixHolder, BucketHolder,
6739+        # ShareWriter), each of which is responsible for a single
6740+        # directory on disk, and have them use reference counting of
6741+        # their children to know when they should do the rmdir. This
6742+        # approach is simpler, but relies on os.rmdir (used by
6743+        # fp_rmdir_if_empty) refusing to delete a non-empty directory.
6744+        # Do *not* use fileutil.fp_remove() here!
6745+        parent = self._home.parent()
6746+        fileutil.fp_rmdir_if_empty(parent)
6747+
6748+        # we also delete the grandparent (prefix) directory, .../ab ,
6749+        # again to avoid leaving directories lying around. This might
6750+        # fail if there is another bucket open that shares a prefix (like
6751+        # ab/abfff).
6752+        fileutil.fp_rmdir_if_empty(parent.parent())
6753+
6754+        # we leave the great-grandparent (incoming/) directory in place.
6755+
6756+        # allow lease changes after closing.
6757+        self._home = self._finalhome
6758+        self._finalhome = None
6759 
6760     def get_used_space(self):
6761hunk ./src/allmydata/storage/backends/disk/immutable.py 132
6762-        return (fileutil.get_used_space(self._home) +
6763-                fileutil.get_used_space(self._incominghome))
6764+        return (fileutil.get_used_space(self._finalhome) +
6765+                fileutil.get_used_space(self._home))
6766 
6767     def get_storage_index(self):
6768         return self._storageindex
6769hunk ./src/allmydata/storage/backends/disk/immutable.py 175
6770         precondition(offset >= 0, offset)
6771         if self._max_size is not None and offset+length > self._max_size:
6772             raise DataTooLargeError(self._max_size, offset, length)
6773-        f = self._incominghome.open(mode='rb+')
6774+        f = self._home.open(mode='rb+')
6775         try:
6776             real_offset = self._data_offset+offset
6777             f.seek(real_offset)
6778hunk ./src/allmydata/storage/backends/disk/immutable.py 205
6779 
6780     # These lease operations are intended for use by disk_backend.py.
6781     # Other clients should not depend on the fact that the disk backend
6782-    # stores leases in share files.
6783+    # stores leases in share files. XXX bucket.py also relies on this.
6784 
6785     def get_leases(self):
6786         """Yields a LeaseInfo instance for all leases."""
6787hunk ./src/allmydata/storage/backends/disk/immutable.py 221
6788             f.close()
6789 
6790     def add_lease(self, lease_info):
6791-        f = self._incominghome.open(mode='rb')
6792+        f = self._home.open(mode='rb+')
6793         try:
6794             num_leases = self._read_num_leases(f)
6795hunk ./src/allmydata/storage/backends/disk/immutable.py 224
6796-        finally:
6797-            f.close()
6798-        f = self._home.open(mode='wb+')
6799-        try:
6800             self._write_lease_record(f, num_leases, lease_info)
6801             self._write_num_leases(f, num_leases+1)
6802         finally:
6803hunk ./src/allmydata/storage/backends/disk/mutable.py 440
6804         pass
6805 
6806 
6807-def create_mutable_disk_share(fp, serverid, write_enabler, parent):
6808-    ms = MutableDiskShare(fp, parent)
6809+def create_mutable_disk_share(storageindex, shnum, fp, serverid, write_enabler, parent):
6810+    ms = MutableDiskShare(storageindex, shnum, fp, parent)
6811     ms.create(serverid, write_enabler)
6812     del ms
6813hunk ./src/allmydata/storage/backends/disk/mutable.py 444
6814-    return MutableDiskShare(fp, parent)
6815+    return MutableDiskShare(storageindex, shnum, fp, parent)
6816hunk ./src/allmydata/storage/bucket.py 44
6817         start = time.time()
6818 
6819         self._share.close()
6820-        filelen = self._share.stat()
6821+        # XXX should this be self._share.get_used_space() ?
6822+        consumed_size = self._share.get_size()
6823         self._share = None
6824 
6825         self.closed = True
6826hunk ./src/allmydata/storage/bucket.py 51
6827         self._canary.dontNotifyOnDisconnect(self._disconnect_marker)
6828 
6829-        self.ss.bucket_writer_closed(self, filelen)
6830+        self.ss.bucket_writer_closed(self, consumed_size)
6831         self.ss.add_latency("close", time.time() - start)
6832         self.ss.count("close")
6833 
6834hunk ./src/allmydata/storage/server.py 182
6835                                 renew_secret, cancel_secret,
6836                                 sharenums, allocated_size,
6837                                 canary, owner_num=0):
6838-        # cancel_secret is no longer used.
6839         # owner_num is not for clients to set, but rather it should be
6840         # curried into a StorageServer instance dedicated to a particular
6841         # owner.
6842hunk ./src/allmydata/storage/server.py 195
6843         # Note that the lease should not be added until the BucketWriter
6844         # has been closed.
6845         expire_time = time.time() + 31*24*60*60
6846-        lease_info = LeaseInfo(owner_num, renew_secret,
6847+        lease_info = LeaseInfo(owner_num, renew_secret, cancel_secret,
6848                                expire_time, self._serverid)
6849 
6850         max_space_per_bucket = allocated_size
6851hunk ./src/allmydata/test/no_network.py 349
6852         return self.g.servers_by_number[i]
6853 
6854     def get_serverdir(self, i):
6855-        return self.g.servers_by_number[i].backend.storedir
6856+        return self.g.servers_by_number[i].backend._storedir
6857 
6858     def remove_server(self, i):
6859         self.g.remove_server(self.g.servers_by_number[i].get_serverid())
6860hunk ./src/allmydata/test/no_network.py 357
6861     def iterate_servers(self):
6862         for i in sorted(self.g.servers_by_number.keys()):
6863             ss = self.g.servers_by_number[i]
6864-            yield (i, ss, ss.backend.storedir)
6865+            yield (i, ss, ss.backend._storedir)
6866 
6867     def find_uri_shares(self, uri):
6868         si = tahoe_uri.from_string(uri).get_storage_index()
6869hunk ./src/allmydata/test/no_network.py 384
6870         return shares
6871 
6872     def copy_share(self, from_share, uri, to_server):
6873-        si = uri.from_string(self.uri).get_storage_index()
6874+        si = tahoe_uri.from_string(uri).get_storage_index()
6875         (i_shnum, i_serverid, i_sharefp) = from_share
6876         shares_dir = to_server.backend.get_shareset(si)._sharehomedir
6877         i_sharefp.copyTo(shares_dir.child(str(i_shnum)))
6878hunk ./src/allmydata/test/test_download.py 127
6879 
6880         return d
6881 
6882-    def _write_shares(self, uri, shares):
6883-        si = uri.from_string(uri).get_storage_index()
6884+    def _write_shares(self, fileuri, shares):
6885+        si = uri.from_string(fileuri).get_storage_index()
6886         for i in shares:
6887             shares_for_server = shares[i]
6888             for shnum in shares_for_server:
6889hunk ./src/allmydata/test/test_hung_server.py 36
6890 
6891     def _hang(self, servers, **kwargs):
6892         for ss in servers:
6893-            self.g.hang_server(ss.get_serverid(), **kwargs)
6894+            self.g.hang_server(ss.original.get_serverid(), **kwargs)
6895 
6896     def _unhang(self, servers, **kwargs):
6897         for ss in servers:
6898hunk ./src/allmydata/test/test_hung_server.py 40
6899-            self.g.unhang_server(ss.get_serverid(), **kwargs)
6900+            self.g.unhang_server(ss.original.get_serverid(), **kwargs)
6901 
6902     def _hang_shares(self, shnums, **kwargs):
6903         # hang all servers who are holding the given shares
6904hunk ./src/allmydata/test/test_hung_server.py 52
6905                     hung_serverids.add(i_serverid)
6906 
6907     def _delete_all_shares_from(self, servers):
6908-        serverids = [ss.get_serverid() for ss in servers]
6909+        serverids = [ss.original.get_serverid() for ss in servers]
6910         for (i_shnum, i_serverid, i_sharefp) in self.shares:
6911             if i_serverid in serverids:
6912                 i_sharefp.remove()
6913hunk ./src/allmydata/test/test_hung_server.py 58
6914 
6915     def _corrupt_all_shares_in(self, servers, corruptor_func):
6916-        serverids = [ss.get_serverid() for ss in servers]
6917+        serverids = [ss.original.get_serverid() for ss in servers]
6918         for (i_shnum, i_serverid, i_sharefp) in self.shares:
6919             if i_serverid in serverids:
6920                 self.corrupt_share((i_shnum, i_serverid, i_sharefp), corruptor_func)
6921hunk ./src/allmydata/test/test_hung_server.py 64
6922 
6923     def _copy_all_shares_from(self, from_servers, to_server):
6924-        serverids = [ss.get_serverid() for ss in from_servers]
6925+        serverids = [ss.original.get_serverid() for ss in from_servers]
6926         for (i_shnum, i_serverid, i_sharefp) in self.shares:
6927             if i_serverid in serverids:
6928                 self.copy_share((i_shnum, i_serverid, i_sharefp), self.uri, to_server)
6929hunk ./src/allmydata/test/test_mutable.py 2990
6930             fso = debug.FindSharesOptions()
6931             storage_index = base32.b2a(n.get_storage_index())
6932             fso.si_s = storage_index
6933-            fso.nodedirs = [unicode(os.path.dirname(os.path.abspath(storedir)))
6934+            fso.nodedirs = [unicode(storedir.parent().path)
6935                             for (i,ss,storedir)
6936                             in self.iterate_servers()]
6937             fso.stdout = StringIO()
6938hunk ./src/allmydata/test/test_upload.py 818
6939         if share_number is not None:
6940             self._copy_share_to_server(share_number, server_number)
6941 
6942-
6943     def _copy_share_to_server(self, share_number, server_number):
6944         ss = self.g.servers_by_number[server_number]
6945hunk ./src/allmydata/test/test_upload.py 820
6946-        self.copy_share(self.shares[share_number], ss)
6947+        self.copy_share(self.shares[share_number], self.uri, ss)
6948 
6949     def _setup_grid(self):
6950         """
6951}
6952[docs/backends: document the configuration options for the pluggable backends scheme. refs #999
6953david-sarah@jacaranda.org**20110920171737
6954 Ignore-this: 5947e864682a43cb04e557334cda7c19
6955] {
6956adddir ./docs/backends
6957addfile ./docs/backends/S3.rst
6958hunk ./docs/backends/S3.rst 1
6959+====================================================
6960+Storing Shares in Amazon Simple Storage Service (S3)
6961+====================================================
6962+
6963+S3 is a commercial storage service provided by Amazon, described at
6964+`<https://aws.amazon.com/s3/>`_.
6965+
6966+The Tahoe-LAFS storage server can be configured to store its shares in
6967+an S3 bucket, rather than on local filesystem. To enable this, add the
6968+following keys to the server's ``tahoe.cfg`` file:
6969+
6970+``[storage]``
6971+
6972+``backend = s3``
6973+
6974+    This turns off the local filesystem backend and enables use of S3.
6975+
6976+``s3.access_key_id = (string, required)``
6977+``s3.secret_access_key = (string, required)``
6978+
6979+    These two give the storage server permission to access your Amazon
6980+    Web Services account, allowing them to upload and download shares
6981+    from S3.
6982+
6983+``s3.bucket = (string, required)``
6984+
6985+    This controls which bucket will be used to hold shares. The Tahoe-LAFS
6986+    storage server will only modify and access objects in the configured S3
6987+    bucket.
6988+
6989+``s3.url = (URL string, optional)``
6990+
6991+    This URL tells the storage server how to access the S3 service. It
6992+    defaults to ``http://s3.amazonaws.com``, but by setting it to something
6993+    else, you may be able to use some other S3-like service if it is
6994+    sufficiently compatible.
6995+
6996+``s3.max_space = (str, optional)``
6997+
6998+    This tells the server to limit how much space can be used in the S3
6999+    bucket. Before each share is uploaded, the server will ask S3 for the
7000+    current bucket usage, and will only accept the share if it does not cause
7001+    the usage to grow above this limit.
7002+
7003+    The string contains a number, with an optional case-insensitive scale
7004+    suffix like "K" or "M" or "G", and an optional "B" or "iB" suffix. So
7005+    "100MB", "100M", "100000000B", "100000000", and "100000kb" all mean the
7006+    same thing. Likewise, "1MiB", "1024KiB", and "1048576B" all mean the same
7007+    thing.
7008+
7009+    If ``s3.max_space`` is omitted, the default behavior is to allow
7010+    unlimited usage.
7011+
7012+
7013+Once configured, the WUI "storage server" page will provide information about
7014+how much space is being used and how many shares are being stored.
7015+
7016+
7017+Issues
7018+------
7019+
7020+Objects in an S3 bucket cannot be read for free. As a result, when Tahoe-LAFS
7021+is configured to store shares in S3 rather than on local disk, some common
7022+operations may behave differently:
7023+
7024+* Lease crawling/expiration is not yet implemented. As a result, shares will
7025+  be retained forever, and the Storage Server status web page will not show
7026+  information about the number of mutable/immutable shares present.
7027+
7028+* Enabling ``s3.max_space`` causes an extra S3 usage query to be sent for
7029+  each share upload, causing the upload process to run slightly slower and
7030+  incur more S3 request charges.
7031addfile ./docs/backends/disk.rst
7032hunk ./docs/backends/disk.rst 1
7033+====================================
7034+Storing Shares on a Local Filesystem
7035+====================================
7036+
7037+The "disk" backend stores shares on the local filesystem. Versions of
7038+Tahoe-LAFS <= 1.9.0 always stored shares in this way.
7039+
7040+``[storage]``
7041+
7042+``backend = disk``
7043+
7044+    This enables use of the disk backend, and is the default.
7045+
7046+``reserved_space = (str, optional)``
7047+
7048+    If provided, this value defines how much disk space is reserved: the
7049+    storage server will not accept any share that causes the amount of free
7050+    disk space to drop below this value. (The free space is measured by a
7051+    call to statvfs(2) on Unix, or GetDiskFreeSpaceEx on Windows, and is the
7052+    space available to the user account under which the storage server runs.)
7053+
7054+    This string contains a number, with an optional case-insensitive scale
7055+    suffix like "K" or "M" or "G", and an optional "B" or "iB" suffix. So
7056+    "100MB", "100M", "100000000B", "100000000", and "100000kb" all mean the
7057+    same thing. Likewise, "1MiB", "1024KiB", and "1048576B" all mean the same
7058+    thing.
7059+
7060+    "``tahoe create-node``" generates a tahoe.cfg with
7061+    "``reserved_space=1G``", but you may wish to raise, lower, or remove the
7062+    reservation to suit your needs.
7063+
7064+``expire.enabled =``
7065+
7066+``expire.mode =``
7067+
7068+``expire.override_lease_duration =``
7069+
7070+``expire.cutoff_date =``
7071+
7072+``expire.immutable =``
7073+
7074+``expire.mutable =``
7075+
7076+    These settings control garbage collection, causing the server to
7077+    delete shares that no longer have an up-to-date lease on them. Please
7078+    see `<garbage-collection.rst>`_ for full details.
7079hunk ./docs/configuration.rst 436
7080     <http://tahoe-lafs.org/trac/tahoe-lafs/ticket/390>`_ for the current
7081     status of this bug. The default value is ``False``.
7082 
7083-``reserved_space = (str, optional)``
7084+``backend = (string, optional)``
7085 
7086hunk ./docs/configuration.rst 438
7087-    If provided, this value defines how much disk space is reserved: the
7088-    storage server will not accept any share that causes the amount of free
7089-    disk space to drop below this value. (The free space is measured by a
7090-    call to statvfs(2) on Unix, or GetDiskFreeSpaceEx on Windows, and is the
7091-    space available to the user account under which the storage server runs.)
7092+    Storage servers can store the data into different "backends". Clients
7093+    need not be aware of which backend is used by a server. The default
7094+    value is ``disk``.
7095 
7096hunk ./docs/configuration.rst 442
7097-    This string contains a number, with an optional case-insensitive scale
7098-    suffix like "K" or "M" or "G", and an optional "B" or "iB" suffix. So
7099-    "100MB", "100M", "100000000B", "100000000", and "100000kb" all mean the
7100-    same thing. Likewise, "1MiB", "1024KiB", and "1048576B" all mean the same
7101-    thing.
7102+``backend = disk``
7103 
7104hunk ./docs/configuration.rst 444
7105-    "``tahoe create-node``" generates a tahoe.cfg with
7106-    "``reserved_space=1G``", but you may wish to raise, lower, or remove the
7107-    reservation to suit your needs.
7108+    The default is to store shares on the local filesystem (in
7109+    BASEDIR/storage/shares/). For configuration details (including how to
7110+    reserve a minimum amount of free space), see `<backends/disk.rst>`_.
7111 
7112hunk ./docs/configuration.rst 448
7113-``expire.enabled =``
7114+``backend = S3``
7115 
7116hunk ./docs/configuration.rst 450
7117-``expire.mode =``
7118-
7119-``expire.override_lease_duration =``
7120-
7121-``expire.cutoff_date =``
7122-
7123-``expire.immutable =``
7124-
7125-``expire.mutable =``
7126-
7127-    These settings control garbage collection, in which the server will
7128-    delete shares that no longer have an up-to-date lease on them. Please see
7129-    `<garbage-collection.rst>`_ for full details.
7130+    The storage server can store all shares to an Amazon Simple Storage
7131+    Service (S3) bucket. For configuration details, see `<backends/S3.rst>`_.
7132 
7133 
7134 Running A Helper
7135}
7136[Fix some incorrect attribute accesses. refs #999
7137david-sarah@jacaranda.org**20110921031207
7138 Ignore-this: f1ea4c3ea191f6d4b719afaebd2b2bcd
7139] {
7140hunk ./src/allmydata/client.py 258
7141 
7142         backend = DiskBackend(storedir, readonly=readonly, reserved_space=reserved,
7143                               discard_storage=discard)
7144-        ss = StorageServer(nodeid, backend, storedir,
7145+        ss = StorageServer(self.nodeid, backend, storedir,
7146                            stats_provider=self.stats_provider,
7147                            expiration_policy=expiration_policy)
7148         self.add_service(ss)
7149hunk ./src/allmydata/interfaces.py 449
7150         Returns the storage index.
7151         """
7152 
7153+    def get_storage_index_string():
7154+        """
7155+        Returns the base32-encoded storage index.
7156+        """
7157+
7158     def get_shnum():
7159         """
7160         Returns the share number.
7161hunk ./src/allmydata/storage/backends/disk/immutable.py 138
7162     def get_storage_index(self):
7163         return self._storageindex
7164 
7165+    def get_storage_index_string(self):
7166+        return si_b2a(self._storageindex)
7167+
7168     def get_shnum(self):
7169         return self._shnum
7170 
7171hunk ./src/allmydata/storage/backends/disk/mutable.py 119
7172     def get_storage_index(self):
7173         return self._storageindex
7174 
7175+    def get_storage_index_string(self):
7176+        return si_b2a(self._storageindex)
7177+
7178     def get_shnum(self):
7179         return self._shnum
7180 
7181hunk ./src/allmydata/storage/bucket.py 86
7182     def __init__(self, ss, share):
7183         self.ss = ss
7184         self._share = share
7185-        self.storageindex = share.storageindex
7186-        self.shnum = share.shnum
7187+        self.storageindex = share.get_storage_index()
7188+        self.shnum = share.get_shnum()
7189 
7190     def __repr__(self):
7191         return "<%s %s %s>" % (self.__class__.__name__,
7192hunk ./src/allmydata/storage/expirer.py 6
7193 from twisted.python import log as twlog
7194 
7195 from allmydata.storage.crawler import ShareCrawler
7196-from allmydata.storage.common import si_b2a, UnknownMutableContainerVersionError, \
7197+from allmydata.storage.common import UnknownMutableContainerVersionError, \
7198      UnknownImmutableContainerVersionError
7199 
7200 
7201hunk ./src/allmydata/storage/expirer.py 124
7202                     struct.error):
7203                 twlog.msg("lease-checker error processing %r" % (share,))
7204                 twlog.err()
7205-                which = (si_b2a(share.storageindex), share.get_shnum())
7206+                which = (share.get_storage_index_string(), share.get_shnum())
7207                 self.state["cycle-to-date"]["corrupt-shares"].append(which)
7208                 wks = (1, 1, 1, "unknown")
7209             would_keep_shares.append(wks)
7210hunk ./src/allmydata/storage/server.py 221
7211         alreadygot = set()
7212         for share in shareset.get_shares():
7213             share.add_or_renew_lease(lease_info)
7214-            alreadygot.add(share.shnum)
7215+            alreadygot.add(share.get_shnum())
7216 
7217         for shnum in sharenums - alreadygot:
7218             if shareset.has_incoming(shnum):
7219hunk ./src/allmydata/storage/server.py 324
7220 
7221         try:
7222             shareset = self.backend.get_shareset(storageindex)
7223-            return shareset.readv(self, shares, readv)
7224+            return shareset.readv(shares, readv)
7225         finally:
7226             self.add_latency("readv", time.time() - start)
7227 
7228hunk ./src/allmydata/storage/shares.py 1
7229-#! /usr/bin/python
7230-
7231-from allmydata.storage.mutable import MutableShareFile
7232-from allmydata.storage.immutable import ShareFile
7233-
7234-def get_share_file(filename):
7235-    f = open(filename, "rb")
7236-    prefix = f.read(32)
7237-    f.close()
7238-    if prefix == MutableShareFile.MAGIC:
7239-        return MutableShareFile(filename)
7240-    # otherwise assume it's immutable
7241-    return ShareFile(filename)
7242-
7243rmfile ./src/allmydata/storage/shares.py
7244hunk ./src/allmydata/test/no_network.py 387
7245         si = tahoe_uri.from_string(uri).get_storage_index()
7246         (i_shnum, i_serverid, i_sharefp) = from_share
7247         shares_dir = to_server.backend.get_shareset(si)._sharehomedir
7248+        fileutil.fp_make_dirs(shares_dir)
7249         i_sharefp.copyTo(shares_dir.child(str(i_shnum)))
7250 
7251     def restore_all_shares(self, shares):
7252hunk ./src/allmydata/test/no_network.py 391
7253-        for share, data in shares.items():
7254-            share.home.setContent(data)
7255+        for sharepath, data in shares.items():
7256+            FilePath(sharepath).setContent(data)
7257 
7258     def delete_share(self, (shnum, serverid, sharefp)):
7259         sharefp.remove()
7260hunk ./src/allmydata/test/test_upload.py 744
7261         servertoshnums = {} # k: server, v: set(shnum)
7262 
7263         for i, c in self.g.servers_by_number.iteritems():
7264-            for (dirp, dirns, fns) in os.walk(c.sharedir):
7265+            for (dirp, dirns, fns) in os.walk(c.backend._sharedir.path):
7266                 for fn in fns:
7267                     try:
7268                         sharenum = int(fn)
7269}
7270[docs/backends/S3.rst: remove Issues section. refs #999
7271david-sarah@jacaranda.org**20110921031625
7272 Ignore-this: c83d8f52b790bc32488869e6ee1df8c2
7273] hunk ./docs/backends/S3.rst 57
7274 
7275 Once configured, the WUI "storage server" page will provide information about
7276 how much space is being used and how many shares are being stored.
7277-
7278-
7279-Issues
7280-------
7281-
7282-Objects in an S3 bucket cannot be read for free. As a result, when Tahoe-LAFS
7283-is configured to store shares in S3 rather than on local disk, some common
7284-operations may behave differently:
7285-
7286-* Lease crawling/expiration is not yet implemented. As a result, shares will
7287-  be retained forever, and the Storage Server status web page will not show
7288-  information about the number of mutable/immutable shares present.
7289-
7290-* Enabling ``s3.max_space`` causes an extra S3 usage query to be sent for
7291-  each share upload, causing the upload process to run slightly slower and
7292-  incur more S3 request charges.
7293[docs/backends/S3.rst, disk.rst: describe type of space settings as 'quantity of space', not 'str'. refs #999
7294david-sarah@jacaranda.org**20110921031705
7295 Ignore-this: a74ed8e01b0a1ab5f07a1487d7bf138
7296] {
7297hunk ./docs/backends/S3.rst 38
7298     else, you may be able to use some other S3-like service if it is
7299     sufficiently compatible.
7300 
7301-``s3.max_space = (str, optional)``
7302+``s3.max_space = (quantity of space, optional)``
7303 
7304     This tells the server to limit how much space can be used in the S3
7305     bucket. Before each share is uploaded, the server will ask S3 for the
7306hunk ./docs/backends/disk.rst 14
7307 
7308     This enables use of the disk backend, and is the default.
7309 
7310-``reserved_space = (str, optional)``
7311+``reserved_space = (quantity of space, optional)``
7312 
7313     If provided, this value defines how much disk space is reserved: the
7314     storage server will not accept any share that causes the amount of free
7315}
7316[More fixes to tests needed for pluggable backends. refs #999
7317david-sarah@jacaranda.org**20110921184649
7318 Ignore-this: 9be0d3a98e350fd4e17a07d2c00bb4ca
7319] {
7320hunk ./src/allmydata/scripts/debug.py 8
7321 from twisted.python import usage, failure
7322 from twisted.internet import defer
7323 from twisted.scripts import trial as twisted_trial
7324+from twisted.python.filepath import FilePath
7325 
7326 
7327 class DumpOptions(usage.Options):
7328hunk ./src/allmydata/scripts/debug.py 38
7329         self['filename'] = argv_to_abspath(filename)
7330 
7331 def dump_share(options):
7332-    from allmydata.storage.mutable import MutableShareFile
7333+    from allmydata.storage.backends.disk.disk_backend import get_share
7334     from allmydata.util.encodingutil import quote_output
7335 
7336     out = options.stdout
7337hunk ./src/allmydata/scripts/debug.py 46
7338     # check the version, to see if we have a mutable or immutable share
7339     print >>out, "share filename: %s" % quote_output(options['filename'])
7340 
7341-    f = open(options['filename'], "rb")
7342-    prefix = f.read(32)
7343-    f.close()
7344-    if prefix == MutableShareFile.MAGIC:
7345-        return dump_mutable_share(options)
7346-    # otherwise assume it's immutable
7347-    return dump_immutable_share(options)
7348-
7349-def dump_immutable_share(options):
7350-    from allmydata.storage.immutable import ShareFile
7351+    share = get_share("", 0, fp)
7352+    if share.sharetype == "mutable":
7353+        return dump_mutable_share(options, share)
7354+    else:
7355+        assert share.sharetype == "immutable", share.sharetype
7356+        return dump_immutable_share(options)
7357 
7358hunk ./src/allmydata/scripts/debug.py 53
7359+def dump_immutable_share(options, share):
7360     out = options.stdout
7361hunk ./src/allmydata/scripts/debug.py 55
7362-    f = ShareFile(options['filename'])
7363     if not options["leases-only"]:
7364hunk ./src/allmydata/scripts/debug.py 56
7365-        dump_immutable_chk_share(f, out, options)
7366-    dump_immutable_lease_info(f, out)
7367+        dump_immutable_chk_share(share, out, options)
7368+    dump_immutable_lease_info(share, out)
7369     print >>out
7370     return 0
7371 
7372hunk ./src/allmydata/scripts/debug.py 166
7373     return when
7374 
7375 
7376-def dump_mutable_share(options):
7377-    from allmydata.storage.mutable import MutableShareFile
7378+def dump_mutable_share(options, m):
7379     from allmydata.util import base32, idlib
7380     out = options.stdout
7381hunk ./src/allmydata/scripts/debug.py 169
7382-    m = MutableShareFile(options['filename'])
7383     f = open(options['filename'], "rb")
7384     WE, nodeid = m._read_write_enabler_and_nodeid(f)
7385     num_extra_leases = m._read_num_extra_leases(f)
7386hunk ./src/allmydata/scripts/debug.py 641
7387     /home/warner/testnet/node-1/storage/shares/44k/44kai1tui348689nrw8fjegc8c/9
7388     /home/warner/testnet/node-2/storage/shares/44k/44kai1tui348689nrw8fjegc8c/2
7389     """
7390-    from allmydata.storage.server import si_a2b, storage_index_to_dir
7391-    from allmydata.util.encodingutil import listdir_unicode
7392+    from allmydata.storage.server import si_a2b
7393+    from allmydata.storage.backends.disk_backend import si_si2dir
7394+    from allmydata.util.encodingutil import quote_filepath
7395 
7396     out = options.stdout
7397hunk ./src/allmydata/scripts/debug.py 646
7398-    sharedir = storage_index_to_dir(si_a2b(options.si_s))
7399-    for d in options.nodedirs:
7400-        d = os.path.join(d, "storage/shares", sharedir)
7401-        if os.path.exists(d):
7402-            for shnum in listdir_unicode(d):
7403-                print >>out, os.path.join(d, shnum)
7404+    si = si_a2b(options.si_s)
7405+    for nodedir in options.nodedirs:
7406+        sharedir = si_si2dir(nodedir.child("storage").child("shares"), si)
7407+        if sharedir.exists():
7408+            for sharefp in sharedir.children():
7409+                print >>out, quote_filepath(sharefp, quotemarks=False)
7410 
7411     return 0
7412 
7413hunk ./src/allmydata/scripts/debug.py 878
7414         print >>err, "Error processing %s" % quote_output(si_dir)
7415         failure.Failure().printTraceback(err)
7416 
7417+
7418 class CorruptShareOptions(usage.Options):
7419     def getSynopsis(self):
7420         return "Usage: tahoe debug corrupt-share SHARE_FILENAME"
7421hunk ./src/allmydata/scripts/debug.py 902
7422 Obviously, this command should not be used in normal operation.
7423 """
7424         return t
7425+
7426     def parseArgs(self, filename):
7427         self['filename'] = filename
7428 
7429hunk ./src/allmydata/scripts/debug.py 907
7430 def corrupt_share(options):
7431+    do_corrupt_share(options.stdout, FilePath(options['filename']), options['offset'])
7432+
7433+def do_corrupt_share(out, fp, offset="block-random"):
7434     import random
7435hunk ./src/allmydata/scripts/debug.py 911
7436-    from allmydata.storage.mutable import MutableShareFile
7437-    from allmydata.storage.immutable import ShareFile
7438+    from allmydata.storage.backends.disk.mutable import MutableDiskShare
7439+    from allmydata.storage.backends.disk.immutable import ImmutableDiskShare
7440     from allmydata.mutable.layout import unpack_header
7441     from allmydata.immutable.layout import ReadBucketProxy
7442hunk ./src/allmydata/scripts/debug.py 915
7443-    out = options.stdout
7444-    fn = options['filename']
7445-    assert options["offset"] == "block-random", "other offsets not implemented"
7446+
7447+    assert offset == "block-random", "other offsets not implemented"
7448+
7449     # first, what kind of share is it?
7450 
7451     def flip_bit(start, end):
7452hunk ./src/allmydata/scripts/debug.py 924
7453         offset = random.randrange(start, end)
7454         bit = random.randrange(0, 8)
7455         print >>out, "[%d..%d):  %d.b%d" % (start, end, offset, bit)
7456-        f = open(fn, "rb+")
7457-        f.seek(offset)
7458-        d = f.read(1)
7459-        d = chr(ord(d) ^ 0x01)
7460-        f.seek(offset)
7461-        f.write(d)
7462-        f.close()
7463+        f = fp.open("rb+")
7464+        try:
7465+            f.seek(offset)
7466+            d = f.read(1)
7467+            d = chr(ord(d) ^ 0x01)
7468+            f.seek(offset)
7469+            f.write(d)
7470+        finally:
7471+            f.close()
7472 
7473hunk ./src/allmydata/scripts/debug.py 934
7474-    f = open(fn, "rb")
7475-    prefix = f.read(32)
7476-    f.close()
7477-    if prefix == MutableShareFile.MAGIC:
7478-        # mutable
7479-        m = MutableShareFile(fn)
7480-        f = open(fn, "rb")
7481-        f.seek(m.DATA_OFFSET)
7482-        data = f.read(2000)
7483-        # make sure this slot contains an SMDF share
7484-        assert data[0] == "\x00", "non-SDMF mutable shares not supported"
7485+    f = fp.open("rb")
7486+    try:
7487+        prefix = f.read(32)
7488+    finally:
7489         f.close()
7490hunk ./src/allmydata/scripts/debug.py 939
7491+    if prefix == MutableDiskShare.MAGIC:
7492+        # mutable
7493+        m = MutableDiskShare("", 0, fp)
7494+        f = fp.open("rb")
7495+        try:
7496+            f.seek(m.DATA_OFFSET)
7497+            data = f.read(2000)
7498+            # make sure this slot contains an SMDF share
7499+            assert data[0] == "\x00", "non-SDMF mutable shares not supported"
7500+        finally:
7501+            f.close()
7502 
7503         (version, ig_seqnum, ig_roothash, ig_IV, ig_k, ig_N, ig_segsize,
7504          ig_datalen, offsets) = unpack_header(data)
7505hunk ./src/allmydata/scripts/debug.py 960
7506         flip_bit(start, end)
7507     else:
7508         # otherwise assume it's immutable
7509-        f = ShareFile(fn)
7510+        f = ImmutableDiskShare("", 0, fp)
7511         bp = ReadBucketProxy(None, None, '')
7512         offsets = bp._parse_offsets(f.read_share_data(0, 0x24))
7513         start = f._data_offset + offsets["data"]
7514hunk ./src/allmydata/storage/backends/base.py 92
7515             (testv, datav, new_length) = test_and_write_vectors[sharenum]
7516             if sharenum in shares:
7517                 if not shares[sharenum].check_testv(testv):
7518-                    self.log("testv failed: [%d]: %r" % (sharenum, testv))
7519+                    storageserver.log("testv failed: [%d]: %r" % (sharenum, testv))
7520                     testv_is_good = False
7521                     break
7522             else:
7523hunk ./src/allmydata/storage/backends/base.py 99
7524                 # compare the vectors against an empty share, in which all
7525                 # reads return empty strings
7526                 if not EmptyShare().check_testv(testv):
7527-                    self.log("testv failed (empty): [%d] %r" % (sharenum,
7528-                                                                testv))
7529+                    storageserver.log("testv failed (empty): [%d] %r" % (sharenum, testv))
7530                     testv_is_good = False
7531                     break
7532 
7533hunk ./src/allmydata/test/test_cli.py 2892
7534             # delete one, corrupt a second
7535             shares = self.find_uri_shares(self.uri)
7536             self.failUnlessReallyEqual(len(shares), 10)
7537-            os.unlink(shares[0][2])
7538-            cso = debug.CorruptShareOptions()
7539-            cso.stdout = StringIO()
7540-            cso.parseOptions([shares[1][2]])
7541+            shares[0][2].remove()
7542+            stdout = StringIO()
7543+            sharefile = shares[1][2]
7544             storage_index = uri.from_string(self.uri).get_storage_index()
7545             self._corrupt_share_line = "  server %s, SI %s, shnum %d" % \
7546                                        (base32.b2a(shares[1][1]),
7547hunk ./src/allmydata/test/test_cli.py 2900
7548                                         base32.b2a(storage_index),
7549                                         shares[1][0])
7550-            debug.corrupt_share(cso)
7551+            debug.do_corrupt_share(stdout, sharefile)
7552         d.addCallback(_clobber_shares)
7553 
7554         d.addCallback(lambda ign: self.do_cli("check", "--verify", self.uri))
7555hunk ./src/allmydata/test/test_cli.py 3017
7556         def _clobber_shares(ignored):
7557             shares = self.find_uri_shares(self.uris[u"g\u00F6\u00F6d"])
7558             self.failUnlessReallyEqual(len(shares), 10)
7559-            os.unlink(shares[0][2])
7560+            shares[0][2].remove()
7561 
7562             shares = self.find_uri_shares(self.uris["mutable"])
7563hunk ./src/allmydata/test/test_cli.py 3020
7564-            cso = debug.CorruptShareOptions()
7565-            cso.stdout = StringIO()
7566-            cso.parseOptions([shares[1][2]])
7567+            stdout = StringIO()
7568+            sharefile = shares[1][2]
7569             storage_index = uri.from_string(self.uris["mutable"]).get_storage_index()
7570             self._corrupt_share_line = " corrupt: server %s, SI %s, shnum %d" % \
7571                                        (base32.b2a(shares[1][1]),
7572hunk ./src/allmydata/test/test_cli.py 3027
7573                                         base32.b2a(storage_index),
7574                                         shares[1][0])
7575-            debug.corrupt_share(cso)
7576+            debug.do_corrupt_share(stdout, sharefile)
7577         d.addCallback(_clobber_shares)
7578 
7579         # root
7580hunk ./src/allmydata/test/test_client.py 90
7581                            "enabled = true\n" + \
7582                            "reserved_space = 1000\n")
7583         c = client.Client(basedir)
7584-        self.failUnlessEqual(c.getServiceNamed("storage").reserved_space, 1000)
7585+        self.failUnlessEqual(c.getServiceNamed("storage").backend._reserved_space, 1000)
7586 
7587     def test_reserved_2(self):
7588         basedir = "client.Basic.test_reserved_2"
7589hunk ./src/allmydata/test/test_client.py 101
7590                            "enabled = true\n" + \
7591                            "reserved_space = 10K\n")
7592         c = client.Client(basedir)
7593-        self.failUnlessEqual(c.getServiceNamed("storage").reserved_space, 10*1000)
7594+        self.failUnlessEqual(c.getServiceNamed("storage").backend._reserved_space, 10*1000)
7595 
7596     def test_reserved_3(self):
7597         basedir = "client.Basic.test_reserved_3"
7598hunk ./src/allmydata/test/test_client.py 112
7599                            "enabled = true\n" + \
7600                            "reserved_space = 5mB\n")
7601         c = client.Client(basedir)
7602-        self.failUnlessEqual(c.getServiceNamed("storage").reserved_space,
7603+        self.failUnlessEqual(c.getServiceNamed("storage").backend._reserved_space,
7604                              5*1000*1000)
7605 
7606     def test_reserved_4(self):
7607hunk ./src/allmydata/test/test_client.py 124
7608                            "enabled = true\n" + \
7609                            "reserved_space = 78Gb\n")
7610         c = client.Client(basedir)
7611-        self.failUnlessEqual(c.getServiceNamed("storage").reserved_space,
7612+        self.failUnlessEqual(c.getServiceNamed("storage").backend._reserved_space,
7613                              78*1000*1000*1000)
7614 
7615     def test_reserved_bad(self):
7616hunk ./src/allmydata/test/test_client.py 136
7617                            "enabled = true\n" + \
7618                            "reserved_space = bogus\n")
7619         c = client.Client(basedir)
7620-        self.failUnlessEqual(c.getServiceNamed("storage").reserved_space, 0)
7621+        self.failUnlessEqual(c.getServiceNamed("storage").backend._reserved_space, 0)
7622 
7623     def _permute(self, sb, key):
7624         return [ s.get_serverid() for s in sb.get_servers_for_psi(key) ]
7625hunk ./src/allmydata/test/test_crawler.py 7
7626 from twisted.trial import unittest
7627 from twisted.application import service
7628 from twisted.internet import defer
7629+from twisted.python.filepath import FilePath
7630 from foolscap.api import eventually, fireEventually
7631 
7632 from allmydata.util import fileutil, hashutil, pollmixin
7633hunk ./src/allmydata/test/test_crawler.py 13
7634 from allmydata.storage.server import StorageServer, si_b2a
7635 from allmydata.storage.crawler import ShareCrawler, TimeSliceExceeded
7636+from allmydata.storage.backends.disk.disk_backend import DiskBackend
7637 
7638 from allmydata.test.test_storage import FakeCanary
7639 from allmydata.test.common_util import StallMixin
7640hunk ./src/allmydata/test/test_crawler.py 115
7641 
7642     def test_immediate(self):
7643         self.basedir = "crawler/Basic/immediate"
7644-        fileutil.make_dirs(self.basedir)
7645         serverid = "\x00" * 20
7646hunk ./src/allmydata/test/test_crawler.py 116
7647-        ss = StorageServer(self.basedir, serverid)
7648+        fp = FilePath(self.basedir)
7649+        backend = DiskBackend(fp)
7650+        ss = StorageServer(serverid, backend, fp)
7651         ss.setServiceParent(self.s)
7652 
7653         sis = [self.write(i, ss, serverid) for i in range(10)]
7654hunk ./src/allmydata/test/test_crawler.py 122
7655-        statefile = os.path.join(self.basedir, "statefile")
7656+        statefp = fp.child("statefile")
7657 
7658hunk ./src/allmydata/test/test_crawler.py 124
7659-        c = BucketEnumeratingCrawler(ss, statefile, allowed_cpu_percentage=.1)
7660+        c = BucketEnumeratingCrawler(backend, statefp, allowed_cpu_percentage=.1)
7661         c.load_state()
7662 
7663         c.start_current_prefix(time.time())
7664hunk ./src/allmydata/test/test_crawler.py 137
7665         self.failUnlessEqual(sorted(sis), sorted(c.all_buckets))
7666 
7667         # check that a new crawler picks up on the state file properly
7668-        c2 = BucketEnumeratingCrawler(ss, statefile)
7669+        c2 = BucketEnumeratingCrawler(backend, statefp)
7670         c2.load_state()
7671 
7672         c2.start_current_prefix(time.time())
7673hunk ./src/allmydata/test/test_crawler.py 145
7674 
7675     def test_service(self):
7676         self.basedir = "crawler/Basic/service"
7677-        fileutil.make_dirs(self.basedir)
7678         serverid = "\x00" * 20
7679hunk ./src/allmydata/test/test_crawler.py 146
7680-        ss = StorageServer(self.basedir, serverid)
7681+        fp = FilePath(self.basedir)
7682+        backend = DiskBackend(fp)
7683+        ss = StorageServer(serverid, backend, fp)
7684         ss.setServiceParent(self.s)
7685 
7686         sis = [self.write(i, ss, serverid) for i in range(10)]
7687hunk ./src/allmydata/test/test_crawler.py 153
7688 
7689-        statefile = os.path.join(self.basedir, "statefile")
7690-        c = BucketEnumeratingCrawler(ss, statefile)
7691+        statefp = fp.child("statefile")
7692+        c = BucketEnumeratingCrawler(backend, statefp)
7693         c.setServiceParent(self.s)
7694 
7695         # it should be legal to call get_state() and get_progress() right
7696hunk ./src/allmydata/test/test_crawler.py 174
7697 
7698     def test_paced(self):
7699         self.basedir = "crawler/Basic/paced"
7700-        fileutil.make_dirs(self.basedir)
7701         serverid = "\x00" * 20
7702hunk ./src/allmydata/test/test_crawler.py 175
7703-        ss = StorageServer(self.basedir, serverid)
7704+        fp = FilePath(self.basedir)
7705+        backend = DiskBackend(fp)
7706+        ss = StorageServer(serverid, backend, fp)
7707         ss.setServiceParent(self.s)
7708 
7709         # put four buckets in each prefixdir
7710hunk ./src/allmydata/test/test_crawler.py 186
7711             for tail in range(4):
7712                 sis.append(self.write(i, ss, serverid, tail))
7713 
7714-        statefile = os.path.join(self.basedir, "statefile")
7715+        statefp = fp.child("statefile")
7716 
7717hunk ./src/allmydata/test/test_crawler.py 188
7718-        c = PacedCrawler(ss, statefile)
7719+        c = PacedCrawler(backend, statefp)
7720         c.load_state()
7721         try:
7722             c.start_current_prefix(time.time())
7723hunk ./src/allmydata/test/test_crawler.py 213
7724         del c
7725 
7726         # start a new crawler, it should start from the beginning
7727-        c = PacedCrawler(ss, statefile)
7728+        c = PacedCrawler(backend, statefp)
7729         c.load_state()
7730         try:
7731             c.start_current_prefix(time.time())
7732hunk ./src/allmydata/test/test_crawler.py 226
7733         c.cpu_slice = PacedCrawler.cpu_slice
7734 
7735         # a third crawler should pick up from where it left off
7736-        c2 = PacedCrawler(ss, statefile)
7737+        c2 = PacedCrawler(backend, statefp)
7738         c2.all_buckets = c.all_buckets[:]
7739         c2.load_state()
7740         c2.countdown = -1
7741hunk ./src/allmydata/test/test_crawler.py 237
7742 
7743         # now stop it at the end of a bucket (countdown=4), to exercise a
7744         # different place that checks the time
7745-        c = PacedCrawler(ss, statefile)
7746+        c = PacedCrawler(backend, statefp)
7747         c.load_state()
7748         c.countdown = 4
7749         try:
7750hunk ./src/allmydata/test/test_crawler.py 256
7751 
7752         # stop it again at the end of the bucket, check that a new checker
7753         # picks up correctly
7754-        c = PacedCrawler(ss, statefile)
7755+        c = PacedCrawler(backend, statefp)
7756         c.load_state()
7757         c.countdown = 4
7758         try:
7759hunk ./src/allmydata/test/test_crawler.py 266
7760         # that should stop at the end of one of the buckets.
7761         c.save_state()
7762 
7763-        c2 = PacedCrawler(ss, statefile)
7764+        c2 = PacedCrawler(backend, statefp)
7765         c2.all_buckets = c.all_buckets[:]
7766         c2.load_state()
7767         c2.countdown = -1
7768hunk ./src/allmydata/test/test_crawler.py 277
7769 
7770     def test_paced_service(self):
7771         self.basedir = "crawler/Basic/paced_service"
7772-        fileutil.make_dirs(self.basedir)
7773         serverid = "\x00" * 20
7774hunk ./src/allmydata/test/test_crawler.py 278
7775-        ss = StorageServer(self.basedir, serverid)
7776+        fp = FilePath(self.basedir)
7777+        backend = DiskBackend(fp)
7778+        ss = StorageServer(serverid, backend, fp)
7779         ss.setServiceParent(self.s)
7780 
7781         sis = [self.write(i, ss, serverid) for i in range(10)]
7782hunk ./src/allmydata/test/test_crawler.py 285
7783 
7784-        statefile = os.path.join(self.basedir, "statefile")
7785-        c = PacedCrawler(ss, statefile)
7786+        statefp = fp.child("statefile")
7787+        c = PacedCrawler(backend, statefp)
7788 
7789         did_check_progress = [False]
7790         def check_progress():
7791hunk ./src/allmydata/test/test_crawler.py 345
7792         # and read the stdout when it runs.
7793 
7794         self.basedir = "crawler/Basic/cpu_usage"
7795-        fileutil.make_dirs(self.basedir)
7796         serverid = "\x00" * 20
7797hunk ./src/allmydata/test/test_crawler.py 346
7798-        ss = StorageServer(self.basedir, serverid)
7799+        fp = FilePath(self.basedir)
7800+        backend = DiskBackend(fp)
7801+        ss = StorageServer(serverid, backend, fp)
7802         ss.setServiceParent(self.s)
7803 
7804         for i in range(10):
7805hunk ./src/allmydata/test/test_crawler.py 354
7806             self.write(i, ss, serverid)
7807 
7808-        statefile = os.path.join(self.basedir, "statefile")
7809-        c = ConsumingCrawler(ss, statefile)
7810+        statefp = fp.child("statefile")
7811+        c = ConsumingCrawler(backend, statefp)
7812         c.setServiceParent(self.s)
7813 
7814         # this will run as fast as it can, consuming about 50ms per call to
7815hunk ./src/allmydata/test/test_crawler.py 391
7816 
7817     def test_empty_subclass(self):
7818         self.basedir = "crawler/Basic/empty_subclass"
7819-        fileutil.make_dirs(self.basedir)
7820         serverid = "\x00" * 20
7821hunk ./src/allmydata/test/test_crawler.py 392
7822-        ss = StorageServer(self.basedir, serverid)
7823+        fp = FilePath(self.basedir)
7824+        backend = DiskBackend(fp)
7825+        ss = StorageServer(serverid, backend, fp)
7826         ss.setServiceParent(self.s)
7827 
7828         for i in range(10):
7829hunk ./src/allmydata/test/test_crawler.py 400
7830             self.write(i, ss, serverid)
7831 
7832-        statefile = os.path.join(self.basedir, "statefile")
7833-        c = ShareCrawler(ss, statefile)
7834+        statefp = fp.child("statefile")
7835+        c = ShareCrawler(backend, statefp)
7836         c.slow_start = 0
7837         c.setServiceParent(self.s)
7838 
7839hunk ./src/allmydata/test/test_crawler.py 417
7840         d.addCallback(_done)
7841         return d
7842 
7843-
7844     def test_oneshot(self):
7845         self.basedir = "crawler/Basic/oneshot"
7846hunk ./src/allmydata/test/test_crawler.py 419
7847-        fileutil.make_dirs(self.basedir)
7848         serverid = "\x00" * 20
7849hunk ./src/allmydata/test/test_crawler.py 420
7850-        ss = StorageServer(self.basedir, serverid)
7851+        fp = FilePath(self.basedir)
7852+        backend = DiskBackend(fp)
7853+        ss = StorageServer(serverid, backend, fp)
7854         ss.setServiceParent(self.s)
7855 
7856         for i in range(30):
7857hunk ./src/allmydata/test/test_crawler.py 428
7858             self.write(i, ss, serverid)
7859 
7860-        statefile = os.path.join(self.basedir, "statefile")
7861-        c = OneShotCrawler(ss, statefile)
7862+        statefp = fp.child("statefile")
7863+        c = OneShotCrawler(backend, statefp)
7864         c.setServiceParent(self.s)
7865 
7866         d = c.finished_d
7867hunk ./src/allmydata/test/test_crawler.py 447
7868             self.failUnlessEqual(s["current-cycle"], None)
7869         d.addCallback(_check)
7870         return d
7871-
7872hunk ./src/allmydata/test/test_deepcheck.py 23
7873      ShouldFailMixin
7874 from allmydata.test.common_util import StallMixin
7875 from allmydata.test.no_network import GridTestMixin
7876+from allmydata.scripts import debug
7877+
7878 
7879 timeout = 2400 # One of these took 1046.091s on Zandr's ARM box.
7880 
7881hunk ./src/allmydata/test/test_deepcheck.py 905
7882         d.addErrback(self.explain_error)
7883         return d
7884 
7885-
7886-
7887     def set_up_damaged_tree(self):
7888         # 6.4s
7889 
7890hunk ./src/allmydata/test/test_deepcheck.py 989
7891 
7892         return d
7893 
7894-    def _run_cli(self, argv):
7895-        stdout, stderr = StringIO(), StringIO()
7896-        # this can only do synchronous operations
7897-        assert argv[0] == "debug"
7898-        runner.runner(argv, run_by_human=False, stdout=stdout, stderr=stderr)
7899-        return stdout.getvalue()
7900-
7901     def _delete_some_shares(self, node):
7902         self.delete_shares_numbered(node.get_uri(), [0,1])
7903 
7904hunk ./src/allmydata/test/test_deepcheck.py 995
7905     def _corrupt_some_shares(self, node):
7906         for (shnum, serverid, sharefile) in self.find_uri_shares(node.get_uri()):
7907             if shnum in (0,1):
7908-                self._run_cli(["debug", "corrupt-share", sharefile])
7909+                debug.do_corrupt_share(StringIO(), sharefile)
7910 
7911     def _delete_most_shares(self, node):
7912         self.delete_shares_numbered(node.get_uri(), range(1,10))
7913hunk ./src/allmydata/test/test_deepcheck.py 1000
7914 
7915-
7916     def check_is_healthy(self, cr, where):
7917         try:
7918             self.failUnless(ICheckResults.providedBy(cr), (cr, type(cr), where))
7919hunk ./src/allmydata/test/test_download.py 134
7920             for shnum in shares_for_server:
7921                 share_dir = self.get_server(i).backend.get_shareset(si)._sharehomedir
7922                 fileutil.fp_make_dirs(share_dir)
7923-                share_dir.child(str(shnum)).setContent(shares[shnum])
7924+                share_dir.child(str(shnum)).setContent(shares_for_server[shnum])
7925 
7926     def load_shares(self, ignored=None):
7927         # this uses the data generated by create_shares() to populate the
7928hunk ./src/allmydata/test/test_hung_server.py 32
7929 
7930     def _break(self, servers):
7931         for ss in servers:
7932-            self.g.break_server(ss.get_serverid())
7933+            self.g.break_server(ss.original.get_serverid())
7934 
7935     def _hang(self, servers, **kwargs):
7936         for ss in servers:
7937hunk ./src/allmydata/test/test_hung_server.py 67
7938         serverids = [ss.original.get_serverid() for ss in from_servers]
7939         for (i_shnum, i_serverid, i_sharefp) in self.shares:
7940             if i_serverid in serverids:
7941-                self.copy_share((i_shnum, i_serverid, i_sharefp), self.uri, to_server)
7942+                self.copy_share((i_shnum, i_serverid, i_sharefp), self.uri, to_server.original)
7943 
7944         self.shares = self.find_uri_shares(self.uri)
7945 
7946hunk ./src/allmydata/test/test_mutable.py 3669
7947         # Now execute each assignment by writing the storage.
7948         for (share, servernum) in assignments:
7949             sharedata = base64.b64decode(self.sdmf_old_shares[share])
7950-            storage_dir = self.get_server(servernum).backend.get_shareset(si).sharehomedir
7951+            storage_dir = self.get_server(servernum).backend.get_shareset(si)._sharehomedir
7952             fileutil.fp_make_dirs(storage_dir)
7953             storage_dir.child("%d" % share).setContent(sharedata)
7954         # ...and verify that the shares are there.
7955hunk ./src/allmydata/test/test_no_network.py 10
7956 from allmydata.immutable.upload import Data
7957 from allmydata.util.consumer import download_to_data
7958 
7959+
7960 class Harness(unittest.TestCase):
7961     def setUp(self):
7962         self.s = service.MultiService()
7963hunk ./src/allmydata/test/test_storage.py 1
7964-import time, os.path, platform, stat, re, simplejson, struct, shutil
7965+import time, os.path, platform, stat, re, simplejson, struct, shutil, itertools
7966 
7967 import mock
7968 
7969hunk ./src/allmydata/test/test_storage.py 6
7970 from twisted.trial import unittest
7971-
7972 from twisted.internet import defer
7973 from twisted.application import service
7974hunk ./src/allmydata/test/test_storage.py 8
7975+from twisted.python.filepath import FilePath
7976 from foolscap.api import fireEventually
7977hunk ./src/allmydata/test/test_storage.py 10
7978-import itertools
7979+
7980 from allmydata import interfaces
7981 from allmydata.util import fileutil, hashutil, base32, pollmixin, time_format
7982 from allmydata.storage.server import StorageServer
7983hunk ./src/allmydata/test/test_storage.py 14
7984+from allmydata.storage.backends.disk.disk_backend import DiskBackend
7985 from allmydata.storage.backends.disk.mutable import MutableDiskShare
7986 from allmydata.storage.bucket import BucketWriter, BucketReader
7987 from allmydata.storage.common import DataTooLargeError, \
7988hunk ./src/allmydata/test/test_storage.py 310
7989         return self.sparent.stopService()
7990 
7991     def workdir(self, name):
7992-        basedir = os.path.join("storage", "Server", name)
7993-        return basedir
7994+        return FilePath("storage").child("Server").child(name)
7995 
7996     def create(self, name, reserved_space=0, klass=StorageServer):
7997         workdir = self.workdir(name)
7998hunk ./src/allmydata/test/test_storage.py 314
7999-        ss = klass(workdir, "\x00" * 20, reserved_space=reserved_space,
8000+        backend = DiskBackend(workdir, readonly=False, reserved_space=reserved_space)
8001+        ss = klass("\x00" * 20, backend, workdir,
8002                    stats_provider=FakeStatsProvider())
8003         ss.setServiceParent(self.sparent)
8004         return ss
8005hunk ./src/allmydata/test/test_storage.py 1386
8006 
8007     def tearDown(self):
8008         self.sparent.stopService()
8009-        shutil.rmtree(self.workdir("MDMFProxies storage test server"))
8010+        fileutil.fp_remove(self.workdir("MDMFProxies storage test server"))
8011 
8012 
8013     def write_enabler(self, we_tag):
8014hunk ./src/allmydata/test/test_storage.py 2781
8015         return self.sparent.stopService()
8016 
8017     def workdir(self, name):
8018-        basedir = os.path.join("storage", "Server", name)
8019-        return basedir
8020+        return FilePath("storage").child("Server").child(name)
8021 
8022     def create(self, name):
8023         workdir = self.workdir(name)
8024hunk ./src/allmydata/test/test_storage.py 2785
8025-        ss = StorageServer(workdir, "\x00" * 20)
8026+        backend = DiskBackend(workdir)
8027+        ss = StorageServer("\x00" * 20, backend, workdir)
8028         ss.setServiceParent(self.sparent)
8029         return ss
8030 
8031hunk ./src/allmydata/test/test_storage.py 4061
8032         }
8033 
8034         basedir = "storage/WebStatus/status_right_disk_stats"
8035-        fileutil.make_dirs(basedir)
8036-        ss = StorageServer(basedir, "\x00" * 20, reserved_space=reserved_space)
8037-        expecteddir = ss.sharedir
8038+        fp = FilePath(basedir)
8039+        backend = DiskBackend(fp, readonly=False, reserved_space=reserved_space)
8040+        ss = StorageServer("\x00" * 20, backend, fp)
8041+        expecteddir = backend._sharedir
8042         ss.setServiceParent(self.s)
8043         w = StorageStatus(ss)
8044         html = w.renderSynchronously()
8045hunk ./src/allmydata/test/test_storage.py 4084
8046 
8047     def test_readonly(self):
8048         basedir = "storage/WebStatus/readonly"
8049-        fileutil.make_dirs(basedir)
8050-        ss = StorageServer(basedir, "\x00" * 20, readonly_storage=True)
8051+        fp = FilePath(basedir)
8052+        backend = DiskBackend(fp, readonly=True)
8053+        ss = StorageServer("\x00" * 20, backend, fp)
8054         ss.setServiceParent(self.s)
8055         w = StorageStatus(ss)
8056         html = w.renderSynchronously()
8057hunk ./src/allmydata/test/test_storage.py 4096
8058 
8059     def test_reserved(self):
8060         basedir = "storage/WebStatus/reserved"
8061-        fileutil.make_dirs(basedir)
8062-        ss = StorageServer(basedir, "\x00" * 20, reserved_space=10e6)
8063-        ss.setServiceParent(self.s)
8064-        w = StorageStatus(ss)
8065-        html = w.renderSynchronously()
8066-        self.failUnlessIn("<h1>Storage Server Status</h1>", html)
8067-        s = remove_tags(html)
8068-        self.failUnlessIn("Reserved space: - 10.00 MB (10000000)", s)
8069-
8070-    def test_huge_reserved(self):
8071-        basedir = "storage/WebStatus/reserved"
8072-        fileutil.make_dirs(basedir)
8073-        ss = StorageServer(basedir, "\x00" * 20, reserved_space=10e6)
8074+        fp = FilePath(basedir)
8075+        backend = DiskBackend(fp, readonly=False, reserved_space=10e6)
8076+        ss = StorageServer("\x00" * 20, backend, fp)
8077         ss.setServiceParent(self.s)
8078         w = StorageStatus(ss)
8079         html = w.renderSynchronously()
8080hunk ./src/allmydata/test/test_upload.py 3
8081 # -*- coding: utf-8 -*-
8082 
8083-import os, shutil
8084+import os
8085 from cStringIO import StringIO
8086 from twisted.trial import unittest
8087 from twisted.python.failure import Failure
8088hunk ./src/allmydata/test/test_upload.py 14
8089 from allmydata import uri, monitor, client
8090 from allmydata.immutable import upload, encode
8091 from allmydata.interfaces import FileTooLargeError, UploadUnhappinessError
8092-from allmydata.util import log
8093+from allmydata.util import log, fileutil
8094 from allmydata.util.assertutil import precondition
8095 from allmydata.util.deferredutil import DeferredListShouldSucceed
8096 from allmydata.test.no_network import GridTestMixin
8097hunk ./src/allmydata/test/test_upload.py 972
8098                                         readonly=True))
8099         # Remove the first share from server 0.
8100         def _remove_share_0_from_server_0():
8101-            share_location = self.shares[0][2]
8102-            os.remove(share_location)
8103+            self.shares[0][2].remove()
8104         d.addCallback(lambda ign:
8105             _remove_share_0_from_server_0())
8106         # Set happy = 4 in the client.
8107hunk ./src/allmydata/test/test_upload.py 1847
8108             self._copy_share_to_server(3, 1)
8109             storedir = self.get_serverdir(0)
8110             # remove the storedir, wiping out any existing shares
8111-            shutil.rmtree(storedir)
8112+            fileutil.fp_remove(storedir)
8113             # create an empty storedir to replace the one we just removed
8114hunk ./src/allmydata/test/test_upload.py 1849
8115-            os.mkdir(storedir)
8116+            storedir.mkdir()
8117             client = self.g.clients[0]
8118             client.DEFAULT_ENCODING_PARAMETERS['happy'] = 4
8119             return client
8120hunk ./src/allmydata/test/test_upload.py 1888
8121             self._copy_share_to_server(3, 1)
8122             storedir = self.get_serverdir(0)
8123             # remove the storedir, wiping out any existing shares
8124-            shutil.rmtree(storedir)
8125+            fileutil.fp_remove(storedir)
8126             # create an empty storedir to replace the one we just removed
8127hunk ./src/allmydata/test/test_upload.py 1890
8128-            os.mkdir(storedir)
8129+            storedir.mkdir()
8130             client = self.g.clients[0]
8131             client.DEFAULT_ENCODING_PARAMETERS['happy'] = 4
8132             return client
8133hunk ./src/allmydata/test/test_web.py 4870
8134         d.addErrback(self.explain_web_error)
8135         return d
8136 
8137-    def _assert_leasecount(self, ignored, which, expected):
8138+    def _assert_leasecount(self, which, expected):
8139         lease_counts = self.count_leases(self.uris[which])
8140         for (fn, num_leases) in lease_counts:
8141             if num_leases != expected:
8142hunk ./src/allmydata/test/test_web.py 4903
8143                 self.fileurls[which] = "uri/" + urllib.quote(self.uris[which])
8144         d.addCallback(_compute_fileurls)
8145 
8146-        d.addCallback(self._assert_leasecount, "one", 1)
8147-        d.addCallback(self._assert_leasecount, "two", 1)
8148-        d.addCallback(self._assert_leasecount, "mutable", 1)
8149+        d.addCallback(lambda ign: self._assert_leasecount("one", 1))
8150+        d.addCallback(lambda ign: self._assert_leasecount("two", 1))
8151+        d.addCallback(lambda ign: self._assert_leasecount("mutable", 1))
8152 
8153         d.addCallback(self.CHECK, "one", "t=check") # no add-lease
8154         def _got_html_good(res):
8155hunk ./src/allmydata/test/test_web.py 4913
8156             self.failIf("Not Healthy" in res, res)
8157         d.addCallback(_got_html_good)
8158 
8159-        d.addCallback(self._assert_leasecount, "one", 1)
8160-        d.addCallback(self._assert_leasecount, "two", 1)
8161-        d.addCallback(self._assert_leasecount, "mutable", 1)
8162+        d.addCallback(lambda ign: self._assert_leasecount("one", 1))
8163+        d.addCallback(lambda ign: self._assert_leasecount("two", 1))
8164+        d.addCallback(lambda ign: self._assert_leasecount("mutable", 1))
8165 
8166         # this CHECK uses the original client, which uses the same
8167         # lease-secrets, so it will just renew the original lease
8168hunk ./src/allmydata/test/test_web.py 4922
8169         d.addCallback(self.CHECK, "one", "t=check&add-lease=true")
8170         d.addCallback(_got_html_good)
8171 
8172-        d.addCallback(self._assert_leasecount, "one", 1)
8173-        d.addCallback(self._assert_leasecount, "two", 1)
8174-        d.addCallback(self._assert_leasecount, "mutable", 1)
8175+        d.addCallback(lambda ign: self._assert_leasecount("one", 1))
8176+        d.addCallback(lambda ign: self._assert_leasecount("two", 1))
8177+        d.addCallback(lambda ign: self._assert_leasecount("mutable", 1))
8178 
8179         # this CHECK uses an alternate client, which adds a second lease
8180         d.addCallback(self.CHECK, "one", "t=check&add-lease=true", clientnum=1)
8181hunk ./src/allmydata/test/test_web.py 4930
8182         d.addCallback(_got_html_good)
8183 
8184-        d.addCallback(self._assert_leasecount, "one", 2)
8185-        d.addCallback(self._assert_leasecount, "two", 1)
8186-        d.addCallback(self._assert_leasecount, "mutable", 1)
8187+        d.addCallback(lambda ign: self._assert_leasecount("one", 2))
8188+        d.addCallback(lambda ign: self._assert_leasecount("two", 1))
8189+        d.addCallback(lambda ign: self._assert_leasecount("mutable", 1))
8190 
8191         d.addCallback(self.CHECK, "mutable", "t=check&add-lease=true")
8192         d.addCallback(_got_html_good)
8193hunk ./src/allmydata/test/test_web.py 4937
8194 
8195-        d.addCallback(self._assert_leasecount, "one", 2)
8196-        d.addCallback(self._assert_leasecount, "two", 1)
8197-        d.addCallback(self._assert_leasecount, "mutable", 1)
8198+        d.addCallback(lambda ign: self._assert_leasecount("one", 2))
8199+        d.addCallback(lambda ign: self._assert_leasecount("two", 1))
8200+        d.addCallback(lambda ign: self._assert_leasecount("mutable", 1))
8201 
8202         d.addCallback(self.CHECK, "mutable", "t=check&add-lease=true",
8203                       clientnum=1)
8204hunk ./src/allmydata/test/test_web.py 4945
8205         d.addCallback(_got_html_good)
8206 
8207-        d.addCallback(self._assert_leasecount, "one", 2)
8208-        d.addCallback(self._assert_leasecount, "two", 1)
8209-        d.addCallback(self._assert_leasecount, "mutable", 2)
8210+        d.addCallback(lambda ign: self._assert_leasecount("one", 2))
8211+        d.addCallback(lambda ign: self._assert_leasecount("two", 1))
8212+        d.addCallback(lambda ign: self._assert_leasecount("mutable", 2))
8213 
8214         d.addErrback(self.explain_web_error)
8215         return d
8216hunk ./src/allmydata/test/test_web.py 4989
8217             self.failUnlessReallyEqual(len(units), 4+1)
8218         d.addCallback(_done)
8219 
8220-        d.addCallback(self._assert_leasecount, "root", 1)
8221-        d.addCallback(self._assert_leasecount, "one", 1)
8222-        d.addCallback(self._assert_leasecount, "mutable", 1)
8223+        d.addCallback(lambda ign: self._assert_leasecount("root", 1))
8224+        d.addCallback(lambda ign: self._assert_leasecount("one", 1))
8225+        d.addCallback(lambda ign: self._assert_leasecount("mutable", 1))
8226 
8227         d.addCallback(self.CHECK, "root", "t=stream-deep-check&add-lease=true")
8228         d.addCallback(_done)
8229hunk ./src/allmydata/test/test_web.py 4996
8230 
8231-        d.addCallback(self._assert_leasecount, "root", 1)
8232-        d.addCallback(self._assert_leasecount, "one", 1)
8233-        d.addCallback(self._assert_leasecount, "mutable", 1)
8234+        d.addCallback(lambda ign: self._assert_leasecount("root", 1))
8235+        d.addCallback(lambda ign: self._assert_leasecount("one", 1))
8236+        d.addCallback(lambda ign: self._assert_leasecount("mutable", 1))
8237 
8238         d.addCallback(self.CHECK, "root", "t=stream-deep-check&add-lease=true",
8239                       clientnum=1)
8240hunk ./src/allmydata/test/test_web.py 5004
8241         d.addCallback(_done)
8242 
8243-        d.addCallback(self._assert_leasecount, "root", 2)
8244-        d.addCallback(self._assert_leasecount, "one", 2)
8245-        d.addCallback(self._assert_leasecount, "mutable", 2)
8246+        d.addCallback(lambda ign: self._assert_leasecount("root", 2))
8247+        d.addCallback(lambda ign: self._assert_leasecount("one", 2))
8248+        d.addCallback(lambda ign: self._assert_leasecount("mutable", 2))
8249 
8250         d.addErrback(self.explain_web_error)
8251         return d
8252}
8253[Fix more shallow bugs, mainly FilePathification. Also, remove the max_space_per_bucket parameter from BucketWriter since it can be obtained from the _max_size attribute of the share (via a new get_allocated_size() accessor). refs #999
8254david-sarah@jacaranda.org**20110921221421
8255 Ignore-this: 600e3ccef8533aa43442fa576c7d88cf
8256] {
8257hunk ./src/allmydata/scripts/debug.py 642
8258     /home/warner/testnet/node-2/storage/shares/44k/44kai1tui348689nrw8fjegc8c/2
8259     """
8260     from allmydata.storage.server import si_a2b
8261-    from allmydata.storage.backends.disk_backend import si_si2dir
8262+    from allmydata.storage.backends.disk.disk_backend import si_si2dir
8263     from allmydata.util.encodingutil import quote_filepath
8264 
8265     out = options.stdout
8266hunk ./src/allmydata/scripts/debug.py 648
8267     si = si_a2b(options.si_s)
8268     for nodedir in options.nodedirs:
8269-        sharedir = si_si2dir(nodedir.child("storage").child("shares"), si)
8270+        sharedir = si_si2dir(FilePath(nodedir).child("storage").child("shares"), si)
8271         if sharedir.exists():
8272             for sharefp in sharedir.children():
8273                 print >>out, quote_filepath(sharefp, quotemarks=False)
8274hunk ./src/allmydata/storage/backends/disk/disk_backend.py 189
8275         incominghome = self._incominghomedir.child(str(shnum))
8276         immsh = ImmutableDiskShare(self.get_storage_index(), shnum, incominghome, finalhome,
8277                                    max_size=max_space_per_bucket)
8278-        bw = BucketWriter(storageserver, immsh, max_space_per_bucket, lease_info, canary)
8279+        bw = BucketWriter(storageserver, immsh, lease_info, canary)
8280         if self._discard_storage:
8281             bw.throw_out_all_data = True
8282         return bw
8283hunk ./src/allmydata/storage/backends/disk/immutable.py 147
8284     def unlink(self):
8285         self._home.remove()
8286 
8287+    def get_allocated_size(self):
8288+        return self._max_size
8289+
8290     def get_size(self):
8291         return self._home.getsize()
8292 
8293hunk ./src/allmydata/storage/bucket.py 15
8294 class BucketWriter(Referenceable):
8295     implements(RIBucketWriter)
8296 
8297-    def __init__(self, ss, immutableshare, max_size, lease_info, canary):
8298+    def __init__(self, ss, immutableshare, lease_info, canary):
8299         self.ss = ss
8300hunk ./src/allmydata/storage/bucket.py 17
8301-        self._max_size = max_size # don't allow the client to write more than this
8302         self._canary = canary
8303         self._disconnect_marker = canary.notifyOnDisconnect(self._disconnected)
8304         self.closed = False
8305hunk ./src/allmydata/storage/bucket.py 27
8306         self._share.add_lease(lease_info)
8307 
8308     def allocated_size(self):
8309-        return self._max_size
8310+        return self._share.get_allocated_size()
8311 
8312     def remote_write(self, offset, data):
8313         start = time.time()
8314hunk ./src/allmydata/storage/crawler.py 480
8315             self.state["bucket-counts"][cycle] = {}
8316         self.state["bucket-counts"][cycle][prefix] = len(sharesets)
8317         if prefix in self.prefixes[:self.num_sample_prefixes]:
8318-            self.state["storage-index-samples"][prefix] = (cycle, sharesets)
8319+            si_strings = [shareset.get_storage_index_string() for shareset in sharesets]
8320+            self.state["storage-index-samples"][prefix] = (cycle, si_strings)
8321 
8322     def finished_cycle(self, cycle):
8323         last_counts = self.state["bucket-counts"].get(cycle, [])
8324hunk ./src/allmydata/storage/ex