Ticket #999: pluggable-backends-davidsarah-v12.darcs.patch

File pluggable-backends-davidsarah-v12.darcs.patch, 516.6 KB (added by davidsarah, at 2011-09-23T20:59:31Z)

Updates to null and S3 backends.

Line 
127 patches for repository http://tahoe-lafs.org/source/tahoe/trunk:
2
3Thu Aug 25 01:32:17 BST 2011  david-sarah@jacaranda.org
4  * interfaces.py: 'which -> that' grammar cleanup.
5
6Tue Sep 20 00:29:26 BST 2011  david-sarah@jacaranda.org
7  * Pluggable backends -- new and moved files, changes to moved files. refs #999
8
9Tue Sep 20 00:32:56 BST 2011  david-sarah@jacaranda.org
10  * Pluggable backends -- all other changes. refs #999
11
12Tue Sep 20 04:38:03 BST 2011  david-sarah@jacaranda.org
13  * Work-in-progress, includes fix to bug involving BucketWriter. refs #999
14
15Tue Sep 20 18:17:37 BST 2011  david-sarah@jacaranda.org
16  * docs/backends: document the configuration options for the pluggable backends scheme. refs #999
17
18Wed Sep 21 04:12:07 BST 2011  david-sarah@jacaranda.org
19  * Fix some incorrect attribute accesses. refs #999
20
21Wed Sep 21 04:16:25 BST 2011  david-sarah@jacaranda.org
22  * docs/backends/S3.rst: remove Issues section. refs #999
23
24Wed Sep 21 04:17:05 BST 2011  david-sarah@jacaranda.org
25  * docs/backends/S3.rst, disk.rst: describe type of space settings as 'quantity of space', not 'str'. refs #999
26
27Wed Sep 21 19:46:49 BST 2011  david-sarah@jacaranda.org
28  * More fixes to tests needed for pluggable backends. refs #999
29
30Wed Sep 21 23:14:21 BST 2011  david-sarah@jacaranda.org
31  * Fix more shallow bugs, mainly FilePathification. Also, remove the max_space_per_bucket parameter from BucketWriter since it can be obtained from the _max_size attribute of the share (via a new get_allocated_size() accessor). refs #999
32
33Wed Sep 21 23:20:38 BST 2011  david-sarah@jacaranda.org
34  * uri.py: resolve a conflict between trunk and the pluggable-backends patches. refs #999
35
36Thu Sep 22 05:54:51 BST 2011  david-sarah@jacaranda.org
37  * Fix some more test failures. refs #999
38
39Thu Sep 22 19:30:08 BST 2011  david-sarah@jacaranda.org
40  * Fix most of the crawler tests. refs #999
41
42Thu Sep 22 19:33:23 BST 2011  david-sarah@jacaranda.org
43  * Reinstate the cancel_lease methods of ImmutableDiskShare and MutableDiskShare, since they are needed for lease expiry. refs #999
44
45Fri Sep 23 02:20:44 BST 2011  david-sarah@jacaranda.org
46  * Blank line cleanups.
47
48Fri Sep 23 05:08:25 BST 2011  david-sarah@jacaranda.org
49  * mutable/publish.py: elements should not be removed from a dictionary while it is being iterated over. refs #393
50
51Fri Sep 23 05:10:03 BST 2011  david-sarah@jacaranda.org
52  * A few comment cleanups. refs #999
53
54Fri Sep 23 05:11:15 BST 2011  david-sarah@jacaranda.org
55  * Move advise_corrupt_share to allmydata/storage/backends/base.py, since it will be common to the disk and S3 backends. refs #999
56
57Fri Sep 23 05:13:14 BST 2011  david-sarah@jacaranda.org
58  * Add incomplete S3 backend. refs #999
59
60Fri Sep 23 21:37:23 BST 2011  david-sarah@jacaranda.org
61  * interfaces.py: add fill_in_space_stats method to IStorageBackend. refs #999
62
63Fri Sep 23 21:44:25 BST 2011  david-sarah@jacaranda.org
64  * Remove redundant si_s argument from check_write_enabler. refs #999
65
66Fri Sep 23 21:46:11 BST 2011  david-sarah@jacaranda.org
67  * Implement readv for immutable shares. refs #999
68
69Fri Sep 23 21:49:14 BST 2011  david-sarah@jacaranda.org
70  * The cancel secret needs to be unique, even if it isn't explicitly provided. refs #999
71
72Fri Sep 23 21:49:45 BST 2011  david-sarah@jacaranda.org
73  * Make EmptyShare.check_testv a simple function. refs #999
74
75Fri Sep 23 21:52:19 BST 2011  david-sarah@jacaranda.org
76  * Update the null backend to take into account interface changes. Also, it now records which shares are present, but not their contents. refs #999
77
78Fri Sep 23 21:53:45 BST 2011  david-sarah@jacaranda.org
79  * Update the S3 backend. refs #999
80
81Fri Sep 23 21:55:10 BST 2011  david-sarah@jacaranda.org
82  * Minor cleanup to disk backend. refs #999
83
84New patches:
85
86[interfaces.py: 'which -> that' grammar cleanup.
87david-sarah@jacaranda.org**20110825003217
88 Ignore-this: a3e15f3676de1b346ad78aabdfb8cac6
89] {
90hunk ./src/allmydata/interfaces.py 38
91     the StubClient. This object doesn't actually offer any services, but the
92     announcement helps the Introducer keep track of which clients are
93     subscribed (so the grid admin can keep track of things like the size of
94-    the grid and the client versions in use. This is the (empty)
95+    the grid and the client versions in use). This is the (empty)
96     RemoteInterface for the StubClient."""
97 
98 class RIBucketWriter(RemoteInterface):
99hunk ./src/allmydata/interfaces.py 276
100         (binary) storage index string, and 'shnum' is the integer share
101         number. 'reason' is a human-readable explanation of the problem,
102         probably including some expected hash values and the computed ones
103-        which did not match. Corruption advisories for mutable shares should
104+        that did not match. Corruption advisories for mutable shares should
105         include a hash of the public key (the same value that appears in the
106         mutable-file verify-cap), since the current share format does not
107         store that on disk.
108hunk ./src/allmydata/interfaces.py 413
109           remote_host: the IAddress, if connected, otherwise None
110 
111         This method is intended for monitoring interfaces, such as a web page
112-        which describes connecting and connected peers.
113+        that describes connecting and connected peers.
114         """
115 
116     def get_all_peerids():
117hunk ./src/allmydata/interfaces.py 515
118 
119     # TODO: rename to get_read_cap()
120     def get_readonly():
121-        """Return another IURI instance, which represents a read-only form of
122+        """Return another IURI instance that represents a read-only form of
123         this one. If is_readonly() is True, this returns self."""
124 
125     def get_verify_cap():
126hunk ./src/allmydata/interfaces.py 542
127         passing into init_from_string."""
128 
129 class IDirnodeURI(Interface):
130-    """I am a URI which represents a dirnode."""
131+    """I am a URI that represents a dirnode."""
132 
133 class IFileURI(Interface):
134hunk ./src/allmydata/interfaces.py 545
135-    """I am a URI which represents a filenode."""
136+    """I am a URI that represents a filenode."""
137     def get_size():
138         """Return the length (in bytes) of the file that I represent."""
139 
140hunk ./src/allmydata/interfaces.py 553
141     pass
142 
143 class IMutableFileURI(Interface):
144-    """I am a URI which represents a mutable filenode."""
145+    """I am a URI that represents a mutable filenode."""
146     def get_extension_params():
147         """Return the extension parameters in the URI"""
148 
149hunk ./src/allmydata/interfaces.py 856
150         """
151 
152 class IFileNode(IFilesystemNode):
153-    """I am a node which represents a file: a sequence of bytes. I am not a
154+    """I am a node that represents a file: a sequence of bytes. I am not a
155     container, like IDirectoryNode."""
156     def get_best_readable_version():
157         """Return a Deferred that fires with an IReadable for the 'best'
158hunk ./src/allmydata/interfaces.py 905
159     multiple versions of a file present in the grid, some of which might be
160     unrecoverable (i.e. have fewer than 'k' shares). These versions are
161     loosely ordered: each has a sequence number and a hash, and any version
162-    with seqnum=N was uploaded by a node which has seen at least one version
163+    with seqnum=N was uploaded by a node that has seen at least one version
164     with seqnum=N-1.
165 
166     The 'servermap' (an instance of IMutableFileServerMap) is used to
167hunk ./src/allmydata/interfaces.py 1014
168         as a guide to where the shares are located.
169 
170         I return a Deferred that fires with the requested contents, or
171-        errbacks with UnrecoverableFileError. Note that a servermap which was
172+        errbacks with UnrecoverableFileError. Note that a servermap that was
173         updated with MODE_ANYTHING or MODE_READ may not know about shares for
174         all versions (those modes stop querying servers as soon as they can
175         fulfil their goals), so you may want to use MODE_CHECK (which checks
176hunk ./src/allmydata/interfaces.py 1073
177     """Upload was unable to satisfy 'servers_of_happiness'"""
178 
179 class UnableToFetchCriticalDownloadDataError(Exception):
180-    """I was unable to fetch some piece of critical data which is supposed to
181+    """I was unable to fetch some piece of critical data that is supposed to
182     be identically present in all shares."""
183 
184 class NoServersError(Exception):
185hunk ./src/allmydata/interfaces.py 1085
186     exists, and overwrite= was set to False."""
187 
188 class NoSuchChildError(Exception):
189-    """A directory node was asked to fetch a child which does not exist."""
190+    """A directory node was asked to fetch a child that does not exist."""
191 
192 class ChildOfWrongTypeError(Exception):
193     """An operation was attempted on a child of the wrong type (file or directory)."""
194hunk ./src/allmydata/interfaces.py 1403
195         if you initially thought you were going to use 10 peers, started
196         encoding, and then two of the peers dropped out: you could use
197         desired_share_ids= to skip the work (both memory and CPU) of
198-        producing shares for the peers which are no longer available.
199+        producing shares for the peers that are no longer available.
200 
201         """
202 
203hunk ./src/allmydata/interfaces.py 1478
204         if you initially thought you were going to use 10 peers, started
205         encoding, and then two of the peers dropped out: you could use
206         desired_share_ids= to skip the work (both memory and CPU) of
207-        producing shares for the peers which are no longer available.
208+        producing shares for the peers that are no longer available.
209 
210         For each call, encode() will return a Deferred that fires with two
211         lists, one containing shares and the other containing the shareids.
212hunk ./src/allmydata/interfaces.py 1535
213         required to be of the same length.  The i'th element of their_shareids
214         is required to be the shareid of the i'th buffer in some_shares.
215 
216-        This returns a Deferred which fires with a sequence of buffers. This
217+        This returns a Deferred that fires with a sequence of buffers. This
218         sequence will contain all of the segments of the original data, in
219         order. The sum of the lengths of all of the buffers will be the
220         'data_size' value passed into the original ICodecEncode.set_params()
221hunk ./src/allmydata/interfaces.py 1582
222         Encoding parameters can be set in three ways. 1: The Encoder class
223         provides defaults (3/7/10). 2: the Encoder can be constructed with
224         an 'options' dictionary, in which the
225-        needed_and_happy_and_total_shares' key can be a (k,d,n) tuple. 3:
226+        'needed_and_happy_and_total_shares' key can be a (k,d,n) tuple. 3:
227         set_params((k,d,n)) can be called.
228 
229         If you intend to use set_params(), you must call it before
230hunk ./src/allmydata/interfaces.py 1780
231         produced, so that the segment hashes can be generated with only a
232         single pass.
233 
234-        This returns a Deferred which fires with a sequence of hashes, using:
235+        This returns a Deferred that fires with a sequence of hashes, using:
236 
237          tuple(segment_hashes[first:last])
238 
239hunk ./src/allmydata/interfaces.py 1796
240     def get_plaintext_hash():
241         """OBSOLETE; Get the hash of the whole plaintext.
242 
243-        This returns a Deferred which fires with a tagged SHA-256 hash of the
244+        This returns a Deferred that fires with a tagged SHA-256 hash of the
245         whole plaintext, obtained from hashutil.plaintext_hash(data).
246         """
247 
248hunk ./src/allmydata/interfaces.py 1856
249         be used to encrypt the data. The key will also be hashed to derive
250         the StorageIndex.
251 
252-        Uploadables which want to achieve convergence should hash their file
253+        Uploadables that want to achieve convergence should hash their file
254         contents and the serialized_encoding_parameters to form the key
255         (which of course requires a full pass over the data). Uploadables can
256         use the upload.ConvergentUploadMixin class to achieve this
257hunk ./src/allmydata/interfaces.py 1862
258         automatically.
259 
260-        Uploadables which do not care about convergence (or do not wish to
261+        Uploadables that do not care about convergence (or do not wish to
262         make multiple passes over the data) can simply return a
263         strongly-random 16 byte string.
264 
265hunk ./src/allmydata/interfaces.py 1872
266 
267     def read(length):
268         """Return a Deferred that fires with a list of strings (perhaps with
269-        only a single element) which, when concatenated together, contain the
270+        only a single element) that, when concatenated together, contain the
271         next 'length' bytes of data. If EOF is near, this may provide fewer
272         than 'length' bytes. The total number of bytes provided by read()
273         before it signals EOF must equal the size provided by get_size().
274hunk ./src/allmydata/interfaces.py 1919
275 
276     def read(length):
277         """
278-        Returns a list of strings which, when concatenated, are the next
279+        Returns a list of strings that, when concatenated, are the next
280         length bytes of the file, or fewer if there are fewer bytes
281         between the current location and the end of the file.
282         """
283hunk ./src/allmydata/interfaces.py 1932
284 
285 class IUploadResults(Interface):
286     """I am returned by upload() methods. I contain a number of public
287-    attributes which can be read to determine the results of the upload. Some
288+    attributes that can be read to determine the results of the upload. Some
289     of these are functional, some are timing information. All of these may be
290     None.
291 
292hunk ./src/allmydata/interfaces.py 1965
293 
294 class IDownloadResults(Interface):
295     """I am created internally by download() methods. I contain a number of
296-    public attributes which contain details about the download process.::
297+    public attributes that contain details about the download process.::
298 
299      .file_size : the size of the file, in bytes
300      .servers_used : set of server peerids that were used during download
301hunk ./src/allmydata/interfaces.py 1991
302 class IUploader(Interface):
303     def upload(uploadable):
304         """Upload the file. 'uploadable' must impement IUploadable. This
305-        returns a Deferred which fires with an IUploadResults instance, from
306+        returns a Deferred that fires with an IUploadResults instance, from
307         which the URI of the file can be obtained as results.uri ."""
308 
309     def upload_ssk(write_capability, new_version, uploadable):
310hunk ./src/allmydata/interfaces.py 2041
311         kind of lease that is obtained (which account number to claim, etc).
312 
313         TODO: any problems seen during checking will be reported to the
314-        health-manager.furl, a centralized object which is responsible for
315+        health-manager.furl, a centralized object that is responsible for
316         figuring out why files are unhealthy so corrective action can be
317         taken.
318         """
319hunk ./src/allmydata/interfaces.py 2056
320         will be put in the check-and-repair results. The Deferred will not
321         fire until the repair is complete.
322 
323-        This returns a Deferred which fires with an instance of
324+        This returns a Deferred that fires with an instance of
325         ICheckAndRepairResults."""
326 
327 class IDeepCheckable(Interface):
328hunk ./src/allmydata/interfaces.py 2141
329                               that was found to be corrupt. Each share
330                               locator is a list of (serverid, storage_index,
331                               sharenum).
332-         count-incompatible-shares: the number of shares which are of a share
333+         count-incompatible-shares: the number of shares that are of a share
334                                     format unknown to this checker
335          list-incompatible-shares: a list of 'share locators', one for each
336                                    share that was found to be of an unknown
337hunk ./src/allmydata/interfaces.py 2148
338                                    format. Each share locator is a list of
339                                    (serverid, storage_index, sharenum).
340          servers-responding: list of (binary) storage server identifiers,
341-                             one for each server which responded to the share
342+                             one for each server that responded to the share
343                              query (even if they said they didn't have
344                              shares, and even if they said they did have
345                              shares but then didn't send them when asked, or
346hunk ./src/allmydata/interfaces.py 2345
347         will use the data in the checker results to guide the repair process,
348         such as which servers provided bad data and should therefore be
349         avoided. The ICheckResults object is inside the
350-        ICheckAndRepairResults object, which is returned by the
351+        ICheckAndRepairResults object that is returned by the
352         ICheckable.check() method::
353 
354          d = filenode.check(repair=False)
355hunk ./src/allmydata/interfaces.py 2436
356         methods to create new objects. I return synchronously."""
357 
358     def create_mutable_file(contents=None, keysize=None):
359-        """I create a new mutable file, and return a Deferred which will fire
360+        """I create a new mutable file, and return a Deferred that will fire
361         with the IMutableFileNode instance when it is ready. If contents= is
362         provided (a bytestring), it will be used as the initial contents of
363         the new file, otherwise the file will contain zero bytes. keysize= is
364hunk ./src/allmydata/interfaces.py 2444
365         usual."""
366 
367     def create_new_mutable_directory(initial_children={}):
368-        """I create a new mutable directory, and return a Deferred which will
369+        """I create a new mutable directory, and return a Deferred that will
370         fire with the IDirectoryNode instance when it is ready. If
371         initial_children= is provided (a dict mapping unicode child name to
372         (childnode, metadata_dict) tuples), the directory will be populated
373hunk ./src/allmydata/interfaces.py 2452
374 
375 class IClientStatus(Interface):
376     def list_all_uploads():
377-        """Return a list of uploader objects, one for each upload which
378+        """Return a list of uploader objects, one for each upload that
379         currently has an object available (tracked with weakrefs). This is
380         intended for debugging purposes."""
381     def list_active_uploads():
382hunk ./src/allmydata/interfaces.py 2462
383         started uploads."""
384 
385     def list_all_downloads():
386-        """Return a list of downloader objects, one for each download which
387+        """Return a list of downloader objects, one for each download that
388         currently has an object available (tracked with weakrefs). This is
389         intended for debugging purposes."""
390     def list_active_downloads():
391hunk ./src/allmydata/interfaces.py 2689
392 
393     def provide(provider=RIStatsProvider, nickname=str):
394         """
395-        @param provider: a stats collector instance which should be polled
396+        @param provider: a stats collector instance that should be polled
397                          periodically by the gatherer to collect stats.
398         @param nickname: a name useful to identify the provided client
399         """
400hunk ./src/allmydata/interfaces.py 2722
401 
402 class IValidatedThingProxy(Interface):
403     def start():
404-        """ Acquire a thing and validate it. Return a deferred which is
405+        """ Acquire a thing and validate it. Return a deferred that is
406         eventually fired with self if the thing is valid or errbacked if it
407         can't be acquired or validated."""
408 
409}
410[Pluggable backends -- new and moved files, changes to moved files. refs #999
411david-sarah@jacaranda.org**20110919232926
412 Ignore-this: ec5d2d1362a092d919e84327d3092424
413] {
414adddir ./src/allmydata/storage/backends
415adddir ./src/allmydata/storage/backends/disk
416move ./src/allmydata/storage/immutable.py ./src/allmydata/storage/backends/disk/immutable.py
417move ./src/allmydata/storage/mutable.py ./src/allmydata/storage/backends/disk/mutable.py
418adddir ./src/allmydata/storage/backends/null
419addfile ./src/allmydata/storage/backends/__init__.py
420addfile ./src/allmydata/storage/backends/base.py
421hunk ./src/allmydata/storage/backends/base.py 1
422+
423+from twisted.application import service
424+
425+from allmydata.storage.common import si_b2a
426+from allmydata.storage.lease import LeaseInfo
427+from allmydata.storage.bucket import BucketReader
428+
429+
430+class Backend(service.MultiService):
431+    def __init__(self):
432+        service.MultiService.__init__(self)
433+
434+
435+class ShareSet(object):
436+    """
437+    This class implements shareset logic that could work for all backends, but
438+    might be useful to override for efficiency.
439+    """
440+
441+    def __init__(self, storageindex):
442+        self.storageindex = storageindex
443+
444+    def get_storage_index(self):
445+        return self.storageindex
446+
447+    def get_storage_index_string(self):
448+        return si_b2a(self.storageindex)
449+
450+    def renew_lease(self, renew_secret, new_expiration_time):
451+        found_shares = False
452+        for share in self.get_shares():
453+            found_shares = True
454+            share.renew_lease(renew_secret, new_expiration_time)
455+
456+        if not found_shares:
457+            raise IndexError("no such lease to renew")
458+
459+    def get_leases(self):
460+        # Since all shares get the same lease data, we just grab the leases
461+        # from the first share.
462+        try:
463+            sf = self.get_shares().next()
464+            return sf.get_leases()
465+        except StopIteration:
466+            return iter([])
467+
468+    def add_or_renew_lease(self, lease_info):
469+        # This implementation assumes that lease data is duplicated in
470+        # all shares of a shareset, which might not be true for all backends.
471+        for share in self.get_shares():
472+            share.add_or_renew_lease(lease_info)
473+
474+    def make_bucket_reader(self, storageserver, share):
475+        return BucketReader(storageserver, share)
476+
477+    def testv_and_readv_and_writev(self, storageserver, secrets,
478+                                   test_and_write_vectors, read_vector,
479+                                   expiration_time):
480+        # The implementation here depends on the following helper methods,
481+        # which must be provided by subclasses:
482+        #
483+        # def _clean_up_after_unlink(self):
484+        #     """clean up resources associated with the shareset after some
485+        #     shares might have been deleted"""
486+        #
487+        # def _create_mutable_share(self, storageserver, shnum, write_enabler):
488+        #     """create a mutable share with the given shnum and write_enabler"""
489+
490+        # secrets might be a triple with cancel_secret in secrets[2], but if
491+        # so we ignore the cancel_secret.
492+        write_enabler = secrets[0]
493+        renew_secret = secrets[1]
494+
495+        si_s = self.get_storage_index_string()
496+        shares = {}
497+        for share in self.get_shares():
498+            # XXX is it correct to ignore immutable shares? Maybe get_shares should
499+            # have a parameter saying what type it's expecting.
500+            if share.sharetype == "mutable":
501+                share.check_write_enabler(write_enabler, si_s)
502+                shares[share.get_shnum()] = share
503+
504+        # write_enabler is good for all existing shares
505+
506+        # now evaluate test vectors
507+        testv_is_good = True
508+        for sharenum in test_and_write_vectors:
509+            (testv, datav, new_length) = test_and_write_vectors[sharenum]
510+            if sharenum in shares:
511+                if not shares[sharenum].check_testv(testv):
512+                    self.log("testv failed: [%d]: %r" % (sharenum, testv))
513+                    testv_is_good = False
514+                    break
515+            else:
516+                # compare the vectors against an empty share, in which all
517+                # reads return empty strings
518+                if not EmptyShare().check_testv(testv):
519+                    self.log("testv failed (empty): [%d] %r" % (sharenum,
520+                                                                testv))
521+                    testv_is_good = False
522+                    break
523+
524+        # gather the read vectors, before we do any writes
525+        read_data = {}
526+        for shnum, share in shares.items():
527+            read_data[shnum] = share.readv(read_vector)
528+
529+        ownerid = 1 # TODO
530+        lease_info = LeaseInfo(ownerid, renew_secret,
531+                               expiration_time, storageserver.get_serverid())
532+
533+        if testv_is_good:
534+            # now apply the write vectors
535+            for shnum in test_and_write_vectors:
536+                (testv, datav, new_length) = test_and_write_vectors[shnum]
537+                if new_length == 0:
538+                    if shnum in shares:
539+                        shares[shnum].unlink()
540+                else:
541+                    if shnum not in shares:
542+                        # allocate a new share
543+                        share = self._create_mutable_share(storageserver, shnum, write_enabler)
544+                        shares[shnum] = share
545+                    shares[shnum].writev(datav, new_length)
546+                    # and update the lease
547+                    shares[shnum].add_or_renew_lease(lease_info)
548+
549+            if new_length == 0:
550+                self._clean_up_after_unlink()
551+
552+        return (testv_is_good, read_data)
553+
554+    def readv(self, wanted_shnums, read_vector):
555+        """
556+        Read a vector from the numbered shares in this shareset. An empty
557+        shares list means to return data from all known shares.
558+
559+        @param wanted_shnums=ListOf(int)
560+        @param read_vector=ReadVector
561+        @return DictOf(int, ReadData): shnum -> results, with one key per share
562+        """
563+        datavs = {}
564+        for share in self.get_shares():
565+            shnum = share.get_shnum()
566+            if not wanted_shnums or shnum in wanted_shnums:
567+                datavs[shnum] = share.readv(read_vector)
568+
569+        return datavs
570+
571+
572+def testv_compare(a, op, b):
573+    assert op in ("lt", "le", "eq", "ne", "ge", "gt")
574+    if op == "lt":
575+        return a < b
576+    if op == "le":
577+        return a <= b
578+    if op == "eq":
579+        return a == b
580+    if op == "ne":
581+        return a != b
582+    if op == "ge":
583+        return a >= b
584+    if op == "gt":
585+        return a > b
586+    # never reached
587+
588+
589+class EmptyShare:
590+    def check_testv(self, testv):
591+        test_good = True
592+        for (offset, length, operator, specimen) in testv:
593+            data = ""
594+            if not testv_compare(data, operator, specimen):
595+                test_good = False
596+                break
597+        return test_good
598+
599addfile ./src/allmydata/storage/backends/disk/__init__.py
600addfile ./src/allmydata/storage/backends/disk/disk_backend.py
601hunk ./src/allmydata/storage/backends/disk/disk_backend.py 1
602+
603+import re
604+
605+from twisted.python.filepath import UnlistableError
606+
607+from zope.interface import implements
608+from allmydata.interfaces import IStorageBackend, IShareSet
609+from allmydata.util import fileutil, log, time_format
610+from allmydata.storage.common import si_b2a, si_a2b
611+from allmydata.storage.bucket import BucketWriter
612+from allmydata.storage.backends.base import Backend, ShareSet
613+from allmydata.storage.backends.disk.immutable import ImmutableDiskShare
614+from allmydata.storage.backends.disk.mutable import MutableDiskShare, create_mutable_disk_share
615+
616+# storage/
617+# storage/shares/incoming
618+#   incoming/ holds temp dirs named $START/$STORAGEINDEX/$SHARENUM which will
619+#   be moved to storage/shares/$START/$STORAGEINDEX/$SHARENUM upon success
620+# storage/shares/$START/$STORAGEINDEX
621+# storage/shares/$START/$STORAGEINDEX/$SHARENUM
622+
623+# Where "$START" denotes the first 10 bits worth of $STORAGEINDEX (that's 2
624+# base-32 chars).
625+# $SHARENUM matches this regex:
626+NUM_RE=re.compile("^[0-9]+$")
627+
628+
629+def si_si2dir(startfp, storageindex):
630+    sia = si_b2a(storageindex)
631+    newfp = startfp.child(sia[:2])
632+    return newfp.child(sia)
633+
634+
635+def get_share(fp):
636+    f = fp.open('rb')
637+    try:
638+        prefix = f.read(32)
639+    finally:
640+        f.close()
641+
642+    if prefix == MutableDiskShare.MAGIC:
643+        return MutableDiskShare(fp)
644+    else:
645+        # assume it's immutable
646+        return ImmutableDiskShare(fp)
647+
648+
649+class DiskBackend(Backend):
650+    implements(IStorageBackend)
651+
652+    def __init__(self, storedir, readonly=False, reserved_space=0, discard_storage=False):
653+        Backend.__init__(self)
654+        self._setup_storage(storedir, readonly, reserved_space, discard_storage)
655+        self._setup_corruption_advisory()
656+
657+    def _setup_storage(self, storedir, readonly, reserved_space, discard_storage):
658+        self._storedir = storedir
659+        self._readonly = readonly
660+        self._reserved_space = int(reserved_space)
661+        self._discard_storage = discard_storage
662+        self._sharedir = self._storedir.child("shares")
663+        fileutil.fp_make_dirs(self._sharedir)
664+        self._incomingdir = self._sharedir.child('incoming')
665+        self._clean_incomplete()
666+        if self._reserved_space and (self.get_available_space() is None):
667+            log.msg("warning: [storage]reserved_space= is set, but this platform does not support an API to get disk statistics (statvfs(2) or GetDiskFreeSpaceEx), so this reservation cannot be honored",
668+                    umid="0wZ27w", level=log.UNUSUAL)
669+
670+    def _clean_incomplete(self):
671+        fileutil.fp_remove(self._incomingdir)
672+        fileutil.fp_make_dirs(self._incomingdir)
673+
674+    def _setup_corruption_advisory(self):
675+        # we don't actually create the corruption-advisory dir until necessary
676+        self._corruption_advisory_dir = self._storedir.child("corruption-advisories")
677+
678+    def _make_shareset(self, sharehomedir):
679+        return self.get_shareset(si_a2b(sharehomedir.basename()))
680+
681+    def get_sharesets_for_prefix(self, prefix):
682+        prefixfp = self._sharedir.child(prefix)
683+        try:
684+            sharesets = map(self._make_shareset, prefixfp.children())
685+            def _by_base32si(b):
686+                return b.get_storage_index_string()
687+            sharesets.sort(key=_by_base32si)
688+        except EnvironmentError:
689+            sharesets = []
690+        return sharesets
691+
692+    def get_shareset(self, storageindex):
693+        sharehomedir = si_si2dir(self._sharedir, storageindex)
694+        incominghomedir = si_si2dir(self._incomingdir, storageindex)
695+        return DiskShareSet(storageindex, sharehomedir, incominghomedir, discard_storage=self._discard_storage)
696+
697+    def fill_in_space_stats(self, stats):
698+        stats['storage_server.reserved_space'] = self._reserved_space
699+        try:
700+            disk = fileutil.get_disk_stats(self._sharedir, self._reserved_space)
701+            writeable = disk['avail'] > 0
702+
703+            # spacetime predictors should use disk_avail / (d(disk_used)/dt)
704+            stats['storage_server.disk_total'] = disk['total']
705+            stats['storage_server.disk_used'] = disk['used']
706+            stats['storage_server.disk_free_for_root'] = disk['free_for_root']
707+            stats['storage_server.disk_free_for_nonroot'] = disk['free_for_nonroot']
708+            stats['storage_server.disk_avail'] = disk['avail']
709+        except AttributeError:
710+            writeable = True
711+        except EnvironmentError:
712+            log.msg("OS call to get disk statistics failed", level=log.UNUSUAL)
713+            writeable = False
714+
715+        if self._readonly:
716+            stats['storage_server.disk_avail'] = 0
717+            writeable = False
718+
719+        stats['storage_server.accepting_immutable_shares'] = int(writeable)
720+
721+    def get_available_space(self):
722+        if self._readonly:
723+            return 0
724+        return fileutil.get_available_space(self._sharedir, self._reserved_space)
725+
726+    def advise_corrupt_share(self, sharetype, storageindex, shnum, reason):
727+        fileutil.fp_make_dirs(self._corruption_advisory_dir)
728+        now = time_format.iso_utc(sep="T")
729+        si_s = si_b2a(storageindex)
730+
731+        # Windows can't handle colons in the filename.
732+        name = ("%s--%s-%d" % (now, si_s, shnum)).replace(":", "")
733+        f = self._corruption_advisory_dir.child(name).open("w")
734+        try:
735+            f.write("report: Share Corruption\n")
736+            f.write("type: %s\n" % sharetype)
737+            f.write("storage_index: %s\n" % si_s)
738+            f.write("share_number: %d\n" % shnum)
739+            f.write("\n")
740+            f.write(reason)
741+            f.write("\n")
742+        finally:
743+            f.close()
744+
745+        log.msg(format=("client claims corruption in (%(share_type)s) " +
746+                        "%(si)s-%(shnum)d: %(reason)s"),
747+                share_type=sharetype, si=si_s, shnum=shnum, reason=reason,
748+                level=log.SCARY, umid="SGx2fA")
749+
750+
751+class DiskShareSet(ShareSet):
752+    implements(IShareSet)
753+
754+    def __init__(self, storageindex, sharehomedir, incominghomedir=None, discard_storage=False):
755+        ShareSet.__init__(self, storageindex)
756+        self._sharehomedir = sharehomedir
757+        self._incominghomedir = incominghomedir
758+        self._discard_storage = discard_storage
759+
760+    def get_overhead(self):
761+        return (fileutil.get_disk_usage(self._sharehomedir) +
762+                fileutil.get_disk_usage(self._incominghomedir))
763+
764+    def get_shares(self):
765+        """
766+        Generate IStorageBackendShare objects for shares we have for this storage index.
767+        ("Shares we have" means completed ones, excluding incoming ones.)
768+        """
769+        try:
770+            for fp in self._sharehomedir.children():
771+                shnumstr = fp.basename()
772+                if not NUM_RE.match(shnumstr):
773+                    continue
774+                sharehome = self._sharehomedir.child(shnumstr)
775+                yield self.get_share(sharehome)
776+        except UnlistableError:
777+            # There is no shares directory at all.
778+            pass
779+
780+    def has_incoming(self, shnum):
781+        if self._incominghomedir is None:
782+            return False
783+        return self._incominghomedir.child(str(shnum)).exists()
784+
785+    def make_bucket_writer(self, storageserver, shnum, max_space_per_bucket, lease_info, canary):
786+        sharehome = self._sharehomedir.child(str(shnum))
787+        incominghome = self._incominghomedir.child(str(shnum))
788+        immsh = ImmutableDiskShare(self.get_storage_index(), shnum, sharehome, incominghome,
789+                                   max_size=max_space_per_bucket, create=True)
790+        bw = BucketWriter(storageserver, immsh, max_space_per_bucket, lease_info, canary)
791+        if self._discard_storage:
792+            bw.throw_out_all_data = True
793+        return bw
794+
795+    def _create_mutable_share(self, storageserver, shnum, write_enabler):
796+        fileutil.fp_make_dirs(self._sharehomedir)
797+        sharehome = self._sharehomedir.child(str(shnum))
798+        serverid = storageserver.get_serverid()
799+        return create_mutable_disk_share(sharehome, serverid, write_enabler, storageserver)
800+
801+    def _clean_up_after_unlink(self):
802+        fileutil.fp_rmdir_if_empty(self._sharehomedir)
803+
804hunk ./src/allmydata/storage/backends/disk/immutable.py 1
805-import os, stat, struct, time
806 
807hunk ./src/allmydata/storage/backends/disk/immutable.py 2
808-from foolscap.api import Referenceable
809+import struct
810 
811 from zope.interface import implements
812hunk ./src/allmydata/storage/backends/disk/immutable.py 5
813-from allmydata.interfaces import RIBucketWriter, RIBucketReader
814-from allmydata.util import base32, fileutil, log
815+
816+from allmydata.interfaces import IStoredShare
817+from allmydata.util import fileutil
818 from allmydata.util.assertutil import precondition
819hunk ./src/allmydata/storage/backends/disk/immutable.py 9
820+from allmydata.util.fileutil import fp_make_dirs
821 from allmydata.util.hashutil import constant_time_compare
822hunk ./src/allmydata/storage/backends/disk/immutable.py 11
823+from allmydata.util.encodingutil import quote_filepath
824+from allmydata.storage.common import si_b2a, UnknownImmutableContainerVersionError, DataTooLargeError
825 from allmydata.storage.lease import LeaseInfo
826hunk ./src/allmydata/storage/backends/disk/immutable.py 14
827-from allmydata.storage.common import UnknownImmutableContainerVersionError, \
828-     DataTooLargeError
829+
830 
831 # each share file (in storage/shares/$SI/$SHNUM) contains lease information
832 # and share data. The share data is accessed by RIBucketWriter.write and
833hunk ./src/allmydata/storage/backends/disk/immutable.py 41
834 # then the value stored in this field will be the actual share data length
835 # modulo 2**32.
836 
837-class ShareFile:
838-    LEASE_SIZE = struct.calcsize(">L32s32sL")
839+class ImmutableDiskShare(object):
840+    implements(IStoredShare)
841+
842     sharetype = "immutable"
843hunk ./src/allmydata/storage/backends/disk/immutable.py 45
844+    LEASE_SIZE = struct.calcsize(">L32s32sL")
845+
846 
847hunk ./src/allmydata/storage/backends/disk/immutable.py 48
848-    def __init__(self, filename, max_size=None, create=False):
849-        """ If max_size is not None then I won't allow more than max_size to be written to me. If create=True and max_size must not be None. """
850+    def __init__(self, storageindex, shnum, finalhome=None, incominghome=None, max_size=None, create=False):
851+        """ If max_size is not None then I won't allow more than
852+        max_size to be written to me. If create=True then max_size
853+        must not be None. """
854         precondition((max_size is not None) or (not create), max_size, create)
855hunk ./src/allmydata/storage/backends/disk/immutable.py 53
856-        self.home = filename
857+        self._storageindex = storageindex
858         self._max_size = max_size
859hunk ./src/allmydata/storage/backends/disk/immutable.py 55
860+        self._incominghome = incominghome
861+        self._home = finalhome
862+        self._shnum = shnum
863         if create:
864             # touch the file, so later callers will see that we're working on
865             # it. Also construct the metadata.
866hunk ./src/allmydata/storage/backends/disk/immutable.py 61
867-            assert not os.path.exists(self.home)
868-            fileutil.make_dirs(os.path.dirname(self.home))
869-            f = open(self.home, 'wb')
870+            assert not finalhome.exists()
871+            fp_make_dirs(self._incominghome.parent())
872             # The second field -- the four-byte share data length -- is no
873             # longer used as of Tahoe v1.3.0, but we continue to write it in
874             # there in case someone downgrades a storage server from >=
875hunk ./src/allmydata/storage/backends/disk/immutable.py 72
876             # the largest length that can fit into the field. That way, even
877             # if this does happen, the old < v1.3.0 server will still allow
878             # clients to read the first part of the share.
879-            f.write(struct.pack(">LLL", 1, min(2**32-1, max_size), 0))
880-            f.close()
881+            self._incominghome.setContent(struct.pack(">LLL", 1, min(2**32-1, max_size), 0) )
882             self._lease_offset = max_size + 0x0c
883             self._num_leases = 0
884         else:
885hunk ./src/allmydata/storage/backends/disk/immutable.py 76
886-            f = open(self.home, 'rb')
887-            filesize = os.path.getsize(self.home)
888-            (version, unused, num_leases) = struct.unpack(">LLL", f.read(0xc))
889-            f.close()
890+            f = self._home.open(mode='rb')
891+            try:
892+                (version, unused, num_leases) = struct.unpack(">LLL", f.read(0xc))
893+            finally:
894+                f.close()
895+            filesize = self._home.getsize()
896             if version != 1:
897                 msg = "sharefile %s had version %d but we wanted 1" % \
898hunk ./src/allmydata/storage/backends/disk/immutable.py 84
899-                      (filename, version)
900+                      (self._home, version)
901                 raise UnknownImmutableContainerVersionError(msg)
902             self._num_leases = num_leases
903             self._lease_offset = filesize - (num_leases * self.LEASE_SIZE)
904hunk ./src/allmydata/storage/backends/disk/immutable.py 90
905         self._data_offset = 0xc
906 
907+    def __repr__(self):
908+        return ("<ImmutableDiskShare %s:%r at %s>"
909+                % (si_b2a(self._storageindex), self._shnum, quote_filepath(self._home)))
910+
911+    def close(self):
912+        fileutil.fp_make_dirs(self._home.parent())
913+        self._incominghome.moveTo(self._home)
914+        try:
915+            # self._incominghome is like storage/shares/incoming/ab/abcde/4 .
916+            # We try to delete the parent (.../ab/abcde) to avoid leaving
917+            # these directories lying around forever, but the delete might
918+            # fail if we're working on another share for the same storage
919+            # index (like ab/abcde/5). The alternative approach would be to
920+            # use a hierarchy of objects (PrefixHolder, BucketHolder,
921+            # ShareWriter), each of which is responsible for a single
922+            # directory on disk, and have them use reference counting of
923+            # their children to know when they should do the rmdir. This
924+            # approach is simpler, but relies on os.rmdir refusing to delete
925+            # a non-empty directory. Do *not* use fileutil.fp_remove() here!
926+            fileutil.fp_rmdir_if_empty(self._incominghome.parent())
927+            # we also delete the grandparent (prefix) directory, .../ab ,
928+            # again to avoid leaving directories lying around. This might
929+            # fail if there is another bucket open that shares a prefix (like
930+            # ab/abfff).
931+            fileutil.fp_rmdir_if_empty(self._incominghome.parent().parent())
932+            # we leave the great-grandparent (incoming/) directory in place.
933+        except EnvironmentError:
934+            # ignore the "can't rmdir because the directory is not empty"
935+            # exceptions, those are normal consequences of the
936+            # above-mentioned conditions.
937+            pass
938+        pass
939+
940+    def get_used_space(self):
941+        return (fileutil.get_used_space(self._home) +
942+                fileutil.get_used_space(self._incominghome))
943+
944+    def get_storage_index(self):
945+        return self._storageindex
946+
947+    def get_shnum(self):
948+        return self._shnum
949+
950     def unlink(self):
951hunk ./src/allmydata/storage/backends/disk/immutable.py 134
952-        os.unlink(self.home)
953+        self._home.remove()
954+
955+    def get_size(self):
956+        return self._home.getsize()
957+
958+    def get_data_length(self):
959+        return self._lease_offset - self._data_offset
960+
961+    #def readv(self, read_vector):
962+    #    ...
963 
964     def read_share_data(self, offset, length):
965         precondition(offset >= 0)
966hunk ./src/allmydata/storage/backends/disk/immutable.py 147
967-        # reads beyond the end of the data are truncated. Reads that start
968+
969+        # Reads beyond the end of the data are truncated. Reads that start
970         # beyond the end of the data return an empty string.
971         seekpos = self._data_offset+offset
972         actuallength = max(0, min(length, self._lease_offset-seekpos))
973hunk ./src/allmydata/storage/backends/disk/immutable.py 154
974         if actuallength == 0:
975             return ""
976-        f = open(self.home, 'rb')
977-        f.seek(seekpos)
978-        return f.read(actuallength)
979+        f = self._home.open(mode='rb')
980+        try:
981+            f.seek(seekpos)
982+            sharedata = f.read(actuallength)
983+        finally:
984+            f.close()
985+        return sharedata
986 
987     def write_share_data(self, offset, data):
988         length = len(data)
989hunk ./src/allmydata/storage/backends/disk/immutable.py 167
990         precondition(offset >= 0, offset)
991         if self._max_size is not None and offset+length > self._max_size:
992             raise DataTooLargeError(self._max_size, offset, length)
993-        f = open(self.home, 'rb+')
994-        real_offset = self._data_offset+offset
995-        f.seek(real_offset)
996-        assert f.tell() == real_offset
997-        f.write(data)
998-        f.close()
999+        f = self._incominghome.open(mode='rb+')
1000+        try:
1001+            real_offset = self._data_offset+offset
1002+            f.seek(real_offset)
1003+            assert f.tell() == real_offset
1004+            f.write(data)
1005+        finally:
1006+            f.close()
1007 
1008     def _write_lease_record(self, f, lease_number, lease_info):
1009         offset = self._lease_offset + lease_number * self.LEASE_SIZE
1010hunk ./src/allmydata/storage/backends/disk/immutable.py 184
1011 
1012     def _read_num_leases(self, f):
1013         f.seek(0x08)
1014-        (num_leases,) = struct.unpack(">L", f.read(4))
1015+        ro = f.read(4)
1016+        (num_leases,) = struct.unpack(">L", ro)
1017         return num_leases
1018 
1019     def _write_num_leases(self, f, num_leases):
1020hunk ./src/allmydata/storage/backends/disk/immutable.py 195
1021     def _truncate_leases(self, f, num_leases):
1022         f.truncate(self._lease_offset + num_leases * self.LEASE_SIZE)
1023 
1024+    # These lease operations are intended for use by disk_backend.py.
1025+    # Other clients should not depend on the fact that the disk backend
1026+    # stores leases in share files.
1027+
1028     def get_leases(self):
1029         """Yields a LeaseInfo instance for all leases."""
1030hunk ./src/allmydata/storage/backends/disk/immutable.py 201
1031-        f = open(self.home, 'rb')
1032-        (version, unused, num_leases) = struct.unpack(">LLL", f.read(0xc))
1033-        f.seek(self._lease_offset)
1034-        for i in range(num_leases):
1035-            data = f.read(self.LEASE_SIZE)
1036-            if data:
1037-                yield LeaseInfo().from_immutable_data(data)
1038+        f = self._home.open(mode='rb')
1039+        try:
1040+            (version, unused, num_leases) = struct.unpack(">LLL", f.read(0xc))
1041+            f.seek(self._lease_offset)
1042+            for i in range(num_leases):
1043+                data = f.read(self.LEASE_SIZE)
1044+                if data:
1045+                    yield LeaseInfo().from_immutable_data(data)
1046+        finally:
1047+            f.close()
1048 
1049     def add_lease(self, lease_info):
1050hunk ./src/allmydata/storage/backends/disk/immutable.py 213
1051-        f = open(self.home, 'rb+')
1052-        num_leases = self._read_num_leases(f)
1053-        self._write_lease_record(f, num_leases, lease_info)
1054-        self._write_num_leases(f, num_leases+1)
1055-        f.close()
1056+        f = self._incominghome.open(mode='rb')
1057+        try:
1058+            num_leases = self._read_num_leases(f)
1059+        finally:
1060+            f.close()
1061+        f = self._home.open(mode='wb+')
1062+        try:
1063+            self._write_lease_record(f, num_leases, lease_info)
1064+            self._write_num_leases(f, num_leases+1)
1065+        finally:
1066+            f.close()
1067 
1068     def renew_lease(self, renew_secret, new_expire_time):
1069hunk ./src/allmydata/storage/backends/disk/immutable.py 226
1070-        for i,lease in enumerate(self.get_leases()):
1071-            if constant_time_compare(lease.renew_secret, renew_secret):
1072-                # yup. See if we need to update the owner time.
1073-                if new_expire_time > lease.expiration_time:
1074-                    # yes
1075-                    lease.expiration_time = new_expire_time
1076-                    f = open(self.home, 'rb+')
1077-                    self._write_lease_record(f, i, lease)
1078-                    f.close()
1079-                return
1080+        try:
1081+            for i, lease in enumerate(self.get_leases()):
1082+                if constant_time_compare(lease.renew_secret, renew_secret):
1083+                    # yup. See if we need to update the owner time.
1084+                    if new_expire_time > lease.expiration_time:
1085+                        # yes
1086+                        lease.expiration_time = new_expire_time
1087+                        f = self._home.open('rb+')
1088+                        try:
1089+                            self._write_lease_record(f, i, lease)
1090+                        finally:
1091+                            f.close()
1092+                    return
1093+        except IndexError, e:
1094+            raise Exception("IndexError: %s" % (e,))
1095         raise IndexError("unable to renew non-existent lease")
1096 
1097     def add_or_renew_lease(self, lease_info):
1098hunk ./src/allmydata/storage/backends/disk/immutable.py 249
1099                              lease_info.expiration_time)
1100         except IndexError:
1101             self.add_lease(lease_info)
1102-
1103-
1104-    def cancel_lease(self, cancel_secret):
1105-        """Remove a lease with the given cancel_secret. If the last lease is
1106-        cancelled, the file will be removed. Return the number of bytes that
1107-        were freed (by truncating the list of leases, and possibly by
1108-        deleting the file. Raise IndexError if there was no lease with the
1109-        given cancel_secret.
1110-        """
1111-
1112-        leases = list(self.get_leases())
1113-        num_leases_removed = 0
1114-        for i,lease in enumerate(leases):
1115-            if constant_time_compare(lease.cancel_secret, cancel_secret):
1116-                leases[i] = None
1117-                num_leases_removed += 1
1118-        if not num_leases_removed:
1119-            raise IndexError("unable to find matching lease to cancel")
1120-        if num_leases_removed:
1121-            # pack and write out the remaining leases. We write these out in
1122-            # the same order as they were added, so that if we crash while
1123-            # doing this, we won't lose any non-cancelled leases.
1124-            leases = [l for l in leases if l] # remove the cancelled leases
1125-            f = open(self.home, 'rb+')
1126-            for i,lease in enumerate(leases):
1127-                self._write_lease_record(f, i, lease)
1128-            self._write_num_leases(f, len(leases))
1129-            self._truncate_leases(f, len(leases))
1130-            f.close()
1131-        space_freed = self.LEASE_SIZE * num_leases_removed
1132-        if not len(leases):
1133-            space_freed += os.stat(self.home)[stat.ST_SIZE]
1134-            self.unlink()
1135-        return space_freed
1136-
1137-
1138-class BucketWriter(Referenceable):
1139-    implements(RIBucketWriter)
1140-
1141-    def __init__(self, ss, incominghome, finalhome, max_size, lease_info, canary):
1142-        self.ss = ss
1143-        self.incominghome = incominghome
1144-        self.finalhome = finalhome
1145-        self._max_size = max_size # don't allow the client to write more than this
1146-        self._canary = canary
1147-        self._disconnect_marker = canary.notifyOnDisconnect(self._disconnected)
1148-        self.closed = False
1149-        self.throw_out_all_data = False
1150-        self._sharefile = ShareFile(incominghome, create=True, max_size=max_size)
1151-        # also, add our lease to the file now, so that other ones can be
1152-        # added by simultaneous uploaders
1153-        self._sharefile.add_lease(lease_info)
1154-
1155-    def allocated_size(self):
1156-        return self._max_size
1157-
1158-    def remote_write(self, offset, data):
1159-        start = time.time()
1160-        precondition(not self.closed)
1161-        if self.throw_out_all_data:
1162-            return
1163-        self._sharefile.write_share_data(offset, data)
1164-        self.ss.add_latency("write", time.time() - start)
1165-        self.ss.count("write")
1166-
1167-    def remote_close(self):
1168-        precondition(not self.closed)
1169-        start = time.time()
1170-
1171-        fileutil.make_dirs(os.path.dirname(self.finalhome))
1172-        fileutil.rename(self.incominghome, self.finalhome)
1173-        try:
1174-            # self.incominghome is like storage/shares/incoming/ab/abcde/4 .
1175-            # We try to delete the parent (.../ab/abcde) to avoid leaving
1176-            # these directories lying around forever, but the delete might
1177-            # fail if we're working on another share for the same storage
1178-            # index (like ab/abcde/5). The alternative approach would be to
1179-            # use a hierarchy of objects (PrefixHolder, BucketHolder,
1180-            # ShareWriter), each of which is responsible for a single
1181-            # directory on disk, and have them use reference counting of
1182-            # their children to know when they should do the rmdir. This
1183-            # approach is simpler, but relies on os.rmdir refusing to delete
1184-            # a non-empty directory. Do *not* use fileutil.rm_dir() here!
1185-            os.rmdir(os.path.dirname(self.incominghome))
1186-            # we also delete the grandparent (prefix) directory, .../ab ,
1187-            # again to avoid leaving directories lying around. This might
1188-            # fail if there is another bucket open that shares a prefix (like
1189-            # ab/abfff).
1190-            os.rmdir(os.path.dirname(os.path.dirname(self.incominghome)))
1191-            # we leave the great-grandparent (incoming/) directory in place.
1192-        except EnvironmentError:
1193-            # ignore the "can't rmdir because the directory is not empty"
1194-            # exceptions, those are normal consequences of the
1195-            # above-mentioned conditions.
1196-            pass
1197-        self._sharefile = None
1198-        self.closed = True
1199-        self._canary.dontNotifyOnDisconnect(self._disconnect_marker)
1200-
1201-        filelen = os.stat(self.finalhome)[stat.ST_SIZE]
1202-        self.ss.bucket_writer_closed(self, filelen)
1203-        self.ss.add_latency("close", time.time() - start)
1204-        self.ss.count("close")
1205-
1206-    def _disconnected(self):
1207-        if not self.closed:
1208-            self._abort()
1209-
1210-    def remote_abort(self):
1211-        log.msg("storage: aborting sharefile %s" % self.incominghome,
1212-                facility="tahoe.storage", level=log.UNUSUAL)
1213-        if not self.closed:
1214-            self._canary.dontNotifyOnDisconnect(self._disconnect_marker)
1215-        self._abort()
1216-        self.ss.count("abort")
1217-
1218-    def _abort(self):
1219-        if self.closed:
1220-            return
1221-
1222-        os.remove(self.incominghome)
1223-        # if we were the last share to be moved, remove the incoming/
1224-        # directory that was our parent
1225-        parentdir = os.path.split(self.incominghome)[0]
1226-        if not os.listdir(parentdir):
1227-            os.rmdir(parentdir)
1228-        self._sharefile = None
1229-
1230-        # We are now considered closed for further writing. We must tell
1231-        # the storage server about this so that it stops expecting us to
1232-        # use the space it allocated for us earlier.
1233-        self.closed = True
1234-        self.ss.bucket_writer_closed(self, 0)
1235-
1236-
1237-class BucketReader(Referenceable):
1238-    implements(RIBucketReader)
1239-
1240-    def __init__(self, ss, sharefname, storage_index=None, shnum=None):
1241-        self.ss = ss
1242-        self._share_file = ShareFile(sharefname)
1243-        self.storage_index = storage_index
1244-        self.shnum = shnum
1245-
1246-    def __repr__(self):
1247-        return "<%s %s %s>" % (self.__class__.__name__,
1248-                               base32.b2a_l(self.storage_index[:8], 60),
1249-                               self.shnum)
1250-
1251-    def remote_read(self, offset, length):
1252-        start = time.time()
1253-        data = self._share_file.read_share_data(offset, length)
1254-        self.ss.add_latency("read", time.time() - start)
1255-        self.ss.count("read")
1256-        return data
1257-
1258-    def remote_advise_corrupt_share(self, reason):
1259-        return self.ss.remote_advise_corrupt_share("immutable",
1260-                                                   self.storage_index,
1261-                                                   self.shnum,
1262-                                                   reason)
1263hunk ./src/allmydata/storage/backends/disk/mutable.py 1
1264-import os, stat, struct
1265 
1266hunk ./src/allmydata/storage/backends/disk/mutable.py 2
1267-from allmydata.interfaces import BadWriteEnablerError
1268-from allmydata.util import idlib, log
1269+import struct
1270+
1271+from zope.interface import implements
1272+
1273+from allmydata.interfaces import IStoredMutableShare, BadWriteEnablerError
1274+from allmydata.util import fileutil, idlib, log
1275 from allmydata.util.assertutil import precondition
1276 from allmydata.util.hashutil import constant_time_compare
1277hunk ./src/allmydata/storage/backends/disk/mutable.py 10
1278-from allmydata.storage.lease import LeaseInfo
1279-from allmydata.storage.common import UnknownMutableContainerVersionError, \
1280+from allmydata.util.encodingutil import quote_filepath
1281+from allmydata.storage.common import si_b2a, UnknownMutableContainerVersionError, \
1282      DataTooLargeError
1283hunk ./src/allmydata/storage/backends/disk/mutable.py 13
1284+from allmydata.storage.lease import LeaseInfo
1285+from allmydata.storage.backends.base import testv_compare
1286 
1287hunk ./src/allmydata/storage/backends/disk/mutable.py 16
1288-# the MutableShareFile is like the ShareFile, but used for mutable data. It
1289-# has a different layout. See docs/mutable.txt for more details.
1290+
1291+# The MutableDiskShare is like the ImmutableDiskShare, but used for mutable data.
1292+# It has a different layout. See docs/mutable.rst for more details.
1293 
1294 # #   offset    size    name
1295 # 1   0         32      magic verstr "tahoe mutable container v1" plus binary
1296hunk ./src/allmydata/storage/backends/disk/mutable.py 31
1297 #                        4    4   expiration timestamp
1298 #                        8   32   renewal token
1299 #                        40  32   cancel token
1300-#                        72  20   nodeid which accepted the tokens
1301+#                        72  20   nodeid that accepted the tokens
1302 # 7   468       (a)     data
1303 # 8   ??        4       count of extra leases
1304 # 9   ??        n*92    extra leases
1305hunk ./src/allmydata/storage/backends/disk/mutable.py 37
1306 
1307 
1308-# The struct module doc says that L's are 4 bytes in size., and that Q's are
1309+# The struct module doc says that L's are 4 bytes in size, and that Q's are
1310 # 8 bytes in size. Since compatibility depends upon this, double-check it.
1311 assert struct.calcsize(">L") == 4, struct.calcsize(">L")
1312 assert struct.calcsize(">Q") == 8, struct.calcsize(">Q")
1313hunk ./src/allmydata/storage/backends/disk/mutable.py 42
1314 
1315-class MutableShareFile:
1316+
1317+class MutableDiskShare(object):
1318+    implements(IStoredMutableShare)
1319 
1320     sharetype = "mutable"
1321     DATA_LENGTH_OFFSET = struct.calcsize(">32s20s32s")
1322hunk ./src/allmydata/storage/backends/disk/mutable.py 54
1323     assert LEASE_SIZE == 92
1324     DATA_OFFSET = HEADER_SIZE + 4*LEASE_SIZE
1325     assert DATA_OFFSET == 468, DATA_OFFSET
1326+
1327     # our sharefiles share with a recognizable string, plus some random
1328     # binary data to reduce the chance that a regular text file will look
1329     # like a sharefile.
1330hunk ./src/allmydata/storage/backends/disk/mutable.py 63
1331     MAX_SIZE = 2*1000*1000*1000 # 2GB, kind of arbitrary
1332     # TODO: decide upon a policy for max share size
1333 
1334-    def __init__(self, filename, parent=None):
1335-        self.home = filename
1336-        if os.path.exists(self.home):
1337+    def __init__(self, storageindex, shnum, home, parent=None):
1338+        self._storageindex = storageindex
1339+        self._shnum = shnum
1340+        self._home = home
1341+        if self._home.exists():
1342             # we don't cache anything, just check the magic
1343hunk ./src/allmydata/storage/backends/disk/mutable.py 69
1344-            f = open(self.home, 'rb')
1345-            data = f.read(self.HEADER_SIZE)
1346-            (magic,
1347-             write_enabler_nodeid, write_enabler,
1348-             data_length, extra_least_offset) = \
1349-             struct.unpack(">32s20s32sQQ", data)
1350-            if magic != self.MAGIC:
1351-                msg = "sharefile %s had magic '%r' but we wanted '%r'" % \
1352-                      (filename, magic, self.MAGIC)
1353-                raise UnknownMutableContainerVersionError(msg)
1354+            f = self._home.open('rb')
1355+            try:
1356+                data = f.read(self.HEADER_SIZE)
1357+                (magic,
1358+                 write_enabler_nodeid, write_enabler,
1359+                 data_length, extra_least_offset) = \
1360+                 struct.unpack(">32s20s32sQQ", data)
1361+                if magic != self.MAGIC:
1362+                    msg = "sharefile %s had magic '%r' but we wanted '%r'" % \
1363+                          (quote_filepath(self._home), magic, self.MAGIC)
1364+                    raise UnknownMutableContainerVersionError(msg)
1365+            finally:
1366+                f.close()
1367         self.parent = parent # for logging
1368 
1369     def log(self, *args, **kwargs):
1370hunk ./src/allmydata/storage/backends/disk/mutable.py 87
1371         return self.parent.log(*args, **kwargs)
1372 
1373-    def create(self, my_nodeid, write_enabler):
1374-        assert not os.path.exists(self.home)
1375+    def create(self, serverid, write_enabler):
1376+        assert not self._home.exists()
1377         data_length = 0
1378         extra_lease_offset = (self.HEADER_SIZE
1379                               + 4 * self.LEASE_SIZE
1380hunk ./src/allmydata/storage/backends/disk/mutable.py 95
1381                               + data_length)
1382         assert extra_lease_offset == self.DATA_OFFSET # true at creation
1383         num_extra_leases = 0
1384-        f = open(self.home, 'wb')
1385-        header = struct.pack(">32s20s32sQQ",
1386-                             self.MAGIC, my_nodeid, write_enabler,
1387-                             data_length, extra_lease_offset,
1388-                             )
1389-        leases = ("\x00"*self.LEASE_SIZE) * 4
1390-        f.write(header + leases)
1391-        # data goes here, empty after creation
1392-        f.write(struct.pack(">L", num_extra_leases))
1393-        # extra leases go here, none at creation
1394-        f.close()
1395+        f = self._home.open('wb')
1396+        try:
1397+            header = struct.pack(">32s20s32sQQ",
1398+                                 self.MAGIC, serverid, write_enabler,
1399+                                 data_length, extra_lease_offset,
1400+                                 )
1401+            leases = ("\x00"*self.LEASE_SIZE) * 4
1402+            f.write(header + leases)
1403+            # data goes here, empty after creation
1404+            f.write(struct.pack(">L", num_extra_leases))
1405+            # extra leases go here, none at creation
1406+        finally:
1407+            f.close()
1408+
1409+    def __repr__(self):
1410+        return ("<MutableDiskShare %s:%r at %s>"
1411+                % (si_b2a(self._storageindex), self._shnum, quote_filepath(self._home)))
1412+
1413+    def get_used_space(self):
1414+        return fileutil.get_used_space(self._home)
1415+
1416+    def get_storage_index(self):
1417+        return self._storageindex
1418+
1419+    def get_shnum(self):
1420+        return self._shnum
1421 
1422     def unlink(self):
1423hunk ./src/allmydata/storage/backends/disk/mutable.py 123
1424-        os.unlink(self.home)
1425+        self._home.remove()
1426 
1427     def _read_data_length(self, f):
1428         f.seek(self.DATA_LENGTH_OFFSET)
1429hunk ./src/allmydata/storage/backends/disk/mutable.py 291
1430 
1431     def get_leases(self):
1432         """Yields a LeaseInfo instance for all leases."""
1433-        f = open(self.home, 'rb')
1434-        for i, lease in self._enumerate_leases(f):
1435-            yield lease
1436-        f.close()
1437+        f = self._home.open('rb')
1438+        try:
1439+            for i, lease in self._enumerate_leases(f):
1440+                yield lease
1441+        finally:
1442+            f.close()
1443 
1444     def _enumerate_leases(self, f):
1445         for i in range(self._get_num_lease_slots(f)):
1446hunk ./src/allmydata/storage/backends/disk/mutable.py 303
1447             try:
1448                 data = self._read_lease_record(f, i)
1449                 if data is not None:
1450-                    yield i,data
1451+                    yield i, data
1452             except IndexError:
1453                 return
1454 
1455hunk ./src/allmydata/storage/backends/disk/mutable.py 307
1456+    # These lease operations are intended for use by disk_backend.py.
1457+    # Other non-test clients should not depend on the fact that the disk
1458+    # backend stores leases in share files.
1459+
1460     def add_lease(self, lease_info):
1461         precondition(lease_info.owner_num != 0) # 0 means "no lease here"
1462hunk ./src/allmydata/storage/backends/disk/mutable.py 313
1463-        f = open(self.home, 'rb+')
1464-        num_lease_slots = self._get_num_lease_slots(f)
1465-        empty_slot = self._get_first_empty_lease_slot(f)
1466-        if empty_slot is not None:
1467-            self._write_lease_record(f, empty_slot, lease_info)
1468-        else:
1469-            self._write_lease_record(f, num_lease_slots, lease_info)
1470-        f.close()
1471+        f = self._home.open('rb+')
1472+        try:
1473+            num_lease_slots = self._get_num_lease_slots(f)
1474+            empty_slot = self._get_first_empty_lease_slot(f)
1475+            if empty_slot is not None:
1476+                self._write_lease_record(f, empty_slot, lease_info)
1477+            else:
1478+                self._write_lease_record(f, num_lease_slots, lease_info)
1479+        finally:
1480+            f.close()
1481 
1482     def renew_lease(self, renew_secret, new_expire_time):
1483         accepting_nodeids = set()
1484hunk ./src/allmydata/storage/backends/disk/mutable.py 326
1485-        f = open(self.home, 'rb+')
1486-        for (leasenum,lease) in self._enumerate_leases(f):
1487-            if constant_time_compare(lease.renew_secret, renew_secret):
1488-                # yup. See if we need to update the owner time.
1489-                if new_expire_time > lease.expiration_time:
1490-                    # yes
1491-                    lease.expiration_time = new_expire_time
1492-                    self._write_lease_record(f, leasenum, lease)
1493-                f.close()
1494-                return
1495-            accepting_nodeids.add(lease.nodeid)
1496-        f.close()
1497+        f = self._home.open('rb+')
1498+        try:
1499+            for (leasenum, lease) in self._enumerate_leases(f):
1500+                if constant_time_compare(lease.renew_secret, renew_secret):
1501+                    # yup. See if we need to update the owner time.
1502+                    if new_expire_time > lease.expiration_time:
1503+                        # yes
1504+                        lease.expiration_time = new_expire_time
1505+                        self._write_lease_record(f, leasenum, lease)
1506+                    return
1507+                accepting_nodeids.add(lease.nodeid)
1508+        finally:
1509+            f.close()
1510         # Return the accepting_nodeids set, to give the client a chance to
1511hunk ./src/allmydata/storage/backends/disk/mutable.py 340
1512-        # update the leases on a share which has been migrated from its
1513+        # update the leases on a share that has been migrated from its
1514         # original server to a new one.
1515         msg = ("Unable to renew non-existent lease. I have leases accepted by"
1516                " nodeids: ")
1517hunk ./src/allmydata/storage/backends/disk/mutable.py 357
1518         except IndexError:
1519             self.add_lease(lease_info)
1520 
1521-    def cancel_lease(self, cancel_secret):
1522-        """Remove any leases with the given cancel_secret. If the last lease
1523-        is cancelled, the file will be removed. Return the number of bytes
1524-        that were freed (by truncating the list of leases, and possibly by
1525-        deleting the file. Raise IndexError if there was no lease with the
1526-        given cancel_secret."""
1527-
1528-        accepting_nodeids = set()
1529-        modified = 0
1530-        remaining = 0
1531-        blank_lease = LeaseInfo(owner_num=0,
1532-                                renew_secret="\x00"*32,
1533-                                cancel_secret="\x00"*32,
1534-                                expiration_time=0,
1535-                                nodeid="\x00"*20)
1536-        f = open(self.home, 'rb+')
1537-        for (leasenum,lease) in self._enumerate_leases(f):
1538-            accepting_nodeids.add(lease.nodeid)
1539-            if constant_time_compare(lease.cancel_secret, cancel_secret):
1540-                self._write_lease_record(f, leasenum, blank_lease)
1541-                modified += 1
1542-            else:
1543-                remaining += 1
1544-        if modified:
1545-            freed_space = self._pack_leases(f)
1546-            f.close()
1547-            if not remaining:
1548-                freed_space += os.stat(self.home)[stat.ST_SIZE]
1549-                self.unlink()
1550-            return freed_space
1551-
1552-        msg = ("Unable to cancel non-existent lease. I have leases "
1553-               "accepted by nodeids: ")
1554-        msg += ",".join([("'%s'" % idlib.nodeid_b2a(anid))
1555-                         for anid in accepting_nodeids])
1556-        msg += " ."
1557-        raise IndexError(msg)
1558-
1559-    def _pack_leases(self, f):
1560-        # TODO: reclaim space from cancelled leases
1561-        return 0
1562-
1563     def _read_write_enabler_and_nodeid(self, f):
1564         f.seek(0)
1565         data = f.read(self.HEADER_SIZE)
1566hunk ./src/allmydata/storage/backends/disk/mutable.py 369
1567 
1568     def readv(self, readv):
1569         datav = []
1570-        f = open(self.home, 'rb')
1571-        for (offset, length) in readv:
1572-            datav.append(self._read_share_data(f, offset, length))
1573-        f.close()
1574+        f = self._home.open('rb')
1575+        try:
1576+            for (offset, length) in readv:
1577+                datav.append(self._read_share_data(f, offset, length))
1578+        finally:
1579+            f.close()
1580         return datav
1581 
1582hunk ./src/allmydata/storage/backends/disk/mutable.py 377
1583-#    def remote_get_length(self):
1584-#        f = open(self.home, 'rb')
1585-#        data_length = self._read_data_length(f)
1586-#        f.close()
1587-#        return data_length
1588+    def get_size(self):
1589+        return self._home.getsize()
1590+
1591+    def get_data_length(self):
1592+        f = self._home.open('rb')
1593+        try:
1594+            data_length = self._read_data_length(f)
1595+        finally:
1596+            f.close()
1597+        return data_length
1598 
1599     def check_write_enabler(self, write_enabler, si_s):
1600hunk ./src/allmydata/storage/backends/disk/mutable.py 389
1601-        f = open(self.home, 'rb+')
1602-        (real_write_enabler, write_enabler_nodeid) = \
1603-                             self._read_write_enabler_and_nodeid(f)
1604-        f.close()
1605+        f = self._home.open('rb+')
1606+        try:
1607+            (real_write_enabler, write_enabler_nodeid) = self._read_write_enabler_and_nodeid(f)
1608+        finally:
1609+            f.close()
1610         # avoid a timing attack
1611         #if write_enabler != real_write_enabler:
1612         if not constant_time_compare(write_enabler, real_write_enabler):
1613hunk ./src/allmydata/storage/backends/disk/mutable.py 410
1614 
1615     def check_testv(self, testv):
1616         test_good = True
1617-        f = open(self.home, 'rb+')
1618-        for (offset, length, operator, specimen) in testv:
1619-            data = self._read_share_data(f, offset, length)
1620-            if not testv_compare(data, operator, specimen):
1621-                test_good = False
1622-                break
1623-        f.close()
1624+        f = self._home.open('rb+')
1625+        try:
1626+            for (offset, length, operator, specimen) in testv:
1627+                data = self._read_share_data(f, offset, length)
1628+                if not testv_compare(data, operator, specimen):
1629+                    test_good = False
1630+                    break
1631+        finally:
1632+            f.close()
1633         return test_good
1634 
1635     def writev(self, datav, new_length):
1636hunk ./src/allmydata/storage/backends/disk/mutable.py 422
1637-        f = open(self.home, 'rb+')
1638-        for (offset, data) in datav:
1639-            self._write_share_data(f, offset, data)
1640-        if new_length is not None:
1641-            cur_length = self._read_data_length(f)
1642-            if new_length < cur_length:
1643-                self._write_data_length(f, new_length)
1644-                # TODO: if we're going to shrink the share file when the
1645-                # share data has shrunk, then call
1646-                # self._change_container_size() here.
1647-        f.close()
1648-
1649-def testv_compare(a, op, b):
1650-    assert op in ("lt", "le", "eq", "ne", "ge", "gt")
1651-    if op == "lt":
1652-        return a < b
1653-    if op == "le":
1654-        return a <= b
1655-    if op == "eq":
1656-        return a == b
1657-    if op == "ne":
1658-        return a != b
1659-    if op == "ge":
1660-        return a >= b
1661-    if op == "gt":
1662-        return a > b
1663-    # never reached
1664+        f = self._home.open('rb+')
1665+        try:
1666+            for (offset, data) in datav:
1667+                self._write_share_data(f, offset, data)
1668+            if new_length is not None:
1669+                cur_length = self._read_data_length(f)
1670+                if new_length < cur_length:
1671+                    self._write_data_length(f, new_length)
1672+                    # TODO: if we're going to shrink the share file when the
1673+                    # share data has shrunk, then call
1674+                    # self._change_container_size() here.
1675+        finally:
1676+            f.close()
1677 
1678hunk ./src/allmydata/storage/backends/disk/mutable.py 436
1679-class EmptyShare:
1680+    def close(self):
1681+        pass
1682 
1683hunk ./src/allmydata/storage/backends/disk/mutable.py 439
1684-    def check_testv(self, testv):
1685-        test_good = True
1686-        for (offset, length, operator, specimen) in testv:
1687-            data = ""
1688-            if not testv_compare(data, operator, specimen):
1689-                test_good = False
1690-                break
1691-        return test_good
1692 
1693hunk ./src/allmydata/storage/backends/disk/mutable.py 440
1694-def create_mutable_sharefile(filename, my_nodeid, write_enabler, parent):
1695-    ms = MutableShareFile(filename, parent)
1696-    ms.create(my_nodeid, write_enabler)
1697+def create_mutable_disk_share(fp, serverid, write_enabler, parent):
1698+    ms = MutableDiskShare(fp, parent)
1699+    ms.create(serverid, write_enabler)
1700     del ms
1701hunk ./src/allmydata/storage/backends/disk/mutable.py 444
1702-    return MutableShareFile(filename, parent)
1703-
1704+    return MutableDiskShare(fp, parent)
1705addfile ./src/allmydata/storage/backends/null/__init__.py
1706addfile ./src/allmydata/storage/backends/null/null_backend.py
1707hunk ./src/allmydata/storage/backends/null/null_backend.py 2
1708 
1709+import os, struct
1710+
1711+from zope.interface import implements
1712+
1713+from allmydata.interfaces import IStorageBackend, IShareSet, IStoredShare, IStoredMutableShare
1714+from allmydata.util.assertutil import precondition
1715+from allmydata.util.hashutil import constant_time_compare
1716+from allmydata.storage.backends.base import Backend, ShareSet
1717+from allmydata.storage.bucket import BucketWriter
1718+from allmydata.storage.common import si_b2a
1719+from allmydata.storage.lease import LeaseInfo
1720+
1721+
1722+class NullBackend(Backend):
1723+    implements(IStorageBackend)
1724+
1725+    def __init__(self):
1726+        Backend.__init__(self)
1727+
1728+    def get_available_space(self, reserved_space):
1729+        return None
1730+
1731+    def get_sharesets_for_prefix(self, prefix):
1732+        pass
1733+
1734+    def get_shareset(self, storageindex):
1735+        return NullShareSet(storageindex)
1736+
1737+    def fill_in_space_stats(self, stats):
1738+        pass
1739+
1740+    def set_storage_server(self, ss):
1741+        self.ss = ss
1742+
1743+    def advise_corrupt_share(self, sharetype, storageindex, shnum, reason):
1744+        pass
1745+
1746+
1747+class NullShareSet(ShareSet):
1748+    implements(IShareSet)
1749+
1750+    def __init__(self, storageindex):
1751+        self.storageindex = storageindex
1752+
1753+    def get_overhead(self):
1754+        return 0
1755+
1756+    def get_incoming_shnums(self):
1757+        return frozenset()
1758+
1759+    def get_shares(self):
1760+        pass
1761+
1762+    def get_share(self, shnum):
1763+        return None
1764+
1765+    def get_storage_index(self):
1766+        return self.storageindex
1767+
1768+    def get_storage_index_string(self):
1769+        return si_b2a(self.storageindex)
1770+
1771+    def make_bucket_writer(self, storageserver, shnum, max_space_per_bucket, lease_info, canary):
1772+        immutableshare = ImmutableNullShare()
1773+        return BucketWriter(self.ss, immutableshare, max_space_per_bucket, lease_info, canary)
1774+
1775+    def _create_mutable_share(self, storageserver, shnum, write_enabler):
1776+        return MutableNullShare()
1777+
1778+    def _clean_up_after_unlink(self):
1779+        pass
1780+
1781+
1782+class ImmutableNullShare:
1783+    implements(IStoredShare)
1784+    sharetype = "immutable"
1785+
1786+    def __init__(self):
1787+        """ If max_size is not None then I won't allow more than
1788+        max_size to be written to me. If create=True then max_size
1789+        must not be None. """
1790+        pass
1791+
1792+    def get_shnum(self):
1793+        return self.shnum
1794+
1795+    def unlink(self):
1796+        os.unlink(self.fname)
1797+
1798+    def read_share_data(self, offset, length):
1799+        precondition(offset >= 0)
1800+        # Reads beyond the end of the data are truncated. Reads that start
1801+        # beyond the end of the data return an empty string.
1802+        seekpos = self._data_offset+offset
1803+        fsize = os.path.getsize(self.fname)
1804+        actuallength = max(0, min(length, fsize-seekpos)) # XXX #1528
1805+        if actuallength == 0:
1806+            return ""
1807+        f = open(self.fname, 'rb')
1808+        f.seek(seekpos)
1809+        return f.read(actuallength)
1810+
1811+    def write_share_data(self, offset, data):
1812+        pass
1813+
1814+    def _write_lease_record(self, f, lease_number, lease_info):
1815+        offset = self._lease_offset + lease_number * self.LEASE_SIZE
1816+        f.seek(offset)
1817+        assert f.tell() == offset
1818+        f.write(lease_info.to_immutable_data())
1819+
1820+    def _read_num_leases(self, f):
1821+        f.seek(0x08)
1822+        (num_leases,) = struct.unpack(">L", f.read(4))
1823+        return num_leases
1824+
1825+    def _write_num_leases(self, f, num_leases):
1826+        f.seek(0x08)
1827+        f.write(struct.pack(">L", num_leases))
1828+
1829+    def _truncate_leases(self, f, num_leases):
1830+        f.truncate(self._lease_offset + num_leases * self.LEASE_SIZE)
1831+
1832+    def get_leases(self):
1833+        """Yields a LeaseInfo instance for all leases."""
1834+        f = open(self.fname, 'rb')
1835+        (version, unused, num_leases) = struct.unpack(">LLL", f.read(0xc))
1836+        f.seek(self._lease_offset)
1837+        for i in range(num_leases):
1838+            data = f.read(self.LEASE_SIZE)
1839+            if data:
1840+                yield LeaseInfo().from_immutable_data(data)
1841+
1842+    def add_lease(self, lease):
1843+        pass
1844+
1845+    def renew_lease(self, renew_secret, new_expire_time):
1846+        for i,lease in enumerate(self.get_leases()):
1847+            if constant_time_compare(lease.renew_secret, renew_secret):
1848+                # yup. See if we need to update the owner time.
1849+                if new_expire_time > lease.expiration_time:
1850+                    # yes
1851+                    lease.expiration_time = new_expire_time
1852+                    f = open(self.fname, 'rb+')
1853+                    self._write_lease_record(f, i, lease)
1854+                    f.close()
1855+                return
1856+        raise IndexError("unable to renew non-existent lease")
1857+
1858+    def add_or_renew_lease(self, lease_info):
1859+        try:
1860+            self.renew_lease(lease_info.renew_secret,
1861+                             lease_info.expiration_time)
1862+        except IndexError:
1863+            self.add_lease(lease_info)
1864+
1865+
1866+class MutableNullShare:
1867+    implements(IStoredMutableShare)
1868+    sharetype = "mutable"
1869+
1870+    """ XXX: TODO """
1871addfile ./src/allmydata/storage/bucket.py
1872hunk ./src/allmydata/storage/bucket.py 1
1873+
1874+import time
1875+
1876+from foolscap.api import Referenceable
1877+
1878+from zope.interface import implements
1879+from allmydata.interfaces import RIBucketWriter, RIBucketReader
1880+from allmydata.util import base32, log
1881+from allmydata.util.assertutil import precondition
1882+
1883+
1884+class BucketWriter(Referenceable):
1885+    implements(RIBucketWriter)
1886+
1887+    def __init__(self, ss, immutableshare, max_size, lease_info, canary):
1888+        self.ss = ss
1889+        self._max_size = max_size # don't allow the client to write more than this
1890+        self._canary = canary
1891+        self._disconnect_marker = canary.notifyOnDisconnect(self._disconnected)
1892+        self.closed = False
1893+        self.throw_out_all_data = False
1894+        self._share = immutableshare
1895+        # also, add our lease to the file now, so that other ones can be
1896+        # added by simultaneous uploaders
1897+        self._share.add_lease(lease_info)
1898+
1899+    def allocated_size(self):
1900+        return self._max_size
1901+
1902+    def remote_write(self, offset, data):
1903+        start = time.time()
1904+        precondition(not self.closed)
1905+        if self.throw_out_all_data:
1906+            return
1907+        self._share.write_share_data(offset, data)
1908+        self.ss.add_latency("write", time.time() - start)
1909+        self.ss.count("write")
1910+
1911+    def remote_close(self):
1912+        precondition(not self.closed)
1913+        start = time.time()
1914+
1915+        self._share.close()
1916+        filelen = self._share.stat()
1917+        self._share = None
1918+
1919+        self.closed = True
1920+        self._canary.dontNotifyOnDisconnect(self._disconnect_marker)
1921+
1922+        self.ss.bucket_writer_closed(self, filelen)
1923+        self.ss.add_latency("close", time.time() - start)
1924+        self.ss.count("close")
1925+
1926+    def _disconnected(self):
1927+        if not self.closed:
1928+            self._abort()
1929+
1930+    def remote_abort(self):
1931+        log.msg("storage: aborting write to share %r" % self._share,
1932+                facility="tahoe.storage", level=log.UNUSUAL)
1933+        if not self.closed:
1934+            self._canary.dontNotifyOnDisconnect(self._disconnect_marker)
1935+        self._abort()
1936+        self.ss.count("abort")
1937+
1938+    def _abort(self):
1939+        if self.closed:
1940+            return
1941+        self._share.unlink()
1942+        self._share = None
1943+
1944+        # We are now considered closed for further writing. We must tell
1945+        # the storage server about this so that it stops expecting us to
1946+        # use the space it allocated for us earlier.
1947+        self.closed = True
1948+        self.ss.bucket_writer_closed(self, 0)
1949+
1950+
1951+class BucketReader(Referenceable):
1952+    implements(RIBucketReader)
1953+
1954+    def __init__(self, ss, share):
1955+        self.ss = ss
1956+        self._share = share
1957+        self.storageindex = share.storageindex
1958+        self.shnum = share.shnum
1959+
1960+    def __repr__(self):
1961+        return "<%s %s %s>" % (self.__class__.__name__,
1962+                               base32.b2a_l(self.storageindex[:8], 60),
1963+                               self.shnum)
1964+
1965+    def remote_read(self, offset, length):
1966+        start = time.time()
1967+        data = self._share.read_share_data(offset, length)
1968+        self.ss.add_latency("read", time.time() - start)
1969+        self.ss.count("read")
1970+        return data
1971+
1972+    def remote_advise_corrupt_share(self, reason):
1973+        return self.ss.remote_advise_corrupt_share("immutable",
1974+                                                   self.storageindex,
1975+                                                   self.shnum,
1976+                                                   reason)
1977addfile ./src/allmydata/test/test_backends.py
1978hunk ./src/allmydata/test/test_backends.py 1
1979+import os, stat
1980+from twisted.trial import unittest
1981+from allmydata.util.log import msg
1982+from allmydata.test.common_util import ReallyEqualMixin
1983+import mock
1984+
1985+# This is the code that we're going to be testing.
1986+from allmydata.storage.server import StorageServer
1987+from allmydata.storage.backends.disk.disk_backend import DiskBackend, si_si2dir
1988+from allmydata.storage.backends.null.null_backend import NullBackend
1989+
1990+# The following share file content was generated with
1991+# storage.immutable.ShareFile from Tahoe-LAFS v1.8.2
1992+# with share data == 'a'. The total size of this input
1993+# is 85 bytes.
1994+shareversionnumber = '\x00\x00\x00\x01'
1995+sharedatalength = '\x00\x00\x00\x01'
1996+numberofleases = '\x00\x00\x00\x01'
1997+shareinputdata = 'a'
1998+ownernumber = '\x00\x00\x00\x00'
1999+renewsecret  = 'x'*32
2000+cancelsecret = 'y'*32
2001+expirationtime = '\x00(\xde\x80'
2002+nextlease = ''
2003+containerdata = shareversionnumber + sharedatalength + numberofleases
2004+client_data = shareinputdata + ownernumber + renewsecret + \
2005+    cancelsecret + expirationtime + nextlease
2006+share_data = containerdata + client_data
2007+testnodeid = 'testnodeidxxxxxxxxxx'
2008+
2009+
2010+class MockFileSystem(unittest.TestCase):
2011+    """ I simulate a filesystem that the code under test can use. I simulate
2012+    just the parts of the filesystem that the current implementation of Disk
2013+    backend needs. """
2014+    def setUp(self):
2015+        # Make patcher, patch, and effects for disk-using functions.
2016+        msg( "%s.setUp()" % (self,))
2017+        self.mockedfilepaths = {}
2018+        # keys are pathnames, values are MockFilePath objects. This is necessary because
2019+        # MockFilePath behavior sometimes depends on the filesystem. Where it does,
2020+        # self.mockedfilepaths has the relevant information.
2021+        self.storedir = MockFilePath('teststoredir', self.mockedfilepaths)
2022+        self.basedir = self.storedir.child('shares')
2023+        self.baseincdir = self.basedir.child('incoming')
2024+        self.sharedirfinalname = self.basedir.child('or').child('orsxg5dtorxxeylhmvpws3temv4a')
2025+        self.sharedirincomingname = self.baseincdir.child('or').child('orsxg5dtorxxeylhmvpws3temv4a')
2026+        self.shareincomingname = self.sharedirincomingname.child('0')
2027+        self.sharefinalname = self.sharedirfinalname.child('0')
2028+
2029+        # FIXME: these patches won't work; disk_backend no longer imports FilePath, BucketCountingCrawler,
2030+        # or LeaseCheckingCrawler.
2031+
2032+        self.FilePathFake = mock.patch('allmydata.storage.backends.disk.disk_backend.FilePath', new = MockFilePath)
2033+        self.FilePathFake.__enter__()
2034+
2035+        self.BCountingCrawler = mock.patch('allmydata.storage.backends.disk.disk_backend.BucketCountingCrawler')
2036+        FakeBCC = self.BCountingCrawler.__enter__()
2037+        FakeBCC.side_effect = self.call_FakeBCC
2038+
2039+        self.LeaseCheckingCrawler = mock.patch('allmydata.storage.backends.disk.disk_backend.LeaseCheckingCrawler')
2040+        FakeLCC = self.LeaseCheckingCrawler.__enter__()
2041+        FakeLCC.side_effect = self.call_FakeLCC
2042+
2043+        self.get_available_space = mock.patch('allmydata.util.fileutil.get_available_space')
2044+        GetSpace = self.get_available_space.__enter__()
2045+        GetSpace.side_effect = self.call_get_available_space
2046+
2047+        self.statforsize = mock.patch('allmydata.storage.backends.disk.core.filepath.stat')
2048+        getsize = self.statforsize.__enter__()
2049+        getsize.side_effect = self.call_statforsize
2050+
2051+    def call_FakeBCC(self, StateFile):
2052+        return MockBCC()
2053+
2054+    def call_FakeLCC(self, StateFile, HistoryFile, ExpirationPolicy):
2055+        return MockLCC()
2056+
2057+    def call_get_available_space(self, storedir, reservedspace):
2058+        # The input vector has an input size of 85.
2059+        return 85 - reservedspace
2060+
2061+    def call_statforsize(self, fakefpname):
2062+        return self.mockedfilepaths[fakefpname].fileobject.size()
2063+
2064+    def tearDown(self):
2065+        msg( "%s.tearDown()" % (self,))
2066+        self.FilePathFake.__exit__()
2067+        self.mockedfilepaths = {}
2068+
2069+
2070+class MockFilePath:
2071+    def __init__(self, pathstring, ffpathsenvironment, existence=False):
2072+        #  I can't just make the values MockFileObjects because they may be directories.
2073+        self.mockedfilepaths = ffpathsenvironment
2074+        self.path = pathstring
2075+        self.existence = existence
2076+        if not self.mockedfilepaths.has_key(self.path):
2077+            #  The first MockFilePath object is special
2078+            self.mockedfilepaths[self.path] = self
2079+            self.fileobject = None
2080+        else:
2081+            self.fileobject = self.mockedfilepaths[self.path].fileobject
2082+        self.spawn = {}
2083+        self.antecedent = os.path.dirname(self.path)
2084+
2085+    def setContent(self, contentstring):
2086+        # This method rewrites the data in the file that corresponds to its path
2087+        # name whether it preexisted or not.
2088+        self.fileobject = MockFileObject(contentstring)
2089+        self.existence = True
2090+        self.mockedfilepaths[self.path].fileobject = self.fileobject
2091+        self.mockedfilepaths[self.path].existence = self.existence
2092+        self.setparents()
2093+
2094+    def create(self):
2095+        # This method chokes if there's a pre-existing file!
2096+        if self.mockedfilepaths[self.path].fileobject:
2097+            raise OSError
2098+        else:
2099+            self.existence = True
2100+            self.mockedfilepaths[self.path].fileobject = self.fileobject
2101+            self.mockedfilepaths[self.path].existence = self.existence
2102+            self.setparents()
2103+
2104+    def open(self, mode='r'):
2105+        # XXX Makes no use of mode.
2106+        if not self.mockedfilepaths[self.path].fileobject:
2107+            # If there's no fileobject there already then make one and put it there.
2108+            self.fileobject = MockFileObject()
2109+            self.existence = True
2110+            self.mockedfilepaths[self.path].fileobject = self.fileobject
2111+            self.mockedfilepaths[self.path].existence = self.existence
2112+        else:
2113+            # Otherwise get a ref to it.
2114+            self.fileobject = self.mockedfilepaths[self.path].fileobject
2115+            self.existence = self.mockedfilepaths[self.path].existence
2116+        return self.fileobject.open(mode)
2117+
2118+    def child(self, childstring):
2119+        arg2child = os.path.join(self.path, childstring)
2120+        child = MockFilePath(arg2child, self.mockedfilepaths)
2121+        return child
2122+
2123+    def children(self):
2124+        childrenfromffs = [ffp for ffp in self.mockedfilepaths.values() if ffp.path.startswith(self.path)]
2125+        childrenfromffs = [ffp for ffp in childrenfromffs if not ffp.path.endswith(self.path)]
2126+        childrenfromffs = [ffp for ffp in childrenfromffs if ffp.exists()]
2127+        self.spawn = frozenset(childrenfromffs)
2128+        return self.spawn
2129+
2130+    def parent(self):
2131+        if self.mockedfilepaths.has_key(self.antecedent):
2132+            parent = self.mockedfilepaths[self.antecedent]
2133+        else:
2134+            parent = MockFilePath(self.antecedent, self.mockedfilepaths)
2135+        return parent
2136+
2137+    def parents(self):
2138+        antecedents = []
2139+        def f(fps, antecedents):
2140+            newfps = os.path.split(fps)[0]
2141+            if newfps:
2142+                antecedents.append(newfps)
2143+                f(newfps, antecedents)
2144+        f(self.path, antecedents)
2145+        return antecedents
2146+
2147+    def setparents(self):
2148+        for fps in self.parents():
2149+            if not self.mockedfilepaths.has_key(fps):
2150+                self.mockedfilepaths[fps] = MockFilePath(fps, self.mockedfilepaths, exists=True)
2151+
2152+    def basename(self):
2153+        return os.path.split(self.path)[1]
2154+
2155+    def moveTo(self, newffp):
2156+        #  XXX Makes no distinction between file and directory arguments, this is deviation from filepath.moveTo
2157+        if self.mockedfilepaths[newffp.path].exists():
2158+            raise OSError
2159+        else:
2160+            self.mockedfilepaths[newffp.path] = self
2161+            self.path = newffp.path
2162+
2163+    def getsize(self):
2164+        return self.fileobject.getsize()
2165+
2166+    def exists(self):
2167+        return self.existence
2168+
2169+    def isdir(self):
2170+        return True
2171+
2172+    def makedirs(self):
2173+        # XXX These methods assume that fp_<FOO> functions in fileutil will be tested elsewhere!
2174+        pass
2175+
2176+    def remove(self):
2177+        pass
2178+
2179+
2180+class MockFileObject:
2181+    def __init__(self, contentstring=''):
2182+        self.buffer = contentstring
2183+        self.pos = 0
2184+    def open(self, mode='r'):
2185+        return self
2186+    def write(self, instring):
2187+        begin = self.pos
2188+        padlen = begin - len(self.buffer)
2189+        if padlen > 0:
2190+            self.buffer += '\x00' * padlen
2191+        end = self.pos + len(instring)
2192+        self.buffer = self.buffer[:begin]+instring+self.buffer[end:]
2193+        self.pos = end
2194+    def close(self):
2195+        self.pos = 0
2196+    def seek(self, pos):
2197+        self.pos = pos
2198+    def read(self, numberbytes):
2199+        return self.buffer[self.pos:self.pos+numberbytes]
2200+    def tell(self):
2201+        return self.pos
2202+    def size(self):
2203+        # XXX This method A: Is not to be found in a real file B: Is part of a wild-mung-up of filepath.stat!
2204+        # XXX Finally we shall hopefully use a getsize method soon, must consult first though.
2205+        # Hmmm...  perhaps we need to sometimes stat the address when there's not a mockfileobject present?
2206+        return {stat.ST_SIZE:len(self.buffer)}
2207+    def getsize(self):
2208+        return len(self.buffer)
2209+
2210+class MockBCC:
2211+    def setServiceParent(self, Parent):
2212+        pass
2213+
2214+
2215+class MockLCC:
2216+    def setServiceParent(self, Parent):
2217+        pass
2218+
2219+
2220+class TestServerWithNullBackend(unittest.TestCase, ReallyEqualMixin):
2221+    """ NullBackend is just for testing and executable documentation, so
2222+    this test is actually a test of StorageServer in which we're using
2223+    NullBackend as helper code for the test, rather than a test of
2224+    NullBackend. """
2225+    def setUp(self):
2226+        self.ss = StorageServer(testnodeid, NullBackend())
2227+
2228+    @mock.patch('os.mkdir')
2229+    @mock.patch('__builtin__.open')
2230+    @mock.patch('os.listdir')
2231+    @mock.patch('os.path.isdir')
2232+    def test_write_share(self, mockisdir, mocklistdir, mockopen, mockmkdir):
2233+        """
2234+        Write a new share. This tests that StorageServer's remote_allocate_buckets
2235+        generates the correct return types when given test-vector arguments. That
2236+        bs is of the correct type is verified by attempting to invoke remote_write
2237+        on bs[0].
2238+        """
2239+        alreadygot, bs = self.ss.remote_allocate_buckets('teststorage_index', 'x'*32, 'y'*32, set((0,)), 1, mock.Mock())
2240+        bs[0].remote_write(0, 'a')
2241+        self.failIf(mockisdir.called)
2242+        self.failIf(mocklistdir.called)
2243+        self.failIf(mockopen.called)
2244+        self.failIf(mockmkdir.called)
2245+
2246+
2247+class TestServerConstruction(MockFileSystem, ReallyEqualMixin):
2248+    def test_create_server_disk_backend(self):
2249+        """ This tests whether a server instance can be constructed with a
2250+        filesystem backend. To pass the test, it mustn't use the filesystem
2251+        outside of its configured storedir. """
2252+        StorageServer(testnodeid, DiskBackend(self.storedir))
2253+
2254+
2255+class TestServerAndDiskBackend(MockFileSystem, ReallyEqualMixin):
2256+    """ This tests both the StorageServer and the Disk backend together. """
2257+    def setUp(self):
2258+        MockFileSystem.setUp(self)
2259+        try:
2260+            self.backend = DiskBackend(self.storedir)
2261+            self.ss = StorageServer(testnodeid, self.backend)
2262+
2263+            self.backendwithreserve = DiskBackend(self.storedir, reserved_space = 1)
2264+            self.sswithreserve = StorageServer(testnodeid, self.backendwithreserve)
2265+        except:
2266+            MockFileSystem.tearDown(self)
2267+            raise
2268+
2269+    @mock.patch('time.time')
2270+    @mock.patch('allmydata.util.fileutil.get_available_space')
2271+    def test_out_of_space(self, mockget_available_space, mocktime):
2272+        mocktime.return_value = 0
2273+
2274+        def call_get_available_space(dir, reserve):
2275+            return 0
2276+
2277+        mockget_available_space.side_effect = call_get_available_space
2278+        alreadygotc, bsc = self.sswithreserve.remote_allocate_buckets('teststorage_index', 'x'*32, 'y'*32, set((0,)), 1, mock.Mock())
2279+        self.failUnlessReallyEqual(bsc, {})
2280+
2281+    @mock.patch('time.time')
2282+    def test_write_and_read_share(self, mocktime):
2283+        """
2284+        Write a new share, read it, and test the server's (and disk backend's)
2285+        handling of simultaneous and successive attempts to write the same
2286+        share.
2287+        """
2288+        mocktime.return_value = 0
2289+        # Inspect incoming and fail unless it's empty.
2290+        incomingset = self.ss.backend.get_incoming_shnums('teststorage_index')
2291+
2292+        self.failUnlessReallyEqual(incomingset, frozenset())
2293+
2294+        # Populate incoming with the sharenum: 0.
2295+        alreadygot, bs = self.ss.remote_allocate_buckets('teststorage_index', 'x'*32, 'y'*32, frozenset((0,)), 1, mock.Mock())
2296+
2297+        # This is a transparent-box test: Inspect incoming and fail unless the sharenum: 0 is listed there.
2298+        self.failUnlessReallyEqual(self.ss.backend.get_incoming_shnums('teststorage_index'), frozenset((0,)))
2299+
2300+
2301+
2302+        # Attempt to create a second share writer with the same sharenum.
2303+        alreadygota, bsa = self.ss.remote_allocate_buckets('teststorage_index', 'x'*32, 'y'*32, frozenset((0,)), 1, mock.Mock())
2304+
2305+        # Show that no sharewriter results from a remote_allocate_buckets
2306+        # with the same si and sharenum, until BucketWriter.remote_close()
2307+        # has been called.
2308+        self.failIf(bsa)
2309+
2310+        # Test allocated size.
2311+        spaceint = self.ss.allocated_size()
2312+        self.failUnlessReallyEqual(spaceint, 1)
2313+
2314+        # Write 'a' to shnum 0. Only tested together with close and read.
2315+        bs[0].remote_write(0, 'a')
2316+
2317+        # Preclose: Inspect final, failUnless nothing there.
2318+        self.failUnlessReallyEqual(len(list(self.backend.get_shares('teststorage_index'))), 0)
2319+        bs[0].remote_close()
2320+
2321+        # Postclose: (Omnibus) failUnless written data is in final.
2322+        sharesinfinal = list(self.backend.get_shares('teststorage_index'))
2323+        self.failUnlessReallyEqual(len(sharesinfinal), 1)
2324+        contents = sharesinfinal[0].read_share_data(0, 73)
2325+        self.failUnlessReallyEqual(contents, client_data)
2326+
2327+        # Exercise the case that the share we're asking to allocate is
2328+        # already (completely) uploaded.
2329+        self.ss.remote_allocate_buckets('teststorage_index', 'x'*32, 'y'*32, set((0,)), 1, mock.Mock())
2330+
2331+
2332+    def test_read_old_share(self):
2333+        """ This tests whether the code correctly finds and reads
2334+        shares written out by old (Tahoe-LAFS <= v1.8.2)
2335+        servers. There is a similar test in test_download, but that one
2336+        is from the perspective of the client and exercises a deeper
2337+        stack of code. This one is for exercising just the
2338+        StorageServer object. """
2339+        # Contruct a file with the appropriate contents in the mockfilesystem.
2340+        datalen = len(share_data)
2341+        finalhome = si_si2dir(self.basedir, 'teststorage_index').child(str(0))
2342+        finalhome.setContent(share_data)
2343+
2344+        # Now begin the test.
2345+        bs = self.ss.remote_get_buckets('teststorage_index')
2346+
2347+        self.failUnlessEqual(len(bs), 1)
2348+        b = bs['0']
2349+        # These should match by definition, the next two cases cover cases without (completely) unambiguous behaviors.
2350+        self.failUnlessReallyEqual(b.remote_read(0, datalen), client_data)
2351+        # If you try to read past the end you get the as much data as is there.
2352+        self.failUnlessReallyEqual(b.remote_read(0, datalen+20), client_data)
2353+        # If you start reading past the end of the file you get the empty string.
2354+        self.failUnlessReallyEqual(b.remote_read(datalen+1, 3), '')
2355}
2356[Pluggable backends -- all other changes. refs #999
2357david-sarah@jacaranda.org**20110919233256
2358 Ignore-this: 1a77b6b5d178b32a9b914b699ba7e957
2359] {
2360hunk ./src/allmydata/client.py 245
2361             sharetypes.append("immutable")
2362         if self.get_config("storage", "expire.mutable", True, boolean=True):
2363             sharetypes.append("mutable")
2364-        expiration_sharetypes = tuple(sharetypes)
2365 
2366hunk ./src/allmydata/client.py 246
2367+        expiration_policy = {
2368+            'enabled': expire,
2369+            'mode': mode,
2370+            'override_lease_duration': o_l_d,
2371+            'cutoff_date': cutoff_date,
2372+            'sharetypes': tuple(sharetypes),
2373+        }
2374         ss = StorageServer(storedir, self.nodeid,
2375                            reserved_space=reserved,
2376                            discard_storage=discard,
2377hunk ./src/allmydata/client.py 258
2378                            readonly_storage=readonly,
2379                            stats_provider=self.stats_provider,
2380-                           expiration_enabled=expire,
2381-                           expiration_mode=mode,
2382-                           expiration_override_lease_duration=o_l_d,
2383-                           expiration_cutoff_date=cutoff_date,
2384-                           expiration_sharetypes=expiration_sharetypes)
2385+                           expiration_policy=expiration_policy)
2386         self.add_service(ss)
2387 
2388         d = self.when_tub_ready()
2389hunk ./src/allmydata/immutable/offloaded.py 306
2390         if os.path.exists(self._encoding_file):
2391             self.log("ciphertext already present, bypassing fetch",
2392                      level=log.UNUSUAL)
2393+            # XXX the following comment is probably stale, since
2394+            # LocalCiphertextReader.get_plaintext_hashtree_leaves does not exist.
2395+            #
2396             # we'll still need the plaintext hashes (when
2397             # LocalCiphertextReader.get_plaintext_hashtree_leaves() is
2398             # called), and currently the easiest way to get them is to ask
2399hunk ./src/allmydata/immutable/upload.py 765
2400             self._status.set_progress(1, progress)
2401         return cryptdata
2402 
2403-
2404     def get_plaintext_hashtree_leaves(self, first, last, num_segments):
2405hunk ./src/allmydata/immutable/upload.py 766
2406+        """OBSOLETE; Get the leaf nodes of a merkle hash tree over the
2407+        plaintext segments, i.e. get the tagged hashes of the given segments.
2408+        The segment size is expected to be generated by the
2409+        IEncryptedUploadable before any plaintext is read or ciphertext
2410+        produced, so that the segment hashes can be generated with only a
2411+        single pass.
2412+
2413+        This returns a Deferred that fires with a sequence of hashes, using:
2414+
2415+         tuple(segment_hashes[first:last])
2416+
2417+        'num_segments' is used to assert that the number of segments that the
2418+        IEncryptedUploadable handled matches the number of segments that the
2419+        encoder was expecting.
2420+
2421+        This method must not be called until the final byte has been read
2422+        from read_encrypted(). Once this method is called, read_encrypted()
2423+        can never be called again.
2424+        """
2425         # this is currently unused, but will live again when we fix #453
2426         if len(self._plaintext_segment_hashes) < num_segments:
2427             # close out the last one
2428hunk ./src/allmydata/immutable/upload.py 803
2429         return defer.succeed(tuple(self._plaintext_segment_hashes[first:last]))
2430 
2431     def get_plaintext_hash(self):
2432+        """OBSOLETE; Get the hash of the whole plaintext.
2433+
2434+        This returns a Deferred that fires with a tagged SHA-256 hash of the
2435+        whole plaintext, obtained from hashutil.plaintext_hash(data).
2436+        """
2437+        # this is currently unused, but will live again when we fix #453
2438         h = self._plaintext_hasher.digest()
2439         return defer.succeed(h)
2440 
2441hunk ./src/allmydata/interfaces.py 29
2442 Number = IntegerConstraint(8) # 2**(8*8) == 16EiB ~= 18e18 ~= 18 exabytes
2443 Offset = Number
2444 ReadSize = int # the 'int' constraint is 2**31 == 2Gib -- large files are processed in not-so-large increments
2445-WriteEnablerSecret = Hash # used to protect mutable bucket modifications
2446-LeaseRenewSecret = Hash # used to protect bucket lease renewal requests
2447-LeaseCancelSecret = Hash # used to protect bucket lease cancellation requests
2448+WriteEnablerSecret = Hash # used to protect mutable share modifications
2449+LeaseRenewSecret = Hash # used to protect lease renewal requests
2450+LeaseCancelSecret = Hash # used to protect lease cancellation requests
2451 
2452 class RIStubClient(RemoteInterface):
2453     """Each client publishes a service announcement for a dummy object called
2454hunk ./src/allmydata/interfaces.py 106
2455                          sharenums=SetOf(int, maxLength=MAX_BUCKETS),
2456                          allocated_size=Offset, canary=Referenceable):
2457         """
2458-        @param storage_index: the index of the bucket to be created or
2459+        @param storage_index: the index of the shareset to be created or
2460                               increfed.
2461         @param sharenums: these are the share numbers (probably between 0 and
2462                           99) that the sender is proposing to store on this
2463hunk ./src/allmydata/interfaces.py 111
2464                           server.
2465-        @param renew_secret: This is the secret used to protect bucket refresh
2466+        @param renew_secret: This is the secret used to protect lease renewal.
2467                              This secret is generated by the client and
2468                              stored for later comparison by the server. Each
2469                              server is given a different secret.
2470hunk ./src/allmydata/interfaces.py 115
2471-        @param cancel_secret: Like renew_secret, but protects bucket decref.
2472-        @param canary: If the canary is lost before close(), the bucket is
2473+        @param cancel_secret: ignored
2474+        @param canary: If the canary is lost before close(), the allocation is
2475                        deleted.
2476         @return: tuple of (alreadygot, allocated), where alreadygot is what we
2477                  already have and allocated is what we hereby agree to accept.
2478hunk ./src/allmydata/interfaces.py 129
2479                   renew_secret=LeaseRenewSecret,
2480                   cancel_secret=LeaseCancelSecret):
2481         """
2482-        Add a new lease on the given bucket. If the renew_secret matches an
2483+        Add a new lease on the given shareset. If the renew_secret matches an
2484         existing lease, that lease will be renewed instead. If there is no
2485hunk ./src/allmydata/interfaces.py 131
2486-        bucket for the given storage_index, return silently. (note that in
2487+        shareset for the given storage_index, return silently. (Note that in
2488         tahoe-1.3.0 and earlier, IndexError was raised if there was no
2489hunk ./src/allmydata/interfaces.py 133
2490-        bucket)
2491+        shareset.)
2492         """
2493         return Any() # returns None now, but future versions might change
2494 
2495hunk ./src/allmydata/interfaces.py 139
2496     def renew_lease(storage_index=StorageIndex, renew_secret=LeaseRenewSecret):
2497         """
2498-        Renew the lease on a given bucket, resetting the timer to 31 days.
2499-        Some networks will use this, some will not. If there is no bucket for
2500+        Renew the lease on a given shareset, resetting the timer to 31 days.
2501+        Some networks will use this, some will not. If there is no shareset for
2502         the given storage_index, IndexError will be raised.
2503 
2504         For mutable shares, if the given renew_secret does not match an
2505hunk ./src/allmydata/interfaces.py 146
2506         existing lease, IndexError will be raised with a note listing the
2507         server-nodeids on the existing leases, so leases on migrated shares
2508-        can be renewed or cancelled. For immutable shares, IndexError
2509-        (without the note) will be raised.
2510+        can be renewed. For immutable shares, IndexError (without the note)
2511+        will be raised.
2512         """
2513         return Any()
2514 
2515hunk ./src/allmydata/interfaces.py 154
2516     def get_buckets(storage_index=StorageIndex):
2517         return DictOf(int, RIBucketReader, maxKeys=MAX_BUCKETS)
2518 
2519-
2520-
2521     def slot_readv(storage_index=StorageIndex,
2522                    shares=ListOf(int), readv=ReadVector):
2523         """Read a vector from the numbered shares associated with the given
2524hunk ./src/allmydata/interfaces.py 163
2525 
2526     def slot_testv_and_readv_and_writev(storage_index=StorageIndex,
2527                                         secrets=TupleOf(WriteEnablerSecret,
2528-                                                        LeaseRenewSecret,
2529-                                                        LeaseCancelSecret),
2530+                                                        LeaseRenewSecret),
2531                                         tw_vectors=TestAndWriteVectorsForShares,
2532                                         r_vector=ReadVector,
2533                                         ):
2534hunk ./src/allmydata/interfaces.py 167
2535-        """General-purpose test-and-set operation for mutable slots. Perform
2536-        a bunch of comparisons against the existing shares. If they all pass,
2537-        then apply a bunch of write vectors to those shares. Then use the
2538-        read vectors to extract data from all the shares and return the data.
2539+        """
2540+        General-purpose atomic test-read-and-set operation for mutable slots.
2541+        Perform a bunch of comparisons against the existing shares. If they
2542+        all pass: use the read vectors to extract data from all the shares,
2543+        then apply a bunch of write vectors to those shares. Return the read
2544+        data, which does not include any modifications made by the writes.
2545 
2546         This method is, um, large. The goal is to allow clients to update all
2547         the shares associated with a mutable file in a single round trip.
2548hunk ./src/allmydata/interfaces.py 177
2549 
2550-        @param storage_index: the index of the bucket to be created or
2551+        @param storage_index: the index of the shareset to be created or
2552                               increfed.
2553         @param write_enabler: a secret that is stored along with the slot.
2554                               Writes are accepted from any caller who can
2555hunk ./src/allmydata/interfaces.py 183
2556                               present the matching secret. A different secret
2557                               should be used for each slot*server pair.
2558-        @param renew_secret: This is the secret used to protect bucket refresh
2559+        @param renew_secret: This is the secret used to protect lease renewal.
2560                              This secret is generated by the client and
2561                              stored for later comparison by the server. Each
2562                              server is given a different secret.
2563hunk ./src/allmydata/interfaces.py 187
2564-        @param cancel_secret: Like renew_secret, but protects bucket decref.
2565+        @param cancel_secret: ignored
2566 
2567hunk ./src/allmydata/interfaces.py 189
2568-        The 'secrets' argument is a tuple of (write_enabler, renew_secret,
2569-        cancel_secret). The first is required to perform any write. The
2570-        latter two are used when allocating new shares. To simply acquire a
2571-        new lease on existing shares, use an empty testv and an empty writev.
2572+        The 'secrets' argument is a tuple with (write_enabler, renew_secret).
2573+        The write_enabler is required to perform any write. The renew_secret
2574+        is used when allocating new shares.
2575 
2576         Each share can have a separate test vector (i.e. a list of
2577         comparisons to perform). If all vectors for all shares pass, then all
2578hunk ./src/allmydata/interfaces.py 280
2579         store that on disk.
2580         """
2581 
2582-class IStorageBucketWriter(Interface):
2583+
2584+class IStorageBackend(Interface):
2585     """
2586hunk ./src/allmydata/interfaces.py 283
2587-    Objects of this kind live on the client side.
2588+    Objects of this kind live on the server side and are used by the
2589+    storage server object.
2590     """
2591hunk ./src/allmydata/interfaces.py 286
2592-    def put_block(segmentnum=int, data=ShareData):
2593-        """@param data: For most segments, this data will be 'blocksize'
2594-        bytes in length. The last segment might be shorter.
2595-        @return: a Deferred that fires (with None) when the operation completes
2596+    def get_available_space():
2597+        """
2598+        Returns available space for share storage in bytes, or
2599+        None if this information is not available or if the available
2600+        space is unlimited.
2601+
2602+        If the backend is configured for read-only mode then this will
2603+        return 0.
2604+        """
2605+
2606+    def get_sharesets_for_prefix(prefix):
2607+        """
2608+        Generates IShareSet objects for all storage indices matching the
2609+        given prefix for which this backend holds shares.
2610+        """
2611+
2612+    def get_shareset(storageindex):
2613+        """
2614+        Get an IShareSet object for the given storage index.
2615+        """
2616+
2617+    def advise_corrupt_share(storageindex, sharetype, shnum, reason):
2618+        """
2619+        Clients who discover hash failures in shares that they have
2620+        downloaded from me will use this method to inform me about the
2621+        failures. I will record their concern so that my operator can
2622+        manually inspect the shares in question.
2623+
2624+        'sharetype' is either 'mutable' or 'immutable'. 'shnum' is the integer
2625+        share number. 'reason' is a human-readable explanation of the problem,
2626+        probably including some expected hash values and the computed ones
2627+        that did not match. Corruption advisories for mutable shares should
2628+        include a hash of the public key (the same value that appears in the
2629+        mutable-file verify-cap), since the current share format does not
2630+        store that on disk.
2631+
2632+        @param storageindex=str
2633+        @param sharetype=str
2634+        @param shnum=int
2635+        @param reason=str
2636+        """
2637+
2638+
2639+class IShareSet(Interface):
2640+    def get_storage_index():
2641+        """
2642+        Returns the storage index for this shareset.
2643+        """
2644+
2645+    def get_storage_index_string():
2646+        """
2647+        Returns the base32-encoded storage index for this shareset.
2648+        """
2649+
2650+    def get_overhead():
2651+        """
2652+        Returns the storage overhead, in bytes, of this shareset (exclusive
2653+        of the space used by its shares).
2654+        """
2655+
2656+    def get_shares():
2657+        """
2658+        Generates the IStoredShare objects held in this shareset.
2659+        """
2660+
2661+    def has_incoming(shnum):
2662+        """
2663+        Returns True if this shareset has an incoming (partial) share with this number, otherwise False.
2664+        """
2665+
2666+    def make_bucket_writer(storageserver, shnum, max_space_per_bucket, lease_info, canary):
2667+        """
2668+        Create a bucket writer that can be used to write data to a given share.
2669+
2670+        @param storageserver=RIStorageServer
2671+        @param shnum=int: A share number in this shareset
2672+        @param max_space_per_bucket=int: The maximum space allocated for the
2673+                 share, in bytes
2674+        @param lease_info=LeaseInfo: The initial lease information
2675+        @param canary=Referenceable: If the canary is lost before close(), the
2676+                 bucket is deleted.
2677+        @return an IStorageBucketWriter for the given share
2678+        """
2679+
2680+    def make_bucket_reader(storageserver, share):
2681+        """
2682+        Create a bucket reader that can be used to read data from a given share.
2683+
2684+        @param storageserver=RIStorageServer
2685+        @param share=IStoredShare
2686+        @return an IStorageBucketReader for the given share
2687+        """
2688+
2689+    def readv(wanted_shnums, read_vector):
2690+        """
2691+        Read a vector from the numbered shares in this shareset. An empty
2692+        wanted_shnums list means to return data from all known shares.
2693+
2694+        @param wanted_shnums=ListOf(int)
2695+        @param read_vector=ReadVector
2696+        @return DictOf(int, ReadData): shnum -> results, with one key per share
2697+        """
2698+
2699+    def testv_and_readv_and_writev(storageserver, secrets, test_and_write_vectors, read_vector, expiration_time):
2700+        """
2701+        General-purpose atomic test-read-and-set operation for mutable slots.
2702+        Perform a bunch of comparisons against the existing shares in this
2703+        shareset. If they all pass: use the read vectors to extract data from
2704+        all the shares, then apply a bunch of write vectors to those shares.
2705+        Return the read data, which does not include any modifications made by
2706+        the writes.
2707+
2708+        See the similar method in RIStorageServer for more detail.
2709+
2710+        @param storageserver=RIStorageServer
2711+        @param secrets=TupleOf(WriteEnablerSecret, LeaseRenewSecret[, ...])
2712+        @param test_and_write_vectors=TestAndWriteVectorsForShares
2713+        @param read_vector=ReadVector
2714+        @param expiration_time=int
2715+        @return TupleOf(bool, DictOf(int, ReadData))
2716+        """
2717+
2718+    def add_or_renew_lease(lease_info):
2719+        """
2720+        Add a new lease on the shares in this shareset. If the renew_secret
2721+        matches an existing lease, that lease will be renewed instead. If
2722+        there are no shares in this shareset, return silently.
2723+
2724+        @param lease_info=LeaseInfo
2725+        """
2726+
2727+    def renew_lease(renew_secret, new_expiration_time):
2728+        """
2729+        Renew a lease on the shares in this shareset, resetting the timer
2730+        to 31 days. Some grids will use this, some will not. If there are no
2731+        shares in this shareset, IndexError will be raised.
2732+
2733+        For mutable shares, if the given renew_secret does not match an
2734+        existing lease, IndexError will be raised with a note listing the
2735+        server-nodeids on the existing leases, so leases on migrated shares
2736+        can be renewed. For immutable shares, IndexError (without the note)
2737+        will be raised.
2738+
2739+        @param renew_secret=LeaseRenewSecret
2740+        """
2741+
2742+
2743+class IStoredShare(Interface):
2744+    """
2745+    This object contains as much as all of the share data.  It is intended
2746+    for lazy evaluation, such that in many use cases substantially less than
2747+    all of the share data will be accessed.
2748+    """
2749+    def close():
2750+        """
2751+        Complete writing to this share.
2752+        """
2753+
2754+    def get_storage_index():
2755+        """
2756+        Returns the storage index.
2757+        """
2758+
2759+    def get_shnum():
2760+        """
2761+        Returns the share number.
2762+        """
2763+
2764+    def get_data_length():
2765+        """
2766+        Returns the data length in bytes.
2767+        """
2768+
2769+    def get_size():
2770+        """
2771+        Returns the size of the share in bytes.
2772+        """
2773+
2774+    def get_used_space():
2775+        """
2776+        Returns the amount of backend storage including overhead, in bytes, used
2777+        by this share.
2778+        """
2779+
2780+    def unlink():
2781+        """
2782+        Signal that this share can be removed from the backend storage. This does
2783+        not guarantee that the share data will be immediately inaccessible, or
2784+        that it will be securely erased.
2785+        """
2786+
2787+    def readv(read_vector):
2788+        """
2789+        XXX
2790+        """
2791+
2792+
2793+class IStoredMutableShare(IStoredShare):
2794+    def check_write_enabler(write_enabler, si_s):
2795+        """
2796+        XXX
2797         """
2798 
2799hunk ./src/allmydata/interfaces.py 489
2800-    def put_plaintext_hashes(hashes=ListOf(Hash)):
2801+    def check_testv(test_vector):
2802+        """
2803+        XXX
2804+        """
2805+
2806+    def writev(datav, new_length):
2807+        """
2808+        XXX
2809+        """
2810+
2811+
2812+class IStorageBucketWriter(Interface):
2813+    """
2814+    Objects of this kind live on the client side.
2815+    """
2816+    def put_block(segmentnum, data):
2817         """
2818hunk ./src/allmydata/interfaces.py 506
2819+        @param segmentnum=int
2820+        @param data=ShareData: For most segments, this data will be 'blocksize'
2821+        bytes in length. The last segment might be shorter.
2822         @return: a Deferred that fires (with None) when the operation completes
2823         """
2824 
2825hunk ./src/allmydata/interfaces.py 512
2826-    def put_crypttext_hashes(hashes=ListOf(Hash)):
2827+    def put_crypttext_hashes(hashes):
2828         """
2829hunk ./src/allmydata/interfaces.py 514
2830+        @param hashes=ListOf(Hash)
2831         @return: a Deferred that fires (with None) when the operation completes
2832         """
2833 
2834hunk ./src/allmydata/interfaces.py 518
2835-    def put_block_hashes(blockhashes=ListOf(Hash)):
2836+    def put_block_hashes(blockhashes):
2837         """
2838hunk ./src/allmydata/interfaces.py 520
2839+        @param blockhashes=ListOf(Hash)
2840         @return: a Deferred that fires (with None) when the operation completes
2841         """
2842 
2843hunk ./src/allmydata/interfaces.py 524
2844-    def put_share_hashes(sharehashes=ListOf(TupleOf(int, Hash))):
2845+    def put_share_hashes(sharehashes):
2846         """
2847hunk ./src/allmydata/interfaces.py 526
2848+        @param sharehashes=ListOf(TupleOf(int, Hash))
2849         @return: a Deferred that fires (with None) when the operation completes
2850         """
2851 
2852hunk ./src/allmydata/interfaces.py 530
2853-    def put_uri_extension(data=URIExtensionData):
2854+    def put_uri_extension(data):
2855         """This block of data contains integrity-checking information (hashes
2856         of plaintext, crypttext, and shares), as well as encoding parameters
2857         that are necessary to recover the data. This is a serialized dict
2858hunk ./src/allmydata/interfaces.py 535
2859         mapping strings to other strings. The hash of this data is kept in
2860-        the URI and verified before any of the data is used. All buckets for
2861-        a given file contain identical copies of this data.
2862+        the URI and verified before any of the data is used. All share
2863+        containers for a given file contain identical copies of this data.
2864 
2865         The serialization format is specified with the following pseudocode:
2866         for k in sorted(dict.keys()):
2867hunk ./src/allmydata/interfaces.py 543
2868             assert re.match(r'^[a-zA-Z_\-]+$', k)
2869             write(k + ':' + netstring(dict[k]))
2870 
2871+        @param data=URIExtensionData
2872         @return: a Deferred that fires (with None) when the operation completes
2873         """
2874 
2875hunk ./src/allmydata/interfaces.py 558
2876 
2877 class IStorageBucketReader(Interface):
2878 
2879-    def get_block_data(blocknum=int, blocksize=int, size=int):
2880+    def get_block_data(blocknum, blocksize, size):
2881         """Most blocks will be the same size. The last block might be shorter
2882         than the others.
2883 
2884hunk ./src/allmydata/interfaces.py 562
2885+        @param blocknum=int
2886+        @param blocksize=int
2887+        @param size=int
2888         @return: ShareData
2889         """
2890 
2891hunk ./src/allmydata/interfaces.py 573
2892         @return: ListOf(Hash)
2893         """
2894 
2895-    def get_block_hashes(at_least_these=SetOf(int)):
2896+    def get_block_hashes(at_least_these=()):
2897         """
2898hunk ./src/allmydata/interfaces.py 575
2899+        @param at_least_these=SetOf(int)
2900         @return: ListOf(Hash)
2901         """
2902 
2903hunk ./src/allmydata/interfaces.py 579
2904-    def get_share_hashes(at_least_these=SetOf(int)):
2905+    def get_share_hashes():
2906         """
2907         @return: ListOf(TupleOf(int, Hash))
2908         """
2909hunk ./src/allmydata/interfaces.py 611
2910         @return: unicode nickname, or None
2911         """
2912 
2913-    # methods moved from IntroducerClient, need review
2914-    def get_all_connections():
2915-        """Return a frozenset of (nodeid, service_name, rref) tuples, one for
2916-        each active connection we've established to a remote service. This is
2917-        mostly useful for unit tests that need to wait until a certain number
2918-        of connections have been made."""
2919-
2920-    def get_all_connectors():
2921-        """Return a dict that maps from (nodeid, service_name) to a
2922-        RemoteServiceConnector instance for all services that we are actively
2923-        trying to connect to. Each RemoteServiceConnector has the following
2924-        public attributes::
2925-
2926-          service_name: the type of service provided, like 'storage'
2927-          announcement_time: when we first heard about this service
2928-          last_connect_time: when we last established a connection
2929-          last_loss_time: when we last lost a connection
2930-
2931-          version: the peer's version, from the most recent connection
2932-          oldest_supported: the peer's oldest supported version, same
2933-
2934-          rref: the RemoteReference, if connected, otherwise None
2935-          remote_host: the IAddress, if connected, otherwise None
2936-
2937-        This method is intended for monitoring interfaces, such as a web page
2938-        that describes connecting and connected peers.
2939-        """
2940-
2941-    def get_all_peerids():
2942-        """Return a frozenset of all peerids to whom we have a connection (to
2943-        one or more services) established. Mostly useful for unit tests."""
2944-
2945-    def get_all_connections_for(service_name):
2946-        """Return a frozenset of (nodeid, service_name, rref) tuples, one
2947-        for each active connection that provides the given SERVICE_NAME."""
2948-
2949-    def get_permuted_peers(service_name, key):
2950-        """Returns an ordered list of (peerid, rref) tuples, selecting from
2951-        the connections that provide SERVICE_NAME, using a hash-based
2952-        permutation keyed by KEY. This randomizes the service list in a
2953-        repeatable way, to distribute load over many peers.
2954-        """
2955-
2956 
2957 class IMutableSlotWriter(Interface):
2958     """
2959hunk ./src/allmydata/interfaces.py 616
2960     The interface for a writer around a mutable slot on a remote server.
2961     """
2962-    def set_checkstring(checkstring, *args):
2963+    def set_checkstring(seqnum_or_checkstring, root_hash=None, salt=None):
2964         """
2965         Set the checkstring that I will pass to the remote server when
2966         writing.
2967hunk ./src/allmydata/interfaces.py 640
2968         Add a block and salt to the share.
2969         """
2970 
2971-    def put_encprivey(encprivkey):
2972+    def put_encprivkey(encprivkey):
2973         """
2974         Add the encrypted private key to the share.
2975         """
2976hunk ./src/allmydata/interfaces.py 645
2977 
2978-    def put_blockhashes(blockhashes=list):
2979+    def put_blockhashes(blockhashes):
2980         """
2981hunk ./src/allmydata/interfaces.py 647
2982+        @param blockhashes=list
2983         Add the block hash tree to the share.
2984         """
2985 
2986hunk ./src/allmydata/interfaces.py 651
2987-    def put_sharehashes(sharehashes=dict):
2988+    def put_sharehashes(sharehashes):
2989         """
2990hunk ./src/allmydata/interfaces.py 653
2991+        @param sharehashes=dict
2992         Add the share hash chain to the share.
2993         """
2994 
2995hunk ./src/allmydata/interfaces.py 739
2996     def get_extension_params():
2997         """Return the extension parameters in the URI"""
2998 
2999-    def set_extension_params():
3000+    def set_extension_params(params):
3001         """Set the extension parameters that should be in the URI"""
3002 
3003 class IDirectoryURI(Interface):
3004hunk ./src/allmydata/interfaces.py 879
3005         writer-visible data using this writekey.
3006         """
3007 
3008-    # TODO: Can this be overwrite instead of replace?
3009-    def replace(new_contents):
3010-        """Replace the contents of the mutable file, provided that no other
3011+    def overwrite(new_contents):
3012+        """Overwrite the contents of the mutable file, provided that no other
3013         node has published (or is attempting to publish, concurrently) a
3014         newer version of the file than this one.
3015 
3016hunk ./src/allmydata/interfaces.py 1346
3017         is empty, the metadata will be an empty dictionary.
3018         """
3019 
3020-    def set_uri(name, writecap, readcap=None, metadata=None, overwrite=True):
3021+    def set_uri(name, writecap, readcap, metadata=None, overwrite=True):
3022         """I add a child (by writecap+readcap) at the specific name. I return
3023         a Deferred that fires when the operation finishes. If overwrite= is
3024         True, I will replace any existing child of the same name, otherwise
3025hunk ./src/allmydata/interfaces.py 1745
3026     Block Hash, and the encoding parameters, both of which must be included
3027     in the URI.
3028 
3029-    I do not choose shareholders, that is left to the IUploader. I must be
3030-    given a dict of RemoteReferences to storage buckets that are ready and
3031-    willing to receive data.
3032+    I do not choose shareholders, that is left to the IUploader.
3033     """
3034 
3035     def set_size(size):
3036hunk ./src/allmydata/interfaces.py 1752
3037         """Specify the number of bytes that will be encoded. This must be
3038         peformed before get_serialized_params() can be called.
3039         """
3040+
3041     def set_params(params):
3042         """Override the default encoding parameters. 'params' is a tuple of
3043         (k,d,n), where 'k' is the number of required shares, 'd' is the
3044hunk ./src/allmydata/interfaces.py 1848
3045     download, validate, decode, and decrypt data from them, writing the
3046     results to an output file.
3047 
3048-    I do not locate the shareholders, that is left to the IDownloader. I must
3049-    be given a dict of RemoteReferences to storage buckets that are ready to
3050-    send data.
3051+    I do not locate the shareholders, that is left to the IDownloader.
3052     """
3053 
3054     def setup(outfile):
3055hunk ./src/allmydata/interfaces.py 1950
3056         resuming an interrupted upload (where we need to compute the
3057         plaintext hashes, but don't need the redundant encrypted data)."""
3058 
3059-    def get_plaintext_hashtree_leaves(first, last, num_segments):
3060-        """OBSOLETE; Get the leaf nodes of a merkle hash tree over the
3061-        plaintext segments, i.e. get the tagged hashes of the given segments.
3062-        The segment size is expected to be generated by the
3063-        IEncryptedUploadable before any plaintext is read or ciphertext
3064-        produced, so that the segment hashes can be generated with only a
3065-        single pass.
3066-
3067-        This returns a Deferred that fires with a sequence of hashes, using:
3068-
3069-         tuple(segment_hashes[first:last])
3070-
3071-        'num_segments' is used to assert that the number of segments that the
3072-        IEncryptedUploadable handled matches the number of segments that the
3073-        encoder was expecting.
3074-
3075-        This method must not be called until the final byte has been read
3076-        from read_encrypted(). Once this method is called, read_encrypted()
3077-        can never be called again.
3078-        """
3079-
3080-    def get_plaintext_hash():
3081-        """OBSOLETE; Get the hash of the whole plaintext.
3082-
3083-        This returns a Deferred that fires with a tagged SHA-256 hash of the
3084-        whole plaintext, obtained from hashutil.plaintext_hash(data).
3085-        """
3086-
3087     def close():
3088         """Just like IUploadable.close()."""
3089 
3090hunk ./src/allmydata/interfaces.py 2144
3091         returns a Deferred that fires with an IUploadResults instance, from
3092         which the URI of the file can be obtained as results.uri ."""
3093 
3094-    def upload_ssk(write_capability, new_version, uploadable):
3095-        """TODO: how should this work?"""
3096-
3097 class ICheckable(Interface):
3098     def check(monitor, verify=False, add_lease=False):
3099         """Check up on my health, optionally repairing any problems.
3100hunk ./src/allmydata/interfaces.py 2505
3101 
3102 class IRepairResults(Interface):
3103     """I contain the results of a repair operation."""
3104-    def get_successful(self):
3105+    def get_successful():
3106         """Returns a boolean: True if the repair made the file healthy, False
3107         if not. Repair failure generally indicates a file that has been
3108         damaged beyond repair."""
3109hunk ./src/allmydata/interfaces.py 2577
3110     Tahoe process will typically have a single NodeMaker, but unit tests may
3111     create simplified/mocked forms for testing purposes.
3112     """
3113-    def create_from_cap(writecap, readcap=None, **kwargs):
3114+    def create_from_cap(writecap, readcap=None, deep_immutable=False, name=u"<unknown name>"):
3115         """I create an IFilesystemNode from the given writecap/readcap. I can
3116         only provide nodes for existing file/directory objects: use my other
3117         methods to create new objects. I return synchronously."""
3118hunk ./src/allmydata/monitor.py 30
3119 
3120     # the following methods are provided for the operation code
3121 
3122-    def is_cancelled(self):
3123+    def is_cancelled():
3124         """Returns True if the operation has been cancelled. If True,
3125         operation code should stop creating new work, and attempt to stop any
3126         work already in progress."""
3127hunk ./src/allmydata/monitor.py 35
3128 
3129-    def raise_if_cancelled(self):
3130+    def raise_if_cancelled():
3131         """Raise OperationCancelledError if the operation has been cancelled.
3132         Operation code that has a robust error-handling path can simply call
3133         this periodically."""
3134hunk ./src/allmydata/monitor.py 40
3135 
3136-    def set_status(self, status):
3137+    def set_status(status):
3138         """Sets the Monitor's 'status' object to an arbitrary value.
3139         Different operations will store different sorts of status information
3140         here. Operation code should use get+modify+set sequences to update
3141hunk ./src/allmydata/monitor.py 46
3142         this."""
3143 
3144-    def get_status(self):
3145+    def get_status():
3146         """Return the status object. If the operation failed, this will be a
3147         Failure instance."""
3148 
3149hunk ./src/allmydata/monitor.py 50
3150-    def finish(self, status):
3151+    def finish(status):
3152         """Call this when the operation is done, successful or not. The
3153         Monitor's lifetime is influenced by the completion of the operation
3154         it is monitoring. The Monitor's 'status' value will be set with the
3155hunk ./src/allmydata/monitor.py 63
3156 
3157     # the following methods are provided for the initiator of the operation
3158 
3159-    def is_finished(self):
3160+    def is_finished():
3161         """Return a boolean, True if the operation is done (whether
3162         successful or failed), False if it is still running."""
3163 
3164hunk ./src/allmydata/monitor.py 67
3165-    def when_done(self):
3166+    def when_done():
3167         """Return a Deferred that fires when the operation is complete. It
3168         will fire with the operation status, the same value as returned by
3169         get_status()."""
3170hunk ./src/allmydata/monitor.py 72
3171 
3172-    def cancel(self):
3173+    def cancel():
3174         """Cancel the operation as soon as possible. is_cancelled() will
3175         start returning True after this is called."""
3176 
3177hunk ./src/allmydata/mutable/filenode.py 753
3178         self._writekey = writekey
3179         self._serializer = defer.succeed(None)
3180 
3181-
3182     def get_sequence_number(self):
3183         """
3184         Get the sequence number of the mutable version that I represent.
3185hunk ./src/allmydata/mutable/filenode.py 759
3186         """
3187         return self._version[0] # verinfo[0] == the sequence number
3188 
3189+    def get_servermap(self):
3190+        return self._servermap
3191 
3192hunk ./src/allmydata/mutable/filenode.py 762
3193-    # TODO: Terminology?
3194     def get_writekey(self):
3195         """
3196         I return a writekey or None if I don't have a writekey.
3197hunk ./src/allmydata/mutable/filenode.py 768
3198         """
3199         return self._writekey
3200 
3201-
3202     def set_downloader_hints(self, hints):
3203         """
3204         I set the downloader hints.
3205hunk ./src/allmydata/mutable/filenode.py 776
3206 
3207         self._downloader_hints = hints
3208 
3209-
3210     def get_downloader_hints(self):
3211         """
3212         I return the downloader hints.
3213hunk ./src/allmydata/mutable/filenode.py 782
3214         """
3215         return self._downloader_hints
3216 
3217-
3218     def overwrite(self, new_contents):
3219         """
3220         I overwrite the contents of this mutable file version with the
3221hunk ./src/allmydata/mutable/filenode.py 791
3222 
3223         return self._do_serialized(self._overwrite, new_contents)
3224 
3225-
3226     def _overwrite(self, new_contents):
3227         assert IMutableUploadable.providedBy(new_contents)
3228         assert self._servermap.last_update_mode == MODE_WRITE
3229hunk ./src/allmydata/mutable/filenode.py 797
3230 
3231         return self._upload(new_contents)
3232 
3233-
3234     def modify(self, modifier, backoffer=None):
3235         """I use a modifier callback to apply a change to the mutable file.
3236         I implement the following pseudocode::
3237hunk ./src/allmydata/mutable/filenode.py 841
3238 
3239         return self._do_serialized(self._modify, modifier, backoffer)
3240 
3241-
3242     def _modify(self, modifier, backoffer):
3243         if backoffer is None:
3244             backoffer = BackoffAgent().delay
3245hunk ./src/allmydata/mutable/filenode.py 846
3246         return self._modify_and_retry(modifier, backoffer, True)
3247 
3248-
3249     def _modify_and_retry(self, modifier, backoffer, first_time):
3250         """
3251         I try to apply modifier to the contents of this version of the
3252hunk ./src/allmydata/mutable/filenode.py 878
3253         d.addErrback(_retry)
3254         return d
3255 
3256-
3257     def _modify_once(self, modifier, first_time):
3258         """
3259         I attempt to apply a modifier to the contents of the mutable
3260hunk ./src/allmydata/mutable/filenode.py 913
3261         d.addCallback(_apply)
3262         return d
3263 
3264-
3265     def is_readonly(self):
3266         """
3267         I return True if this MutableFileVersion provides no write
3268hunk ./src/allmydata/mutable/filenode.py 921
3269         """
3270         return self._writekey is None
3271 
3272-
3273     def is_mutable(self):
3274         """
3275         I return True, since mutable files are always mutable by
3276hunk ./src/allmydata/mutable/filenode.py 928
3277         """
3278         return True
3279 
3280-
3281     def get_storage_index(self):
3282         """
3283         I return the storage index of the reference that I encapsulate.
3284hunk ./src/allmydata/mutable/filenode.py 934
3285         """
3286         return self._storage_index
3287 
3288-
3289     def get_size(self):
3290         """
3291         I return the length, in bytes, of this readable object.
3292hunk ./src/allmydata/mutable/filenode.py 940
3293         """
3294         return self._servermap.size_of_version(self._version)
3295 
3296-
3297     def download_to_data(self, fetch_privkey=False):
3298         """
3299         I return a Deferred that fires with the contents of this
3300hunk ./src/allmydata/mutable/filenode.py 951
3301         d.addCallback(lambda mc: "".join(mc.chunks))
3302         return d
3303 
3304-
3305     def _try_to_download_data(self):
3306         """
3307         I am an unserialized cousin of download_to_data; I am called
3308hunk ./src/allmydata/mutable/filenode.py 963
3309         d.addCallback(lambda mc: "".join(mc.chunks))
3310         return d
3311 
3312-
3313     def read(self, consumer, offset=0, size=None, fetch_privkey=False):
3314         """
3315         I read a portion (possibly all) of the mutable file that I
3316hunk ./src/allmydata/mutable/filenode.py 971
3317         return self._do_serialized(self._read, consumer, offset, size,
3318                                    fetch_privkey)
3319 
3320-
3321     def _read(self, consumer, offset=0, size=None, fetch_privkey=False):
3322         """
3323         I am the serialized companion of read.
3324hunk ./src/allmydata/mutable/filenode.py 981
3325         d = r.download(consumer, offset, size)
3326         return d
3327 
3328-
3329     def _do_serialized(self, cb, *args, **kwargs):
3330         # note: to avoid deadlock, this callable is *not* allowed to invoke
3331         # other serialized methods within this (or any other)
3332hunk ./src/allmydata/mutable/filenode.py 999
3333         self._serializer.addErrback(log.err)
3334         return d
3335 
3336-
3337     def _upload(self, new_contents):
3338         #assert self._pubkey, "update_servermap must be called before publish"
3339         p = Publish(self._node, self._storage_broker, self._servermap)
3340hunk ./src/allmydata/mutable/filenode.py 1009
3341         d.addCallback(self._did_upload, new_contents.get_size())
3342         return d
3343 
3344-
3345     def _did_upload(self, res, size):
3346         self._most_recent_size = size
3347         return res
3348hunk ./src/allmydata/mutable/filenode.py 1029
3349         """
3350         return self._do_serialized(self._update, data, offset)
3351 
3352-
3353     def _update(self, data, offset):
3354         """
3355         I update the mutable file version represented by this particular
3356hunk ./src/allmydata/mutable/filenode.py 1058
3357         d.addCallback(self._build_uploadable_and_finish, data, offset)
3358         return d
3359 
3360-
3361     def _do_modify_update(self, data, offset):
3362         """
3363         I perform a file update by modifying the contents of the file
3364hunk ./src/allmydata/mutable/filenode.py 1073
3365             return new
3366         return self._modify(m, None)
3367 
3368-
3369     def _do_update_update(self, data, offset):
3370         """
3371         I start the Servermap update that gets us the data we need to
3372hunk ./src/allmydata/mutable/filenode.py 1108
3373         return self._update_servermap(update_range=(start_segment,
3374                                                     end_segment))
3375 
3376-
3377     def _decode_and_decrypt_segments(self, ignored, data, offset):
3378         """
3379         After the servermap update, I take the encrypted and encoded
3380hunk ./src/allmydata/mutable/filenode.py 1148
3381         d3 = defer.succeed(blockhashes)
3382         return deferredutil.gatherResults([d1, d2, d3])
3383 
3384-
3385     def _build_uploadable_and_finish(self, segments_and_bht, data, offset):
3386         """
3387         After the process has the plaintext segments, I build the
3388hunk ./src/allmydata/mutable/filenode.py 1163
3389         p = Publish(self._node, self._storage_broker, self._servermap)
3390         return p.update(u, offset, segments_and_bht[2], self._version)
3391 
3392-
3393     def _update_servermap(self, mode=MODE_WRITE, update_range=None):
3394         """
3395         I update the servermap. I return a Deferred that fires when the
3396hunk ./src/allmydata/storage/common.py 1
3397-
3398-import os.path
3399 from allmydata.util import base32
3400 
3401 class DataTooLargeError(Exception):
3402hunk ./src/allmydata/storage/common.py 5
3403     pass
3404+
3405 class UnknownMutableContainerVersionError(Exception):
3406     pass
3407hunk ./src/allmydata/storage/common.py 8
3408+
3409 class UnknownImmutableContainerVersionError(Exception):
3410     pass
3411 
3412hunk ./src/allmydata/storage/common.py 18
3413 
3414 def si_a2b(ascii_storageindex):
3415     return base32.a2b(ascii_storageindex)
3416-
3417-def storage_index_to_dir(storageindex):
3418-    sia = si_b2a(storageindex)
3419-    return os.path.join(sia[:2], sia)
3420hunk ./src/allmydata/storage/crawler.py 2
3421 
3422-import os, time, struct
3423+import time, struct
3424 import cPickle as pickle
3425 from twisted.internet import reactor
3426 from twisted.application import service
3427hunk ./src/allmydata/storage/crawler.py 6
3428+
3429+from allmydata.util.assertutil import precondition
3430+from allmydata.interfaces import IStorageBackend
3431 from allmydata.storage.common import si_b2a
3432hunk ./src/allmydata/storage/crawler.py 10
3433-from allmydata.util import fileutil
3434+
3435 
3436 class TimeSliceExceeded(Exception):
3437     pass
3438hunk ./src/allmydata/storage/crawler.py 15
3439 
3440+
3441 class ShareCrawler(service.MultiService):
3442hunk ./src/allmydata/storage/crawler.py 17
3443-    """A ShareCrawler subclass is attached to a StorageServer, and
3444-    periodically walks all of its shares, processing each one in some
3445-    fashion. This crawl is rate-limited, to reduce the IO burden on the host,
3446-    since large servers can easily have a terabyte of shares, in several
3447-    million files, which can take hours or days to read.
3448+    """
3449+    An instance of a subclass of ShareCrawler is attached to a storage
3450+    backend, and periodically walks the backend's shares, processing them
3451+    in some fashion. This crawl is rate-limited to reduce the I/O burden on
3452+    the host, since large servers can easily have a terabyte of shares in
3453+    several million files, which can take hours or days to read.
3454 
3455     Once the crawler starts a cycle, it will proceed at a rate limited by the
3456     allowed_cpu_percentage= and cpu_slice= parameters: yielding the reactor
3457hunk ./src/allmydata/storage/crawler.py 33
3458     long enough to ensure that 'minimum_cycle_time' elapses between the start
3459     of two consecutive cycles.
3460 
3461-    We assume that the normal upload/download/get_buckets traffic of a tahoe
3462+    We assume that the normal upload/download/DYHB traffic of a Tahoe-LAFS
3463     grid will cause the prefixdir contents to be mostly cached in the kernel,
3464hunk ./src/allmydata/storage/crawler.py 35
3465-    or that the number of buckets in each prefixdir will be small enough to
3466-    load quickly. A 1TB allmydata.com server was measured to have 2.56M
3467-    buckets, spread into the 1024 prefixdirs, with about 2500 buckets per
3468+    or that the number of sharesets in each prefixdir will be small enough to
3469+    load quickly. A 1TB allmydata.com server was measured to have 2.56 million
3470+    sharesets, spread into the 1024 prefixdirs, with about 2500 sharesets per
3471     prefix. On this server, each prefixdir took 130ms-200ms to list the first
3472     time, and 17ms to list the second time.
3473 
3474hunk ./src/allmydata/storage/crawler.py 41
3475-    To use a crawler, create a subclass which implements the process_bucket()
3476-    method. It will be called with a prefixdir and a base32 storage index
3477-    string. process_bucket() must run synchronously. Any keys added to
3478-    self.state will be preserved. Override add_initial_state() to set up
3479-    initial state keys. Override finished_cycle() to perform additional
3480-    processing when the cycle is complete. Any status that the crawler
3481-    produces should be put in the self.state dictionary. Status renderers
3482-    (like a web page which describes the accomplishments of your crawler)
3483-    will use crawler.get_state() to retrieve this dictionary; they can
3484-    present the contents as they see fit.
3485+    To implement a crawler, create a subclass that implements the
3486+    process_shareset() method. It will be called with a prefixdir and an
3487+    object providing the IShareSet interface. process_shareset() must run
3488+    synchronously. Any keys added to self.state will be preserved. Override
3489+    add_initial_state() to set up initial state keys. Override
3490+    finished_cycle() to perform additional processing when the cycle is
3491+    complete. Any status that the crawler produces should be put in the
3492+    self.state dictionary. Status renderers (like a web page describing the
3493+    accomplishments of your crawler) will use crawler.get_state() to retrieve
3494+    this dictionary; they can present the contents as they see fit.
3495 
3496hunk ./src/allmydata/storage/crawler.py 52
3497-    Then create an instance, with a reference to a StorageServer and a
3498-    filename where it can store persistent state. The statefile is used to
3499-    keep track of how far around the ring the process has travelled, as well
3500-    as timing history to allow the pace to be predicted and controlled. The
3501-    statefile will be updated and written to disk after each time slice (just
3502-    before the crawler yields to the reactor), and also after each cycle is
3503-    finished, and also when stopService() is called. Note that this means
3504-    that a crawler which is interrupted with SIGKILL while it is in the
3505-    middle of a time slice will lose progress: the next time the node is
3506-    started, the crawler will repeat some unknown amount of work.
3507+    Then create an instance, with a reference to a backend object providing
3508+    the IStorageBackend interface, and a filename where it can store
3509+    persistent state. The statefile is used to keep track of how far around
3510+    the ring the process has travelled, as well as timing history to allow
3511+    the pace to be predicted and controlled. The statefile will be updated
3512+    and written to disk after each time slice (just before the crawler yields
3513+    to the reactor), and also after each cycle is finished, and also when
3514+    stopService() is called. Note that this means that a crawler that is
3515+    interrupted with SIGKILL while it is in the middle of a time slice will
3516+    lose progress: the next time the node is started, the crawler will repeat
3517+    some unknown amount of work.
3518 
3519     The crawler instance must be started with startService() before it will
3520hunk ./src/allmydata/storage/crawler.py 65
3521-    do any work. To make it stop doing work, call stopService().
3522+    do any work. To make it stop doing work, call stopService(). A crawler
3523+    is usually a child service of a StorageServer, although it should not
3524+    depend on that.
3525+
3526+    For historical reasons, some dictionary key names use the term "bucket"
3527+    for what is now preferably called a "shareset" (the set of shares that a
3528+    server holds under a given storage index).
3529     """
3530 
3531     slow_start = 300 # don't start crawling for 5 minutes after startup
3532hunk ./src/allmydata/storage/crawler.py 80
3533     cpu_slice = 1.0 # use up to 1.0 seconds before yielding
3534     minimum_cycle_time = 300 # don't run a cycle faster than this
3535 
3536-    def __init__(self, server, statefile, allowed_cpu_percentage=None):
3537+    def __init__(self, backend, statefp, allowed_cpu_percentage=None):
3538+        precondition(IStorageBackend.providedBy(backend), backend)
3539         service.MultiService.__init__(self)
3540hunk ./src/allmydata/storage/crawler.py 83
3541+        self.backend = backend
3542+        self.statefp = statefp
3543         if allowed_cpu_percentage is not None:
3544             self.allowed_cpu_percentage = allowed_cpu_percentage
3545hunk ./src/allmydata/storage/crawler.py 87
3546-        self.server = server
3547-        self.sharedir = server.sharedir
3548-        self.statefile = statefile
3549         self.prefixes = [si_b2a(struct.pack(">H", i << (16-10)))[:2]
3550                          for i in range(2**10)]
3551         self.prefixes.sort()
3552hunk ./src/allmydata/storage/crawler.py 91
3553         self.timer = None
3554-        self.bucket_cache = (None, [])
3555+        self.shareset_cache = (None, [])
3556         self.current_sleep_time = None
3557         self.next_wake_time = None
3558         self.last_prefix_finished_time = None
3559hunk ./src/allmydata/storage/crawler.py 154
3560                 left = len(self.prefixes) - self.last_complete_prefix_index
3561                 remaining = left * self.last_prefix_elapsed_time
3562                 # TODO: remainder of this prefix: we need to estimate the
3563-                # per-bucket time, probably by measuring the time spent on
3564-                # this prefix so far, divided by the number of buckets we've
3565+                # per-shareset time, probably by measuring the time spent on
3566+                # this prefix so far, divided by the number of sharesets we've
3567                 # processed.
3568             d["estimated-cycle-complete-time-left"] = remaining
3569             # it's possible to call get_progress() from inside a crawler's
3570hunk ./src/allmydata/storage/crawler.py 175
3571         state dictionary.
3572 
3573         If we are not currently sleeping (i.e. get_state() was called from
3574-        inside the process_prefixdir, process_bucket, or finished_cycle()
3575+        inside the process_prefixdir, process_shareset, or finished_cycle()
3576         methods, or if startService has not yet been called on this crawler),
3577         these two keys will be None.
3578 
3579hunk ./src/allmydata/storage/crawler.py 188
3580     def load_state(self):
3581         # we use this to store state for both the crawler's internals and
3582         # anything the subclass-specific code needs. The state is stored
3583-        # after each bucket is processed, after each prefixdir is processed,
3584+        # after each shareset is processed, after each prefixdir is processed,
3585         # and after a cycle is complete. The internal keys we use are:
3586         #  ["version"]: int, always 1
3587         #  ["last-cycle-finished"]: int, or None if we have not yet finished
3588hunk ./src/allmydata/storage/crawler.py 202
3589         #                            are sleeping between cycles, or if we
3590         #                            have not yet finished any prefixdir since
3591         #                            a cycle was started
3592-        #  ["last-complete-bucket"]: str, base32 storage index bucket name
3593-        #                            of the last bucket to be processed, or
3594-        #                            None if we are sleeping between cycles
3595+        #  ["last-complete-bucket"]: str, base32 storage index of the last
3596+        #                            shareset to be processed, or None if we
3597+        #                            are sleeping between cycles
3598         try:
3599hunk ./src/allmydata/storage/crawler.py 206
3600-            f = open(self.statefile, "rb")
3601-            state = pickle.load(f)
3602-            f.close()
3603+            state = pickle.loads(self.statefp.getContent())
3604         except EnvironmentError:
3605             state = {"version": 1,
3606                      "last-cycle-finished": None,
3607hunk ./src/allmydata/storage/crawler.py 242
3608         else:
3609             last_complete_prefix = self.prefixes[lcpi]
3610         self.state["last-complete-prefix"] = last_complete_prefix
3611-        tmpfile = self.statefile + ".tmp"
3612-        f = open(tmpfile, "wb")
3613-        pickle.dump(self.state, f)
3614-        f.close()
3615-        fileutil.move_into_place(tmpfile, self.statefile)
3616+        self.statefp.setContent(pickle.dumps(self.state))
3617 
3618     def startService(self):
3619         # arrange things to look like we were just sleeping, so
3620hunk ./src/allmydata/storage/crawler.py 284
3621         sleep_time = (this_slice / self.allowed_cpu_percentage) - this_slice
3622         # if the math gets weird, or a timequake happens, don't sleep
3623         # forever. Note that this means that, while a cycle is running, we
3624-        # will process at least one bucket every 5 minutes, no matter how
3625-        # long that bucket takes.
3626+        # will process at least one shareset every 5 minutes, no matter how
3627+        # long that shareset takes.
3628         sleep_time = max(0.0, min(sleep_time, 299))
3629         if finished_cycle:
3630             # how long should we sleep between cycles? Don't run faster than
3631hunk ./src/allmydata/storage/crawler.py 315
3632         for i in range(self.last_complete_prefix_index+1, len(self.prefixes)):
3633             # if we want to yield earlier, just raise TimeSliceExceeded()
3634             prefix = self.prefixes[i]
3635-            prefixdir = os.path.join(self.sharedir, prefix)
3636-            if i == self.bucket_cache[0]:
3637-                buckets = self.bucket_cache[1]
3638+            if i == self.shareset_cache[0]:
3639+                sharesets = self.shareset_cache[1]
3640             else:
3641hunk ./src/allmydata/storage/crawler.py 318
3642-                try:
3643-                    buckets = os.listdir(prefixdir)
3644-                    buckets.sort()
3645-                except EnvironmentError:
3646-                    buckets = []
3647-                self.bucket_cache = (i, buckets)
3648-            self.process_prefixdir(cycle, prefix, prefixdir,
3649-                                   buckets, start_slice)
3650+                sharesets = self.backend.get_sharesets_for_prefix(prefix)
3651+                self.shareset_cache = (i, sharesets)
3652+            self.process_prefixdir(cycle, prefix, sharesets, start_slice)
3653             self.last_complete_prefix_index = i
3654 
3655             now = time.time()
3656hunk ./src/allmydata/storage/crawler.py 345
3657         self.finished_cycle(cycle)
3658         self.save_state()
3659 
3660-    def process_prefixdir(self, cycle, prefix, prefixdir, buckets, start_slice):
3661-        """This gets a list of bucket names (i.e. storage index strings,
3662+    def process_prefixdir(self, cycle, prefix, sharesets, start_slice):
3663+        """
3664+        This gets a list of shareset names (i.e. storage index strings,
3665         base32-encoded) in sorted order.
3666 
3667         You can override this if your crawler doesn't care about the actual
3668hunk ./src/allmydata/storage/crawler.py 352
3669         shares, for example a crawler which merely keeps track of how many
3670-        buckets are being managed by this server.
3671+        sharesets are being managed by this server.
3672 
3673hunk ./src/allmydata/storage/crawler.py 354
3674-        Subclasses which *do* care about actual bucket should leave this
3675-        method along, and implement process_bucket() instead.
3676+        Subclasses which *do* care about actual shareset should leave this
3677+        method alone, and implement process_shareset() instead.
3678         """
3679 
3680hunk ./src/allmydata/storage/crawler.py 358
3681-        for bucket in buckets:
3682-            if bucket <= self.state["last-complete-bucket"]:
3683+        for shareset in sharesets:
3684+            base32si = shareset.get_storage_index_string()
3685+            if base32si <= self.state["last-complete-bucket"]:
3686                 continue
3687hunk ./src/allmydata/storage/crawler.py 362
3688-            self.process_bucket(cycle, prefix, prefixdir, bucket)
3689-            self.state["last-complete-bucket"] = bucket
3690+            self.process_shareset(cycle, prefix, shareset)
3691+            self.state["last-complete-bucket"] = base32si
3692             if time.time() >= start_slice + self.cpu_slice:
3693                 raise TimeSliceExceeded()
3694 
3695hunk ./src/allmydata/storage/crawler.py 370
3696     # the remaining methods are explictly for subclasses to implement.
3697 
3698     def started_cycle(self, cycle):
3699-        """Notify a subclass that the crawler is about to start a cycle.
3700+        """
3701+        Notify a subclass that the crawler is about to start a cycle.
3702 
3703         This method is for subclasses to override. No upcall is necessary.
3704         """
3705hunk ./src/allmydata/storage/crawler.py 377
3706         pass
3707 
3708-    def process_bucket(self, cycle, prefix, prefixdir, storage_index_b32):
3709-        """Examine a single bucket. Subclasses should do whatever they want
3710+    def process_shareset(self, cycle, prefix, shareset):
3711+        """
3712+        Examine a single shareset. Subclasses should do whatever they want
3713         to do to the shares therein, then update self.state as necessary.
3714 
3715         If the crawler is never interrupted by SIGKILL, this method will be
3716hunk ./src/allmydata/storage/crawler.py 383
3717-        called exactly once per share (per cycle). If it *is* interrupted,
3718+        called exactly once per shareset (per cycle). If it *is* interrupted,
3719         then the next time the node is started, some amount of work will be
3720         duplicated, according to when self.save_state() was last called. By
3721         default, save_state() is called at the end of each timeslice, and
3722hunk ./src/allmydata/storage/crawler.py 391
3723 
3724         To reduce the chance of duplicate work (i.e. to avoid adding multiple
3725         records to a database), you can call save_state() at the end of your
3726-        process_bucket() method. This will reduce the maximum duplicated work
3727-        to one bucket per SIGKILL. It will also add overhead, probably 1-20ms
3728-        per bucket (and some disk writes), which will count against your
3729-        allowed_cpu_percentage, and which may be considerable if
3730-        process_bucket() runs quickly.
3731+        process_shareset() method. This will reduce the maximum duplicated
3732+        work to one shareset per SIGKILL. It will also add overhead, probably
3733+        1-20ms per shareset (and some disk writes), which will count against
3734+        your allowed_cpu_percentage, and which may be considerable if
3735+        process_shareset() runs quickly.
3736 
3737         This method is for subclasses to override. No upcall is necessary.
3738         """
3739hunk ./src/allmydata/storage/crawler.py 402
3740         pass
3741 
3742     def finished_prefix(self, cycle, prefix):
3743-        """Notify a subclass that the crawler has just finished processing a
3744-        prefix directory (all buckets with the same two-character/10bit
3745+        """
3746+        Notify a subclass that the crawler has just finished processing a
3747+        prefix directory (all sharesets with the same two-character/10-bit
3748         prefix). To impose a limit on how much work might be duplicated by a
3749         SIGKILL that occurs during a timeslice, you can call
3750         self.save_state() here, but be aware that it may represent a
3751hunk ./src/allmydata/storage/crawler.py 415
3752         pass
3753 
3754     def finished_cycle(self, cycle):
3755-        """Notify subclass that a cycle (one complete traversal of all
3756+        """
3757+        Notify subclass that a cycle (one complete traversal of all
3758         prefixdirs) has just finished. 'cycle' is the number of the cycle
3759         that just finished. This method should perform summary work and
3760         update self.state to publish information to status displays.
3761hunk ./src/allmydata/storage/crawler.py 433
3762         pass
3763 
3764     def yielding(self, sleep_time):
3765-        """The crawler is about to sleep for 'sleep_time' seconds. This
3766+        """
3767+        The crawler is about to sleep for 'sleep_time' seconds. This
3768         method is mostly for the convenience of unit tests.
3769 
3770         This method is for subclasses to override. No upcall is necessary.
3771hunk ./src/allmydata/storage/crawler.py 443
3772 
3773 
3774 class BucketCountingCrawler(ShareCrawler):
3775-    """I keep track of how many buckets are being managed by this server.
3776-    This is equivalent to the number of distributed files and directories for
3777-    which I am providing storage. The actual number of files+directories in
3778-    the full grid is probably higher (especially when there are more servers
3779-    than 'N', the number of generated shares), because some files+directories
3780-    will have shares on other servers instead of me. Also note that the
3781-    number of buckets will differ from the number of shares in small grids,
3782-    when more than one share is placed on a single server.
3783+    """
3784+    I keep track of how many sharesets, each corresponding to a storage index,
3785+    are being managed by this server. This is equivalent to the number of
3786+    distributed files and directories for which I am providing storage. The
3787+    actual number of files and directories in the full grid is probably higher
3788+    (especially when there are more servers than 'N', the number of generated
3789+    shares), because some files and directories will have shares on other
3790+    servers instead of me. Also note that the number of sharesets will differ
3791+    from the number of shares in small grids, when more than one share is
3792+    placed on a single server.
3793     """
3794 
3795     minimum_cycle_time = 60*60 # we don't need this more than once an hour
3796hunk ./src/allmydata/storage/crawler.py 457
3797 
3798-    def __init__(self, server, statefile, num_sample_prefixes=1):
3799-        ShareCrawler.__init__(self, server, statefile)
3800+    def __init__(self, backend, statefp, num_sample_prefixes=1):
3801+        ShareCrawler.__init__(self, backend, statefp)
3802         self.num_sample_prefixes = num_sample_prefixes
3803 
3804     def add_initial_state(self):
3805hunk ./src/allmydata/storage/crawler.py 471
3806         self.state.setdefault("last-complete-bucket-count", None)
3807         self.state.setdefault("storage-index-samples", {})
3808 
3809-    def process_prefixdir(self, cycle, prefix, prefixdir, buckets, start_slice):
3810+    def process_prefixdir(self, cycle, prefix, sharesets, start_slice):
3811         # we override process_prefixdir() because we don't want to look at
3812hunk ./src/allmydata/storage/crawler.py 473
3813-        # the individual buckets. We'll save state after each one. On my
3814+        # the individual sharesets. We'll save state after each one. On my
3815         # laptop, a mostly-empty storage server can process about 70
3816         # prefixdirs in a 1.0s slice.
3817         if cycle not in self.state["bucket-counts"]:
3818hunk ./src/allmydata/storage/crawler.py 478
3819             self.state["bucket-counts"][cycle] = {}
3820-        self.state["bucket-counts"][cycle][prefix] = len(buckets)
3821+        self.state["bucket-counts"][cycle][prefix] = len(sharesets)
3822         if prefix in self.prefixes[:self.num_sample_prefixes]:
3823hunk ./src/allmydata/storage/crawler.py 480
3824-            self.state["storage-index-samples"][prefix] = (cycle, buckets)
3825+            self.state["storage-index-samples"][prefix] = (cycle, sharesets)
3826 
3827     def finished_cycle(self, cycle):
3828         last_counts = self.state["bucket-counts"].get(cycle, [])
3829hunk ./src/allmydata/storage/crawler.py 486
3830         if len(last_counts) == len(self.prefixes):
3831             # great, we have a whole cycle.
3832-            num_buckets = sum(last_counts.values())
3833-            self.state["last-complete-bucket-count"] = num_buckets
3834+            num_sharesets = sum(last_counts.values())
3835+            self.state["last-complete-bucket-count"] = num_sharesets
3836             # get rid of old counts
3837             for old_cycle in list(self.state["bucket-counts"].keys()):
3838                 if old_cycle != cycle:
3839hunk ./src/allmydata/storage/crawler.py 494
3840                     del self.state["bucket-counts"][old_cycle]
3841         # get rid of old samples too
3842         for prefix in list(self.state["storage-index-samples"].keys()):
3843-            old_cycle,buckets = self.state["storage-index-samples"][prefix]
3844+            old_cycle, storage_indices = self.state["storage-index-samples"][prefix]
3845             if old_cycle != cycle:
3846                 del self.state["storage-index-samples"][prefix]
3847hunk ./src/allmydata/storage/crawler.py 497
3848-
3849hunk ./src/allmydata/storage/expirer.py 1
3850-import time, os, pickle, struct
3851+
3852+import time, pickle, struct
3853+from twisted.python import log as twlog
3854+
3855 from allmydata.storage.crawler import ShareCrawler
3856hunk ./src/allmydata/storage/expirer.py 6
3857-from allmydata.storage.shares import get_share_file
3858-from allmydata.storage.common import UnknownMutableContainerVersionError, \
3859+from allmydata.storage.common import si_b2a, UnknownMutableContainerVersionError, \
3860      UnknownImmutableContainerVersionError
3861hunk ./src/allmydata/storage/expirer.py 8
3862-from twisted.python import log as twlog
3863+
3864 
3865 class LeaseCheckingCrawler(ShareCrawler):
3866     """I examine the leases on all shares, determining which are still valid
3867hunk ./src/allmydata/storage/expirer.py 17
3868     removed.
3869 
3870     I collect statistics on the leases and make these available to a web
3871-    status page, including::
3872+    status page, including:
3873 
3874     Space recovered during this cycle-so-far:
3875      actual (only if expiration_enabled=True):
3876hunk ./src/allmydata/storage/expirer.py 21
3877-      num-buckets, num-shares, sum of share sizes, real disk usage
3878+      num-storage-indices, num-shares, sum of share sizes, real disk usage
3879       ('real disk usage' means we use stat(fn).st_blocks*512 and include any
3880        space used by the directory)
3881      what it would have been with the original lease expiration time
3882hunk ./src/allmydata/storage/expirer.py 32
3883 
3884     Space recovered during the last 10 cycles  <-- saved in separate pickle
3885 
3886-    Shares/buckets examined:
3887+    Shares/storage-indices examined:
3888      this cycle-so-far
3889      prediction of rest of cycle
3890      during last 10 cycles <-- separate pickle
3891hunk ./src/allmydata/storage/expirer.py 42
3892     Histogram of leases-per-share:
3893      this-cycle-to-date
3894      last 10 cycles <-- separate pickle
3895-    Histogram of lease ages, buckets = 1day
3896+    Histogram of lease ages, storage-indices over 1 day
3897      cycle-to-date
3898      last 10 cycles <-- separate pickle
3899 
3900hunk ./src/allmydata/storage/expirer.py 53
3901     slow_start = 360 # wait 6 minutes after startup
3902     minimum_cycle_time = 12*60*60 # not more than twice per day
3903 
3904-    def __init__(self, server, statefile, historyfile,
3905-                 expiration_enabled, mode,
3906-                 override_lease_duration, # used if expiration_mode=="age"
3907-                 cutoff_date, # used if expiration_mode=="cutoff-date"
3908-                 sharetypes):
3909-        self.historyfile = historyfile
3910-        self.expiration_enabled = expiration_enabled
3911-        self.mode = mode
3912+    def __init__(self, backend, statefp, historyfp, expiration_policy):
3913+        # ShareCrawler.__init__ will call add_initial_state, so self.historyfp has to be set first.
3914+        self.historyfp = historyfp
3915+        ShareCrawler.__init__(self, backend, statefp)
3916+
3917+        self.expiration_enabled = expiration_policy['enabled']
3918+        self.mode = expiration_policy['mode']
3919         self.override_lease_duration = None
3920         self.cutoff_date = None
3921         if self.mode == "age":
3922hunk ./src/allmydata/storage/expirer.py 63
3923-            assert isinstance(override_lease_duration, (int, type(None)))
3924-            self.override_lease_duration = override_lease_duration # seconds
3925+            assert isinstance(expiration_policy['override_lease_duration'], (int, type(None)))
3926+            self.override_lease_duration = expiration_policy['override_lease_duration'] # seconds
3927         elif self.mode == "cutoff-date":
3928hunk ./src/allmydata/storage/expirer.py 66
3929-            assert isinstance(cutoff_date, int) # seconds-since-epoch
3930-            assert cutoff_date is not None
3931-            self.cutoff_date = cutoff_date
3932+            assert isinstance(expiration_policy['cutoff_date'], int) # seconds-since-epoch
3933+            self.cutoff_date = expiration_policy['cutoff_date']
3934         else:
3935hunk ./src/allmydata/storage/expirer.py 69
3936-            raise ValueError("GC mode '%s' must be 'age' or 'cutoff-date'" % mode)
3937-        self.sharetypes_to_expire = sharetypes
3938-        ShareCrawler.__init__(self, server, statefile)
3939+            raise ValueError("GC mode '%s' must be 'age' or 'cutoff-date'" % expiration_policy['mode'])
3940+        self.sharetypes_to_expire = expiration_policy['sharetypes']
3941 
3942     def add_initial_state(self):
3943         # we fill ["cycle-to-date"] here (even though they will be reset in
3944hunk ./src/allmydata/storage/expirer.py 84
3945             self.state["cycle-to-date"].setdefault(k, so_far[k])
3946 
3947         # initialize history
3948-        if not os.path.exists(self.historyfile):
3949+        if not self.historyfp.exists():
3950             history = {} # cyclenum -> dict
3951hunk ./src/allmydata/storage/expirer.py 86
3952-            f = open(self.historyfile, "wb")
3953-            pickle.dump(history, f)
3954-            f.close()
3955+            self.historyfp.setContent(pickle.dumps(history))
3956 
3957     def create_empty_cycle_dict(self):
3958         recovered = self.create_empty_recovered_dict()
3959hunk ./src/allmydata/storage/expirer.py 99
3960 
3961     def create_empty_recovered_dict(self):
3962         recovered = {}
3963+        # "buckets" is ambiguous; here it means the number of sharesets (one per storage index per server)
3964         for a in ("actual", "original", "configured", "examined"):
3965             for b in ("buckets", "shares", "sharebytes", "diskbytes"):
3966                 recovered[a+"-"+b] = 0
3967hunk ./src/allmydata/storage/expirer.py 110
3968     def started_cycle(self, cycle):
3969         self.state["cycle-to-date"] = self.create_empty_cycle_dict()
3970 
3971-    def stat(self, fn):
3972-        return os.stat(fn)
3973-
3974-    def process_bucket(self, cycle, prefix, prefixdir, storage_index_b32):
3975-        bucketdir = os.path.join(prefixdir, storage_index_b32)
3976-        s = self.stat(bucketdir)
3977+    def process_storage_index(self, cycle, prefix, container):
3978         would_keep_shares = []
3979         wks = None
3980hunk ./src/allmydata/storage/expirer.py 113
3981+        sharetype = None
3982 
3983hunk ./src/allmydata/storage/expirer.py 115
3984-        for fn in os.listdir(bucketdir):
3985-            try:
3986-                shnum = int(fn)
3987-            except ValueError:
3988-                continue # non-numeric means not a sharefile
3989-            sharefile = os.path.join(bucketdir, fn)
3990+        for share in container.get_shares():
3991+            sharetype = share.sharetype
3992             try:
3993hunk ./src/allmydata/storage/expirer.py 118
3994-                wks = self.process_share(sharefile)
3995+                wks = self.process_share(share)
3996             except (UnknownMutableContainerVersionError,
3997                     UnknownImmutableContainerVersionError,
3998                     struct.error):
3999hunk ./src/allmydata/storage/expirer.py 122
4000-                twlog.msg("lease-checker error processing %s" % sharefile)
4001+                twlog.msg("lease-checker error processing %r" % (share,))
4002                 twlog.err()
4003hunk ./src/allmydata/storage/expirer.py 124
4004-                which = (storage_index_b32, shnum)
4005+                which = (si_b2a(share.storageindex), share.get_shnum())
4006                 self.state["cycle-to-date"]["corrupt-shares"].append(which)
4007                 wks = (1, 1, 1, "unknown")
4008             would_keep_shares.append(wks)
4009hunk ./src/allmydata/storage/expirer.py 129
4010 
4011-        sharetype = None
4012+        container_type = None
4013         if wks:
4014hunk ./src/allmydata/storage/expirer.py 131
4015-            # use the last share's sharetype as the buckettype
4016-            sharetype = wks[3]
4017+            # use the last share's sharetype as the container type
4018+            container_type = wks[3]
4019         rec = self.state["cycle-to-date"]["space-recovered"]
4020         self.increment(rec, "examined-buckets", 1)
4021         if sharetype:
4022hunk ./src/allmydata/storage/expirer.py 136
4023-            self.increment(rec, "examined-buckets-"+sharetype, 1)
4024+            self.increment(rec, "examined-buckets-"+container_type, 1)
4025+
4026+        container_diskbytes = container.get_overhead()
4027 
4028hunk ./src/allmydata/storage/expirer.py 140
4029-        try:
4030-            bucket_diskbytes = s.st_blocks * 512
4031-        except AttributeError:
4032-            bucket_diskbytes = 0 # no stat().st_blocks on windows
4033         if sum([wks[0] for wks in would_keep_shares]) == 0:
4034hunk ./src/allmydata/storage/expirer.py 141
4035-            self.increment_bucketspace("original", bucket_diskbytes, sharetype)
4036+            self.increment_container_space("original", container_diskbytes, sharetype)
4037         if sum([wks[1] for wks in would_keep_shares]) == 0:
4038hunk ./src/allmydata/storage/expirer.py 143
4039-            self.increment_bucketspace("configured", bucket_diskbytes, sharetype)
4040+            self.increment_container_space("configured", container_diskbytes, sharetype)
4041         if sum([wks[2] for wks in would_keep_shares]) == 0:
4042hunk ./src/allmydata/storage/expirer.py 145
4043-            self.increment_bucketspace("actual", bucket_diskbytes, sharetype)
4044+            self.increment_container_space("actual", container_diskbytes, sharetype)
4045 
4046hunk ./src/allmydata/storage/expirer.py 147
4047-    def process_share(self, sharefilename):
4048-        # first, find out what kind of a share it is
4049-        sf = get_share_file(sharefilename)
4050-        sharetype = sf.sharetype
4051+    def process_share(self, share):
4052+        sharetype = share.sharetype
4053         now = time.time()
4054hunk ./src/allmydata/storage/expirer.py 150
4055-        s = self.stat(sharefilename)
4056+        sharebytes = share.get_size()
4057+        diskbytes = share.get_used_space()
4058 
4059         num_leases = 0
4060         num_valid_leases_original = 0
4061hunk ./src/allmydata/storage/expirer.py 158
4062         num_valid_leases_configured = 0
4063         expired_leases_configured = []
4064 
4065-        for li in sf.get_leases():
4066+        for li in share.get_leases():
4067             num_leases += 1
4068             original_expiration_time = li.get_expiration_time()
4069             grant_renew_time = li.get_grant_renew_time_time()
4070hunk ./src/allmydata/storage/expirer.py 171
4071 
4072             #  expired-or-not according to our configured age limit
4073             expired = False
4074-            if self.mode == "age":
4075-                age_limit = original_expiration_time
4076-                if self.override_lease_duration is not None:
4077-                    age_limit = self.override_lease_duration
4078-                if age > age_limit:
4079-                    expired = True
4080-            else:
4081-                assert self.mode == "cutoff-date"
4082-                if grant_renew_time < self.cutoff_date:
4083-                    expired = True
4084-            if sharetype not in self.sharetypes_to_expire:
4085-                expired = False
4086+            if sharetype in self.sharetypes_to_expire:
4087+                if self.mode == "age":
4088+                    age_limit = original_expiration_time
4089+                    if self.override_lease_duration is not None:
4090+                        age_limit = self.override_lease_duration
4091+                    if age > age_limit:
4092+                        expired = True
4093+                else:
4094+                    assert self.mode == "cutoff-date"
4095+                    if grant_renew_time < self.cutoff_date:
4096+                        expired = True
4097 
4098             if expired:
4099                 expired_leases_configured.append(li)
4100hunk ./src/allmydata/storage/expirer.py 190
4101 
4102         so_far = self.state["cycle-to-date"]
4103         self.increment(so_far["leases-per-share-histogram"], num_leases, 1)
4104-        self.increment_space("examined", s, sharetype)
4105+        self.increment_space("examined", diskbytes, sharetype)
4106 
4107         would_keep_share = [1, 1, 1, sharetype]
4108 
4109hunk ./src/allmydata/storage/expirer.py 196
4110         if self.expiration_enabled:
4111             for li in expired_leases_configured:
4112-                sf.cancel_lease(li.cancel_secret)
4113+                share.cancel_lease(li.cancel_secret)
4114 
4115         if num_valid_leases_original == 0:
4116             would_keep_share[0] = 0
4117hunk ./src/allmydata/storage/expirer.py 200
4118-            self.increment_space("original", s, sharetype)
4119+            self.increment_space("original", sharebytes, diskbytes, sharetype)
4120 
4121         if num_valid_leases_configured == 0:
4122             would_keep_share[1] = 0
4123hunk ./src/allmydata/storage/expirer.py 204
4124-            self.increment_space("configured", s, sharetype)
4125+            self.increment_space("configured", sharebytes, diskbytes, sharetype)
4126             if self.expiration_enabled:
4127                 would_keep_share[2] = 0
4128hunk ./src/allmydata/storage/expirer.py 207
4129-                self.increment_space("actual", s, sharetype)
4130+                self.increment_space("actual", sharebytes, diskbytes, sharetype)
4131 
4132         return would_keep_share
4133 
4134hunk ./src/allmydata/storage/expirer.py 211
4135-    def increment_space(self, a, s, sharetype):
4136-        sharebytes = s.st_size
4137-        try:
4138-            # note that stat(2) says that st_blocks is 512 bytes, and that
4139-            # st_blksize is "optimal file sys I/O ops blocksize", which is
4140-            # independent of the block-size that st_blocks uses.
4141-            diskbytes = s.st_blocks * 512
4142-        except AttributeError:
4143-            # the docs say that st_blocks is only on linux. I also see it on
4144-            # MacOS. But it isn't available on windows.
4145-            diskbytes = sharebytes
4146+    def increment_space(self, a, sharebytes, diskbytes, sharetype):
4147         so_far_sr = self.state["cycle-to-date"]["space-recovered"]
4148         self.increment(so_far_sr, a+"-shares", 1)
4149         self.increment(so_far_sr, a+"-sharebytes", sharebytes)
4150hunk ./src/allmydata/storage/expirer.py 221
4151             self.increment(so_far_sr, a+"-sharebytes-"+sharetype, sharebytes)
4152             self.increment(so_far_sr, a+"-diskbytes-"+sharetype, diskbytes)
4153 
4154-    def increment_bucketspace(self, a, bucket_diskbytes, sharetype):
4155+    def increment_container_space(self, a, container_diskbytes, container_type):
4156         rec = self.state["cycle-to-date"]["space-recovered"]
4157hunk ./src/allmydata/storage/expirer.py 223
4158-        self.increment(rec, a+"-diskbytes", bucket_diskbytes)
4159+        self.increment(rec, a+"-diskbytes", container_diskbytes)
4160         self.increment(rec, a+"-buckets", 1)
4161hunk ./src/allmydata/storage/expirer.py 225
4162-        if sharetype:
4163-            self.increment(rec, a+"-diskbytes-"+sharetype, bucket_diskbytes)
4164-            self.increment(rec, a+"-buckets-"+sharetype, 1)
4165+        if container_type:
4166+            self.increment(rec, a+"-diskbytes-"+container_type, container_diskbytes)
4167+            self.increment(rec, a+"-buckets-"+container_type, 1)
4168 
4169     def increment(self, d, k, delta=1):
4170         if k not in d:
4171hunk ./src/allmydata/storage/expirer.py 281
4172         # copy() needs to become a deepcopy
4173         h["space-recovered"] = s["space-recovered"].copy()
4174 
4175-        history = pickle.load(open(self.historyfile, "rb"))
4176+        history = pickle.load(self.historyfp.getContent())
4177         history[cycle] = h
4178         while len(history) > 10:
4179             oldcycles = sorted(history.keys())
4180hunk ./src/allmydata/storage/expirer.py 286
4181             del history[oldcycles[0]]
4182-        f = open(self.historyfile, "wb")
4183-        pickle.dump(history, f)
4184-        f.close()
4185+        self.historyfp.setContent(pickle.dumps(history))
4186 
4187     def get_state(self):
4188         """In addition to the crawler state described in
4189hunk ./src/allmydata/storage/expirer.py 355
4190         progress = self.get_progress()
4191 
4192         state = ShareCrawler.get_state(self) # does a shallow copy
4193-        history = pickle.load(open(self.historyfile, "rb"))
4194+        history = pickle.load(self.historyfp.getContent())
4195         state["history"] = history
4196 
4197         if not progress["cycle-in-progress"]:
4198hunk ./src/allmydata/storage/lease.py 3
4199 import struct, time
4200 
4201+
4202+class NonExistentLeaseError(Exception):
4203+    pass
4204+
4205 class LeaseInfo:
4206     def __init__(self, owner_num=None, renew_secret=None, cancel_secret=None,
4207                  expiration_time=None, nodeid=None):
4208hunk ./src/allmydata/storage/lease.py 21
4209 
4210     def get_expiration_time(self):
4211         return self.expiration_time
4212+
4213     def get_grant_renew_time_time(self):
4214         # hack, based upon fixed 31day expiration period
4215         return self.expiration_time - 31*24*60*60
4216hunk ./src/allmydata/storage/lease.py 25
4217+
4218     def get_age(self):
4219         return time.time() - self.get_grant_renew_time_time()
4220 
4221hunk ./src/allmydata/storage/lease.py 36
4222          self.expiration_time) = struct.unpack(">L32s32sL", data)
4223         self.nodeid = None
4224         return self
4225+
4226     def to_immutable_data(self):
4227         return struct.pack(">L32s32sL",
4228                            self.owner_num,
4229hunk ./src/allmydata/storage/lease.py 49
4230                            int(self.expiration_time),
4231                            self.renew_secret, self.cancel_secret,
4232                            self.nodeid)
4233+
4234     def from_mutable_data(self, data):
4235         (self.owner_num,
4236          self.expiration_time,
4237hunk ./src/allmydata/storage/server.py 1
4238-import os, re, weakref, struct, time
4239+import weakref, time
4240 
4241 from foolscap.api import Referenceable
4242 from twisted.application import service
4243hunk ./src/allmydata/storage/server.py 7
4244 
4245 from zope.interface import implements
4246-from allmydata.interfaces import RIStorageServer, IStatsProducer
4247-from allmydata.util import fileutil, idlib, log, time_format
4248+from allmydata.interfaces import RIStorageServer, IStatsProducer, IStorageBackend
4249+from allmydata.util.assertutil import precondition
4250+from allmydata.util import idlib, log
4251 import allmydata # for __full_version__
4252 
4253hunk ./src/allmydata/storage/server.py 12
4254-from allmydata.storage.common import si_b2a, si_a2b, storage_index_to_dir
4255-_pyflakes_hush = [si_b2a, si_a2b, storage_index_to_dir] # re-exported
4256+from allmydata.storage.common import si_a2b, si_b2a
4257+[si_a2b]  # hush pyflakes
4258 from allmydata.storage.lease import LeaseInfo
4259hunk ./src/allmydata/storage/server.py 15
4260-from allmydata.storage.mutable import MutableShareFile, EmptyShare, \
4261-     create_mutable_sharefile
4262-from allmydata.storage.immutable import ShareFile, BucketWriter, BucketReader
4263-from allmydata.storage.crawler import BucketCountingCrawler
4264 from allmydata.storage.expirer import LeaseCheckingCrawler
4265hunk ./src/allmydata/storage/server.py 16
4266-
4267-# storage/
4268-# storage/shares/incoming
4269-#   incoming/ holds temp dirs named $START/$STORAGEINDEX/$SHARENUM which will
4270-#   be moved to storage/shares/$START/$STORAGEINDEX/$SHARENUM upon success
4271-# storage/shares/$START/$STORAGEINDEX
4272-# storage/shares/$START/$STORAGEINDEX/$SHARENUM
4273-
4274-# Where "$START" denotes the first 10 bits worth of $STORAGEINDEX (that's 2
4275-# base-32 chars).
4276-
4277-# $SHARENUM matches this regex:
4278-NUM_RE=re.compile("^[0-9]+$")
4279-
4280+from allmydata.storage.crawler import BucketCountingCrawler
4281 
4282 
4283 class StorageServer(service.MultiService, Referenceable):
4284hunk ./src/allmydata/storage/server.py 21
4285     implements(RIStorageServer, IStatsProducer)
4286+
4287     name = 'storage'
4288     LeaseCheckerClass = LeaseCheckingCrawler
4289hunk ./src/allmydata/storage/server.py 24
4290+    DEFAULT_EXPIRATION_POLICY = {
4291+        'enabled': False,
4292+        'mode': 'age',
4293+        'override_lease_duration': None,
4294+        'cutoff_date': None,
4295+        'sharetypes': ('mutable', 'immutable'),
4296+    }
4297 
4298hunk ./src/allmydata/storage/server.py 32
4299-    def __init__(self, storedir, nodeid, reserved_space=0,
4300-                 discard_storage=False, readonly_storage=False,
4301+    def __init__(self, serverid, backend, statedir,
4302                  stats_provider=None,
4303hunk ./src/allmydata/storage/server.py 34
4304-                 expiration_enabled=False,
4305-                 expiration_mode="age",
4306-                 expiration_override_lease_duration=None,
4307-                 expiration_cutoff_date=None,
4308-                 expiration_sharetypes=("mutable", "immutable")):
4309+                 expiration_policy=None):
4310         service.MultiService.__init__(self)
4311hunk ./src/allmydata/storage/server.py 36
4312-        assert isinstance(nodeid, str)
4313-        assert len(nodeid) == 20
4314-        self.my_nodeid = nodeid
4315-        self.storedir = storedir
4316-        sharedir = os.path.join(storedir, "shares")
4317-        fileutil.make_dirs(sharedir)
4318-        self.sharedir = sharedir
4319-        # we don't actually create the corruption-advisory dir until necessary
4320-        self.corruption_advisory_dir = os.path.join(storedir,
4321-                                                    "corruption-advisories")
4322-        self.reserved_space = int(reserved_space)
4323-        self.no_storage = discard_storage
4324-        self.readonly_storage = readonly_storage
4325+        precondition(IStorageBackend.providedBy(backend), backend)
4326+        precondition(isinstance(serverid, str), serverid)
4327+        precondition(len(serverid) == 20, serverid)
4328+
4329+        self._serverid = serverid
4330         self.stats_provider = stats_provider
4331         if self.stats_provider:
4332             self.stats_provider.register_producer(self)
4333hunk ./src/allmydata/storage/server.py 44
4334-        self.incomingdir = os.path.join(sharedir, 'incoming')
4335-        self._clean_incomplete()
4336-        fileutil.make_dirs(self.incomingdir)
4337         self._active_writers = weakref.WeakKeyDictionary()
4338hunk ./src/allmydata/storage/server.py 45
4339+        self.backend = backend
4340+        self.backend.setServiceParent(self)
4341+        self._statedir = statedir
4342         log.msg("StorageServer created", facility="tahoe.storage")
4343 
4344hunk ./src/allmydata/storage/server.py 50
4345-        if reserved_space:
4346-            if self.get_available_space() is None:
4347-                log.msg("warning: [storage]reserved_space= is set, but this platform does not support an API to get disk statistics (statvfs(2) or GetDiskFreeSpaceEx), so this reservation cannot be honored",
4348-                        umin="0wZ27w", level=log.UNUSUAL)
4349-
4350         self.latencies = {"allocate": [], # immutable
4351                           "write": [],
4352                           "close": [],
4353hunk ./src/allmydata/storage/server.py 61
4354                           "renew": [],
4355                           "cancel": [],
4356                           }
4357-        self.add_bucket_counter()
4358-
4359-        statefile = os.path.join(self.storedir, "lease_checker.state")
4360-        historyfile = os.path.join(self.storedir, "lease_checker.history")
4361-        klass = self.LeaseCheckerClass
4362-        self.lease_checker = klass(self, statefile, historyfile,
4363-                                   expiration_enabled, expiration_mode,
4364-                                   expiration_override_lease_duration,
4365-                                   expiration_cutoff_date,
4366-                                   expiration_sharetypes)
4367-        self.lease_checker.setServiceParent(self)
4368+        self._setup_bucket_counter()
4369+        self._setup_lease_checker(expiration_policy or self.DEFAULT_EXPIRATION_POLICY)
4370 
4371     def __repr__(self):
4372hunk ./src/allmydata/storage/server.py 65
4373-        return "<StorageServer %s>" % (idlib.shortnodeid_b2a(self.my_nodeid),)
4374+        return "<StorageServer %s>" % (idlib.shortnodeid_b2a(self._serverid),)
4375 
4376hunk ./src/allmydata/storage/server.py 67
4377-    def add_bucket_counter(self):
4378-        statefile = os.path.join(self.storedir, "bucket_counter.state")
4379-        self.bucket_counter = BucketCountingCrawler(self, statefile)
4380+    def _setup_bucket_counter(self):
4381+        statefp = self._statedir.child("bucket_counter.state")
4382+        self.bucket_counter = BucketCountingCrawler(self.backend, statefp)
4383         self.bucket_counter.setServiceParent(self)
4384 
4385hunk ./src/allmydata/storage/server.py 72
4386+    def _setup_lease_checker(self, expiration_policy):
4387+        statefp = self._statedir.child("lease_checker.state")
4388+        historyfp = self._statedir.child("lease_checker.history")
4389+        self.lease_checker = self.LeaseCheckerClass(self.backend, statefp, historyfp, expiration_policy)
4390+        self.lease_checker.setServiceParent(self)
4391+
4392     def count(self, name, delta=1):
4393         if self.stats_provider:
4394             self.stats_provider.count("storage_server." + name, delta)
4395hunk ./src/allmydata/storage/server.py 92
4396         """Return a dict, indexed by category, that contains a dict of
4397         latency numbers for each category. If there are sufficient samples
4398         for unambiguous interpretation, each dict will contain the
4399-        following keys: mean, 01_0_percentile, 10_0_percentile,
4400+        following keys: samplesize, mean, 01_0_percentile, 10_0_percentile,
4401         50_0_percentile (median), 90_0_percentile, 95_0_percentile,
4402         99_0_percentile, 99_9_percentile.  If there are insufficient
4403         samples for a given percentile to be interpreted unambiguously
4404hunk ./src/allmydata/storage/server.py 114
4405             else:
4406                 stats["mean"] = None
4407 
4408-            orderstatlist = [(0.01, "01_0_percentile", 100), (0.1, "10_0_percentile", 10),\
4409-                             (0.50, "50_0_percentile", 10), (0.90, "90_0_percentile", 10),\
4410-                             (0.95, "95_0_percentile", 20), (0.99, "99_0_percentile", 100),\
4411+            orderstatlist = [(0.1, "10_0_percentile", 10), (0.5, "50_0_percentile", 10), \
4412+                             (0.9, "90_0_percentile", 10), (0.95, "95_0_percentile", 20), \
4413+                             (0.01, "01_0_percentile", 100),  (0.99, "99_0_percentile", 100),\
4414                              (0.999, "99_9_percentile", 1000)]
4415 
4416             for percentile, percentilestring, minnumtoobserve in orderstatlist:
4417hunk ./src/allmydata/storage/server.py 133
4418             kwargs["facility"] = "tahoe.storage"
4419         return log.msg(*args, **kwargs)
4420 
4421-    def _clean_incomplete(self):
4422-        fileutil.rm_dir(self.incomingdir)
4423+    def get_serverid(self):
4424+        return self._serverid
4425 
4426     def get_stats(self):
4427         # remember: RIStatsProvider requires that our return dict
4428hunk ./src/allmydata/storage/server.py 138
4429-        # contains numeric values.
4430+        # contains numeric, or None values.
4431         stats = { 'storage_server.allocated': self.allocated_size(), }
4432hunk ./src/allmydata/storage/server.py 140
4433-        stats['storage_server.reserved_space'] = self.reserved_space
4434         for category,ld in self.get_latencies().items():
4435             for name,v in ld.items():
4436                 stats['storage_server.latencies.%s.%s' % (category, name)] = v
4437hunk ./src/allmydata/storage/server.py 144
4438 
4439-        try:
4440-            disk = fileutil.get_disk_stats(self.sharedir, self.reserved_space)
4441-            writeable = disk['avail'] > 0
4442-
4443-            # spacetime predictors should use disk_avail / (d(disk_used)/dt)
4444-            stats['storage_server.disk_total'] = disk['total']
4445-            stats['storage_server.disk_used'] = disk['used']
4446-            stats['storage_server.disk_free_for_root'] = disk['free_for_root']
4447-            stats['storage_server.disk_free_for_nonroot'] = disk['free_for_nonroot']
4448-            stats['storage_server.disk_avail'] = disk['avail']
4449-        except AttributeError:
4450-            writeable = True
4451-        except EnvironmentError:
4452-            log.msg("OS call to get disk statistics failed", level=log.UNUSUAL)
4453-            writeable = False
4454-
4455-        if self.readonly_storage:
4456-            stats['storage_server.disk_avail'] = 0
4457-            writeable = False
4458+        self.backend.fill_in_space_stats(stats)
4459 
4460hunk ./src/allmydata/storage/server.py 146
4461-        stats['storage_server.accepting_immutable_shares'] = int(writeable)
4462         s = self.bucket_counter.get_state()
4463         bucket_count = s.get("last-complete-bucket-count")
4464         if bucket_count:
4465hunk ./src/allmydata/storage/server.py 153
4466         return stats
4467 
4468     def get_available_space(self):
4469-        """Returns available space for share storage in bytes, or None if no
4470-        API to get this information is available."""
4471-
4472-        if self.readonly_storage:
4473-            return 0
4474-        return fileutil.get_available_space(self.sharedir, self.reserved_space)
4475+        return self.backend.get_available_space()
4476 
4477     def allocated_size(self):
4478         space = 0
4479hunk ./src/allmydata/storage/server.py 162
4480         return space
4481 
4482     def remote_get_version(self):
4483-        remaining_space = self.get_available_space()
4484+        remaining_space = self.backend.get_available_space()
4485         if remaining_space is None:
4486             # We're on a platform that has no API to get disk stats.
4487             remaining_space = 2**64
4488hunk ./src/allmydata/storage/server.py 178
4489                     }
4490         return version
4491 
4492-    def remote_allocate_buckets(self, storage_index,
4493+    def remote_allocate_buckets(self, storageindex,
4494                                 renew_secret, cancel_secret,
4495                                 sharenums, allocated_size,
4496                                 canary, owner_num=0):
4497hunk ./src/allmydata/storage/server.py 182
4498+        # cancel_secret is no longer used.
4499         # owner_num is not for clients to set, but rather it should be
4500hunk ./src/allmydata/storage/server.py 184
4501-        # curried into the PersonalStorageServer instance that is dedicated
4502-        # to a particular owner.
4503+        # curried into a StorageServer instance dedicated to a particular
4504+        # owner.
4505         start = time.time()
4506         self.count("allocate")
4507hunk ./src/allmydata/storage/server.py 188
4508-        alreadygot = set()
4509         bucketwriters = {} # k: shnum, v: BucketWriter
4510hunk ./src/allmydata/storage/server.py 189
4511-        si_dir = storage_index_to_dir(storage_index)
4512-        si_s = si_b2a(storage_index)
4513 
4514hunk ./src/allmydata/storage/server.py 190
4515+        si_s = si_b2a(storageindex)
4516         log.msg("storage: allocate_buckets %s" % si_s)
4517 
4518hunk ./src/allmydata/storage/server.py 193
4519-        # in this implementation, the lease information (including secrets)
4520-        # goes into the share files themselves. It could also be put into a
4521-        # separate database. Note that the lease should not be added until
4522-        # the BucketWriter has been closed.
4523+        # Note that the lease should not be added until the BucketWriter
4524+        # has been closed.
4525         expire_time = time.time() + 31*24*60*60
4526hunk ./src/allmydata/storage/server.py 196
4527-        lease_info = LeaseInfo(owner_num,
4528-                               renew_secret, cancel_secret,
4529-                               expire_time, self.my_nodeid)
4530+        lease_info = LeaseInfo(owner_num, renew_secret,
4531+                               expire_time, self._serverid)
4532 
4533         max_space_per_bucket = allocated_size
4534 
4535hunk ./src/allmydata/storage/server.py 201
4536-        remaining_space = self.get_available_space()
4537+        remaining_space = self.backend.get_available_space()
4538         limited = remaining_space is not None
4539         if limited:
4540hunk ./src/allmydata/storage/server.py 204
4541-            # this is a bit conservative, since some of this allocated_size()
4542-            # has already been written to disk, where it will show up in
4543+            # This is a bit conservative, since some of this allocated_size()
4544+            # has already been written to the backend, where it will show up in
4545             # get_available_space.
4546             remaining_space -= self.allocated_size()
4547hunk ./src/allmydata/storage/server.py 208
4548-        # self.readonly_storage causes remaining_space <= 0
4549+            # If the backend is read-only, remaining_space will be <= 0.
4550+
4551+        shareset = self.backend.get_shareset(storageindex)
4552 
4553hunk ./src/allmydata/storage/server.py 212
4554-        # fill alreadygot with all shares that we have, not just the ones
4555+        # Fill alreadygot with all shares that we have, not just the ones
4556         # they asked about: this will save them a lot of work. Add or update
4557         # leases for all of them: if they want us to hold shares for this
4558hunk ./src/allmydata/storage/server.py 215
4559-        # file, they'll want us to hold leases for this file.
4560-        for (shnum, fn) in self._get_bucket_shares(storage_index):
4561-            alreadygot.add(shnum)
4562-            sf = ShareFile(fn)
4563-            sf.add_or_renew_lease(lease_info)
4564+        # file, they'll want us to hold leases for all the shares of it.
4565+        #
4566+        # XXX should we be making the assumption here that lease info is
4567+        # duplicated in all shares?
4568+        alreadygot = set()
4569+        for share in shareset.get_shares():
4570+            share.add_or_renew_lease(lease_info)
4571+            alreadygot.add(share.shnum)
4572 
4573hunk ./src/allmydata/storage/server.py 224
4574-        for shnum in sharenums:
4575-            incominghome = os.path.join(self.incomingdir, si_dir, "%d" % shnum)
4576-            finalhome = os.path.join(self.sharedir, si_dir, "%d" % shnum)
4577-            if os.path.exists(finalhome):
4578-                # great! we already have it. easy.
4579-                pass
4580-            elif os.path.exists(incominghome):
4581+        for shnum in sharenums - alreadygot:
4582+            if shareset.has_incoming(shnum):
4583                 # Note that we don't create BucketWriters for shnums that
4584                 # have a partial share (in incoming/), so if a second upload
4585                 # occurs while the first is still in progress, the second
4586hunk ./src/allmydata/storage/server.py 232
4587                 # uploader will use different storage servers.
4588                 pass
4589             elif (not limited) or (remaining_space >= max_space_per_bucket):
4590-                # ok! we need to create the new share file.
4591-                bw = BucketWriter(self, incominghome, finalhome,
4592-                                  max_space_per_bucket, lease_info, canary)
4593-                if self.no_storage:
4594-                    bw.throw_out_all_data = True
4595+                bw = shareset.make_bucket_writer(self, shnum, max_space_per_bucket,
4596+                                                 lease_info, canary)
4597                 bucketwriters[shnum] = bw
4598                 self._active_writers[bw] = 1
4599                 if limited:
4600hunk ./src/allmydata/storage/server.py 239
4601                     remaining_space -= max_space_per_bucket
4602             else:
4603-                # bummer! not enough space to accept this bucket
4604+                # Bummer not enough space to accept this share.
4605                 pass
4606 
4607hunk ./src/allmydata/storage/server.py 242
4608-        if bucketwriters:
4609-            fileutil.make_dirs(os.path.join(self.sharedir, si_dir))
4610-
4611         self.add_latency("allocate", time.time() - start)
4612         return alreadygot, bucketwriters
4613 
4614hunk ./src/allmydata/storage/server.py 245
4615-    def _iter_share_files(self, storage_index):
4616-        for shnum, filename in self._get_bucket_shares(storage_index):
4617-            f = open(filename, 'rb')
4618-            header = f.read(32)
4619-            f.close()
4620-            if header[:32] == MutableShareFile.MAGIC:
4621-                sf = MutableShareFile(filename, self)
4622-                # note: if the share has been migrated, the renew_lease()
4623-                # call will throw an exception, with information to help the
4624-                # client update the lease.
4625-            elif header[:4] == struct.pack(">L", 1):
4626-                sf = ShareFile(filename)
4627-            else:
4628-                continue # non-sharefile
4629-            yield sf
4630-
4631-    def remote_add_lease(self, storage_index, renew_secret, cancel_secret,
4632+    def remote_add_lease(self, storageindex, renew_secret, cancel_secret,
4633                          owner_num=1):
4634hunk ./src/allmydata/storage/server.py 247
4635+        # cancel_secret is no longer used.
4636         start = time.time()
4637         self.count("add-lease")
4638         new_expire_time = time.time() + 31*24*60*60
4639hunk ./src/allmydata/storage/server.py 251
4640-        lease_info = LeaseInfo(owner_num,
4641-                               renew_secret, cancel_secret,
4642-                               new_expire_time, self.my_nodeid)
4643-        for sf in self._iter_share_files(storage_index):
4644-            sf.add_or_renew_lease(lease_info)
4645-        self.add_latency("add-lease", time.time() - start)
4646-        return None
4647+        lease_info = LeaseInfo(owner_num, renew_secret,
4648+                               new_expire_time, self._serverid)
4649 
4650hunk ./src/allmydata/storage/server.py 254
4651-    def remote_renew_lease(self, storage_index, renew_secret):
4652+        try:
4653+            self.backend.add_or_renew_lease(lease_info)
4654+        finally:
4655+            self.add_latency("add-lease", time.time() - start)
4656+
4657+    def remote_renew_lease(self, storageindex, renew_secret):
4658         start = time.time()
4659         self.count("renew")
4660hunk ./src/allmydata/storage/server.py 262
4661-        new_expire_time = time.time() + 31*24*60*60
4662-        found_buckets = False
4663-        for sf in self._iter_share_files(storage_index):
4664-            found_buckets = True
4665-            sf.renew_lease(renew_secret, new_expire_time)
4666-        self.add_latency("renew", time.time() - start)
4667-        if not found_buckets:
4668-            raise IndexError("no such lease to renew")
4669+
4670+        try:
4671+            shareset = self.backend.get_shareset(storageindex)
4672+            new_expiration_time = start + 31*24*60*60   # one month from now
4673+            shareset.renew_lease(renew_secret, new_expiration_time)
4674+        finally:
4675+            self.add_latency("renew", time.time() - start)
4676 
4677     def bucket_writer_closed(self, bw, consumed_size):
4678         if self.stats_provider:
4679hunk ./src/allmydata/storage/server.py 275
4680             self.stats_provider.count('storage_server.bytes_added', consumed_size)
4681         del self._active_writers[bw]
4682 
4683-    def _get_bucket_shares(self, storage_index):
4684-        """Return a list of (shnum, pathname) tuples for files that hold
4685-        shares for this storage_index. In each tuple, 'shnum' will always be
4686-        the integer form of the last component of 'pathname'."""
4687-        storagedir = os.path.join(self.sharedir, storage_index_to_dir(storage_index))
4688-        try:
4689-            for f in os.listdir(storagedir):
4690-                if NUM_RE.match(f):
4691-                    filename = os.path.join(storagedir, f)
4692-                    yield (int(f), filename)
4693-        except OSError:
4694-            # Commonly caused by there being no buckets at all.
4695-            pass
4696-
4697-    def remote_get_buckets(self, storage_index):
4698+    def remote_get_buckets(self, storageindex):
4699         start = time.time()
4700         self.count("get")
4701hunk ./src/allmydata/storage/server.py 278
4702-        si_s = si_b2a(storage_index)
4703+        si_s = si_b2a(storageindex)
4704         log.msg("storage: get_buckets %s" % si_s)
4705         bucketreaders = {} # k: sharenum, v: BucketReader
4706hunk ./src/allmydata/storage/server.py 281
4707-        for shnum, filename in self._get_bucket_shares(storage_index):
4708-            bucketreaders[shnum] = BucketReader(self, filename,
4709-                                                storage_index, shnum)
4710-        self.add_latency("get", time.time() - start)
4711-        return bucketreaders
4712 
4713hunk ./src/allmydata/storage/server.py 282
4714-    def get_leases(self, storage_index):
4715-        """Provide an iterator that yields all of the leases attached to this
4716-        bucket. Each lease is returned as a LeaseInfo instance.
4717+        try:
4718+            shareset = self.backend.get_shareset(storageindex)
4719+            for share in shareset.get_shares():
4720+                bucketreaders[share.get_shnum()] = shareset.make_bucket_reader(self, share)
4721+            return bucketreaders
4722+        finally:
4723+            self.add_latency("get", time.time() - start)
4724 
4725hunk ./src/allmydata/storage/server.py 290
4726-        This method is not for client use.
4727+    def get_leases(self, storageindex):
4728         """
4729hunk ./src/allmydata/storage/server.py 292
4730+        Provide an iterator that yields all of the leases attached to this
4731+        bucket. Each lease is returned as a LeaseInfo instance.
4732 
4733hunk ./src/allmydata/storage/server.py 295
4734-        # since all shares get the same lease data, we just grab the leases
4735-        # from the first share
4736-        try:
4737-            shnum, filename = self._get_bucket_shares(storage_index).next()
4738-            sf = ShareFile(filename)
4739-            return sf.get_leases()
4740-        except StopIteration:
4741-            return iter([])
4742+        This method is not for client use. XXX do we need it at all?
4743+        """
4744+        return self.backend.get_shareset(storageindex).get_leases()
4745 
4746hunk ./src/allmydata/storage/server.py 299
4747-    def remote_slot_testv_and_readv_and_writev(self, storage_index,
4748+    def remote_slot_testv_and_readv_and_writev(self, storageindex,
4749                                                secrets,
4750                                                test_and_write_vectors,
4751                                                read_vector):
4752hunk ./src/allmydata/storage/server.py 305
4753         start = time.time()
4754         self.count("writev")
4755-        si_s = si_b2a(storage_index)
4756+        si_s = si_b2a(storageindex)
4757         log.msg("storage: slot_writev %s" % si_s)
4758hunk ./src/allmydata/storage/server.py 307
4759-        si_dir = storage_index_to_dir(storage_index)
4760-        (write_enabler, renew_secret, cancel_secret) = secrets
4761-        # shares exist if there is a file for them
4762-        bucketdir = os.path.join(self.sharedir, si_dir)
4763-        shares = {}
4764-        if os.path.isdir(bucketdir):
4765-            for sharenum_s in os.listdir(bucketdir):
4766-                try:
4767-                    sharenum = int(sharenum_s)
4768-                except ValueError:
4769-                    continue
4770-                filename = os.path.join(bucketdir, sharenum_s)
4771-                msf = MutableShareFile(filename, self)
4772-                msf.check_write_enabler(write_enabler, si_s)
4773-                shares[sharenum] = msf
4774-        # write_enabler is good for all existing shares.
4775-
4776-        # Now evaluate test vectors.
4777-        testv_is_good = True
4778-        for sharenum in test_and_write_vectors:
4779-            (testv, datav, new_length) = test_and_write_vectors[sharenum]
4780-            if sharenum in shares:
4781-                if not shares[sharenum].check_testv(testv):
4782-                    self.log("testv failed: [%d]: %r" % (sharenum, testv))
4783-                    testv_is_good = False
4784-                    break
4785-            else:
4786-                # compare the vectors against an empty share, in which all
4787-                # reads return empty strings.
4788-                if not EmptyShare().check_testv(testv):
4789-                    self.log("testv failed (empty): [%d] %r" % (sharenum,
4790-                                                                testv))
4791-                    testv_is_good = False
4792-                    break
4793-
4794-        # now gather the read vectors, before we do any writes
4795-        read_data = {}
4796-        for sharenum, share in shares.items():
4797-            read_data[sharenum] = share.readv(read_vector)
4798-
4799-        ownerid = 1 # TODO
4800-        expire_time = time.time() + 31*24*60*60   # one month
4801-        lease_info = LeaseInfo(ownerid,
4802-                               renew_secret, cancel_secret,
4803-                               expire_time, self.my_nodeid)
4804-
4805-        if testv_is_good:
4806-            # now apply the write vectors
4807-            for sharenum in test_and_write_vectors:
4808-                (testv, datav, new_length) = test_and_write_vectors[sharenum]
4809-                if new_length == 0:
4810-                    if sharenum in shares:
4811-                        shares[sharenum].unlink()
4812-                else:
4813-                    if sharenum not in shares:
4814-                        # allocate a new share
4815-                        allocated_size = 2000 # arbitrary, really
4816-                        share = self._allocate_slot_share(bucketdir, secrets,
4817-                                                          sharenum,
4818-                                                          allocated_size,
4819-                                                          owner_num=0)
4820-                        shares[sharenum] = share
4821-                    shares[sharenum].writev(datav, new_length)
4822-                    # and update the lease
4823-                    shares[sharenum].add_or_renew_lease(lease_info)
4824-
4825-            if new_length == 0:
4826-                # delete empty bucket directories
4827-                if not os.listdir(bucketdir):
4828-                    os.rmdir(bucketdir)
4829 
4830hunk ./src/allmydata/storage/server.py 308
4831+        try:
4832+            shareset = self.backend.get_shareset(storageindex)
4833+            expiration_time = start + 31*24*60*60   # one month from now
4834+            return shareset.testv_and_readv_and_writev(self, secrets, test_and_write_vectors,
4835+                                                       read_vector, expiration_time)
4836+        finally:
4837+            self.add_latency("writev", time.time() - start)
4838 
4839hunk ./src/allmydata/storage/server.py 316
4840-        # all done
4841-        self.add_latency("writev", time.time() - start)
4842-        return (testv_is_good, read_data)
4843-
4844-    def _allocate_slot_share(self, bucketdir, secrets, sharenum,
4845-                             allocated_size, owner_num=0):
4846-        (write_enabler, renew_secret, cancel_secret) = secrets
4847-        my_nodeid = self.my_nodeid
4848-        fileutil.make_dirs(bucketdir)
4849-        filename = os.path.join(bucketdir, "%d" % sharenum)
4850-        share = create_mutable_sharefile(filename, my_nodeid, write_enabler,
4851-                                         self)
4852-        return share
4853-
4854-    def remote_slot_readv(self, storage_index, shares, readv):
4855+    def remote_slot_readv(self, storageindex, shares, readv):
4856         start = time.time()
4857         self.count("readv")
4858hunk ./src/allmydata/storage/server.py 319
4859-        si_s = si_b2a(storage_index)
4860-        lp = log.msg("storage: slot_readv %s %s" % (si_s, shares),
4861-                     facility="tahoe.storage", level=log.OPERATIONAL)
4862-        si_dir = storage_index_to_dir(storage_index)
4863-        # shares exist if there is a file for them
4864-        bucketdir = os.path.join(self.sharedir, si_dir)
4865-        if not os.path.isdir(bucketdir):
4866+        si_s = si_b2a(storageindex)
4867+        log.msg("storage: slot_readv %s %s" % (si_s, shares),
4868+                facility="tahoe.storage", level=log.OPERATIONAL)
4869+
4870+        try:
4871+            shareset = self.backend.get_shareset(storageindex)
4872+            return shareset.readv(self, shares, readv)
4873+        finally:
4874             self.add_latency("readv", time.time() - start)
4875hunk ./src/allmydata/storage/server.py 328
4876-            return {}
4877-        datavs = {}
4878-        for sharenum_s in os.listdir(bucketdir):
4879-            try:
4880-                sharenum = int(sharenum_s)
4881-            except ValueError:
4882-                continue
4883-            if sharenum in shares or not shares:
4884-                filename = os.path.join(bucketdir, sharenum_s)
4885-                msf = MutableShareFile(filename, self)
4886-                datavs[sharenum] = msf.readv(readv)
4887-        log.msg("returning shares %s" % (datavs.keys(),),
4888-                facility="tahoe.storage", level=log.NOISY, parent=lp)
4889-        self.add_latency("readv", time.time() - start)
4890-        return datavs
4891 
4892hunk ./src/allmydata/storage/server.py 329
4893-    def remote_advise_corrupt_share(self, share_type, storage_index, shnum,
4894-                                    reason):
4895-        fileutil.make_dirs(self.corruption_advisory_dir)
4896-        now = time_format.iso_utc(sep="T")
4897-        si_s = si_b2a(storage_index)
4898-        # windows can't handle colons in the filename
4899-        fn = os.path.join(self.corruption_advisory_dir,
4900-                          "%s--%s-%d" % (now, si_s, shnum)).replace(":","")
4901-        f = open(fn, "w")
4902-        f.write("report: Share Corruption\n")
4903-        f.write("type: %s\n" % share_type)
4904-        f.write("storage_index: %s\n" % si_s)
4905-        f.write("share_number: %d\n" % shnum)
4906-        f.write("\n")
4907-        f.write(reason)
4908-        f.write("\n")
4909-        f.close()
4910-        log.msg(format=("client claims corruption in (%(share_type)s) " +
4911-                        "%(si)s-%(shnum)d: %(reason)s"),
4912-                share_type=share_type, si=si_s, shnum=shnum, reason=reason,
4913-                level=log.SCARY, umid="SGx2fA")
4914-        return None
4915+    def remote_advise_corrupt_share(self, share_type, storage_index, shnum, reason):
4916+        self.backend.advise_corrupt_share(share_type, storage_index, shnum, reason)
4917hunk ./src/allmydata/test/common.py 20
4918 from allmydata.mutable.common import CorruptShareError
4919 from allmydata.mutable.layout import unpack_header
4920 from allmydata.mutable.publish import MutableData
4921-from allmydata.storage.mutable import MutableShareFile
4922+from allmydata.storage.backends.disk.mutable import MutableDiskShare
4923 from allmydata.util import hashutil, log, fileutil, pollmixin
4924 from allmydata.util.assertutil import precondition
4925 from allmydata.util.consumer import download_to_data
4926hunk ./src/allmydata/test/common.py 1297
4927 
4928 def _corrupt_mutable_share_data(data, debug=False):
4929     prefix = data[:32]
4930-    assert prefix == MutableShareFile.MAGIC, "This function is designed to corrupt mutable shares of v1, and the magic number doesn't look right: %r vs %r" % (prefix, MutableShareFile.MAGIC)
4931-    data_offset = MutableShareFile.DATA_OFFSET
4932+    assert prefix == MutableDiskShare.MAGIC, "This function is designed to corrupt mutable shares of v1, and the magic number doesn't look right: %r vs %r" % (prefix, MutableDiskShare.MAGIC)
4933+    data_offset = MutableDiskShare.DATA_OFFSET
4934     sharetype = data[data_offset:data_offset+1]
4935     assert sharetype == "\x00", "non-SDMF mutable shares not supported"
4936     (version, ig_seqnum, ig_roothash, ig_IV, ig_k, ig_N, ig_segsize,
4937hunk ./src/allmydata/test/no_network.py 21
4938 from twisted.application import service
4939 from twisted.internet import defer, reactor
4940 from twisted.python.failure import Failure
4941+from twisted.python.filepath import FilePath
4942 from foolscap.api import Referenceable, fireEventually, RemoteException
4943 from base64 import b32encode
4944hunk ./src/allmydata/test/no_network.py 24
4945+
4946 from allmydata import uri as tahoe_uri
4947 from allmydata.client import Client
4948hunk ./src/allmydata/test/no_network.py 27
4949-from allmydata.storage.server import StorageServer, storage_index_to_dir
4950+from allmydata.storage.server import StorageServer
4951+from allmydata.storage.backends.disk.disk_backend import DiskBackend
4952 from allmydata.util import fileutil, idlib, hashutil
4953 from allmydata.util.hashutil import sha1
4954 from allmydata.test.common_web import HTTPClientGETFactory
4955hunk ./src/allmydata/test/no_network.py 155
4956             seed = server.get_permutation_seed()
4957             return sha1(peer_selection_index + seed).digest()
4958         return sorted(self.get_connected_servers(), key=_permuted)
4959+
4960     def get_connected_servers(self):
4961         return self.client._servers
4962hunk ./src/allmydata/test/no_network.py 158
4963+
4964     def get_nickname_for_serverid(self, serverid):
4965         return None
4966 
4967hunk ./src/allmydata/test/no_network.py 162
4968+    def get_known_servers(self):
4969+        return self.get_connected_servers()
4970+
4971+    def get_all_serverids(self):
4972+        return self.client.get_all_serverids()
4973+
4974+
4975 class NoNetworkClient(Client):
4976     def create_tub(self):
4977         pass
4978hunk ./src/allmydata/test/no_network.py 262
4979 
4980     def make_server(self, i, readonly=False):
4981         serverid = hashutil.tagged_hash("serverid", str(i))[:20]
4982-        serverdir = os.path.join(self.basedir, "servers",
4983-                                 idlib.shortnodeid_b2a(serverid), "storage")
4984-        fileutil.make_dirs(serverdir)
4985-        ss = StorageServer(serverdir, serverid, stats_provider=SimpleStats(),
4986-                           readonly_storage=readonly)
4987+        storagedir = FilePath(self.basedir).child("servers").child(idlib.shortnodeid_b2a(serverid)).child("storage")
4988+
4989+        # The backend will make the storage directory and any necessary parents.
4990+        backend = DiskBackend(storagedir, readonly=readonly)
4991+        ss = StorageServer(serverid, backend, storagedir, stats_provider=SimpleStats())
4992         ss._no_network_server_number = i
4993         return ss
4994 
4995hunk ./src/allmydata/test/no_network.py 276
4996         middleman = service.MultiService()
4997         middleman.setServiceParent(self)
4998         ss.setServiceParent(middleman)
4999-        serverid = ss.my_nodeid
5000+        serverid = ss.get_serverid()
5001         self.servers_by_number[i] = ss
5002         wrapper = wrap_storage_server(ss)
5003         self.wrappers_by_id[serverid] = wrapper
5004hunk ./src/allmydata/test/no_network.py 295
5005         # it's enough to remove the server from c._servers (we don't actually
5006         # have to detach and stopService it)
5007         for i,ss in self.servers_by_number.items():
5008-            if ss.my_nodeid == serverid:
5009+            if ss.get_serverid() == serverid:
5010                 del self.servers_by_number[i]
5011                 break
5012         del self.wrappers_by_id[serverid]
5013hunk ./src/allmydata/test/no_network.py 345
5014     def get_clientdir(self, i=0):
5015         return self.g.clients[i].basedir
5016 
5017+    def get_server(self, i):
5018+        return self.g.servers_by_number[i]
5019+
5020     def get_serverdir(self, i):
5021hunk ./src/allmydata/test/no_network.py 349
5022-        return self.g.servers_by_number[i].storedir
5023+        return self.g.servers_by_number[i].backend.storedir
5024+
5025+    def remove_server(self, i):
5026+        self.g.remove_server(self.g.servers_by_number[i].get_serverid())
5027 
5028     def iterate_servers(self):
5029         for i in sorted(self.g.servers_by_number.keys()):
5030hunk ./src/allmydata/test/no_network.py 357
5031             ss = self.g.servers_by_number[i]
5032-            yield (i, ss, ss.storedir)
5033+            yield (i, ss, ss.backend.storedir)
5034 
5035     def find_uri_shares(self, uri):
5036         si = tahoe_uri.from_string(uri).get_storage_index()
5037hunk ./src/allmydata/test/no_network.py 361
5038-        prefixdir = storage_index_to_dir(si)
5039         shares = []
5040         for i,ss in self.g.servers_by_number.items():
5041hunk ./src/allmydata/test/no_network.py 363
5042-            serverid = ss.my_nodeid
5043-            basedir = os.path.join(ss.sharedir, prefixdir)
5044-            if not os.path.exists(basedir):
5045-                continue
5046-            for f in os.listdir(basedir):
5047-                try:
5048-                    shnum = int(f)
5049-                    shares.append((shnum, serverid, os.path.join(basedir, f)))
5050-                except ValueError:
5051-                    pass
5052+            for share in ss.backend.get_shareset(si).get_shares():
5053+                shares.append((share.get_shnum(), ss.get_serverid(), share._home))
5054         return sorted(shares)
5055 
5056hunk ./src/allmydata/test/no_network.py 367
5057+    def count_leases(self, uri):
5058+        """Return (filename, leasecount) pairs in arbitrary order."""
5059+        si = tahoe_uri.from_string(uri).get_storage_index()
5060+        lease_counts = []
5061+        for i,ss in self.g.servers_by_number.items():
5062+            for share in ss.backend.get_shareset(si).get_shares():
5063+                num_leases = len(list(share.get_leases()))
5064+                lease_counts.append( (share._home.path, num_leases) )
5065+        return lease_counts
5066+
5067     def copy_shares(self, uri):
5068         shares = {}
5069hunk ./src/allmydata/test/no_network.py 379
5070-        for (shnum, serverid, sharefile) in self.find_uri_shares(uri):
5071-            shares[sharefile] = open(sharefile, "rb").read()
5072+        for (shnum, serverid, sharefp) in self.find_uri_shares(uri):
5073+            shares[sharefp.path] = sharefp.getContent()
5074         return shares
5075 
5076hunk ./src/allmydata/test/no_network.py 383
5077+    def copy_share(self, from_share, uri, to_server):
5078+        si = uri.from_string(self.uri).get_storage_index()
5079+        (i_shnum, i_serverid, i_sharefp) = from_share
5080+        shares_dir = to_server.backend.get_shareset(si)._sharehomedir
5081+        i_sharefp.copyTo(shares_dir.child(str(i_shnum)))
5082+
5083     def restore_all_shares(self, shares):
5084hunk ./src/allmydata/test/no_network.py 390
5085-        for sharefile, data in shares.items():
5086-            open(sharefile, "wb").write(data)
5087+        for share, data in shares.items():
5088+            share.home.setContent(data)
5089 
5090hunk ./src/allmydata/test/no_network.py 393
5091-    def delete_share(self, (shnum, serverid, sharefile)):
5092-        os.unlink(sharefile)
5093+    def delete_share(self, (shnum, serverid, sharefp)):
5094+        sharefp.remove()
5095 
5096     def delete_shares_numbered(self, uri, shnums):
5097hunk ./src/allmydata/test/no_network.py 397
5098-        for (i_shnum, i_serverid, i_sharefile) in self.find_uri_shares(uri):
5099+        for (i_shnum, i_serverid, i_sharefp) in self.find_uri_shares(uri):
5100             if i_shnum in shnums:
5101hunk ./src/allmydata/test/no_network.py 399
5102-                os.unlink(i_sharefile)
5103+                i_sharefp.remove()
5104 
5105hunk ./src/allmydata/test/no_network.py 401
5106-    def corrupt_share(self, (shnum, serverid, sharefile), corruptor_function):
5107-        sharedata = open(sharefile, "rb").read()
5108-        corruptdata = corruptor_function(sharedata)
5109-        open(sharefile, "wb").write(corruptdata)
5110+    def corrupt_share(self, (shnum, serverid, sharefp), corruptor_function, debug=False):
5111+        sharedata = sharefp.getContent()
5112+        corruptdata = corruptor_function(sharedata, debug=debug)
5113+        sharefp.setContent(corruptdata)
5114 
5115     def corrupt_shares_numbered(self, uri, shnums, corruptor, debug=False):
5116hunk ./src/allmydata/test/no_network.py 407
5117-        for (i_shnum, i_serverid, i_sharefile) in self.find_uri_shares(uri):
5118+        for (i_shnum, i_serverid, i_sharefp) in self.find_uri_shares(uri):
5119             if i_shnum in shnums:
5120hunk ./src/allmydata/test/no_network.py 409
5121-                sharedata = open(i_sharefile, "rb").read()
5122-                corruptdata = corruptor(sharedata, debug=debug)
5123-                open(i_sharefile, "wb").write(corruptdata)
5124+                self.corrupt_share((i_shnum, i_serverid, i_sharefp), corruptor, debug=debug)
5125 
5126     def corrupt_all_shares(self, uri, corruptor, debug=False):
5127hunk ./src/allmydata/test/no_network.py 412
5128-        for (i_shnum, i_serverid, i_sharefile) in self.find_uri_shares(uri):
5129-            sharedata = open(i_sharefile, "rb").read()
5130-            corruptdata = corruptor(sharedata, debug=debug)
5131-            open(i_sharefile, "wb").write(corruptdata)
5132+        for (i_shnum, i_serverid, i_sharefp) in self.find_uri_shares(uri):
5133+            self.corrupt_share((i_shnum, i_serverid, i_sharefp), corruptor, debug=debug)
5134 
5135     def GET(self, urlpath, followRedirect=False, return_response=False,
5136             method="GET", clientnum=0, **kwargs):
5137hunk ./src/allmydata/test/test_download.py 6
5138 # a previous run. This asserts that the current code is capable of decoding
5139 # shares from a previous version.
5140 
5141-import os
5142 from twisted.trial import unittest
5143 from twisted.internet import defer, reactor
5144 from allmydata import uri
5145hunk ./src/allmydata/test/test_download.py 9
5146-from allmydata.storage.server import storage_index_to_dir
5147 from allmydata.util import base32, fileutil, spans, log, hashutil
5148 from allmydata.util.consumer import download_to_data, MemoryConsumer
5149 from allmydata.immutable import upload, layout
5150hunk ./src/allmydata/test/test_download.py 85
5151         u = upload.Data(plaintext, None)
5152         d = self.c0.upload(u)
5153         f = open("stored_shares.py", "w")
5154-        def _created_immutable(ur):
5155-            # write the generated shares and URI to a file, which can then be
5156-            # incorporated into this one next time.
5157-            f.write('immutable_uri = "%s"\n' % ur.uri)
5158-            f.write('immutable_shares = {\n')
5159-            si = uri.from_string(ur.uri).get_storage_index()
5160-            si_dir = storage_index_to_dir(si)
5161+
5162+        def _write_py(uri):
5163+            si = uri.from_string(uri).get_storage_index()
5164             for (i,ss,ssdir) in self.iterate_servers():
5165hunk ./src/allmydata/test/test_download.py 89
5166-                sharedir = os.path.join(ssdir, "shares", si_dir)
5167                 shares = {}
5168hunk ./src/allmydata/test/test_download.py 90
5169-                for fn in os.listdir(sharedir):
5170-                    shnum = int(fn)
5171-                    sharedata = open(os.path.join(sharedir, fn), "rb").read()
5172-                    shares[shnum] = sharedata
5173-                fileutil.rm_dir(sharedir)
5174+                shareset = ss.backend.get_shareset(si)
5175+                for share in shareset.get_shares():
5176+                    sharedata = share._home.getContent()
5177+                    shares[share.get_shnum()] = sharedata
5178+
5179+                fileutil.fp_remove(shareset._sharehomedir)
5180                 if shares:
5181                     f.write(' %d: { # client[%d]\n' % (i, i))
5182                     for shnum in sorted(shares.keys()):
5183hunk ./src/allmydata/test/test_download.py 103
5184                                 (shnum, base32.b2a(shares[shnum])))
5185                     f.write('    },\n')
5186             f.write('}\n')
5187-            f.write('\n')
5188 
5189hunk ./src/allmydata/test/test_download.py 104
5190+        def _created_immutable(ur):
5191+            # write the generated shares and URI to a file, which can then be
5192+            # incorporated into this one next time.
5193+            f.write('immutable_uri = "%s"\n' % ur.uri)
5194+            f.write('immutable_shares = {\n')
5195+            _write_py(ur.uri)
5196+            f.write('\n')
5197         d.addCallback(_created_immutable)
5198 
5199         d.addCallback(lambda ignored:
5200hunk ./src/allmydata/test/test_download.py 118
5201         def _created_mutable(n):
5202             f.write('mutable_uri = "%s"\n' % n.get_uri())
5203             f.write('mutable_shares = {\n')
5204-            si = uri.from_string(n.get_uri()).get_storage_index()
5205-            si_dir = storage_index_to_dir(si)
5206-            for (i,ss,ssdir) in self.iterate_servers():
5207-                sharedir = os.path.join(ssdir, "shares", si_dir)
5208-                shares = {}
5209-                for fn in os.listdir(sharedir):
5210-                    shnum = int(fn)
5211-                    sharedata = open(os.path.join(sharedir, fn), "rb").read()
5212-                    shares[shnum] = sharedata
5213-                fileutil.rm_dir(sharedir)
5214-                if shares:
5215-                    f.write(' %d: { # client[%d]\n' % (i, i))
5216-                    for shnum in sorted(shares.keys()):
5217-                        f.write('  %d: base32.a2b("%s"),\n' %
5218-                                (shnum, base32.b2a(shares[shnum])))
5219-                    f.write('    },\n')
5220-            f.write('}\n')
5221-
5222-            f.close()
5223+            _write_py(n.get_uri())
5224         d.addCallback(_created_mutable)
5225 
5226         def _done(ignored):
5227hunk ./src/allmydata/test/test_download.py 123
5228             f.close()
5229-        d.addCallback(_done)
5230+        d.addBoth(_done)
5231 
5232         return d
5233 
5234hunk ./src/allmydata/test/test_download.py 127
5235+    def _write_shares(self, uri, shares):
5236+        si = uri.from_string(uri).get_storage_index()
5237+        for i in shares:
5238+            shares_for_server = shares[i]
5239+            for shnum in shares_for_server:
5240+                share_dir = self.get_server(i).backend.get_shareset(si)._sharehomedir
5241+                fileutil.fp_make_dirs(share_dir)
5242+                share_dir.child(str(shnum)).setContent(shares[shnum])
5243+
5244     def load_shares(self, ignored=None):
5245         # this uses the data generated by create_shares() to populate the
5246         # storage servers with pre-generated shares
5247hunk ./src/allmydata/test/test_download.py 139
5248-        si = uri.from_string(immutable_uri).get_storage_index()
5249-        si_dir = storage_index_to_dir(si)
5250-        for i in immutable_shares:
5251-            shares = immutable_shares[i]
5252-            for shnum in shares:
5253-                dn = os.path.join(self.get_serverdir(i), "shares", si_dir)
5254-                fileutil.make_dirs(dn)
5255-                fn = os.path.join(dn, str(shnum))
5256-                f = open(fn, "wb")
5257-                f.write(shares[shnum])
5258-                f.close()
5259-
5260-        si = uri.from_string(mutable_uri).get_storage_index()
5261-        si_dir = storage_index_to_dir(si)
5262-        for i in mutable_shares:
5263-            shares = mutable_shares[i]
5264-            for shnum in shares:
5265-                dn = os.path.join(self.get_serverdir(i), "shares", si_dir)
5266-                fileutil.make_dirs(dn)
5267-                fn = os.path.join(dn, str(shnum))
5268-                f = open(fn, "wb")
5269-                f.write(shares[shnum])
5270-                f.close()
5271+        self._write_shares(immutable_uri, immutable_shares)
5272+        self._write_shares(mutable_uri, mutable_shares)
5273 
5274     def download_immutable(self, ignored=None):
5275         n = self.c0.create_node_from_uri(immutable_uri)
5276hunk ./src/allmydata/test/test_download.py 183
5277 
5278         self.load_shares()
5279         si = uri.from_string(immutable_uri).get_storage_index()
5280-        si_dir = storage_index_to_dir(si)
5281 
5282         n = self.c0.create_node_from_uri(immutable_uri)
5283         d = download_to_data(n)
5284hunk ./src/allmydata/test/test_download.py 198
5285                 for clientnum in immutable_shares:
5286                     for shnum in immutable_shares[clientnum]:
5287                         if s._shnum == shnum:
5288-                            fn = os.path.join(self.get_serverdir(clientnum),
5289-                                              "shares", si_dir, str(shnum))
5290-                            os.unlink(fn)
5291+                            share_dir = self.get_server(clientnum).backend.get_shareset(si)._sharehomedir
5292+                            share_dir.child(str(shnum)).remove()
5293         d.addCallback(_clobber_some_shares)
5294         d.addCallback(lambda ign: download_to_data(n))
5295         d.addCallback(_got_data)
5296hunk ./src/allmydata/test/test_download.py 212
5297                 for shnum in immutable_shares[clientnum]:
5298                     if shnum == save_me:
5299                         continue
5300-                    fn = os.path.join(self.get_serverdir(clientnum),
5301-                                      "shares", si_dir, str(shnum))
5302-                    if os.path.exists(fn):
5303-                        os.unlink(fn)
5304+                    share_dir = self.get_server(clientnum).backend.get_shareset(si)._sharehomedir
5305+                    fileutil.fp_remove(share_dir.child(str(shnum)))
5306             # now the download should fail with NotEnoughSharesError
5307             return self.shouldFail(NotEnoughSharesError, "1shares", None,
5308                                    download_to_data, n)
5309hunk ./src/allmydata/test/test_download.py 223
5310             # delete the last remaining share
5311             for clientnum in immutable_shares:
5312                 for shnum in immutable_shares[clientnum]:
5313-                    fn = os.path.join(self.get_serverdir(clientnum),
5314-                                      "shares", si_dir, str(shnum))
5315-                    if os.path.exists(fn):
5316-                        os.unlink(fn)
5317+                    share_dir = self.get_server(clientnum).backend.get_shareset(si)._sharehomedir
5318+                    share_dir.child(str(shnum)).remove()
5319             # now a new download should fail with NoSharesError. We want a
5320             # new ImmutableFileNode so it will forget about the old shares.
5321             # If we merely called create_node_from_uri() without first
5322hunk ./src/allmydata/test/test_download.py 801
5323         # will report two shares, and the ShareFinder will handle the
5324         # duplicate by attaching both to the same CommonShare instance.
5325         si = uri.from_string(immutable_uri).get_storage_index()
5326-        si_dir = storage_index_to_dir(si)
5327-        sh0_file = [sharefile
5328-                    for (shnum, serverid, sharefile)
5329-                    in self.find_uri_shares(immutable_uri)
5330-                    if shnum == 0][0]
5331-        sh0_data = open(sh0_file, "rb").read()
5332+        sh0_fp = [sharefp for (shnum, serverid, sharefp)
5333+                          in self.find_uri_shares(immutable_uri)
5334+                          if shnum == 0][0]
5335+        sh0_data = sh0_fp.getContent()
5336         for clientnum in immutable_shares:
5337             if 0 in immutable_shares[clientnum]:
5338                 continue
5339hunk ./src/allmydata/test/test_download.py 808
5340-            cdir = self.get_serverdir(clientnum)
5341-            target = os.path.join(cdir, "shares", si_dir, "0")
5342-            outf = open(target, "wb")
5343-            outf.write(sh0_data)
5344-            outf.close()
5345+            cdir = self.get_server(clientnum).backend.get_shareset(si)._sharehomedir
5346+            fileutil.fp_make_dirs(cdir)
5347+            cdir.child(str(shnum)).setContent(sh0_data)
5348 
5349         d = self.download_immutable()
5350         return d
5351hunk ./src/allmydata/test/test_encode.py 134
5352         d.addCallback(_try)
5353         return d
5354 
5355-    def get_share_hashes(self, at_least_these=()):
5356+    def get_share_hashes(self):
5357         d = self._start()
5358         def _try(unused=None):
5359             if self.mode == "bad sharehash":
5360hunk ./src/allmydata/test/test_hung_server.py 3
5361 # -*- coding: utf-8 -*-
5362 
5363-import os, shutil
5364 from twisted.trial import unittest
5365 from twisted.internet import defer
5366hunk ./src/allmydata/test/test_hung_server.py 5
5367-from allmydata import uri
5368+
5369 from allmydata.util.consumer import download_to_data
5370 from allmydata.immutable import upload
5371 from allmydata.mutable.common import UnrecoverableFileError
5372hunk ./src/allmydata/test/test_hung_server.py 10
5373 from allmydata.mutable.publish import MutableData
5374-from allmydata.storage.common import storage_index_to_dir
5375 from allmydata.test.no_network import GridTestMixin
5376 from allmydata.test.common import ShouldFailMixin
5377 from allmydata.util.pollmixin import PollMixin
5378hunk ./src/allmydata/test/test_hung_server.py 18
5379 immutable_plaintext = "data" * 10000
5380 mutable_plaintext = "muta" * 10000
5381 
5382+
5383 class HungServerDownloadTest(GridTestMixin, ShouldFailMixin, PollMixin,
5384                              unittest.TestCase):
5385     # Many of these tests take around 60 seconds on François's ARM buildslave:
5386hunk ./src/allmydata/test/test_hung_server.py 31
5387     timeout = 240
5388 
5389     def _break(self, servers):
5390-        for (id, ss) in servers:
5391-            self.g.break_server(id)
5392+        for ss in servers:
5393+            self.g.break_server(ss.get_serverid())
5394 
5395     def _hang(self, servers, **kwargs):
5396hunk ./src/allmydata/test/test_hung_server.py 35
5397-        for (id, ss) in servers:
5398-            self.g.hang_server(id, **kwargs)
5399+        for ss in servers:
5400+            self.g.hang_server(ss.get_serverid(), **kwargs)
5401 
5402     def _unhang(self, servers, **kwargs):
5403hunk ./src/allmydata/test/test_hung_server.py 39
5404-        for (id, ss) in servers:
5405-            self.g.unhang_server(id, **kwargs)
5406+        for ss in servers:
5407+            self.g.unhang_server(ss.get_serverid(), **kwargs)
5408 
5409     def _hang_shares(self, shnums, **kwargs):
5410         # hang all servers who are holding the given shares
5411hunk ./src/allmydata/test/test_hung_server.py 52
5412                     hung_serverids.add(i_serverid)
5413 
5414     def _delete_all_shares_from(self, servers):
5415-        serverids = [id for (id, ss) in servers]
5416-        for (i_shnum, i_serverid, i_sharefile) in self.shares:
5417+        serverids = [ss.get_serverid() for ss in servers]
5418+        for (i_shnum, i_serverid, i_sharefp) in self.shares:
5419             if i_serverid in serverids:
5420hunk ./src/allmydata/test/test_hung_server.py 55
5421-                os.unlink(i_sharefile)
5422+                i_sharefp.remove()
5423 
5424     def _corrupt_all_shares_in(self, servers, corruptor_func):
5425hunk ./src/allmydata/test/test_hung_server.py 58
5426-        serverids = [id for (id, ss) in servers]
5427-        for (i_shnum, i_serverid, i_sharefile) in self.shares:
5428+        serverids = [ss.get_serverid() for ss in servers]
5429+        for (i_shnum, i_serverid, i_sharefp) in self.shares:
5430             if i_serverid in serverids:
5431hunk ./src/allmydata/test/test_hung_server.py 61
5432-                self._corrupt_share((i_shnum, i_sharefile), corruptor_func)
5433+                self.corrupt_share((i_shnum, i_serverid, i_sharefp), corruptor_func)
5434 
5435     def _copy_all_shares_from(self, from_servers, to_server):
5436hunk ./src/allmydata/test/test_hung_server.py 64
5437-        serverids = [id for (id, ss) in from_servers]
5438-        for (i_shnum, i_serverid, i_sharefile) in self.shares:
5439+        serverids = [ss.get_serverid() for ss in from_servers]
5440+        for (i_shnum, i_serverid, i_sharefp) in self.shares:
5441             if i_serverid in serverids:
5442hunk ./src/allmydata/test/test_hung_server.py 67
5443-                self._copy_share((i_shnum, i_sharefile), to_server)
5444+                self.copy_share((i_shnum, i_serverid, i_sharefp), self.uri, to_server)
5445 
5446hunk ./src/allmydata/test/test_hung_server.py 69
5447-    def _copy_share(self, share, to_server):
5448-        (sharenum, sharefile) = share
5449-        (id, ss) = to_server
5450-        shares_dir = os.path.join(ss.original.storedir, "shares")
5451-        si = uri.from_string(self.uri).get_storage_index()
5452-        si_dir = os.path.join(shares_dir, storage_index_to_dir(si))
5453-        if not os.path.exists(si_dir):
5454-            os.makedirs(si_dir)
5455-        new_sharefile = os.path.join(si_dir, str(sharenum))
5456-        shutil.copy(sharefile, new_sharefile)
5457         self.shares = self.find_uri_shares(self.uri)
5458hunk ./src/allmydata/test/test_hung_server.py 70
5459-        # Make sure that the storage server has the share.
5460-        self.failUnless((sharenum, ss.original.my_nodeid, new_sharefile)
5461-                        in self.shares)
5462-
5463-    def _corrupt_share(self, share, corruptor_func):
5464-        (sharenum, sharefile) = share
5465-        data = open(sharefile, "rb").read()
5466-        newdata = corruptor_func(data)
5467-        os.unlink(sharefile)
5468-        wf = open(sharefile, "wb")
5469-        wf.write(newdata)
5470-        wf.close()
5471 
5472     def _set_up(self, mutable, testdir, num_clients=1, num_servers=10):
5473         self.mutable = mutable
5474hunk ./src/allmydata/test/test_hung_server.py 82
5475 
5476         self.c0 = self.g.clients[0]
5477         nm = self.c0.nodemaker
5478-        self.servers = sorted([(s.get_serverid(), s.get_rref())
5479-                               for s in nm.storage_broker.get_connected_servers()])
5480+        unsorted = [(s.get_serverid(), s.get_rref()) for s in nm.storage_broker.get_connected_servers()]
5481+        self.servers = [ss for (id, ss) in sorted(unsorted)]
5482         self.servers = self.servers[5:] + self.servers[:5]
5483 
5484         if mutable:
5485hunk ./src/allmydata/test/test_hung_server.py 244
5486             # stuck-but-not-overdue, and 4 live requests. All 4 live requests
5487             # will retire before the download is complete and the ShareFinder
5488             # is shut off. That will leave 4 OVERDUE and 1
5489-            # stuck-but-not-overdue, for a total of 5 requests in in
5490+            # stuck-but-not-overdue, for a total of 5 requests in
5491             # _sf.pending_requests
5492             for t in self._sf.overdue_timers.values()[:4]:
5493                 t.reset(-1.0)
5494hunk ./src/allmydata/test/test_mutable.py 21
5495 from foolscap.api import eventually, fireEventually
5496 from foolscap.logging import log
5497 from allmydata.storage_client import StorageFarmBroker
5498-from allmydata.storage.common import storage_index_to_dir
5499 from allmydata.scripts import debug
5500 
5501 from allmydata.mutable.filenode import MutableFileNode, BackoffAgent
5502hunk ./src/allmydata/test/test_mutable.py 3670
5503         # Now execute each assignment by writing the storage.
5504         for (share, servernum) in assignments:
5505             sharedata = base64.b64decode(self.sdmf_old_shares[share])
5506-            storedir = self.get_serverdir(servernum)
5507-            storage_path = os.path.join(storedir, "shares",
5508-                                        storage_index_to_dir(si))
5509-            fileutil.make_dirs(storage_path)
5510-            fileutil.write(os.path.join(storage_path, "%d" % share),
5511-                           sharedata)
5512+            storage_dir = self.get_server(servernum).backend.get_shareset(si).sharehomedir
5513+            fileutil.fp_make_dirs(storage_dir)
5514+            storage_dir.child("%d" % share).setContent(sharedata)
5515         # ...and verify that the shares are there.
5516         shares = self.find_uri_shares(self.sdmf_old_cap)
5517         assert len(shares) == 10
5518hunk ./src/allmydata/test/test_provisioning.py 13
5519 from nevow import inevow
5520 from zope.interface import implements
5521 
5522-class MyRequest:
5523+class MockRequest:
5524     implements(inevow.IRequest)
5525     pass
5526 
5527hunk ./src/allmydata/test/test_provisioning.py 26
5528     def test_load(self):
5529         pt = provisioning.ProvisioningTool()
5530         self.fields = {}
5531-        #r = MyRequest()
5532+        #r = MockRequest()
5533         #r.fields = self.fields
5534         #ctx = RequestContext()
5535         #unfilled = pt.renderSynchronously(ctx)
5536hunk ./src/allmydata/test/test_repairer.py 537
5537         # happiness setting.
5538         def _delete_some_servers(ignored):
5539             for i in xrange(7):
5540-                self.g.remove_server(self.g.servers_by_number[i].my_nodeid)
5541+                self.remove_server(i)
5542 
5543             assert len(self.g.servers_by_number) == 3
5544 
5545hunk ./src/allmydata/test/test_storage.py 14
5546 from allmydata import interfaces
5547 from allmydata.util import fileutil, hashutil, base32, pollmixin, time_format
5548 from allmydata.storage.server import StorageServer
5549-from allmydata.storage.mutable import MutableShareFile
5550-from allmydata.storage.immutable import BucketWriter, BucketReader
5551-from allmydata.storage.common import DataTooLargeError, storage_index_to_dir, \
5552+from allmydata.storage.backends.disk.mutable import MutableDiskShare
5553+from allmydata.storage.bucket import BucketWriter, BucketReader
5554+from allmydata.storage.common import DataTooLargeError, \
5555      UnknownMutableContainerVersionError, UnknownImmutableContainerVersionError
5556 from allmydata.storage.lease import LeaseInfo
5557 from allmydata.storage.crawler import BucketCountingCrawler
5558hunk ./src/allmydata/test/test_storage.py 474
5559         w[0].remote_write(0, "\xff"*10)
5560         w[0].remote_close()
5561 
5562-        fn = os.path.join(ss.sharedir, storage_index_to_dir("si1"), "0")
5563-        f = open(fn, "rb+")
5564+        fp = ss.backend.get_shareset("si1").sharehomedir.child("0")
5565+        f = fp.open("rb+")
5566         f.seek(0)
5567         f.write(struct.pack(">L", 0)) # this is invalid: minimum used is v1
5568         f.close()
5569hunk ./src/allmydata/test/test_storage.py 814
5570     def test_bad_magic(self):
5571         ss = self.create("test_bad_magic")
5572         self.allocate(ss, "si1", "we1", self._lease_secret.next(), set([0]), 10)
5573-        fn = os.path.join(ss.sharedir, storage_index_to_dir("si1"), "0")
5574-        f = open(fn, "rb+")
5575+        fp = ss.backend.get_shareset("si1").sharehomedir.child("0")
5576+        f = fp.open("rb+")
5577         f.seek(0)
5578         f.write("BAD MAGIC")
5579         f.close()
5580hunk ./src/allmydata/test/test_storage.py 842
5581 
5582         # Trying to make the container too large (by sending a write vector
5583         # whose offset is too high) will raise an exception.
5584-        TOOBIG = MutableShareFile.MAX_SIZE + 10
5585+        TOOBIG = MutableDiskShare.MAX_SIZE + 10
5586         self.failUnlessRaises(DataTooLargeError,
5587                               rstaraw, "si1", secrets,
5588                               {0: ([], [(TOOBIG,data)], None)},
5589hunk ./src/allmydata/test/test_storage.py 1229
5590 
5591         # create a random non-numeric file in the bucket directory, to
5592         # exercise the code that's supposed to ignore those.
5593-        bucket_dir = os.path.join(self.workdir("test_leases"),
5594-                                  "shares", storage_index_to_dir("si1"))
5595-        f = open(os.path.join(bucket_dir, "ignore_me.txt"), "w")
5596-        f.write("you ought to be ignoring me\n")
5597-        f.close()
5598+        bucket_dir = ss.backend.get_shareset("si1").sharehomedir
5599+        bucket_dir.child("ignore_me.txt").setContent("you ought to be ignoring me\n")
5600 
5601hunk ./src/allmydata/test/test_storage.py 1232
5602-        s0 = MutableShareFile(os.path.join(bucket_dir, "0"))
5603+        s0 = MutableDiskShare(os.path.join(bucket_dir, "0"))
5604         self.failUnlessEqual(len(list(s0.get_leases())), 1)
5605 
5606         # add-lease on a missing storage index is silently ignored
5607hunk ./src/allmydata/test/test_storage.py 3118
5608         [immutable_si_0, immutable_si_1, mutable_si_2, mutable_si_3] = self.sis
5609 
5610         # add a non-sharefile to exercise another code path
5611-        fn = os.path.join(ss.sharedir,
5612-                          storage_index_to_dir(immutable_si_0),
5613-                          "not-a-share")
5614-        f = open(fn, "wb")
5615-        f.write("I am not a share.\n")
5616-        f.close()
5617+        fp = ss.backend.get_shareset(immutable_si_0).sharehomedir.child("not-a-share")
5618+        fp.setContent("I am not a share.\n")
5619 
5620         # this is before the crawl has started, so we're not in a cycle yet
5621         initial_state = lc.get_state()
5622hunk ./src/allmydata/test/test_storage.py 3282
5623     def test_expire_age(self):
5624         basedir = "storage/LeaseCrawler/expire_age"
5625         fileutil.make_dirs(basedir)
5626-        # setting expiration_time to 2000 means that any lease which is more
5627-        # than 2000s old will be expired.
5628-        ss = InstrumentedStorageServer(basedir, "\x00" * 20,
5629-                                       expiration_enabled=True,
5630-                                       expiration_mode="age",
5631-                                       expiration_override_lease_duration=2000)
5632+        # setting 'override_lease_duration' to 2000 means that any lease that
5633+        # is more than 2000 seconds old will be expired.
5634+        expiration_policy = {
5635+            'enabled': True,
5636+            'mode': 'age',
5637+            'override_lease_duration': 2000,
5638+            'sharetypes': ('mutable', 'immutable'),
5639+        }
5640+        ss = InstrumentedStorageServer(basedir, "\x00" * 20, expiration_policy)
5641         # make it start sooner than usual.
5642         lc = ss.lease_checker
5643         lc.slow_start = 0
5644hunk ./src/allmydata/test/test_storage.py 3423
5645     def test_expire_cutoff_date(self):
5646         basedir = "storage/LeaseCrawler/expire_cutoff_date"
5647         fileutil.make_dirs(basedir)
5648-        # setting cutoff-date to 2000 seconds ago means that any lease which
5649-        # is more than 2000s old will be expired.
5650+        # setting 'cutoff_date' to 2000 seconds ago means that any lease that
5651+        # is more than 2000 seconds old will be expired.
5652         now = time.time()
5653         then = int(now - 2000)
5654hunk ./src/allmydata/test/test_storage.py 3427
5655-        ss = InstrumentedStorageServer(basedir, "\x00" * 20,
5656-                                       expiration_enabled=True,
5657-                                       expiration_mode="cutoff-date",
5658-                                       expiration_cutoff_date=then)
5659+        expiration_policy = {
5660+            'enabled': True,
5661+            'mode': 'cutoff-date',
5662+            'cutoff_date': then,
5663+            'sharetypes': ('mutable', 'immutable'),
5664+        }
5665+        ss = InstrumentedStorageServer(basedir, "\x00" * 20, expiration_policy)
5666         # make it start sooner than usual.
5667         lc = ss.lease_checker
5668         lc.slow_start = 0
5669hunk ./src/allmydata/test/test_storage.py 3575
5670     def test_only_immutable(self):
5671         basedir = "storage/LeaseCrawler/only_immutable"
5672         fileutil.make_dirs(basedir)
5673+        # setting 'cutoff_date' to 2000 seconds ago means that any lease that
5674+        # is more than 2000 seconds old will be expired.
5675         now = time.time()
5676         then = int(now - 2000)
5677hunk ./src/allmydata/test/test_storage.py 3579
5678-        ss = StorageServer(basedir, "\x00" * 20,
5679-                           expiration_enabled=True,
5680-                           expiration_mode="cutoff-date",
5681-                           expiration_cutoff_date=then,
5682-                           expiration_sharetypes=("immutable",))
5683+        expiration_policy = {
5684+            'enabled': True,
5685+            'mode': 'cutoff-date',
5686+            'cutoff_date': then,
5687+            'sharetypes': ('immutable',),
5688+        }
5689+        ss = StorageServer(basedir, "\x00" * 20, expiration_policy)
5690         lc = ss.lease_checker
5691         lc.slow_start = 0
5692         webstatus = StorageStatus(ss)
5693hunk ./src/allmydata/test/test_storage.py 3636
5694     def test_only_mutable(self):
5695         basedir = "storage/LeaseCrawler/only_mutable"
5696         fileutil.make_dirs(basedir)
5697+        # setting 'cutoff_date' to 2000 seconds ago means that any lease that
5698+        # is more than 2000 seconds old will be expired.
5699         now = time.time()
5700         then = int(now - 2000)
5701hunk ./src/allmydata/test/test_storage.py 3640
5702-        ss = StorageServer(basedir, "\x00" * 20,
5703-                           expiration_enabled=True,
5704-                           expiration_mode="cutoff-date",
5705-                           expiration_cutoff_date=then,
5706-                           expiration_sharetypes=("mutable",))
5707+        expiration_policy = {
5708+            'enabled': True,
5709+            'mode': 'cutoff-date',
5710+            'cutoff_date': then,
5711+            'sharetypes': ('mutable',),
5712+        }
5713+        ss = StorageServer(basedir, "\x00" * 20, expiration_policy)
5714         lc = ss.lease_checker
5715         lc.slow_start = 0
5716         webstatus = StorageStatus(ss)
5717hunk ./src/allmydata/test/test_storage.py 3819
5718     def test_no_st_blocks(self):
5719         basedir = "storage/LeaseCrawler/no_st_blocks"
5720         fileutil.make_dirs(basedir)
5721-        ss = No_ST_BLOCKS_StorageServer(basedir, "\x00" * 20,
5722-                                        expiration_mode="age",
5723-                                        expiration_override_lease_duration=-1000)
5724-        # a negative expiration_time= means the "configured-"
5725+        # A negative 'override_lease_duration' means that the "configured-"
5726         # space-recovered counts will be non-zero, since all shares will have
5727hunk ./src/allmydata/test/test_storage.py 3821
5728-        # expired by then
5729+        # expired by then.
5730+        expiration_policy = {
5731+            'enabled': True,
5732+            'mode': 'age',
5733+            'override_lease_duration': -1000,
5734+            'sharetypes': ('mutable', 'immutable'),
5735+        }
5736+        ss = No_ST_BLOCKS_StorageServer(basedir, "\x00" * 20, expiration_policy)
5737 
5738         # make it start sooner than usual.
5739         lc = ss.lease_checker
5740hunk ./src/allmydata/test/test_storage.py 3877
5741         [immutable_si_0, immutable_si_1, mutable_si_2, mutable_si_3] = self.sis
5742         first = min(self.sis)
5743         first_b32 = base32.b2a(first)
5744-        fn = os.path.join(ss.sharedir, storage_index_to_dir(first), "0")
5745-        f = open(fn, "rb+")
5746+        fp = ss.backend.get_shareset(first).sharehomedir.child("0")
5747+        f = fp.open("rb+")
5748         f.seek(0)
5749         f.write("BAD MAGIC")
5750         f.close()
5751hunk ./src/allmydata/test/test_storage.py 3890
5752 
5753         # also create an empty bucket
5754         empty_si = base32.b2a("\x04"*16)
5755-        empty_bucket_dir = os.path.join(ss.sharedir,
5756-                                        storage_index_to_dir(empty_si))
5757-        fileutil.make_dirs(empty_bucket_dir)
5758+        empty_bucket_dir = ss.backend.get_shareset(empty_si).sharehomedir
5759+        fileutil.fp_make_dirs(empty_bucket_dir)
5760 
5761         ss.setServiceParent(self.s)
5762 
5763hunk ./src/allmydata/test/test_system.py 10
5764 
5765 import allmydata
5766 from allmydata import uri
5767-from allmydata.storage.mutable import MutableShareFile
5768+from allmydata.storage.backends.disk.mutable import MutableDiskShare
5769 from allmydata.storage.server import si_a2b
5770 from allmydata.immutable import offloaded, upload
5771 from allmydata.immutable.literal import LiteralFileNode
5772hunk ./src/allmydata/test/test_system.py 421
5773         return shares
5774 
5775     def _corrupt_mutable_share(self, filename, which):
5776-        msf = MutableShareFile(filename)
5777+        msf = MutableDiskShare(filename)
5778         datav = msf.readv([ (0, 1000000) ])
5779         final_share = datav[0]
5780         assert len(final_share) < 1000000 # ought to be truncated
5781hunk ./src/allmydata/test/test_upload.py 22
5782 from allmydata.util.happinessutil import servers_of_happiness, \
5783                                          shares_by_server, merge_servers
5784 from allmydata.storage_client import StorageFarmBroker
5785-from allmydata.storage.server import storage_index_to_dir
5786 
5787 MiB = 1024*1024
5788 
5789hunk ./src/allmydata/test/test_upload.py 821
5790 
5791     def _copy_share_to_server(self, share_number, server_number):
5792         ss = self.g.servers_by_number[server_number]
5793-        # Copy share i from the directory associated with the first
5794-        # storage server to the directory associated with this one.
5795-        assert self.g, "I tried to find a grid at self.g, but failed"
5796-        assert self.shares, "I tried to find shares at self.shares, but failed"
5797-        old_share_location = self.shares[share_number][2]
5798-        new_share_location = os.path.join(ss.storedir, "shares")
5799-        si = uri.from_string(self.uri).get_storage_index()
5800-        new_share_location = os.path.join(new_share_location,
5801-                                          storage_index_to_dir(si))
5802-        if not os.path.exists(new_share_location):
5803-            os.makedirs(new_share_location)
5804-        new_share_location = os.path.join(new_share_location,
5805-                                          str(share_number))
5806-        if old_share_location != new_share_location:
5807-            shutil.copy(old_share_location, new_share_location)
5808-        shares = self.find_uri_shares(self.uri)
5809-        # Make sure that the storage server has the share.
5810-        self.failUnless((share_number, ss.my_nodeid, new_share_location)
5811-                        in shares)
5812+        self.copy_share(self.shares[share_number], ss)
5813 
5814     def _setup_grid(self):
5815         """
5816hunk ./src/allmydata/test/test_upload.py 1103
5817                 self._copy_share_to_server(i, 2)
5818         d.addCallback(_copy_shares)
5819         # Remove the first server, and add a placeholder with share 0
5820-        d.addCallback(lambda ign:
5821-            self.g.remove_server(self.g.servers_by_number[0].my_nodeid))
5822+        d.addCallback(lambda ign: self.remove_server(0))
5823         d.addCallback(lambda ign:
5824             self._add_server_with_share(server_number=4, share_number=0))
5825         # Now try uploading.
5826hunk ./src/allmydata/test/test_upload.py 1134
5827         d.addCallback(lambda ign:
5828             self._add_server(server_number=4))
5829         d.addCallback(_copy_shares)
5830-        d.addCallback(lambda ign:
5831-            self.g.remove_server(self.g.servers_by_number[0].my_nodeid))
5832+        d.addCallback(lambda ign: self.remove_server(0))
5833         d.addCallback(_reset_encoding_parameters)
5834         d.addCallback(lambda client:
5835             client.upload(upload.Data("data" * 10000, convergence="")))
5836hunk ./src/allmydata/test/test_upload.py 1196
5837                 self._copy_share_to_server(i, 2)
5838         d.addCallback(_copy_shares)
5839         # Remove server 0, and add another in its place
5840-        d.addCallback(lambda ign:
5841-            self.g.remove_server(self.g.servers_by_number[0].my_nodeid))
5842+        d.addCallback(lambda ign: self.remove_server(0))
5843         d.addCallback(lambda ign:
5844             self._add_server_with_share(server_number=4, share_number=0,
5845                                         readonly=True))
5846hunk ./src/allmydata/test/test_upload.py 1237
5847             for i in xrange(1, 10):
5848                 self._copy_share_to_server(i, 2)
5849         d.addCallback(_copy_shares)
5850-        d.addCallback(lambda ign:
5851-            self.g.remove_server(self.g.servers_by_number[0].my_nodeid))
5852+        d.addCallback(lambda ign: self.remove_server(0))
5853         def _reset_encoding_parameters(ign, happy=4):
5854             client = self.g.clients[0]
5855             client.DEFAULT_ENCODING_PARAMETERS['happy'] = happy
5856hunk ./src/allmydata/test/test_upload.py 1273
5857         # remove the original server
5858         # (necessary to ensure that the Tahoe2ServerSelector will distribute
5859         #  all the shares)
5860-        def _remove_server(ign):
5861-            server = self.g.servers_by_number[0]
5862-            self.g.remove_server(server.my_nodeid)
5863-        d.addCallback(_remove_server)
5864+        d.addCallback(lambda ign: self.remove_server(0))
5865         # This should succeed; we still have 4 servers, and the
5866         # happiness of the upload is 4.
5867         d.addCallback(lambda ign:
5868hunk ./src/allmydata/test/test_upload.py 1285
5869         d.addCallback(lambda ign:
5870             self._setup_and_upload())
5871         d.addCallback(_do_server_setup)
5872-        d.addCallback(_remove_server)
5873+        d.addCallback(lambda ign: self.remove_server(0))
5874         d.addCallback(lambda ign:
5875             self.shouldFail(UploadUnhappinessError,
5876                             "test_dropped_servers_in_encoder",
5877hunk ./src/allmydata/test/test_upload.py 1307
5878             self._add_server_with_share(4, 7, readonly=True)
5879             self._add_server_with_share(5, 8, readonly=True)
5880         d.addCallback(_do_server_setup_2)
5881-        d.addCallback(_remove_server)
5882+        d.addCallback(lambda ign: self.remove_server(0))
5883         d.addCallback(lambda ign:
5884             self._do_upload_with_broken_servers(1))
5885         d.addCallback(_set_basedir)
5886hunk ./src/allmydata/test/test_upload.py 1314
5887         d.addCallback(lambda ign:
5888             self._setup_and_upload())
5889         d.addCallback(_do_server_setup_2)
5890-        d.addCallback(_remove_server)
5891+        d.addCallback(lambda ign: self.remove_server(0))
5892         d.addCallback(lambda ign:
5893             self.shouldFail(UploadUnhappinessError,
5894                             "test_dropped_servers_in_encoder",
5895hunk ./src/allmydata/test/test_upload.py 1528
5896             for i in xrange(1, 10):
5897                 self._copy_share_to_server(i, 1)
5898         d.addCallback(_copy_shares)
5899-        d.addCallback(lambda ign:
5900-            self.g.remove_server(self.g.servers_by_number[0].my_nodeid))
5901+        d.addCallback(lambda ign: self.remove_server(0))
5902         def _prepare_client(ign):
5903             client = self.g.clients[0]
5904             client.DEFAULT_ENCODING_PARAMETERS['happy'] = 4
5905hunk ./src/allmydata/test/test_upload.py 1550
5906         def _setup(ign):
5907             for i in xrange(1, 11):
5908                 self._add_server(server_number=i)
5909-            self.g.remove_server(self.g.servers_by_number[0].my_nodeid)
5910+            self.remove_server(0)
5911             c = self.g.clients[0]
5912             # We set happy to an unsatisfiable value so that we can check the
5913             # counting in the exception message. The same progress message
5914hunk ./src/allmydata/test/test_upload.py 1577
5915                 self._add_server(server_number=i)
5916             self._add_server(server_number=11, readonly=True)
5917             self._add_server(server_number=12, readonly=True)
5918-            self.g.remove_server(self.g.servers_by_number[0].my_nodeid)
5919+            self.remove_server(0)
5920             c = self.g.clients[0]
5921             c.DEFAULT_ENCODING_PARAMETERS['happy'] = 45
5922             return c
5923hunk ./src/allmydata/test/test_upload.py 1605
5924             # the first one that the selector sees.
5925             for i in xrange(10):
5926                 self._copy_share_to_server(i, 9)
5927-            # Remove server 0, and its contents
5928-            self.g.remove_server(self.g.servers_by_number[0].my_nodeid)
5929+            self.remove_server(0)
5930             # Make happiness unsatisfiable
5931             c = self.g.clients[0]
5932             c.DEFAULT_ENCODING_PARAMETERS['happy'] = 45
5933hunk ./src/allmydata/test/test_upload.py 1625
5934         def _then(ign):
5935             for i in xrange(1, 11):
5936                 self._add_server(server_number=i, readonly=True)
5937-            self.g.remove_server(self.g.servers_by_number[0].my_nodeid)
5938+            self.remove_server(0)
5939             c = self.g.clients[0]
5940             c.DEFAULT_ENCODING_PARAMETERS['k'] = 2
5941             c.DEFAULT_ENCODING_PARAMETERS['happy'] = 4
5942hunk ./src/allmydata/test/test_upload.py 1661
5943             self._add_server(server_number=4, readonly=True))
5944         d.addCallback(lambda ign:
5945             self._add_server(server_number=5, readonly=True))
5946-        d.addCallback(lambda ign:
5947-            self.g.remove_server(self.g.servers_by_number[0].my_nodeid))
5948+        d.addCallback(lambda ign: self.remove_server(0))
5949         def _reset_encoding_parameters(ign, happy=4):
5950             client = self.g.clients[0]
5951             client.DEFAULT_ENCODING_PARAMETERS['happy'] = happy
5952hunk ./src/allmydata/test/test_upload.py 1696
5953         d.addCallback(lambda ign:
5954             self._add_server(server_number=2))
5955         def _break_server_2(ign):
5956-            serverid = self.g.servers_by_number[2].my_nodeid
5957+            serverid = self.get_server(2).get_serverid()
5958             self.g.break_server(serverid)
5959         d.addCallback(_break_server_2)
5960         d.addCallback(lambda ign:
5961hunk ./src/allmydata/test/test_upload.py 1705
5962             self._add_server(server_number=4, readonly=True))
5963         d.addCallback(lambda ign:
5964             self._add_server(server_number=5, readonly=True))
5965-        d.addCallback(lambda ign:
5966-            self.g.remove_server(self.g.servers_by_number[0].my_nodeid))
5967+        d.addCallback(lambda ign: self.remove_server(0))
5968         d.addCallback(_reset_encoding_parameters)
5969         d.addCallback(lambda client:
5970             self.shouldFail(UploadUnhappinessError, "test_selection_exceptions",
5971hunk ./src/allmydata/test/test_upload.py 1816
5972             # Copy shares
5973             self._copy_share_to_server(1, 1)
5974             self._copy_share_to_server(2, 1)
5975-            # Remove server 0
5976-            self.g.remove_server(self.g.servers_by_number[0].my_nodeid)
5977+            self.remove_server(0)
5978             client = self.g.clients[0]
5979             client.DEFAULT_ENCODING_PARAMETERS['happy'] = 3
5980             return client
5981hunk ./src/allmydata/test/test_upload.py 1930
5982                                         readonly=True)
5983             self._add_server_with_share(server_number=4, share_number=3,
5984                                         readonly=True)
5985-            # Remove server 0.
5986-            self.g.remove_server(self.g.servers_by_number[0].my_nodeid)
5987+            self.remove_server(0)
5988             # Set the client appropriately
5989             c = self.g.clients[0]
5990             c.DEFAULT_ENCODING_PARAMETERS['happy'] = 4
5991hunk ./src/allmydata/test/test_util.py 9
5992 from twisted.trial import unittest
5993 from twisted.internet import defer, reactor
5994 from twisted.python.failure import Failure
5995+from twisted.python.filepath import FilePath
5996 from twisted.python import log
5997 from pycryptopp.hash.sha256 import SHA256 as _hash
5998 
5999hunk ./src/allmydata/test/test_util.py 508
6000                 os.chdir(saved_cwd)
6001 
6002     def test_disk_stats(self):
6003-        avail = fileutil.get_available_space('.', 2**14)
6004+        avail = fileutil.get_available_space(FilePath('.'), 2**14)
6005         if avail == 0:
6006             raise unittest.SkipTest("This test will spuriously fail there is no disk space left.")
6007 
6008hunk ./src/allmydata/test/test_util.py 512
6009-        disk = fileutil.get_disk_stats('.', 2**13)
6010+        disk = fileutil.get_disk_stats(FilePath('.'), 2**13)
6011         self.failUnless(disk['total'] > 0, disk['total'])
6012         self.failUnless(disk['used'] > 0, disk['used'])
6013         self.failUnless(disk['free_for_root'] > 0, disk['free_for_root'])
6014hunk ./src/allmydata/test/test_util.py 521
6015 
6016     def test_disk_stats_avail_nonnegative(self):
6017         # This test will spuriously fail if you have more than 2^128
6018-        # bytes of available space on your filesystem.
6019-        disk = fileutil.get_disk_stats('.', 2**128)
6020+        # bytes of available space on your filesystem (lucky you).
6021+        disk = fileutil.get_disk_stats(FilePath('.'), 2**128)
6022         self.failUnlessEqual(disk['avail'], 0)
6023 
6024 class PollMixinTests(unittest.TestCase):
6025hunk ./src/allmydata/test/test_web.py 12
6026 from twisted.python import failure, log
6027 from nevow import rend
6028 from allmydata import interfaces, uri, webish, dirnode
6029-from allmydata.storage.shares import get_share_file
6030 from allmydata.storage_client import StorageFarmBroker
6031 from allmydata.immutable import upload
6032 from allmydata.immutable.downloader.status import DownloadStatus
6033hunk ./src/allmydata/test/test_web.py 4111
6034             good_shares = self.find_uri_shares(self.uris["good"])
6035             self.failUnlessReallyEqual(len(good_shares), 10)
6036             sick_shares = self.find_uri_shares(self.uris["sick"])
6037-            os.unlink(sick_shares[0][2])
6038+            sick_shares[0][2].remove()
6039             dead_shares = self.find_uri_shares(self.uris["dead"])
6040             for i in range(1, 10):
6041hunk ./src/allmydata/test/test_web.py 4114
6042-                os.unlink(dead_shares[i][2])
6043+                dead_shares[i][2].remove()
6044             c_shares = self.find_uri_shares(self.uris["corrupt"])
6045             cso = CorruptShareOptions()
6046             cso.stdout = StringIO()
6047hunk ./src/allmydata/test/test_web.py 4118
6048-            cso.parseOptions([c_shares[0][2]])
6049+            cso.parseOptions([c_shares[0][2].path])
6050             corrupt_share(cso)
6051         d.addCallback(_clobber_shares)
6052 
6053hunk ./src/allmydata/test/test_web.py 4253
6054             good_shares = self.find_uri_shares(self.uris["good"])
6055             self.failUnlessReallyEqual(len(good_shares), 10)
6056             sick_shares = self.find_uri_shares(self.uris["sick"])
6057-            os.unlink(sick_shares[0][2])
6058+            sick_shares[0][2].remove()
6059             dead_shares = self.find_uri_shares(self.uris["dead"])
6060             for i in range(1, 10):
6061hunk ./src/allmydata/test/test_web.py 4256
6062-                os.unlink(dead_shares[i][2])
6063+                dead_shares[i][2].remove()
6064             c_shares = self.find_uri_shares(self.uris["corrupt"])
6065             cso = CorruptShareOptions()
6066             cso.stdout = StringIO()
6067hunk ./src/allmydata/test/test_web.py 4260
6068-            cso.parseOptions([c_shares[0][2]])
6069+            cso.parseOptions([c_shares[0][2].path])
6070             corrupt_share(cso)
6071         d.addCallback(_clobber_shares)
6072 
6073hunk ./src/allmydata/test/test_web.py 4319
6074 
6075         def _clobber_shares(ignored):
6076             sick_shares = self.find_uri_shares(self.uris["sick"])
6077-            os.unlink(sick_shares[0][2])
6078+            sick_shares[0][2].remove()
6079         d.addCallback(_clobber_shares)
6080 
6081         d.addCallback(self.CHECK, "sick", "t=check&repair=true&output=json")
6082hunk ./src/allmydata/test/test_web.py 4811
6083             good_shares = self.find_uri_shares(self.uris["good"])
6084             self.failUnlessReallyEqual(len(good_shares), 10)
6085             sick_shares = self.find_uri_shares(self.uris["sick"])
6086-            os.unlink(sick_shares[0][2])
6087+            sick_shares[0][2].remove()
6088             #dead_shares = self.find_uri_shares(self.uris["dead"])
6089             #for i in range(1, 10):
6090hunk ./src/allmydata/test/test_web.py 4814
6091-            #    os.unlink(dead_shares[i][2])
6092+            #    dead_shares[i][2].remove()
6093 
6094             #c_shares = self.find_uri_shares(self.uris["corrupt"])
6095             #cso = CorruptShareOptions()
6096hunk ./src/allmydata/test/test_web.py 4819
6097             #cso.stdout = StringIO()
6098-            #cso.parseOptions([c_shares[0][2]])
6099+            #cso.parseOptions([c_shares[0][2].path])
6100             #corrupt_share(cso)
6101         d.addCallback(_clobber_shares)
6102 
6103hunk ./src/allmydata/test/test_web.py 4870
6104         d.addErrback(self.explain_web_error)
6105         return d
6106 
6107-    def _count_leases(self, ignored, which):
6108-        u = self.uris[which]
6109-        shares = self.find_uri_shares(u)
6110-        lease_counts = []
6111-        for shnum, serverid, fn in shares:
6112-            sf = get_share_file(fn)
6113-            num_leases = len(list(sf.get_leases()))
6114-            lease_counts.append( (fn, num_leases) )
6115-        return lease_counts
6116-
6117-    def _assert_leasecount(self, lease_counts, expected):
6118+    def _assert_leasecount(self, ignored, which, expected):
6119+        lease_counts = self.count_leases(self.uris[which])
6120         for (fn, num_leases) in lease_counts:
6121             if num_leases != expected:
6122                 self.fail("expected %d leases, have %d, on %s" %
6123hunk ./src/allmydata/test/test_web.py 4903
6124                 self.fileurls[which] = "uri/" + urllib.quote(self.uris[which])
6125         d.addCallback(_compute_fileurls)
6126 
6127-        d.addCallback(self._count_leases, "one")
6128-        d.addCallback(self._assert_leasecount, 1)
6129-        d.addCallback(self._count_leases, "two")
6130-        d.addCallback(self._assert_leasecount, 1)
6131-        d.addCallback(self._count_leases, "mutable")
6132-        d.addCallback(self._assert_leasecount, 1)
6133+        d.addCallback(self._assert_leasecount, "one", 1)
6134+        d.addCallback(self._assert_leasecount, "two", 1)
6135+        d.addCallback(self._assert_leasecount, "mutable", 1)
6136 
6137         d.addCallback(self.CHECK, "one", "t=check") # no add-lease
6138         def _got_html_good(res):
6139hunk ./src/allmydata/test/test_web.py 4913
6140             self.failIf("Not Healthy" in res, res)
6141         d.addCallback(_got_html_good)
6142 
6143-        d.addCallback(self._count_leases, "one")
6144-        d.addCallback(self._assert_leasecount, 1)
6145-        d.addCallback(self._count_leases, "two")
6146-        d.addCallback(self._assert_leasecount, 1)
6147-        d.addCallback(self._count_leases, "mutable")
6148-        d.addCallback(self._assert_leasecount, 1)
6149+        d.addCallback(self._assert_leasecount, "one", 1)
6150+        d.addCallback(self._assert_leasecount, "two", 1)
6151+        d.addCallback(self._assert_leasecount, "mutable", 1)
6152 
6153         # this CHECK uses the original client, which uses the same
6154         # lease-secrets, so it will just renew the original lease
6155hunk ./src/allmydata/test/test_web.py 4922
6156         d.addCallback(self.CHECK, "one", "t=check&add-lease=true")
6157         d.addCallback(_got_html_good)
6158 
6159-        d.addCallback(self._count_leases, "one")
6160-        d.addCallback(self._assert_leasecount, 1)
6161-        d.addCallback(self._count_leases, "two")
6162-        d.addCallback(self._assert_leasecount, 1)
6163-        d.addCallback(self._count_leases, "mutable")
6164-        d.addCallback(self._assert_leasecount, 1)
6165+        d.addCallback(self._assert_leasecount, "one", 1)
6166+        d.addCallback(self._assert_leasecount, "two", 1)
6167+        d.addCallback(self._assert_leasecount, "mutable", 1)
6168 
6169         # this CHECK uses an alternate client, which adds a second lease
6170         d.addCallback(self.CHECK, "one", "t=check&add-lease=true", clientnum=1)
6171hunk ./src/allmydata/test/test_web.py 4930
6172         d.addCallback(_got_html_good)
6173 
6174-        d.addCallback(self._count_leases, "one")
6175-        d.addCallback(self._assert_leasecount, 2)
6176-        d.addCallback(self._count_leases, "two")
6177-        d.addCallback(self._assert_leasecount, 1)
6178-        d.addCallback(self._count_leases, "mutable")
6179-        d.addCallback(self._assert_leasecount, 1)
6180+        d.addCallback(self._assert_leasecount, "one", 2)
6181+        d.addCallback(self._assert_leasecount, "two", 1)
6182+        d.addCallback(self._assert_leasecount, "mutable", 1)
6183 
6184         d.addCallback(self.CHECK, "mutable", "t=check&add-lease=true")
6185         d.addCallback(_got_html_good)
6186hunk ./src/allmydata/test/test_web.py 4937
6187 
6188-        d.addCallback(self._count_leases, "one")
6189-        d.addCallback(self._assert_leasecount, 2)
6190-        d.addCallback(self._count_leases, "two")
6191-        d.addCallback(self._assert_leasecount, 1)
6192-        d.addCallback(self._count_leases, "mutable")
6193-        d.addCallback(self._assert_leasecount, 1)
6194+        d.addCallback(self._assert_leasecount, "one", 2)
6195+        d.addCallback(self._assert_leasecount, "two", 1)
6196+        d.addCallback(self._assert_leasecount, "mutable", 1)
6197 
6198         d.addCallback(self.CHECK, "mutable", "t=check&add-lease=true",
6199                       clientnum=1)
6200hunk ./src/allmydata/test/test_web.py 4945
6201         d.addCallback(_got_html_good)
6202 
6203-        d.addCallback(self._count_leases, "one")
6204-        d.addCallback(self._assert_leasecount, 2)
6205-        d.addCallback(self._count_leases, "two")
6206-        d.addCallback(self._assert_leasecount, 1)
6207-        d.addCallback(self._count_leases, "mutable")
6208-        d.addCallback(self._assert_leasecount, 2)
6209+        d.addCallback(self._assert_leasecount, "one", 2)
6210+        d.addCallback(self._assert_leasecount, "two", 1)
6211+        d.addCallback(self._assert_leasecount, "mutable", 2)
6212 
6213         d.addErrback(self.explain_web_error)
6214         return d
6215hunk ./src/allmydata/test/test_web.py 4989
6216             self.failUnlessReallyEqual(len(units), 4+1)
6217         d.addCallback(_done)
6218 
6219-        d.addCallback(self._count_leases, "root")
6220-        d.addCallback(self._assert_leasecount, 1)
6221-        d.addCallback(self._count_leases, "one")
6222-        d.addCallback(self._assert_leasecount, 1)
6223-        d.addCallback(self._count_leases, "mutable")
6224-        d.addCallback(self._assert_leasecount, 1)
6225+        d.addCallback(self._assert_leasecount, "root", 1)
6226+        d.addCallback(self._assert_leasecount, "one", 1)
6227+        d.addCallback(self._assert_leasecount, "mutable", 1)
6228 
6229         d.addCallback(self.CHECK, "root", "t=stream-deep-check&add-lease=true")
6230         d.addCallback(_done)
6231hunk ./src/allmydata/test/test_web.py 4996
6232 
6233-        d.addCallback(self._count_leases, "root")
6234-        d.addCallback(self._assert_leasecount, 1)
6235-        d.addCallback(self._count_leases, "one")
6236-        d.addCallback(self._assert_leasecount, 1)
6237-        d.addCallback(self._count_leases, "mutable")
6238-        d.addCallback(self._assert_leasecount, 1)
6239+        d.addCallback(self._assert_leasecount, "root", 1)
6240+        d.addCallback(self._assert_leasecount, "one", 1)
6241+        d.addCallback(self._assert_leasecount, "mutable", 1)
6242 
6243         d.addCallback(self.CHECK, "root", "t=stream-deep-check&add-lease=true",
6244                       clientnum=1)
6245hunk ./src/allmydata/test/test_web.py 5004
6246         d.addCallback(_done)
6247 
6248-        d.addCallback(self._count_leases, "root")
6249-        d.addCallback(self._assert_leasecount, 2)
6250-        d.addCallback(self._count_leases, "one")
6251-        d.addCallback(self._assert_leasecount, 2)
6252-        d.addCallback(self._count_leases, "mutable")
6253-        d.addCallback(self._assert_leasecount, 2)
6254+        d.addCallback(self._assert_leasecount, "root", 2)
6255+        d.addCallback(self._assert_leasecount, "one", 2)
6256+        d.addCallback(self._assert_leasecount, "mutable", 2)
6257 
6258         d.addErrback(self.explain_web_error)
6259         return d
6260merger 0.0 (
6261hunk ./src/allmydata/uri.py 829
6262+    def is_readonly(self):
6263+        return True
6264+
6265+    def get_readonly(self):
6266+        return self
6267+
6268+
6269hunk ./src/allmydata/uri.py 829
6270+    def is_readonly(self):
6271+        return True
6272+
6273+    def get_readonly(self):
6274+        return self
6275+
6276+
6277)
6278merger 0.0 (
6279hunk ./src/allmydata/uri.py 848
6280+    def is_readonly(self):
6281+        return True
6282+
6283+    def get_readonly(self):
6284+        return self
6285+
6286hunk ./src/allmydata/uri.py 848
6287+    def is_readonly(self):
6288+        return True
6289+
6290+    def get_readonly(self):
6291+        return self
6292+
6293)
6294hunk ./src/allmydata/util/encodingutil.py 221
6295 def quote_path(path, quotemarks=True):
6296     return quote_output("/".join(map(to_str, path)), quotemarks=quotemarks)
6297 
6298+def quote_filepath(fp, quotemarks=True, encoding=None):
6299+    path = fp.path
6300+    if isinstance(path, str):
6301+        try:
6302+            path = path.decode(filesystem_encoding)
6303+        except UnicodeDecodeError:
6304+            return 'b"%s"' % (ESCAPABLE_8BIT.sub(_str_escape, path),)
6305+
6306+    return quote_output(path, quotemarks=quotemarks, encoding=encoding)
6307+
6308 
6309 def unicode_platform():
6310     """
6311hunk ./src/allmydata/util/fileutil.py 5
6312 Futz with files like a pro.
6313 """
6314 
6315-import sys, exceptions, os, stat, tempfile, time, binascii
6316+import errno, sys, exceptions, os, stat, tempfile, time, binascii
6317+
6318+from allmydata.util.assertutil import precondition
6319 
6320 from twisted.python import log
6321hunk ./src/allmydata/util/fileutil.py 10
6322+from twisted.python.filepath import FilePath, UnlistableError
6323 
6324 from pycryptopp.cipher.aes import AES
6325 
6326hunk ./src/allmydata/util/fileutil.py 189
6327             raise tx
6328         raise exceptions.IOError, "unknown error prevented creation of directory, or deleted the directory immediately after creation: %s" % dirname # careful not to construct an IOError with a 2-tuple, as that has a special meaning...
6329 
6330-def rm_dir(dirname):
6331+def fp_make_dirs(dirfp):
6332+    """
6333+    An idempotent version of FilePath.makedirs().  If the dir already
6334+    exists, do nothing and return without raising an exception.  If this
6335+    call creates the dir, return without raising an exception.  If there is
6336+    an error that prevents creation or if the directory gets deleted after
6337+    fp_make_dirs() creates it and before fp_make_dirs() checks that it
6338+    exists, raise an exception.
6339+    """
6340+    log.msg( "xxx 0 %s" % (dirfp,))
6341+    tx = None
6342+    try:
6343+        dirfp.makedirs()
6344+    except OSError, x:
6345+        tx = x
6346+
6347+    if not dirfp.isdir():
6348+        if tx:
6349+            raise tx
6350+        raise exceptions.IOError, "unknown error prevented creation of directory, or deleted the directory immediately after creation: %s" % dirfp # careful not to construct an IOError with a 2-tuple, as that has a special meaning...
6351+
6352+def fp_rmdir_if_empty(dirfp):
6353+    """ Remove the directory if it is empty. """
6354+    try:
6355+        os.rmdir(dirfp.path)
6356+    except OSError, e:
6357+        if e.errno != errno.ENOTEMPTY:
6358+            raise
6359+    else:
6360+        dirfp.changed()
6361+
6362+def rmtree(dirname):
6363     """
6364     A threadsafe and idempotent version of shutil.rmtree().  If the dir is
6365     already gone, do nothing and return without raising an exception.  If this
6366hunk ./src/allmydata/util/fileutil.py 239
6367             else:
6368                 remove(fullname)
6369         os.rmdir(dirname)
6370-    except Exception, le:
6371-        # Ignore "No such file or directory"
6372-        if (not isinstance(le, OSError)) or le.args[0] != 2:
6373+    except EnvironmentError, le:
6374+        # Ignore "No such file or directory", collect any other exception.
6375+        if (le.args[0] != 2 and le.args[0] != 3) or (le.args[0] != errno.ENOENT):
6376             excs.append(le)
6377hunk ./src/allmydata/util/fileutil.py 243
6378+    except Exception, le:
6379+        excs.append(le)
6380 
6381     # Okay, now we've recursively removed everything, ignoring any "No
6382     # such file or directory" errors, and collecting any other errors.
6383hunk ./src/allmydata/util/fileutil.py 256
6384             raise OSError, "Failed to remove dir for unknown reason."
6385         raise OSError, excs
6386 
6387+def fp_remove(fp):
6388+    """
6389+    An idempotent version of shutil.rmtree().  If the file/dir is already
6390+    gone, do nothing and return without raising an exception.  If this call
6391+    removes the file/dir, return without raising an exception.  If there is
6392+    an error that prevents removal, or if a file or directory at the same
6393+    path gets created again by someone else after this deletes it and before
6394+    this checks that it is gone, raise an exception.
6395+    """
6396+    try:
6397+        fp.remove()
6398+    except UnlistableError, e:
6399+        if e.originalException.errno != errno.ENOENT:
6400+            raise
6401+    except OSError, e:
6402+        if e.errno != errno.ENOENT:
6403+            raise
6404+
6405+def rm_dir(dirname):
6406+    # Renamed to be like shutil.rmtree and unlike rmdir.
6407+    return rmtree(dirname)
6408 
6409 def remove_if_possible(f):
6410     try:
6411hunk ./src/allmydata/util/fileutil.py 387
6412         import traceback
6413         traceback.print_exc()
6414 
6415-def get_disk_stats(whichdir, reserved_space=0):
6416+def get_disk_stats(whichdirfp, reserved_space=0):
6417     """Return disk statistics for the storage disk, in the form of a dict
6418     with the following fields.
6419       total:            total bytes on disk
6420hunk ./src/allmydata/util/fileutil.py 408
6421     you can pass how many bytes you would like to leave unused on this
6422     filesystem as reserved_space.
6423     """
6424+    precondition(isinstance(whichdirfp, FilePath), whichdirfp)
6425 
6426     if have_GetDiskFreeSpaceExW:
6427         # If this is a Windows system and GetDiskFreeSpaceExW is available, use it.
6428hunk ./src/allmydata/util/fileutil.py 419
6429         n_free_for_nonroot = c_ulonglong(0)
6430         n_total            = c_ulonglong(0)
6431         n_free_for_root    = c_ulonglong(0)
6432-        retval = GetDiskFreeSpaceExW(whichdir, byref(n_free_for_nonroot),
6433-                                               byref(n_total),
6434-                                               byref(n_free_for_root))
6435+        retval = GetDiskFreeSpaceExW(whichdirfp.path, byref(n_free_for_nonroot),
6436+                                                      byref(n_total),
6437+                                                      byref(n_free_for_root))
6438         if retval == 0:
6439             raise OSError("Windows error %d attempting to get disk statistics for %r"
6440hunk ./src/allmydata/util/fileutil.py 424
6441-                          % (GetLastError(), whichdir))
6442+                          % (GetLastError(), whichdirfp.path))
6443         free_for_nonroot = n_free_for_nonroot.value
6444         total            = n_total.value
6445         free_for_root    = n_free_for_root.value
6446hunk ./src/allmydata/util/fileutil.py 433
6447         # <http://docs.python.org/library/os.html#os.statvfs>
6448         # <http://opengroup.org/onlinepubs/7990989799/xsh/fstatvfs.html>
6449         # <http://opengroup.org/onlinepubs/7990989799/xsh/sysstatvfs.h.html>
6450-        s = os.statvfs(whichdir)
6451+        s = os.statvfs(whichdirfp.path)
6452 
6453         # on my mac laptop:
6454         #  statvfs(2) is a wrapper around statfs(2).
6455hunk ./src/allmydata/util/fileutil.py 460
6456              'avail': avail,
6457            }
6458 
6459-def get_available_space(whichdir, reserved_space):
6460+def get_available_space(whichdirfp, reserved_space):
6461     """Returns available space for share storage in bytes, or None if no
6462     API to get this information is available.
6463 
6464hunk ./src/allmydata/util/fileutil.py 472
6465     you can pass how many bytes you would like to leave unused on this
6466     filesystem as reserved_space.
6467     """
6468+    precondition(isinstance(whichdirfp, FilePath), whichdirfp)
6469     try:
6470hunk ./src/allmydata/util/fileutil.py 474
6471-        return get_disk_stats(whichdir, reserved_space)['avail']
6472+        return get_disk_stats(whichdirfp, reserved_space)['avail']
6473     except AttributeError:
6474         return None
6475hunk ./src/allmydata/util/fileutil.py 477
6476-    except EnvironmentError:
6477-        log.msg("OS call to get disk statistics failed")
6478+
6479+
6480+def get_used_space(fp):
6481+    if fp is None:
6482         return 0
6483hunk ./src/allmydata/util/fileutil.py 482
6484+    try:
6485+        s = os.stat(fp.path)
6486+    except EnvironmentError:
6487+        if not fp.exists():
6488+            return 0
6489+        raise
6490+    else:
6491+        # POSIX defines st_blocks (originally a BSDism):
6492+        #   <http://pubs.opengroup.org/onlinepubs/009695399/basedefs/sys/stat.h.html>
6493+        # but does not require stat() to give it a "meaningful value"
6494+        #   <http://pubs.opengroup.org/onlinepubs/009695399/functions/stat.html>
6495+        # and says:
6496+        #   "The unit for the st_blocks member of the stat structure is not defined
6497+        #    within IEEE Std 1003.1-2001. In some implementations it is 512 bytes.
6498+        #    It may differ on a file system basis. There is no correlation between
6499+        #    values of the st_blocks and st_blksize, and the f_bsize (from <sys/statvfs.h>)
6500+        #    structure members."
6501+        #
6502+        # The Linux docs define it as "the number of blocks allocated to the file,
6503+        # [in] 512-byte units." It is also defined that way on MacOS X. Python does
6504+        # not set the attribute on Windows.
6505+        #
6506+        # We consider platforms that define st_blocks but give it a wrong value, or
6507+        # measure it in a unit other than 512 bytes, to be broken. See also
6508+        # <http://bugs.python.org/issue12350>.
6509+
6510+        if hasattr(s, 'st_blocks'):
6511+            return s.st_blocks * 512
6512+        else:
6513+            return s.st_size
6514}
6515[Work-in-progress, includes fix to bug involving BucketWriter. refs #999
6516david-sarah@jacaranda.org**20110920033803
6517 Ignore-this: 64e9e019421454e4d08141d10b6e4eed
6518] {
6519hunk ./src/allmydata/client.py 9
6520 from twisted.internet import reactor, defer
6521 from twisted.application import service
6522 from twisted.application.internet import TimerService
6523+from twisted.python.filepath import FilePath
6524 from foolscap.api import Referenceable
6525 from pycryptopp.publickey import rsa
6526 
6527hunk ./src/allmydata/client.py 15
6528 import allmydata
6529 from allmydata.storage.server import StorageServer
6530+from allmydata.storage.backends.disk.disk_backend import DiskBackend
6531 from allmydata import storage_client
6532 from allmydata.immutable.upload import Uploader
6533 from allmydata.immutable.offloaded import Helper
6534hunk ./src/allmydata/client.py 213
6535             return
6536         readonly = self.get_config("storage", "readonly", False, boolean=True)
6537 
6538-        storedir = os.path.join(self.basedir, self.STOREDIR)
6539+        storedir = FilePath(self.basedir).child(self.STOREDIR)
6540 
6541         data = self.get_config("storage", "reserved_space", None)
6542         reserved = None
6543hunk ./src/allmydata/client.py 255
6544             'cutoff_date': cutoff_date,
6545             'sharetypes': tuple(sharetypes),
6546         }
6547-        ss = StorageServer(storedir, self.nodeid,
6548-                           reserved_space=reserved,
6549-                           discard_storage=discard,
6550-                           readonly_storage=readonly,
6551+
6552+        backend = DiskBackend(storedir, readonly=readonly, reserved_space=reserved,
6553+                              discard_storage=discard)
6554+        ss = StorageServer(nodeid, backend, storedir,
6555                            stats_provider=self.stats_provider,
6556                            expiration_policy=expiration_policy)
6557         self.add_service(ss)
6558hunk ./src/allmydata/interfaces.py 348
6559 
6560     def get_shares():
6561         """
6562-        Generates the IStoredShare objects held in this shareset.
6563+        Generates IStoredShare objects for all completed shares in this shareset.
6564         """
6565 
6566     def has_incoming(shnum):
6567hunk ./src/allmydata/storage/backends/base.py 69
6568         # def _create_mutable_share(self, storageserver, shnum, write_enabler):
6569         #     """create a mutable share with the given shnum and write_enabler"""
6570 
6571-        # secrets might be a triple with cancel_secret in secrets[2], but if
6572-        # so we ignore the cancel_secret.
6573         write_enabler = secrets[0]
6574         renew_secret = secrets[1]
6575hunk ./src/allmydata/storage/backends/base.py 71
6576+        cancel_secret = '\x00'*32
6577+        if len(secrets) > 2:
6578+            cancel_secret = secrets[2]
6579 
6580         si_s = self.get_storage_index_string()
6581         shares = {}
6582hunk ./src/allmydata/storage/backends/base.py 110
6583             read_data[shnum] = share.readv(read_vector)
6584 
6585         ownerid = 1 # TODO
6586-        lease_info = LeaseInfo(ownerid, renew_secret,
6587+        lease_info = LeaseInfo(ownerid, renew_secret, cancel_secret,
6588                                expiration_time, storageserver.get_serverid())
6589 
6590         if testv_is_good:
6591hunk ./src/allmydata/storage/backends/disk/disk_backend.py 34
6592     return newfp.child(sia)
6593 
6594 
6595-def get_share(fp):
6596+def get_share(storageindex, shnum, fp):
6597     f = fp.open('rb')
6598     try:
6599         prefix = f.read(32)
6600hunk ./src/allmydata/storage/backends/disk/disk_backend.py 42
6601         f.close()
6602 
6603     if prefix == MutableDiskShare.MAGIC:
6604-        return MutableDiskShare(fp)
6605+        return MutableDiskShare(storageindex, shnum, fp)
6606     else:
6607         # assume it's immutable
6608hunk ./src/allmydata/storage/backends/disk/disk_backend.py 45
6609-        return ImmutableDiskShare(fp)
6610+        return ImmutableDiskShare(storageindex, shnum, fp)
6611 
6612 
6613 class DiskBackend(Backend):
6614hunk ./src/allmydata/storage/backends/disk/disk_backend.py 174
6615                 if not NUM_RE.match(shnumstr):
6616                     continue
6617                 sharehome = self._sharehomedir.child(shnumstr)
6618-                yield self.get_share(sharehome)
6619+                yield get_share(self.get_storage_index(), int(shnumstr), sharehome)
6620         except UnlistableError:
6621             # There is no shares directory at all.
6622             pass
6623hunk ./src/allmydata/storage/backends/disk/disk_backend.py 185
6624         return self._incominghomedir.child(str(shnum)).exists()
6625 
6626     def make_bucket_writer(self, storageserver, shnum, max_space_per_bucket, lease_info, canary):
6627-        sharehome = self._sharehomedir.child(str(shnum))
6628+        finalhome = self._sharehomedir.child(str(shnum))
6629         incominghome = self._incominghomedir.child(str(shnum))
6630hunk ./src/allmydata/storage/backends/disk/disk_backend.py 187
6631-        immsh = ImmutableDiskShare(self.get_storage_index(), shnum, sharehome, incominghome,
6632-                                   max_size=max_space_per_bucket, create=True)
6633+        immsh = ImmutableDiskShare(self.get_storage_index(), shnum, incominghome, finalhome,
6634+                                   max_size=max_space_per_bucket)
6635         bw = BucketWriter(storageserver, immsh, max_space_per_bucket, lease_info, canary)
6636         if self._discard_storage:
6637             bw.throw_out_all_data = True
6638hunk ./src/allmydata/storage/backends/disk/disk_backend.py 198
6639         fileutil.fp_make_dirs(self._sharehomedir)
6640         sharehome = self._sharehomedir.child(str(shnum))
6641         serverid = storageserver.get_serverid()
6642-        return create_mutable_disk_share(sharehome, serverid, write_enabler, storageserver)
6643+        return create_mutable_disk_share(self.get_storage_index(), shnum, sharehome, serverid, write_enabler, storageserver)
6644 
6645     def _clean_up_after_unlink(self):
6646         fileutil.fp_rmdir_if_empty(self._sharehomedir)
6647hunk ./src/allmydata/storage/backends/disk/immutable.py 48
6648     LEASE_SIZE = struct.calcsize(">L32s32sL")
6649 
6650 
6651-    def __init__(self, storageindex, shnum, finalhome=None, incominghome=None, max_size=None, create=False):
6652-        """ If max_size is not None then I won't allow more than
6653-        max_size to be written to me. If create=True then max_size
6654-        must not be None. """
6655-        precondition((max_size is not None) or (not create), max_size, create)
6656+    def __init__(self, storageindex, shnum, home, finalhome=None, max_size=None):
6657+        """
6658+        If max_size is not None then I won't allow more than max_size to be written to me.
6659+        If finalhome is not None (meaning that we are creating the share) then max_size
6660+        must not be None.
6661+        """
6662+        precondition((max_size is not None) or (finalhome is None), max_size, finalhome)
6663         self._storageindex = storageindex
6664         self._max_size = max_size
6665hunk ./src/allmydata/storage/backends/disk/immutable.py 57
6666-        self._incominghome = incominghome
6667-        self._home = finalhome
6668+
6669+        # If we are creating the share, _finalhome refers to the final path and
6670+        # _home to the incoming path. Otherwise, _finalhome is None.
6671+        self._finalhome = finalhome
6672+        self._home = home
6673         self._shnum = shnum
6674hunk ./src/allmydata/storage/backends/disk/immutable.py 63
6675-        if create:
6676-            # touch the file, so later callers will see that we're working on
6677+
6678+        if self._finalhome is not None:
6679+            # Touch the file, so later callers will see that we're working on
6680             # it. Also construct the metadata.
6681hunk ./src/allmydata/storage/backends/disk/immutable.py 67
6682-            assert not finalhome.exists()
6683-            fp_make_dirs(self._incominghome.parent())
6684+            assert not self._finalhome.exists()
6685+            fp_make_dirs(self._home.parent())
6686             # The second field -- the four-byte share data length -- is no
6687             # longer used as of Tahoe v1.3.0, but we continue to write it in
6688             # there in case someone downgrades a storage server from >=
6689hunk ./src/allmydata/storage/backends/disk/immutable.py 78
6690             # the largest length that can fit into the field. That way, even
6691             # if this does happen, the old < v1.3.0 server will still allow
6692             # clients to read the first part of the share.
6693-            self._incominghome.setContent(struct.pack(">LLL", 1, min(2**32-1, max_size), 0) )
6694+            self._home.setContent(struct.pack(">LLL", 1, min(2**32-1, max_size), 0) )
6695             self._lease_offset = max_size + 0x0c
6696             self._num_leases = 0
6697         else:
6698hunk ./src/allmydata/storage/backends/disk/immutable.py 101
6699                 % (si_b2a(self._storageindex), self._shnum, quote_filepath(self._home)))
6700 
6701     def close(self):
6702-        fileutil.fp_make_dirs(self._home.parent())
6703-        self._incominghome.moveTo(self._home)
6704-        try:
6705-            # self._incominghome is like storage/shares/incoming/ab/abcde/4 .
6706-            # We try to delete the parent (.../ab/abcde) to avoid leaving
6707-            # these directories lying around forever, but the delete might
6708-            # fail if we're working on another share for the same storage
6709-            # index (like ab/abcde/5). The alternative approach would be to
6710-            # use a hierarchy of objects (PrefixHolder, BucketHolder,
6711-            # ShareWriter), each of which is responsible for a single
6712-            # directory on disk, and have them use reference counting of
6713-            # their children to know when they should do the rmdir. This
6714-            # approach is simpler, but relies on os.rmdir refusing to delete
6715-            # a non-empty directory. Do *not* use fileutil.fp_remove() here!
6716-            fileutil.fp_rmdir_if_empty(self._incominghome.parent())
6717-            # we also delete the grandparent (prefix) directory, .../ab ,
6718-            # again to avoid leaving directories lying around. This might
6719-            # fail if there is another bucket open that shares a prefix (like
6720-            # ab/abfff).
6721-            fileutil.fp_rmdir_if_empty(self._incominghome.parent().parent())
6722-            # we leave the great-grandparent (incoming/) directory in place.
6723-        except EnvironmentError:
6724-            # ignore the "can't rmdir because the directory is not empty"
6725-            # exceptions, those are normal consequences of the
6726-            # above-mentioned conditions.
6727-            pass
6728-        pass
6729+        fileutil.fp_make_dirs(self._finalhome.parent())
6730+        self._home.moveTo(self._finalhome)
6731+
6732+        # self._home is like storage/shares/incoming/ab/abcde/4 .
6733+        # We try to delete the parent (.../ab/abcde) to avoid leaving
6734+        # these directories lying around forever, but the delete might
6735+        # fail if we're working on another share for the same storage
6736+        # index (like ab/abcde/5). The alternative approach would be to
6737+        # use a hierarchy of objects (PrefixHolder, BucketHolder,
6738+        # ShareWriter), each of which is responsible for a single
6739+        # directory on disk, and have them use reference counting of
6740+        # their children to know when they should do the rmdir. This
6741+        # approach is simpler, but relies on os.rmdir (used by
6742+        # fp_rmdir_if_empty) refusing to delete a non-empty directory.
6743+        # Do *not* use fileutil.fp_remove() here!
6744+        parent = self._home.parent()
6745+        fileutil.fp_rmdir_if_empty(parent)
6746+
6747+        # we also delete the grandparent (prefix) directory, .../ab ,
6748+        # again to avoid leaving directories lying around. This might
6749+        # fail if there is another bucket open that shares a prefix (like
6750+        # ab/abfff).
6751+        fileutil.fp_rmdir_if_empty(parent.parent())
6752+
6753+        # we leave the great-grandparent (incoming/) directory in place.
6754+
6755+        # allow lease changes after closing.
6756+        self._home = self._finalhome
6757+        self._finalhome = None
6758 
6759     def get_used_space(self):
6760hunk ./src/allmydata/storage/backends/disk/immutable.py 132
6761-        return (fileutil.get_used_space(self._home) +
6762-                fileutil.get_used_space(self._incominghome))
6763+        return (fileutil.get_used_space(self._finalhome) +
6764+                fileutil.get_used_space(self._home))
6765 
6766     def get_storage_index(self):
6767         return self._storageindex
6768hunk ./src/allmydata/storage/backends/disk/immutable.py 175
6769         precondition(offset >= 0, offset)
6770         if self._max_size is not None and offset+length > self._max_size:
6771             raise DataTooLargeError(self._max_size, offset, length)
6772-        f = self._incominghome.open(mode='rb+')
6773+        f = self._home.open(mode='rb+')
6774         try:
6775             real_offset = self._data_offset+offset
6776             f.seek(real_offset)
6777hunk ./src/allmydata/storage/backends/disk/immutable.py 205
6778 
6779     # These lease operations are intended for use by disk_backend.py.
6780     # Other clients should not depend on the fact that the disk backend
6781-    # stores leases in share files.
6782+    # stores leases in share files. XXX bucket.py also relies on this.
6783 
6784     def get_leases(self):
6785         """Yields a LeaseInfo instance for all leases."""
6786hunk ./src/allmydata/storage/backends/disk/immutable.py 221
6787             f.close()
6788 
6789     def add_lease(self, lease_info):
6790-        f = self._incominghome.open(mode='rb')
6791+        f = self._home.open(mode='rb+')
6792         try:
6793             num_leases = self._read_num_leases(f)
6794hunk ./src/allmydata/storage/backends/disk/immutable.py 224
6795-        finally:
6796-            f.close()
6797-        f = self._home.open(mode='wb+')
6798-        try:
6799             self._write_lease_record(f, num_leases, lease_info)
6800             self._write_num_leases(f, num_leases+1)
6801         finally:
6802hunk ./src/allmydata/storage/backends/disk/mutable.py 440
6803         pass
6804 
6805 
6806-def create_mutable_disk_share(fp, serverid, write_enabler, parent):
6807-    ms = MutableDiskShare(fp, parent)
6808+def create_mutable_disk_share(storageindex, shnum, fp, serverid, write_enabler, parent):
6809+    ms = MutableDiskShare(storageindex, shnum, fp, parent)
6810     ms.create(serverid, write_enabler)
6811     del ms
6812hunk ./src/allmydata/storage/backends/disk/mutable.py 444
6813-    return MutableDiskShare(fp, parent)
6814+    return MutableDiskShare(storageindex, shnum, fp, parent)
6815hunk ./src/allmydata/storage/bucket.py 44
6816         start = time.time()
6817 
6818         self._share.close()
6819-        filelen = self._share.stat()
6820+        # XXX should this be self._share.get_used_space() ?
6821+        consumed_size = self._share.get_size()
6822         self._share = None
6823 
6824         self.closed = True
6825hunk ./src/allmydata/storage/bucket.py 51
6826         self._canary.dontNotifyOnDisconnect(self._disconnect_marker)
6827 
6828-        self.ss.bucket_writer_closed(self, filelen)
6829+        self.ss.bucket_writer_closed(self, consumed_size)
6830         self.ss.add_latency("close", time.time() - start)
6831         self.ss.count("close")
6832 
6833hunk ./src/allmydata/storage/server.py 182
6834                                 renew_secret, cancel_secret,
6835                                 sharenums, allocated_size,
6836                                 canary, owner_num=0):
6837-        # cancel_secret is no longer used.
6838         # owner_num is not for clients to set, but rather it should be
6839         # curried into a StorageServer instance dedicated to a particular
6840         # owner.
6841hunk ./src/allmydata/storage/server.py 195
6842         # Note that the lease should not be added until the BucketWriter
6843         # has been closed.
6844         expire_time = time.time() + 31*24*60*60
6845-        lease_info = LeaseInfo(owner_num, renew_secret,
6846+        lease_info = LeaseInfo(owner_num, renew_secret, cancel_secret,
6847                                expire_time, self._serverid)
6848 
6849         max_space_per_bucket = allocated_size
6850hunk ./src/allmydata/test/no_network.py 349
6851         return self.g.servers_by_number[i]
6852 
6853     def get_serverdir(self, i):
6854-        return self.g.servers_by_number[i].backend.storedir
6855+        return self.g.servers_by_number[i].backend._storedir
6856 
6857     def remove_server(self, i):
6858         self.g.remove_server(self.g.servers_by_number[i].get_serverid())
6859hunk ./src/allmydata/test/no_network.py 357
6860     def iterate_servers(self):
6861         for i in sorted(self.g.servers_by_number.keys()):
6862             ss = self.g.servers_by_number[i]
6863-            yield (i, ss, ss.backend.storedir)
6864+            yield (i, ss, ss.backend._storedir)
6865 
6866     def find_uri_shares(self, uri):
6867         si = tahoe_uri.from_string(uri).get_storage_index()
6868hunk ./src/allmydata/test/no_network.py 384
6869         return shares
6870 
6871     def copy_share(self, from_share, uri, to_server):
6872-        si = uri.from_string(self.uri).get_storage_index()
6873+        si = tahoe_uri.from_string(uri).get_storage_index()
6874         (i_shnum, i_serverid, i_sharefp) = from_share
6875         shares_dir = to_server.backend.get_shareset(si)._sharehomedir
6876         i_sharefp.copyTo(shares_dir.child(str(i_shnum)))
6877hunk ./src/allmydata/test/test_download.py 127
6878 
6879         return d
6880 
6881-    def _write_shares(self, uri, shares):
6882-        si = uri.from_string(uri).get_storage_index()
6883+    def _write_shares(self, fileuri, shares):
6884+        si = uri.from_string(fileuri).get_storage_index()
6885         for i in shares:
6886             shares_for_server = shares[i]
6887             for shnum in shares_for_server:
6888hunk ./src/allmydata/test/test_hung_server.py 36
6889 
6890     def _hang(self, servers, **kwargs):
6891         for ss in servers:
6892-            self.g.hang_server(ss.get_serverid(), **kwargs)
6893+            self.g.hang_server(ss.original.get_serverid(), **kwargs)
6894 
6895     def _unhang(self, servers, **kwargs):
6896         for ss in servers:
6897hunk ./src/allmydata/test/test_hung_server.py 40
6898-            self.g.unhang_server(ss.get_serverid(), **kwargs)
6899+            self.g.unhang_server(ss.original.get_serverid(), **kwargs)
6900 
6901     def _hang_shares(self, shnums, **kwargs):
6902         # hang all servers who are holding the given shares
6903hunk ./src/allmydata/test/test_hung_server.py 52
6904                     hung_serverids.add(i_serverid)
6905 
6906     def _delete_all_shares_from(self, servers):
6907-        serverids = [ss.get_serverid() for ss in servers]
6908+        serverids = [ss.original.get_serverid() for ss in servers]
6909         for (i_shnum, i_serverid, i_sharefp) in self.shares:
6910             if i_serverid in serverids:
6911                 i_sharefp.remove()
6912hunk ./src/allmydata/test/test_hung_server.py 58
6913 
6914     def _corrupt_all_shares_in(self, servers, corruptor_func):
6915-        serverids = [ss.get_serverid() for ss in servers]
6916+        serverids = [ss.original.get_serverid() for ss in servers]
6917         for (i_shnum, i_serverid, i_sharefp) in self.shares:
6918             if i_serverid in serverids:
6919                 self.corrupt_share((i_shnum, i_serverid, i_sharefp), corruptor_func)
6920hunk ./src/allmydata/test/test_hung_server.py 64
6921 
6922     def _copy_all_shares_from(self, from_servers, to_server):
6923-        serverids = [ss.get_serverid() for ss in from_servers]
6924+        serverids = [ss.original.get_serverid() for ss in from_servers]
6925         for (i_shnum, i_serverid, i_sharefp) in self.shares:
6926             if i_serverid in serverids:
6927                 self.copy_share((i_shnum, i_serverid, i_sharefp), self.uri, to_server)
6928hunk ./src/allmydata/test/test_mutable.py 2991
6929             fso = debug.FindSharesOptions()
6930             storage_index = base32.b2a(n.get_storage_index())
6931             fso.si_s = storage_index
6932-            fso.nodedirs = [unicode(os.path.dirname(os.path.abspath(storedir)))
6933+            fso.nodedirs = [unicode(storedir.parent().path)
6934                             for (i,ss,storedir)
6935                             in self.iterate_servers()]
6936             fso.stdout = StringIO()
6937hunk ./src/allmydata/test/test_upload.py 818
6938         if share_number is not None:
6939             self._copy_share_to_server(share_number, server_number)
6940 
6941-
6942     def _copy_share_to_server(self, share_number, server_number):
6943         ss = self.g.servers_by_number[server_number]
6944hunk ./src/allmydata/test/test_upload.py 820
6945-        self.copy_share(self.shares[share_number], ss)
6946+        self.copy_share(self.shares[share_number], self.uri, ss)
6947 
6948     def _setup_grid(self):
6949         """
6950}
6951[docs/backends: document the configuration options for the pluggable backends scheme. refs #999
6952david-sarah@jacaranda.org**20110920171737
6953 Ignore-this: 5947e864682a43cb04e557334cda7c19
6954] {
6955adddir ./docs/backends
6956addfile ./docs/backends/S3.rst
6957hunk ./docs/backends/S3.rst 1
6958+====================================================
6959+Storing Shares in Amazon Simple Storage Service (S3)
6960+====================================================
6961+
6962+S3 is a commercial storage service provided by Amazon, described at
6963+`<https://aws.amazon.com/s3/>`_.
6964+
6965+The Tahoe-LAFS storage server can be configured to store its shares in
6966+an S3 bucket, rather than on local filesystem. To enable this, add the
6967+following keys to the server's ``tahoe.cfg`` file:
6968+
6969+``[storage]``
6970+
6971+``backend = s3``
6972+
6973+    This turns off the local filesystem backend and enables use of S3.
6974+
6975+``s3.access_key_id = (string, required)``
6976+``s3.secret_access_key = (string, required)``
6977+
6978+    These two give the storage server permission to access your Amazon
6979+    Web Services account, allowing them to upload and download shares
6980+    from S3.
6981+
6982+``s3.bucket = (string, required)``
6983+
6984+    This controls which bucket will be used to hold shares. The Tahoe-LAFS
6985+    storage server will only modify and access objects in the configured S3
6986+    bucket.
6987+
6988+``s3.url = (URL string, optional)``
6989+
6990+    This URL tells the storage server how to access the S3 service. It
6991+    defaults to ``http://s3.amazonaws.com``, but by setting it to something
6992+    else, you may be able to use some other S3-like service if it is
6993+    sufficiently compatible.
6994+
6995+``s3.max_space = (str, optional)``
6996+
6997+    This tells the server to limit how much space can be used in the S3
6998+    bucket. Before each share is uploaded, the server will ask S3 for the
6999+    current bucket usage, and will only accept the share if it does not cause
7000+    the usage to grow above this limit.
7001+
7002+    The string contains a number, with an optional case-insensitive scale
7003+    suffix like "K" or "M" or "G", and an optional "B" or "iB" suffix. So
7004+    "100MB", "100M", "100000000B", "100000000", and "100000kb" all mean the
7005+    same thing. Likewise, "1MiB", "1024KiB", and "1048576B" all mean the same
7006+    thing.
7007+
7008+    If ``s3.max_space`` is omitted, the default behavior is to allow
7009+    unlimited usage.
7010+
7011+
7012+Once configured, the WUI "storage server" page will provide information about
7013+how much space is being used and how many shares are being stored.
7014+
7015+
7016+Issues
7017+------
7018+
7019+Objects in an S3 bucket cannot be read for free. As a result, when Tahoe-LAFS
7020+is configured to store shares in S3 rather than on local disk, some common
7021+operations may behave differently:
7022+
7023+* Lease crawling/expiration is not yet implemented. As a result, shares will
7024+  be retained forever, and the Storage Server status web page will not show
7025+  information about the number of mutable/immutable shares present.
7026+
7027+* Enabling ``s3.max_space`` causes an extra S3 usage query to be sent for
7028+  each share upload, causing the upload process to run slightly slower and
7029+  incur more S3 request charges.
7030addfile ./docs/backends/disk.rst
7031hunk ./docs/backends/disk.rst 1
7032+====================================
7033+Storing Shares on a Local Filesystem
7034+====================================
7035+
7036+The "disk" backend stores shares on the local filesystem. Versions of
7037+Tahoe-LAFS <= 1.9.0 always stored shares in this way.
7038+
7039+``[storage]``
7040+
7041+``backend = disk``
7042+
7043+    This enables use of the disk backend, and is the default.
7044+
7045+``reserved_space = (str, optional)``
7046+
7047+    If provided, this value defines how much disk space is reserved: the
7048+    storage server will not accept any share that causes the amount of free
7049+    disk space to drop below this value. (The free space is measured by a
7050+    call to statvfs(2) on Unix, or GetDiskFreeSpaceEx on Windows, and is the
7051+    space available to the user account under which the storage server runs.)
7052+
7053+    This string contains a number, with an optional case-insensitive scale
7054+    suffix like "K" or "M" or "G", and an optional "B" or "iB" suffix. So
7055+    "100MB", "100M", "100000000B", "100000000", and "100000kb" all mean the
7056+    same thing. Likewise, "1MiB", "1024KiB", and "1048576B" all mean the same
7057+    thing.
7058+
7059+    "``tahoe create-node``" generates a tahoe.cfg with
7060+    "``reserved_space=1G``", but you may wish to raise, lower, or remove the
7061+    reservation to suit your needs.
7062+
7063+``expire.enabled =``
7064+
7065+``expire.mode =``
7066+
7067+``expire.override_lease_duration =``
7068+
7069+``expire.cutoff_date =``
7070+
7071+``expire.immutable =``
7072+
7073+``expire.mutable =``
7074+
7075+    These settings control garbage collection, causing the server to
7076+    delete shares that no longer have an up-to-date lease on them. Please
7077+    see `<garbage-collection.rst>`_ for full details.
7078hunk ./docs/configuration.rst 412
7079     <http://tahoe-lafs.org/trac/tahoe-lafs/ticket/390>`_ for the current
7080     status of this bug. The default value is ``False``.
7081 
7082-``reserved_space = (str, optional)``
7083+``backend = (string, optional)``
7084 
7085hunk ./docs/configuration.rst 414
7086-    If provided, this value defines how much disk space is reserved: the
7087-    storage server will not accept any share that causes the amount of free
7088-    disk space to drop below this value. (The free space is measured by a
7089-    call to statvfs(2) on Unix, or GetDiskFreeSpaceEx on Windows, and is the
7090-    space available to the user account under which the storage server runs.)
7091+    Storage servers can store the data into different "backends". Clients
7092+    need not be aware of which backend is used by a server. The default
7093+    value is ``disk``.
7094 
7095hunk ./docs/configuration.rst 418
7096-    This string contains a number, with an optional case-insensitive scale
7097-    suffix like "K" or "M" or "G", and an optional "B" or "iB" suffix. So
7098-    "100MB", "100M", "100000000B", "100000000", and "100000kb" all mean the
7099-    same thing. Likewise, "1MiB", "1024KiB", and "1048576B" all mean the same
7100-    thing.
7101+``backend = disk``
7102 
7103hunk ./docs/configuration.rst 420
7104-    "``tahoe create-node``" generates a tahoe.cfg with
7105-    "``reserved_space=1G``", but you may wish to raise, lower, or remove the
7106-    reservation to suit your needs.
7107+    The default is to store shares on the local filesystem (in
7108+    BASEDIR/storage/shares/). For configuration details (including how to
7109+    reserve a minimum amount of free space), see `<backends/disk.rst>`_.
7110 
7111hunk ./docs/configuration.rst 424
7112-``expire.enabled =``
7113+``backend = S3``
7114 
7115hunk ./docs/configuration.rst 426
7116-``expire.mode =``
7117-
7118-``expire.override_lease_duration =``
7119-
7120-``expire.cutoff_date =``
7121-
7122-``expire.immutable =``
7123-
7124-``expire.mutable =``
7125-
7126-    These settings control garbage collection, in which the server will
7127-    delete shares that no longer have an up-to-date lease on them. Please see
7128-    `<garbage-collection.rst>`_ for full details.
7129+    The storage server can store all shares to an Amazon Simple Storage
7130+    Service (S3) bucket. For configuration details, see `<backends/S3.rst>`_.
7131 
7132 
7133 Running A Helper
7134}
7135[Fix some incorrect attribute accesses. refs #999
7136david-sarah@jacaranda.org**20110921031207
7137 Ignore-this: f1ea4c3ea191f6d4b719afaebd2b2bcd
7138] {
7139hunk ./src/allmydata/client.py 258
7140 
7141         backend = DiskBackend(storedir, readonly=readonly, reserved_space=reserved,
7142                               discard_storage=discard)
7143-        ss = StorageServer(nodeid, backend, storedir,
7144+        ss = StorageServer(self.nodeid, backend, storedir,
7145                            stats_provider=self.stats_provider,
7146                            expiration_policy=expiration_policy)
7147         self.add_service(ss)
7148hunk ./src/allmydata/interfaces.py 449
7149         Returns the storage index.
7150         """
7151 
7152+    def get_storage_index_string():
7153+        """
7154+        Returns the base32-encoded storage index.
7155+        """
7156+
7157     def get_shnum():
7158         """
7159         Returns the share number.
7160hunk ./src/allmydata/storage/backends/disk/immutable.py 138
7161     def get_storage_index(self):
7162         return self._storageindex
7163 
7164+    def get_storage_index_string(self):
7165+        return si_b2a(self._storageindex)
7166+
7167     def get_shnum(self):
7168         return self._shnum
7169 
7170hunk ./src/allmydata/storage/backends/disk/mutable.py 119
7171     def get_storage_index(self):
7172         return self._storageindex
7173 
7174+    def get_storage_index_string(self):
7175+        return si_b2a(self._storageindex)
7176+
7177     def get_shnum(self):
7178         return self._shnum
7179 
7180hunk ./src/allmydata/storage/bucket.py 86
7181     def __init__(self, ss, share):
7182         self.ss = ss
7183         self._share = share
7184-        self.storageindex = share.storageindex
7185-        self.shnum = share.shnum
7186+        self.storageindex = share.get_storage_index()
7187+        self.shnum = share.get_shnum()
7188 
7189     def __repr__(self):
7190         return "<%s %s %s>" % (self.__class__.__name__,
7191hunk ./src/allmydata/storage/expirer.py 6
7192 from twisted.python import log as twlog
7193 
7194 from allmydata.storage.crawler import ShareCrawler
7195-from allmydata.storage.common import si_b2a, UnknownMutableContainerVersionError, \
7196+from allmydata.storage.common import UnknownMutableContainerVersionError, \
7197      UnknownImmutableContainerVersionError
7198 
7199 
7200hunk ./src/allmydata/storage/expirer.py 124
7201                     struct.error):
7202                 twlog.msg("lease-checker error processing %r" % (share,))
7203                 twlog.err()
7204-                which = (si_b2a(share.storageindex), share.get_shnum())
7205+                which = (share.get_storage_index_string(), share.get_shnum())
7206                 self.state["cycle-to-date"]["corrupt-shares"].append(which)
7207                 wks = (1, 1, 1, "unknown")
7208             would_keep_shares.append(wks)
7209hunk ./src/allmydata/storage/server.py 221
7210         alreadygot = set()
7211         for share in shareset.get_shares():
7212             share.add_or_renew_lease(lease_info)
7213-            alreadygot.add(share.shnum)
7214+            alreadygot.add(share.get_shnum())
7215 
7216         for shnum in sharenums - alreadygot:
7217             if shareset.has_incoming(shnum):
7218hunk ./src/allmydata/storage/server.py 324
7219 
7220         try:
7221             shareset = self.backend.get_shareset(storageindex)
7222-            return shareset.readv(self, shares, readv)
7223+            return shareset.readv(shares, readv)
7224         finally:
7225             self.add_latency("readv", time.time() - start)
7226 
7227hunk ./src/allmydata/storage/shares.py 1
7228-#! /usr/bin/python
7229-
7230-from allmydata.storage.mutable import MutableShareFile
7231-from allmydata.storage.immutable import ShareFile
7232-
7233-def get_share_file(filename):
7234-    f = open(filename, "rb")
7235-    prefix = f.read(32)
7236-    f.close()
7237-    if prefix == MutableShareFile.MAGIC:
7238-        return MutableShareFile(filename)
7239-    # otherwise assume it's immutable
7240-    return ShareFile(filename)
7241-
7242rmfile ./src/allmydata/storage/shares.py
7243hunk ./src/allmydata/test/no_network.py 387
7244         si = tahoe_uri.from_string(uri).get_storage_index()
7245         (i_shnum, i_serverid, i_sharefp) = from_share
7246         shares_dir = to_server.backend.get_shareset(si)._sharehomedir
7247+        fileutil.fp_make_dirs(shares_dir)
7248         i_sharefp.copyTo(shares_dir.child(str(i_shnum)))
7249 
7250     def restore_all_shares(self, shares):
7251hunk ./src/allmydata/test/no_network.py 391
7252-        for share, data in shares.items():
7253-            share.home.setContent(data)
7254+        for sharepath, data in shares.items():
7255+            FilePath(sharepath).setContent(data)
7256 
7257     def delete_share(self, (shnum, serverid, sharefp)):
7258         sharefp.remove()
7259hunk ./src/allmydata/test/test_upload.py 744
7260         servertoshnums = {} # k: server, v: set(shnum)
7261 
7262         for i, c in self.g.servers_by_number.iteritems():
7263-            for (dirp, dirns, fns) in os.walk(c.sharedir):
7264+            for (dirp, dirns, fns) in os.walk(c.backend._sharedir.path):
7265                 for fn in fns:
7266                     try:
7267                         sharenum = int(fn)
7268}
7269[docs/backends/S3.rst: remove Issues section. refs #999
7270david-sarah@jacaranda.org**20110921031625
7271 Ignore-this: c83d8f52b790bc32488869e6ee1df8c2
7272] hunk ./docs/backends/S3.rst 57
7273 
7274 Once configured, the WUI "storage server" page will provide information about
7275 how much space is being used and how many shares are being stored.
7276-
7277-
7278-Issues
7279-------
7280-
7281-Objects in an S3 bucket cannot be read for free. As a result, when Tahoe-LAFS
7282-is configured to store shares in S3 rather than on local disk, some common
7283-operations may behave differently:
7284-
7285-* Lease crawling/expiration is not yet implemented. As a result, shares will
7286-  be retained forever, and the Storage Server status web page will not show
7287-  information about the number of mutable/immutable shares present.
7288-
7289-* Enabling ``s3.max_space`` causes an extra S3 usage query to be sent for
7290-  each share upload, causing the upload process to run slightly slower and
7291-  incur more S3 request charges.
7292[docs/backends/S3.rst, disk.rst: describe type of space settings as 'quantity of space', not 'str'. refs #999
7293david-sarah@jacaranda.org**20110921031705
7294 Ignore-this: a74ed8e01b0a1ab5f07a1487d7bf138
7295] {
7296hunk ./docs/backends/S3.rst 38
7297     else, you may be able to use some other S3-like service if it is
7298     sufficiently compatible.
7299 
7300-``s3.max_space = (str, optional)``
7301+``s3.max_space = (quantity of space, optional)``
7302 
7303     This tells the server to limit how much space can be used in the S3
7304     bucket. Before each share is uploaded, the server will ask S3 for the
7305hunk ./docs/backends/disk.rst 14
7306 
7307     This enables use of the disk backend, and is the default.
7308 
7309-``reserved_space = (str, optional)``
7310+``reserved_space = (quantity of space, optional)``
7311 
7312     If provided, this value defines how much disk space is reserved: the
7313     storage server will not accept any share that causes the amount of free
7314}
7315[More fixes to tests needed for pluggable backends. refs #999
7316david-sarah@jacaranda.org**20110921184649
7317 Ignore-this: 9be0d3a98e350fd4e17a07d2c00bb4ca
7318] {
7319hunk ./src/allmydata/scripts/debug.py 8
7320 from twisted.python import usage, failure
7321 from twisted.internet import defer
7322 from twisted.scripts import trial as twisted_trial
7323+from twisted.python.filepath import FilePath
7324 
7325 
7326 class DumpOptions(usage.Options):
7327hunk ./src/allmydata/scripts/debug.py 38
7328         self['filename'] = argv_to_abspath(filename)
7329 
7330 def dump_share(options):
7331-    from allmydata.storage.mutable import MutableShareFile
7332+    from allmydata.storage.backends.disk.disk_backend import get_share
7333     from allmydata.util.encodingutil import quote_output
7334 
7335     out = options.stdout
7336hunk ./src/allmydata/scripts/debug.py 46
7337     # check the version, to see if we have a mutable or immutable share
7338     print >>out, "share filename: %s" % quote_output(options['filename'])
7339 
7340-    f = open(options['filename'], "rb")
7341-    prefix = f.read(32)
7342-    f.close()
7343-    if prefix == MutableShareFile.MAGIC:
7344-        return dump_mutable_share(options)
7345-    # otherwise assume it's immutable
7346-    return dump_immutable_share(options)
7347-
7348-def dump_immutable_share(options):
7349-    from allmydata.storage.immutable import ShareFile
7350+    share = get_share("", 0, fp)
7351+    if share.sharetype == "mutable":
7352+        return dump_mutable_share(options, share)
7353+    else:
7354+        assert share.sharetype == "immutable", share.sharetype
7355+        return dump_immutable_share(options)
7356 
7357hunk ./src/allmydata/scripts/debug.py 53
7358+def dump_immutable_share(options, share):
7359     out = options.stdout
7360hunk ./src/allmydata/scripts/debug.py 55
7361-    f = ShareFile(options['filename'])
7362     if not options["leases-only"]:
7363hunk ./src/allmydata/scripts/debug.py 56
7364-        dump_immutable_chk_share(f, out, options)
7365-    dump_immutable_lease_info(f, out)
7366+        dump_immutable_chk_share(share, out, options)
7367+    dump_immutable_lease_info(share, out)
7368     print >>out
7369     return 0
7370 
7371hunk ./src/allmydata/scripts/debug.py 166
7372     return when
7373 
7374 
7375-def dump_mutable_share(options):
7376-    from allmydata.storage.mutable import MutableShareFile
7377+def dump_mutable_share(options, m):
7378     from allmydata.util import base32, idlib
7379     out = options.stdout
7380hunk ./src/allmydata/scripts/debug.py 169
7381-    m = MutableShareFile(options['filename'])
7382     f = open(options['filename'], "rb")
7383     WE, nodeid = m._read_write_enabler_and_nodeid(f)
7384     num_extra_leases = m._read_num_extra_leases(f)
7385hunk ./src/allmydata/scripts/debug.py 641
7386     /home/warner/testnet/node-1/storage/shares/44k/44kai1tui348689nrw8fjegc8c/9
7387     /home/warner/testnet/node-2/storage/shares/44k/44kai1tui348689nrw8fjegc8c/2
7388     """
7389-    from allmydata.storage.server import si_a2b, storage_index_to_dir
7390-    from allmydata.util.encodingutil import listdir_unicode
7391+    from allmydata.storage.server import si_a2b
7392+    from allmydata.storage.backends.disk_backend import si_si2dir
7393+    from allmydata.util.encodingutil import quote_filepath
7394 
7395     out = options.stdout
7396hunk ./src/allmydata/scripts/debug.py 646
7397-    sharedir = storage_index_to_dir(si_a2b(options.si_s))
7398-    for d in options.nodedirs:
7399-        d = os.path.join(d, "storage/shares", sharedir)
7400-        if os.path.exists(d):
7401-            for shnum in listdir_unicode(d):
7402-                print >>out, os.path.join(d, shnum)
7403+    si = si_a2b(options.si_s)
7404+    for nodedir in options.nodedirs:
7405+        sharedir = si_si2dir(nodedir.child("storage").child("shares"), si)
7406+        if sharedir.exists():
7407+            for sharefp in sharedir.children():
7408+                print >>out, quote_filepath(sharefp, quotemarks=False)
7409 
7410     return 0
7411 
7412hunk ./src/allmydata/scripts/debug.py 878
7413         print >>err, "Error processing %s" % quote_output(si_dir)
7414         failure.Failure().printTraceback(err)
7415 
7416+
7417 class CorruptShareOptions(usage.Options):
7418     def getSynopsis(self):
7419         return "Usage: tahoe debug corrupt-share SHARE_FILENAME"
7420hunk ./src/allmydata/scripts/debug.py 902
7421 Obviously, this command should not be used in normal operation.
7422 """
7423         return t
7424+
7425     def parseArgs(self, filename):
7426         self['filename'] = filename
7427 
7428hunk ./src/allmydata/scripts/debug.py 907
7429 def corrupt_share(options):
7430+    do_corrupt_share(options.stdout, FilePath(options['filename']), options['offset'])
7431+
7432+def do_corrupt_share(out, fp, offset="block-random"):
7433     import random
7434hunk ./src/allmydata/scripts/debug.py 911
7435-    from allmydata.storage.mutable import MutableShareFile
7436-    from allmydata.storage.immutable import ShareFile
7437+    from allmydata.storage.backends.disk.mutable import MutableDiskShare
7438+    from allmydata.storage.backends.disk.immutable import ImmutableDiskShare
7439     from allmydata.mutable.layout import unpack_header
7440     from allmydata.immutable.layout import ReadBucketProxy
7441hunk ./src/allmydata/scripts/debug.py 915
7442-    out = options.stdout
7443-    fn = options['filename']
7444-    assert options["offset"] == "block-random", "other offsets not implemented"
7445+
7446+    assert offset == "block-random", "other offsets not implemented"
7447+
7448     # first, what kind of share is it?
7449 
7450     def flip_bit(start, end):
7451hunk ./src/allmydata/scripts/debug.py 924
7452         offset = random.randrange(start, end)
7453         bit = random.randrange(0, 8)
7454         print >>out, "[%d..%d):  %d.b%d" % (start, end, offset, bit)
7455-        f = open(fn, "rb+")
7456-        f.seek(offset)
7457-        d = f.read(1)
7458-        d = chr(ord(d) ^ 0x01)
7459-        f.seek(offset)
7460-        f.write(d)
7461-        f.close()
7462+        f = fp.open("rb+")
7463+        try:
7464+            f.seek(offset)
7465+            d = f.read(1)
7466+            d = chr(ord(d) ^ 0x01)
7467+            f.seek(offset)
7468+            f.write(d)
7469+        finally:
7470+            f.close()
7471 
7472hunk ./src/allmydata/scripts/debug.py 934
7473-    f = open(fn, "rb")
7474-    prefix = f.read(32)
7475-    f.close()
7476-    if prefix == MutableShareFile.MAGIC:
7477-        # mutable
7478-        m = MutableShareFile(fn)
7479-        f = open(fn, "rb")
7480-        f.seek(m.DATA_OFFSET)
7481-        data = f.read(2000)
7482-        # make sure this slot contains an SMDF share
7483-        assert data[0] == "\x00", "non-SDMF mutable shares not supported"
7484+    f = fp.open("rb")
7485+    try:
7486+        prefix = f.read(32)
7487+    finally:
7488         f.close()
7489hunk ./src/allmydata/scripts/debug.py 939
7490+    if prefix == MutableDiskShare.MAGIC:
7491+        # mutable
7492+        m = MutableDiskShare("", 0, fp)
7493+        f = fp.open("rb")
7494+        try:
7495+            f.seek(m.DATA_OFFSET)
7496+            data = f.read(2000)
7497+            # make sure this slot contains an SMDF share
7498+            assert data[0] == "\x00", "non-SDMF mutable shares not supported"
7499+        finally:
7500+            f.close()
7501 
7502         (version, ig_seqnum, ig_roothash, ig_IV, ig_k, ig_N, ig_segsize,
7503          ig_datalen, offsets) = unpack_header(data)
7504hunk ./src/allmydata/scripts/debug.py 960
7505         flip_bit(start, end)
7506     else:
7507         # otherwise assume it's immutable
7508-        f = ShareFile(fn)
7509+        f = ImmutableDiskShare("", 0, fp)
7510         bp = ReadBucketProxy(None, None, '')
7511         offsets = bp._parse_offsets(f.read_share_data(0, 0x24))
7512         start = f._data_offset + offsets["data"]
7513hunk ./src/allmydata/storage/backends/base.py 92
7514             (testv, datav, new_length) = test_and_write_vectors[sharenum]
7515             if sharenum in shares:
7516                 if not shares[sharenum].check_testv(testv):
7517-                    self.log("testv failed: [%d]: %r" % (sharenum, testv))
7518+                    storageserver.log("testv failed: [%d]: %r" % (sharenum, testv))
7519                     testv_is_good = False
7520                     break
7521             else:
7522hunk ./src/allmydata/storage/backends/base.py 99
7523                 # compare the vectors against an empty share, in which all
7524                 # reads return empty strings
7525                 if not EmptyShare().check_testv(testv):
7526-                    self.log("testv failed (empty): [%d] %r" % (sharenum,
7527-                                                                testv))
7528+                    storageserver.log("testv failed (empty): [%d] %r" % (sharenum, testv))
7529                     testv_is_good = False
7530                     break
7531 
7532hunk ./src/allmydata/test/test_cli.py 2892
7533             # delete one, corrupt a second
7534             shares = self.find_uri_shares(self.uri)
7535             self.failUnlessReallyEqual(len(shares), 10)
7536-            os.unlink(shares[0][2])
7537-            cso = debug.CorruptShareOptions()
7538-            cso.stdout = StringIO()
7539-            cso.parseOptions([shares[1][2]])
7540+            shares[0][2].remove()
7541+            stdout = StringIO()
7542+            sharefile = shares[1][2]
7543             storage_index = uri.from_string(self.uri).get_storage_index()
7544             self._corrupt_share_line = "  server %s, SI %s, shnum %d" % \
7545                                        (base32.b2a(shares[1][1]),
7546hunk ./src/allmydata/test/test_cli.py 2900
7547                                         base32.b2a(storage_index),
7548                                         shares[1][0])
7549-            debug.corrupt_share(cso)
7550+            debug.do_corrupt_share(stdout, sharefile)
7551         d.addCallback(_clobber_shares)
7552 
7553         d.addCallback(lambda ign: self.do_cli("check", "--verify", self.uri))
7554hunk ./src/allmydata/test/test_cli.py 3017
7555         def _clobber_shares(ignored):
7556             shares = self.find_uri_shares(self.uris[u"g\u00F6\u00F6d"])
7557             self.failUnlessReallyEqual(len(shares), 10)
7558-            os.unlink(shares[0][2])
7559+            shares[0][2].remove()
7560 
7561             shares = self.find_uri_shares(self.uris["mutable"])
7562hunk ./src/allmydata/test/test_cli.py 3020
7563-            cso = debug.CorruptShareOptions()
7564-            cso.stdout = StringIO()
7565-            cso.parseOptions([shares[1][2]])
7566+            stdout = StringIO()
7567+            sharefile = shares[1][2]
7568             storage_index = uri.from_string(self.uris["mutable"]).get_storage_index()
7569             self._corrupt_share_line = " corrupt: server %s, SI %s, shnum %d" % \
7570                                        (base32.b2a(shares[1][1]),
7571hunk ./src/allmydata/test/test_cli.py 3027
7572                                         base32.b2a(storage_index),
7573                                         shares[1][0])
7574-            debug.corrupt_share(cso)
7575+            debug.do_corrupt_share(stdout, sharefile)
7576         d.addCallback(_clobber_shares)
7577 
7578         # root
7579hunk ./src/allmydata/test/test_client.py 90
7580                            "enabled = true\n" + \
7581                            "reserved_space = 1000\n")
7582         c = client.Client(basedir)
7583-        self.failUnlessEqual(c.getServiceNamed("storage").reserved_space, 1000)
7584+        self.failUnlessEqual(c.getServiceNamed("storage").backend._reserved_space, 1000)
7585 
7586     def test_reserved_2(self):
7587         basedir = "client.Basic.test_reserved_2"
7588hunk ./src/allmydata/test/test_client.py 101
7589                            "enabled = true\n" + \
7590                            "reserved_space = 10K\n")
7591         c = client.Client(basedir)
7592-        self.failUnlessEqual(c.getServiceNamed("storage").reserved_space, 10*1000)
7593+        self.failUnlessEqual(c.getServiceNamed("storage").backend._reserved_space, 10*1000)
7594 
7595     def test_reserved_3(self):
7596         basedir = "client.Basic.test_reserved_3"
7597hunk ./src/allmydata/test/test_client.py 112
7598                            "enabled = true\n" + \
7599                            "reserved_space = 5mB\n")
7600         c = client.Client(basedir)
7601-        self.failUnlessEqual(c.getServiceNamed("storage").reserved_space,
7602+        self.failUnlessEqual(c.getServiceNamed("storage").backend._reserved_space,
7603                              5*1000*1000)
7604 
7605     def test_reserved_4(self):
7606hunk ./src/allmydata/test/test_client.py 124
7607                            "enabled = true\n" + \
7608                            "reserved_space = 78Gb\n")
7609         c = client.Client(basedir)
7610-        self.failUnlessEqual(c.getServiceNamed("storage").reserved_space,
7611+        self.failUnlessEqual(c.getServiceNamed("storage").backend._reserved_space,
7612                              78*1000*1000*1000)
7613 
7614     def test_reserved_bad(self):
7615hunk ./src/allmydata/test/test_client.py 136
7616                            "enabled = true\n" + \
7617                            "reserved_space = bogus\n")
7618         c = client.Client(basedir)
7619-        self.failUnlessEqual(c.getServiceNamed("storage").reserved_space, 0)
7620+        self.failUnlessEqual(c.getServiceNamed("storage").backend._reserved_space, 0)
7621 
7622     def _permute(self, sb, key):
7623         return [ s.get_serverid() for s in sb.get_servers_for_psi(key) ]
7624hunk ./src/allmydata/test/test_crawler.py 7
7625 from twisted.trial import unittest
7626 from twisted.application import service
7627 from twisted.internet import defer
7628+from twisted.python.filepath import FilePath
7629 from foolscap.api import eventually, fireEventually
7630 
7631 from allmydata.util import fileutil, hashutil, pollmixin
7632hunk ./src/allmydata/test/test_crawler.py 13
7633 from allmydata.storage.server import StorageServer, si_b2a
7634 from allmydata.storage.crawler import ShareCrawler, TimeSliceExceeded
7635+from allmydata.storage.backends.disk.disk_backend import DiskBackend
7636 
7637 from allmydata.test.test_storage import FakeCanary
7638 from allmydata.test.common_util import StallMixin
7639hunk ./src/allmydata/test/test_crawler.py 115
7640 
7641     def test_immediate(self):
7642         self.basedir = "crawler/Basic/immediate"
7643-        fileutil.make_dirs(self.basedir)
7644         serverid = "\x00" * 20
7645hunk ./src/allmydata/test/test_crawler.py 116
7646-        ss = StorageServer(self.basedir, serverid)
7647+        fp = FilePath(self.basedir)
7648+        backend = DiskBackend(fp)
7649+        ss = StorageServer(serverid, backend, fp)
7650         ss.setServiceParent(self.s)
7651 
7652         sis = [self.write(i, ss, serverid) for i in range(10)]
7653hunk ./src/allmydata/test/test_crawler.py 122
7654-        statefile = os.path.join(self.basedir, "statefile")
7655+        statefp = fp.child("statefile")
7656 
7657hunk ./src/allmydata/test/test_crawler.py 124
7658-        c = BucketEnumeratingCrawler(ss, statefile, allowed_cpu_percentage=.1)
7659+        c = BucketEnumeratingCrawler(backend, statefp, allowed_cpu_percentage=.1)
7660         c.load_state()
7661 
7662         c.start_current_prefix(time.time())
7663hunk ./src/allmydata/test/test_crawler.py 137
7664         self.failUnlessEqual(sorted(sis), sorted(c.all_buckets))
7665 
7666         # check that a new crawler picks up on the state file properly
7667-        c2 = BucketEnumeratingCrawler(ss, statefile)
7668+        c2 = BucketEnumeratingCrawler(backend, statefp)
7669         c2.load_state()
7670 
7671         c2.start_current_prefix(time.time())
7672hunk ./src/allmydata/test/test_crawler.py 145
7673 
7674     def test_service(self):
7675         self.basedir = "crawler/Basic/service"
7676-        fileutil.make_dirs(self.basedir)
7677         serverid = "\x00" * 20
7678hunk ./src/allmydata/test/test_crawler.py 146
7679-        ss = StorageServer(self.basedir, serverid)
7680+        fp = FilePath(self.basedir)
7681+        backend = DiskBackend(fp)
7682+        ss = StorageServer(serverid, backend, fp)
7683         ss.setServiceParent(self.s)
7684 
7685         sis = [self.write(i, ss, serverid) for i in range(10)]
7686hunk ./src/allmydata/test/test_crawler.py 153
7687 
7688-        statefile = os.path.join(self.basedir, "statefile")
7689-        c = BucketEnumeratingCrawler(ss, statefile)
7690+        statefp = fp.child("statefile")
7691+        c = BucketEnumeratingCrawler(backend, statefp)
7692         c.setServiceParent(self.s)
7693 
7694         # it should be legal to call get_state() and get_progress() right
7695hunk ./src/allmydata/test/test_crawler.py 174
7696 
7697     def test_paced(self):
7698         self.basedir = "crawler/Basic/paced"
7699-        fileutil.make_dirs(self.basedir)
7700         serverid = "\x00" * 20
7701hunk ./src/allmydata/test/test_crawler.py 175
7702-        ss = StorageServer(self.basedir, serverid)
7703+        fp = FilePath(self.basedir)
7704+        backend = DiskBackend(fp)
7705+        ss = StorageServer(serverid, backend, fp)
7706         ss.setServiceParent(self.s)
7707 
7708         # put four buckets in each prefixdir
7709hunk ./src/allmydata/test/test_crawler.py 186
7710             for tail in range(4):
7711                 sis.append(self.write(i, ss, serverid, tail))
7712 
7713-        statefile = os.path.join(self.basedir, "statefile")
7714+        statefp = fp.child("statefile")
7715 
7716hunk ./src/allmydata/test/test_crawler.py 188
7717-        c = PacedCrawler(ss, statefile)
7718+        c = PacedCrawler(backend, statefp)
7719         c.load_state()
7720         try:
7721             c.start_current_prefix(time.time())
7722hunk ./src/allmydata/test/test_crawler.py 213
7723         del c
7724 
7725         # start a new crawler, it should start from the beginning
7726-        c = PacedCrawler(ss, statefile)
7727+        c = PacedCrawler(backend, statefp)
7728         c.load_state()
7729         try:
7730             c.start_current_prefix(time.time())
7731hunk ./src/allmydata/test/test_crawler.py 226
7732         c.cpu_slice = PacedCrawler.cpu_slice
7733 
7734         # a third crawler should pick up from where it left off
7735-        c2 = PacedCrawler(ss, statefile)
7736+        c2 = PacedCrawler(backend, statefp)
7737         c2.all_buckets = c.all_buckets[:]
7738         c2.load_state()
7739         c2.countdown = -1
7740hunk ./src/allmydata/test/test_crawler.py 237
7741 
7742         # now stop it at the end of a bucket (countdown=4), to exercise a
7743         # different place that checks the time
7744-        c = PacedCrawler(ss, statefile)
7745+        c = PacedCrawler(backend, statefp)
7746         c.load_state()
7747         c.countdown = 4
7748         try:
7749hunk ./src/allmydata/test/test_crawler.py 256
7750 
7751         # stop it again at the end of the bucket, check that a new checker
7752         # picks up correctly
7753-        c = PacedCrawler(ss, statefile)
7754+        c = PacedCrawler(backend, statefp)
7755         c.load_state()
7756         c.countdown = 4
7757         try:
7758hunk ./src/allmydata/test/test_crawler.py 266
7759         # that should stop at the end of one of the buckets.
7760         c.save_state()
7761 
7762-        c2 = PacedCrawler(ss, statefile)
7763+        c2 = PacedCrawler(backend, statefp)
7764         c2.all_buckets = c.all_buckets[:]
7765         c2.load_state()
7766         c2.countdown = -1
7767hunk ./src/allmydata/test/test_crawler.py 277
7768 
7769     def test_paced_service(self):
7770         self.basedir = "crawler/Basic/paced_service"
7771-        fileutil.make_dirs(self.basedir)
7772         serverid = "\x00" * 20
7773hunk ./src/allmydata/test/test_crawler.py 278
7774-        ss = StorageServer(self.basedir, serverid)
7775+        fp = FilePath(self.basedir)
7776+        backend = DiskBackend(fp)
7777+        ss = StorageServer(serverid, backend, fp)
7778         ss.setServiceParent(self.s)
7779 
7780         sis = [self.write(i, ss, serverid) for i in range(10)]
7781hunk ./src/allmydata/test/test_crawler.py 285
7782 
7783-        statefile = os.path.join(self.basedir, "statefile")
7784-        c = PacedCrawler(ss, statefile)
7785+        statefp = fp.child("statefile")
7786+        c = PacedCrawler(backend, statefp)
7787 
7788         did_check_progress = [False]
7789         def check_progress():
7790hunk ./src/allmydata/test/test_crawler.py 345
7791         # and read the stdout when it runs.
7792 
7793         self.basedir = "crawler/Basic/cpu_usage"
7794-        fileutil.make_dirs(self.basedir)
7795         serverid = "\x00" * 20
7796hunk ./src/allmydata/test/test_crawler.py 346
7797-        ss = StorageServer(self.basedir, serverid)
7798+        fp = FilePath(self.basedir)
7799+        backend = DiskBackend(fp)
7800+        ss = StorageServer(serverid, backend, fp)
7801         ss.setServiceParent(self.s)
7802 
7803         for i in range(10):
7804hunk ./src/allmydata/test/test_crawler.py 354
7805             self.write(i, ss, serverid)
7806 
7807-        statefile = os.path.join(self.basedir, "statefile")
7808-        c = ConsumingCrawler(ss, statefile)
7809+        statefp = fp.child("statefile")
7810+        c = ConsumingCrawler(backend, statefp)
7811         c.setServiceParent(self.s)
7812 
7813         # this will run as fast as it can, consuming about 50ms per call to
7814hunk ./src/allmydata/test/test_crawler.py 391
7815 
7816     def test_empty_subclass(self):
7817         self.basedir = "crawler/Basic/empty_subclass"
7818-        fileutil.make_dirs(self.basedir)
7819         serverid = "\x00" * 20
7820hunk ./src/allmydata/test/test_crawler.py 392
7821-        ss = StorageServer(self.basedir, serverid)
7822+        fp = FilePath(self.basedir)
7823+        backend = DiskBackend(fp)
7824+        ss = StorageServer(serverid, backend, fp)
7825         ss.setServiceParent(self.s)
7826 
7827         for i in range(10):
7828hunk ./src/allmydata/test/test_crawler.py 400
7829             self.write(i, ss, serverid)
7830 
7831-        statefile = os.path.join(self.basedir, "statefile")
7832-        c = ShareCrawler(ss, statefile)
7833+        statefp = fp.child("statefile")
7834+        c = ShareCrawler(backend, statefp)
7835         c.slow_start = 0
7836         c.setServiceParent(self.s)
7837 
7838hunk ./src/allmydata/test/test_crawler.py 417
7839         d.addCallback(_done)
7840         return d
7841 
7842-
7843     def test_oneshot(self):
7844         self.basedir = "crawler/Basic/oneshot"
7845hunk ./src/allmydata/test/test_crawler.py 419
7846-        fileutil.make_dirs(self.basedir)
7847         serverid = "\x00" * 20
7848hunk ./src/allmydata/test/test_crawler.py 420
7849-        ss = StorageServer(self.basedir, serverid)
7850+        fp = FilePath(self.basedir)
7851+        backend = DiskBackend(fp)
7852+        ss = StorageServer(serverid, backend, fp)
7853         ss.setServiceParent(self.s)
7854 
7855         for i in range(30):
7856hunk ./src/allmydata/test/test_crawler.py 428
7857             self.write(i, ss, serverid)
7858 
7859-        statefile = os.path.join(self.basedir, "statefile")
7860-        c = OneShotCrawler(ss, statefile)
7861+        statefp = fp.child("statefile")
7862+        c = OneShotCrawler(backend, statefp)
7863         c.setServiceParent(self.s)
7864 
7865         d = c.finished_d
7866hunk ./src/allmydata/test/test_crawler.py 447
7867             self.failUnlessEqual(s["current-cycle"], None)
7868         d.addCallback(_check)
7869         return d
7870-
7871hunk ./src/allmydata/test/test_deepcheck.py 23
7872      ShouldFailMixin
7873 from allmydata.test.common_util import StallMixin
7874 from allmydata.test.no_network import GridTestMixin
7875+from allmydata.scripts import debug
7876+
7877 
7878 timeout = 2400 # One of these took 1046.091s on Zandr's ARM box.
7879 
7880hunk ./src/allmydata/test/test_deepcheck.py 905
7881         d.addErrback(self.explain_error)
7882         return d
7883 
7884-
7885-
7886     def set_up_damaged_tree(self):
7887         # 6.4s
7888 
7889hunk ./src/allmydata/test/test_deepcheck.py 989
7890 
7891         return d
7892 
7893-    def _run_cli(self, argv):
7894-        stdout, stderr = StringIO(), StringIO()
7895-        # this can only do synchronous operations
7896-        assert argv[0] == "debug"
7897-        runner.runner(argv, run_by_human=False, stdout=stdout, stderr=stderr)
7898-        return stdout.getvalue()
7899-
7900     def _delete_some_shares(self, node):
7901         self.delete_shares_numbered(node.get_uri(), [0,1])
7902 
7903hunk ./src/allmydata/test/test_deepcheck.py 995
7904     def _corrupt_some_shares(self, node):
7905         for (shnum, serverid, sharefile) in self.find_uri_shares(node.get_uri()):
7906             if shnum in (0,1):
7907-                self._run_cli(["debug", "corrupt-share", sharefile])
7908+                debug.do_corrupt_share(StringIO(), sharefile)
7909 
7910     def _delete_most_shares(self, node):
7911         self.delete_shares_numbered(node.get_uri(), range(1,10))
7912hunk ./src/allmydata/test/test_deepcheck.py 1000
7913 
7914-
7915     def check_is_healthy(self, cr, where):
7916         try:
7917             self.failUnless(ICheckResults.providedBy(cr), (cr, type(cr), where))
7918hunk ./src/allmydata/test/test_download.py 134
7919             for shnum in shares_for_server:
7920                 share_dir = self.get_server(i).backend.get_shareset(si)._sharehomedir
7921                 fileutil.fp_make_dirs(share_dir)
7922-                share_dir.child(str(shnum)).setContent(shares[shnum])
7923+                share_dir.child(str(shnum)).setContent(shares_for_server[shnum])
7924 
7925     def load_shares(self, ignored=None):
7926         # this uses the data generated by create_shares() to populate the
7927hunk ./src/allmydata/test/test_hung_server.py 32
7928 
7929     def _break(self, servers):
7930         for ss in servers:
7931-            self.g.break_server(ss.get_serverid())
7932+            self.g.break_server(ss.original.get_serverid())
7933 
7934     def _hang(self, servers, **kwargs):
7935         for ss in servers:
7936hunk ./src/allmydata/test/test_hung_server.py 67
7937         serverids = [ss.original.get_serverid() for ss in from_servers]
7938         for (i_shnum, i_serverid, i_sharefp) in self.shares:
7939             if i_serverid in serverids:
7940-                self.copy_share((i_shnum, i_serverid, i_sharefp), self.uri, to_server)
7941+                self.copy_share((i_shnum, i_serverid, i_sharefp), self.uri, to_server.original)
7942 
7943         self.shares = self.find_uri_shares(self.uri)
7944 
7945hunk ./src/allmydata/test/test_mutable.py 3670
7946         # Now execute each assignment by writing the storage.
7947         for (share, servernum) in assignments:
7948             sharedata = base64.b64decode(self.sdmf_old_shares[share])
7949-            storage_dir = self.get_server(servernum).backend.get_shareset(si).sharehomedir
7950+            storage_dir = self.get_server(servernum).backend.get_shareset(si)._sharehomedir
7951             fileutil.fp_make_dirs(storage_dir)
7952             storage_dir.child("%d" % share).setContent(sharedata)
7953         # ...and verify that the shares are there.
7954hunk ./src/allmydata/test/test_no_network.py 10
7955 from allmydata.immutable.upload import Data
7956 from allmydata.util.consumer import download_to_data
7957 
7958+
7959 class Harness(unittest.TestCase):
7960     def setUp(self):
7961         self.s = service.MultiService()
7962hunk ./src/allmydata/test/test_storage.py 1
7963-import time, os.path, platform, stat, re, simplejson, struct, shutil
7964+import time, os.path, platform, stat, re, simplejson, struct, shutil, itertools
7965 
7966 import mock
7967 
7968hunk ./src/allmydata/test/test_storage.py 6
7969 from twisted.trial import unittest
7970-
7971 from twisted.internet import defer
7972 from twisted.application import service
7973hunk ./src/allmydata/test/test_storage.py 8
7974+from twisted.python.filepath import FilePath
7975 from foolscap.api import fireEventually
7976hunk ./src/allmydata/test/test_storage.py 10
7977-import itertools
7978+
7979 from allmydata import interfaces
7980 from allmydata.util import fileutil, hashutil, base32, pollmixin, time_format
7981 from allmydata.storage.server import StorageServer
7982hunk ./src/allmydata/test/test_storage.py 14
7983+from allmydata.storage.backends.disk.disk_backend import DiskBackend
7984 from allmydata.storage.backends.disk.mutable import MutableDiskShare
7985 from allmydata.storage.bucket import BucketWriter, BucketReader
7986 from allmydata.storage.common import DataTooLargeError, \
7987hunk ./src/allmydata/test/test_storage.py 310
7988         return self.sparent.stopService()
7989 
7990     def workdir(self, name):
7991-        basedir = os.path.join("storage", "Server", name)
7992-        return basedir
7993+        return FilePath("storage").child("Server").child(name)
7994 
7995     def create(self, name, reserved_space=0, klass=StorageServer):
7996         workdir = self.workdir(name)
7997hunk ./src/allmydata/test/test_storage.py 314
7998-        ss = klass(workdir, "\x00" * 20, reserved_space=reserved_space,
7999+        backend = DiskBackend(workdir, readonly=False, reserved_space=reserved_space)
8000+        ss = klass("\x00" * 20, backend, workdir,
8001                    stats_provider=FakeStatsProvider())
8002         ss.setServiceParent(self.sparent)
8003         return ss
8004hunk ./src/allmydata/test/test_storage.py 1386
8005 
8006     def tearDown(self):
8007         self.sparent.stopService()
8008-        shutil.rmtree(self.workdir("MDMFProxies storage test server"))
8009+        fileutil.fp_remove(self.workdir("MDMFProxies storage test server"))
8010 
8011 
8012     def write_enabler(self, we_tag):
8013hunk ./src/allmydata/test/test_storage.py 2781
8014         return self.sparent.stopService()
8015 
8016     def workdir(self, name):
8017-        basedir = os.path.join("storage", "Server", name)
8018-        return basedir
8019+        return FilePath("storage").child("Server").child(name)
8020 
8021     def create(self, name):
8022         workdir = self.workdir(name)
8023hunk ./src/allmydata/test/test_storage.py 2785
8024-        ss = StorageServer(workdir, "\x00" * 20)
8025+        backend = DiskBackend(workdir)
8026+        ss = StorageServer("\x00" * 20, backend, workdir)
8027         ss.setServiceParent(self.sparent)
8028         return ss
8029 
8030hunk ./src/allmydata/test/test_storage.py 4061
8031         }
8032 
8033         basedir = "storage/WebStatus/status_right_disk_stats"
8034-        fileutil.make_dirs(basedir)
8035-        ss = StorageServer(basedir, "\x00" * 20, reserved_space=reserved_space)
8036-        expecteddir = ss.sharedir
8037+        fp = FilePath(basedir)
8038+        backend = DiskBackend(fp, readonly=False, reserved_space=reserved_space)
8039+        ss = StorageServer("\x00" * 20, backend, fp)
8040+        expecteddir = backend._sharedir
8041         ss.setServiceParent(self.s)
8042         w = StorageStatus(ss)
8043         html = w.renderSynchronously()
8044hunk ./src/allmydata/test/test_storage.py 4084
8045 
8046     def test_readonly(self):
8047         basedir = "storage/WebStatus/readonly"
8048-        fileutil.make_dirs(basedir)
8049-        ss = StorageServer(basedir, "\x00" * 20, readonly_storage=True)
8050+        fp = FilePath(basedir)
8051+        backend = DiskBackend(fp, readonly=True)
8052+        ss = StorageServer("\x00" * 20, backend, fp)
8053         ss.setServiceParent(self.s)
8054         w = StorageStatus(ss)
8055         html = w.renderSynchronously()
8056hunk ./src/allmydata/test/test_storage.py 4096
8057 
8058     def test_reserved(self):
8059         basedir = "storage/WebStatus/reserved"
8060-        fileutil.make_dirs(basedir)
8061-        ss = StorageServer(basedir, "\x00" * 20, reserved_space=10e6)
8062-        ss.setServiceParent(self.s)
8063-        w = StorageStatus(ss)
8064-        html = w.renderSynchronously()
8065-        self.failUnlessIn("<h1>Storage Server Status</h1>", html)
8066-        s = remove_tags(html)
8067-        self.failUnlessIn("Reserved space: - 10.00 MB (10000000)", s)
8068-
8069-    def test_huge_reserved(self):
8070-        basedir = "storage/WebStatus/reserved"
8071-        fileutil.make_dirs(basedir)
8072-        ss = StorageServer(basedir, "\x00" * 20, reserved_space=10e6)
8073+        fp = FilePath(basedir)
8074+        backend = DiskBackend(fp, readonly=False, reserved_space=10e6)
8075+        ss = StorageServer("\x00" * 20, backend, fp)
8076         ss.setServiceParent(self.s)
8077         w = StorageStatus(ss)
8078         html = w.renderSynchronously()
8079hunk ./src/allmydata/test/test_upload.py 3
8080 # -*- coding: utf-8 -*-
8081 
8082-import os, shutil
8083+import os
8084 from cStringIO import StringIO
8085 from twisted.trial import unittest
8086 from twisted.python.failure import Failure
8087hunk ./src/allmydata/test/test_upload.py 14
8088 from allmydata import uri, monitor, client
8089 from allmydata.immutable import upload, encode
8090 from allmydata.interfaces import FileTooLargeError, UploadUnhappinessError
8091-from allmydata.util import log
8092+from allmydata.util import log, fileutil
8093 from allmydata.util.assertutil import precondition
8094 from allmydata.util.deferredutil import DeferredListShouldSucceed
8095 from allmydata.test.no_network import GridTestMixin
8096hunk ./src/allmydata/test/test_upload.py 972
8097                                         readonly=True))
8098         # Remove the first share from server 0.
8099         def _remove_share_0_from_server_0():
8100-            share_location = self.shares[0][2]
8101-            os.remove(share_location)
8102+            self.shares[0][2].remove()
8103         d.addCallback(lambda ign:
8104             _remove_share_0_from_server_0())
8105         # Set happy = 4 in the client.
8106hunk ./src/allmydata/test/test_upload.py 1847
8107             self._copy_share_to_server(3, 1)
8108             storedir = self.get_serverdir(0)
8109             # remove the storedir, wiping out any existing shares
8110-            shutil.rmtree(storedir)
8111+            fileutil.fp_remove(storedir)
8112             # create an empty storedir to replace the one we just removed
8113hunk ./src/allmydata/test/test_upload.py 1849
8114-            os.mkdir(storedir)
8115+            storedir.mkdir()
8116             client = self.g.clients[0]
8117             client.DEFAULT_ENCODING_PARAMETERS['happy'] = 4
8118             return client
8119hunk ./src/allmydata/test/test_upload.py 1888
8120             self._copy_share_to_server(3, 1)
8121             storedir = self.get_serverdir(0)
8122             # remove the storedir, wiping out any existing shares
8123-            shutil.rmtree(storedir)
8124+            fileutil.fp_remove(storedir)
8125             # create an empty storedir to replace the one we just removed
8126hunk ./src/allmydata/test/test_upload.py 1890
8127-            os.mkdir(storedir)
8128+            storedir.mkdir()
8129             client = self.g.clients[0]
8130             client.DEFAULT_ENCODING_PARAMETERS['happy'] = 4
8131             return client
8132hunk ./src/allmydata/test/test_web.py 4870
8133         d.addErrback(self.explain_web_error)
8134         return d
8135 
8136-    def _assert_leasecount(self, ignored, which, expected):
8137+    def _assert_leasecount(self, which, expected):
8138         lease_counts = self.count_leases(self.uris[which])
8139         for (fn, num_leases) in lease_counts:
8140             if num_leases != expected:
8141hunk ./src/allmydata/test/test_web.py 4903
8142                 self.fileurls[which] = "uri/" + urllib.quote(self.uris[which])
8143         d.addCallback(_compute_fileurls)
8144 
8145-        d.addCallback(self._assert_leasecount, "one", 1)
8146-        d.addCallback(self._assert_leasecount, "two", 1)
8147-        d.addCallback(self._assert_leasecount, "mutable", 1)
8148+        d.addCallback(lambda ign: self._assert_leasecount("one", 1))
8149+        d.addCallback(lambda ign: self._assert_leasecount("two", 1))
8150+        d.addCallback(lambda ign: self._assert_leasecount("mutable", 1))
8151 
8152         d.addCallback(self.CHECK, "one", "t=check") # no add-lease
8153         def _got_html_good(res):
8154hunk ./src/allmydata/test/test_web.py 4913
8155             self.failIf("Not Healthy" in res, res)
8156         d.addCallback(_got_html_good)
8157 
8158-        d.addCallback(self._assert_leasecount, "one", 1)
8159-        d.addCallback(self._assert_leasecount, "two", 1)
8160-        d.addCallback(self._assert_leasecount, "mutable", 1)
8161+        d.addCallback(lambda ign: self._assert_leasecount("one", 1))
8162+        d.addCallback(lambda ign: self._assert_leasecount("two", 1))
8163+        d.addCallback(lambda ign: self._assert_leasecount("mutable", 1))
8164 
8165         # this CHECK uses the original client, which uses the same
8166         # lease-secrets, so it will just renew the original lease
8167hunk ./src/allmydata/test/test_web.py 4922
8168         d.addCallback(self.CHECK, "one", "t=check&add-lease=true")
8169         d.addCallback(_got_html_good)
8170 
8171-        d.addCallback(self._assert_leasecount, "one", 1)
8172-        d.addCallback(self._assert_leasecount, "two", 1)
8173-        d.addCallback(self._assert_leasecount, "mutable", 1)
8174+        d.addCallback(lambda ign: self._assert_leasecount("one", 1))
8175+        d.addCallback(lambda ign: self._assert_leasecount("two", 1))
8176+        d.addCallback(lambda ign: self._assert_leasecount("mutable", 1))
8177 
8178         # this CHECK uses an alternate client, which adds a second lease
8179         d.addCallback(self.CHECK, "one", "t=check&add-lease=true", clientnum=1)
8180hunk ./src/allmydata/test/test_web.py 4930
8181         d.addCallback(_got_html_good)
8182 
8183-        d.addCallback(self._assert_leasecount, "one", 2)
8184-        d.addCallback(self._assert_leasecount, "two", 1)
8185-        d.addCallback(self._assert_leasecount, "mutable", 1)
8186+        d.addCallback(lambda ign: self._assert_leasecount("one", 2))
8187+        d.addCallback(lambda ign: self._assert_leasecount("two", 1))
8188+        d.addCallback(lambda ign: self._assert_leasecount("mutable", 1))
8189 
8190         d.addCallback(self.CHECK, "mutable", "t=check&add-lease=true")
8191         d.addCallback(_got_html_good)
8192hunk ./src/allmydata/test/test_web.py 4937
8193 
8194-        d.addCallback(self._assert_leasecount, "one", 2)
8195-        d.addCallback(self._assert_leasecount, "two", 1)
8196-        d.addCallback(self._assert_leasecount, "mutable", 1)
8197+        d.addCallback(lambda ign: self._assert_leasecount("one", 2))
8198+        d.addCallback(lambda ign: self._assert_leasecount("two", 1))
8199+        d.addCallback(lambda ign: self._assert_leasecount("mutable", 1))
8200 
8201         d.addCallback(self.CHECK, "mutable", "t=check&add-lease=true",
8202                       clientnum=1)
8203hunk ./src/allmydata/test/test_web.py 4945
8204         d.addCallback(_got_html_good)
8205 
8206-        d.addCallback(self._assert_leasecount, "one", 2)
8207-        d.addCallback(self._assert_leasecount, "two", 1)
8208-        d.addCallback(self._assert_leasecount, "mutable", 2)
8209+        d.addCallback(lambda ign: self._assert_leasecount("one", 2))
8210+        d.addCallback(lambda ign: self._assert_leasecount("two", 1))
8211+        d.addCallback(lambda ign: self._assert_leasecount("mutable", 2))
8212 
8213         d.addErrback(self.explain_web_error)
8214         return d
8215hunk ./src/allmydata/test/test_web.py 4989
8216             self.failUnlessReallyEqual(len(units), 4+1)
8217         d.addCallback(_done)
8218 
8219-        d.addCallback(self._assert_leasecount, "root", 1)
8220-        d.addCallback(self._assert_leasecount, "one", 1)
8221-        d.addCallback(self._assert_leasecount, "mutable", 1)
8222+        d.addCallback(lambda ign: self._assert_leasecount("root", 1))
8223+        d.addCallback(lambda ign: self._assert_leasecount("one", 1))
8224+        d.addCallback(lambda ign: self._assert_leasecount("mutable", 1))
8225 
8226         d.addCallback(self.CHECK, "root", "t=stream-deep-check&add-lease=true")
8227         d.addCallback(_done)
8228hunk ./src/allmydata/test/test_web.py 4996
8229 
8230-        d.addCallback(self._assert_leasecount, "root", 1)
8231-        d.addCallback(self._assert_leasecount, "one", 1)
8232-        d.addCallback(self._assert_leasecount, "mutable", 1)
8233+        d.addCallback(lambda ign: self._assert_leasecount("root", 1))
8234+        d.addCallback(lambda ign: self._assert_leasecount("one", 1))
8235+        d.addCallback(lambda ign: self._assert_leasecount("mutable", 1))
8236 
8237         d.addCallback(self.CHECK, "root", "t=stream-deep-check&add-lease=true",
8238                       clientnum=1)
8239hunk ./src/allmydata/test/test_web.py 5004
8240         d.addCallback(_done)
8241 
8242-        d.addCallback(self._assert_leasecount, "root", 2)
8243-        d.addCallback(self._assert_leasecount, "one", 2)
8244-        d.addCallback(self._assert_leasecount, "mutable", 2)
8245+        d.addCallback(lambda ign: self._assert_leasecount("root", 2))
8246+        d.addCallback(lambda ign: self._assert_leasecount("one", 2))
8247+        d.addCallback(lambda ign: self._assert_leasecount("mutable", 2))
8248 
8249         d.addErrback(self.explain_web_error)
8250         return d
8251}
8252[Fix more shallow bugs, mainly FilePathification. Also, remove the max_space_per_bucket parameter from BucketWriter since it can be obtained from the _max_size attribute of the share (via a new get_allocated_size() accessor). refs #999
8253david-sarah@jacaranda.org**20110921221421
8254 Ignore-this: 600e3ccef8533aa43442fa576c7d88cf
8255] {
8256hunk ./src/allmydata/scripts/debug.py 642
8257     /home/warner/testnet/node-2/storage/shares/44k/44kai1tui348689nrw8fjegc8c/2
8258     """
8259     from allmydata.storage.server import si_a2b
8260-    from allmydata.storage.backends.disk_backend import si_si2dir
8261+    from allmydata.storage.backends.disk.disk_backend import si_si2dir
8262     from allmydata.util.encodingutil import quote_filepath
8263 
8264     out = options.stdout
8265hunk ./src/allmydata/scripts/debug.py 648
8266     si = si_a2b(options.si_s)
8267     for nodedir in options.nodedirs:
8268-        sharedir = si_si2dir(nodedir.child("storage").child("shares"), si)
8269+        sharedir = si_si2dir(FilePath(nodedir).child("storage").child("shares"), si)
8270         if sharedir.exists():
8271             for sharefp in sharedir.children():
8272                 print >>out, quote_filepath(sharefp, quotemarks=False)
8273hunk ./src/allmydata/storage/backends/disk/disk_backend.py 189
8274         incominghome = self._incominghomedir.child(str(shnum))
8275         immsh = ImmutableDiskShare(self.get_storage_index(), shnum, incominghome, finalhome,
8276                                    max_size=max_space_per_bucket)
8277-        bw = BucketWriter(storageserver, immsh, max_space_per_bucket, lease_info, canary)
8278+        bw = BucketWriter(storageserver, immsh, lease_info, canary)
8279         if self._discard_storage:
8280             bw.throw_out_all_data = True
8281         return bw
8282hunk ./src/allmydata/storage/backends/disk/immutable.py 147
8283     def unlink(self):
8284         self._home.remove()
8285 
8286+    def get_allocated_size(self):
8287+        return self._max_size
8288+
8289     def get_size(self):
8290         return self._home.getsize()
8291 
8292hunk ./src/allmydata/storage/bucket.py 15
8293 class BucketWriter(Referenceable):
8294     implements(RIBucketWriter)
8295 
8296-    def __init__(self, ss, immutableshare, max_size, lease_info, canary):
8297+    def __init__(self, ss, immutableshare, lease_info, canary):
8298         self.ss = ss
8299hunk ./src/allmydata/storage/bucket.py 17
8300-        self._max_size = max_size # don't allow the client to write more than this
8301         self._canary = canary
8302         self._disconnect_marker = canary.notifyOnDisconnect(self._disconnected)
8303         self.closed = False
8304hunk ./src/allmydata/storage/bucket.py 27
8305         self._share.add_lease(lease_info)
8306 
8307     def allocated_size(self):
8308-        return self._max_size
8309+        return self._share.get_allocated_size()
8310 
8311     def remote_write(self, offset, data):
8312         start = time.time()
8313hunk ./src/allmydata/storage/crawler.py 480
8314             self.state["bucket-counts"][cycle] = {}
8315         self.state["bucket-counts"][cycle][prefix] = len(sharesets)
8316         if prefix in self.prefixes[:self.num_sample_prefixes]:
8317-            self.state["storage-index-samples"][prefix] = (cycle, sharesets)
8318+            si_strings = [shareset.get_storage_index_string() for shareset in sharesets]
8319+            self.state["storage-index-samples"][prefix] = (cycle, si_strings)
8320 
8321     def finished_cycle(self, cycle):
8322         last_counts = self.state["bucket-counts"].get(cycle, [])
8323hunk ./src/allmydata/storage/expirer.py 281
8324         # copy() needs to become a deepcopy
8325         h["space-recovered"] = s["space-recovered"].copy()
8326 
8327-        history = pickle.load(self.historyfp.getContent())
8328+        history = pickle.loads(self.historyfp.getContent())
8329         history[cycle] = h
8330         while len(history) > 10:
8331             oldcycles = sorted(history.keys())
8332hunk ./src/allmydata/storage/expirer.py 355
8333         progress = self.get_progress()
8334 
8335         state = ShareCrawler.get_state(self) # does a shallow copy
8336-        history = pickle.load(self.historyfp.getContent())
8337+        history = pickle.loads(self.historyfp.getContent())
8338         state["history"] = history
8339 
8340         if not progress["cycle-in-progress"]:
8341hunk ./src/allmydata/test/test_download.py 199
8342                     for shnum in immutable_shares[clientnum]:
8343                         if s._shnum == shnum:
8344                             share_dir = self.get_server(clientnum).backend.get_shareset(si)._sharehomedir
8345-                            share_dir.child(str(shnum)).remove()
8346+                            fileutil.fp_remove(share_dir.child(str(shnum)))
8347         d.addCallback(_clobber_some_shares)
8348         d.addCallback(lambda ign: download_to_data(n))
8349         d.addCallback(_got_data)
8350hunk ./src/allmydata/test/test_download.py 224
8351             for clientnum in immutable_shares:
8352                 for shnum in immutable_shares[clientnum]:
8353                     share_dir = self.get_server(clientnum).backend.get_shareset(si)._sharehomedir
8354-                    share_dir.child(str(shnum)).remove()
8355+                    fileutil.fp_remove(share_dir.child(str(shnum)))
8356             # now a new download should fail with NoSharesError. We want a
8357             # new ImmutableFileNode so it will forget about the old shares.
8358             # If we merely called create_node_from_uri() without first
8359hunk ./src/allmydata/test/test_repairer.py 415
8360         def _test_corrupt(ignored):
8361             olddata = {}
8362             shares = self.find_uri_shares(self.uri)
8363-            for (shnum, serverid, sharefile) in shares:
8364-                olddata[ (shnum, serverid) ] = open(sharefile, "rb").read()
8365+            for (shnum, serverid, sharefp) in shares:
8366+                olddata[ (shnum, serverid) ] = sharefp.getContent()
8367             for sh in shares:
8368                 self.corrupt_share(sh, common._corrupt_uri_extension)
8369hunk ./src/allmydata/test/test_repairer.py 419
8370-            for (shnum, serverid, sharefile) in shares:
8371-                newdata = open(sharefile, "rb").read()
8372+            for (shnum, serverid, sharefp) in shares:
8373+                newdata = sharefp.getContent()
8374                 self.failIfEqual(olddata[ (shnum, serverid) ], newdata)
8375         d.addCallback(_test_corrupt)
8376 
8377hunk ./src/allmydata/test/test_storage.py 63
8378 
8379 class Bucket(unittest.TestCase):
8380     def make_workdir(self, name):
8381-        basedir = os.path.join("storage", "Bucket", name)
8382-        incoming = os.path.join(basedir, "tmp", "bucket")
8383-        final = os.path.join(basedir, "bucket")
8384-        fileutil.make_dirs(basedir)
8385-        fileutil.make_dirs(os.path.join(basedir, "tmp"))
8386+        basedir = FilePath("storage").child("Bucket").child(name)
8387+        tmpdir = basedir.child("tmp")
8388+        tmpdir.makedirs()
8389+        incoming = tmpdir.child("bucket")
8390+        final = basedir.child("bucket")
8391         return incoming, final
8392 
8393     def bucket_writer_closed(self, bw, consumed):
8394hunk ./src/allmydata/test/test_storage.py 87
8395 
8396     def test_create(self):
8397         incoming, final = self.make_workdir("test_create")
8398-        bw = BucketWriter(self, incoming, final, 200, self.make_lease(),
8399-                          FakeCanary())
8400+        share = ImmutableDiskShare("", 0, incoming, final, 200)
8401+        bw = BucketWriter(self, share, self.make_lease(), FakeCanary())
8402         bw.remote_write(0, "a"*25)
8403         bw.remote_write(25, "b"*25)
8404         bw.remote_write(50, "c"*25)
8405hunk ./src/allmydata/test/test_storage.py 97
8406 
8407     def test_readwrite(self):
8408         incoming, final = self.make_workdir("test_readwrite")
8409-        bw = BucketWriter(self, incoming, final, 200, self.make_lease(),
8410-                          FakeCanary())
8411+        share = ImmutableDiskShare("", 0, incoming, 200)
8412+        bw = BucketWriter(self, share, self.make_lease(), FakeCanary())
8413         bw.remote_write(0, "a"*25)
8414         bw.remote_write(25, "b"*25)
8415         bw.remote_write(50, "c"*7) # last block may be short
8416hunk ./src/allmydata/test/test_storage.py 140
8417 
8418         incoming, final = self.make_workdir("test_read_past_end_of_share_data")
8419 
8420-        fileutil.write(final, share_file_data)
8421+        final.setContent(share_file_data)
8422 
8423         mockstorageserver = mock.Mock()
8424 
8425hunk ./src/allmydata/test/test_storage.py 179
8426 
8427 class BucketProxy(unittest.TestCase):
8428     def make_bucket(self, name, size):
8429-        basedir = os.path.join("storage", "BucketProxy", name)
8430-        incoming = os.path.join(basedir, "tmp", "bucket")
8431-        final = os.path.join(basedir, "bucket")
8432-        fileutil.make_dirs(basedir)
8433-        fileutil.make_dirs(os.path.join(basedir, "tmp"))
8434-        bw = BucketWriter(self, incoming, final, size, self.make_lease(),
8435-                          FakeCanary())
8436+        basedir = FilePath("storage").child("BucketProxy").child(name)
8437+        tmpdir = basedir.child("tmp")
8438+        tmpdir.makedirs()
8439+        incoming = tmpdir.child("bucket")
8440+        final = basedir.child("bucket")
8441+        share = ImmutableDiskShare("", 0, incoming, final, size)
8442+        bw = BucketWriter(self, share, self.make_lease(), FakeCanary())
8443         rb = RemoteBucket()
8444         rb.target = bw
8445         return bw, rb, final
8446hunk ./src/allmydata/test/test_storage.py 206
8447         pass
8448 
8449     def test_create(self):
8450-        bw, rb, sharefname = self.make_bucket("test_create", 500)
8451+        bw, rb, sharefp = self.make_bucket("test_create", 500)
8452         bp = WriteBucketProxy(rb, None,
8453                               data_size=300,
8454                               block_size=10,
8455hunk ./src/allmydata/test/test_storage.py 237
8456                         for i in (1,9,13)]
8457         uri_extension = "s" + "E"*498 + "e"
8458 
8459-        bw, rb, sharefname = self.make_bucket(name, sharesize)
8460+        bw, rb, sharefp = self.make_bucket(name, sharesize)
8461         bp = wbp_class(rb, None,
8462                        data_size=95,
8463                        block_size=25,
8464hunk ./src/allmydata/test/test_storage.py 258
8465 
8466         # now read everything back
8467         def _start_reading(res):
8468-            br = BucketReader(self, sharefname)
8469+            br = BucketReader(self, sharefp)
8470             rb = RemoteBucket()
8471             rb.target = br
8472             server = NoNetworkServer("abc", None)
8473hunk ./src/allmydata/test/test_storage.py 373
8474         for i, wb in writers.items():
8475             wb.remote_write(0, "%10d" % i)
8476             wb.remote_close()
8477-        storedir = os.path.join(self.workdir("test_dont_overfill_dirs"),
8478-                                "shares")
8479-        children_of_storedir = set(os.listdir(storedir))
8480+        storedir = self.workdir("test_dont_overfill_dirs").child("shares")
8481+        children_of_storedir = sorted([child.basename() for child in storedir.children()])
8482 
8483         # Now store another one under another storageindex that has leading
8484         # chars the same as the first storageindex.
8485hunk ./src/allmydata/test/test_storage.py 382
8486         for i, wb in writers.items():
8487             wb.remote_write(0, "%10d" % i)
8488             wb.remote_close()
8489-        storedir = os.path.join(self.workdir("test_dont_overfill_dirs"),
8490-                                "shares")
8491-        new_children_of_storedir = set(os.listdir(storedir))
8492+        storedir = self.workdir("test_dont_overfill_dirs").child("shares")
8493+        new_children_of_storedir = sorted([child.basename() for child in storedir.children()])
8494         self.failUnlessEqual(children_of_storedir, new_children_of_storedir)
8495 
8496     def test_remove_incoming(self):
8497hunk ./src/allmydata/test/test_storage.py 390
8498         ss = self.create("test_remove_incoming")
8499         already, writers = self.allocate(ss, "vid", range(3), 10)
8500         for i,wb in writers.items():
8501+            incoming_share_home = wb._share._home
8502             wb.remote_write(0, "%10d" % i)
8503             wb.remote_close()
8504hunk ./src/allmydata/test/test_storage.py 393
8505-        incoming_share_dir = wb.incominghome
8506-        incoming_bucket_dir = os.path.dirname(incoming_share_dir)
8507-        incoming_prefix_dir = os.path.dirname(incoming_bucket_dir)
8508-        incoming_dir = os.path.dirname(incoming_prefix_dir)
8509-        self.failIf(os.path.exists(incoming_bucket_dir), incoming_bucket_dir)
8510-        self.failIf(os.path.exists(incoming_prefix_dir), incoming_prefix_dir)
8511-        self.failUnless(os.path.exists(incoming_dir), incoming_dir)
8512+        incoming_bucket_dir = incoming_share_home.parent()
8513+        incoming_prefix_dir = incoming_bucket_dir.parent()
8514+        incoming_dir = incoming_prefix_dir.parent()
8515+        self.failIf(incoming_bucket_dir.exists(), incoming_bucket_dir)
8516+        self.failIf(incoming_prefix_dir.exists(), incoming_prefix_dir)
8517+        self.failUnless(incoming_dir.exists(), incoming_dir)
8518 
8519     def test_abort(self):
8520         # remote_abort, when called on a writer, should make sure that
8521hunk ./src/allmydata/test/test_upload.py 1849
8522             # remove the storedir, wiping out any existing shares
8523             fileutil.fp_remove(storedir)
8524             # create an empty storedir to replace the one we just removed
8525-            storedir.mkdir()
8526+            storedir.makedirs()
8527             client = self.g.clients[0]
8528             client.DEFAULT_ENCODING_PARAMETERS['happy'] = 4
8529             return client
8530hunk ./src/allmydata/test/test_upload.py 1890
8531             # remove the storedir, wiping out any existing shares
8532             fileutil.fp_remove(storedir)
8533             # create an empty storedir to replace the one we just removed
8534-            storedir.mkdir()
8535+            storedir.makedirs()
8536             client = self.g.clients[0]
8537             client.DEFAULT_ENCODING_PARAMETERS['happy'] = 4
8538             return client
8539}
8540[uri.py: resolve a conflict between trunk and the pluggable-backends patches. refs #999
8541david-sarah@jacaranda.org**20110921222038
8542 Ignore-this: ffeeab60d8e71a6a29a002d024d76fcf
8543] {
8544hunk ./src/allmydata/uri.py 829
8545     def is_mutable(self):
8546         return False
8547 
8548+    def is_readonly(self):
8549+        return True
8550+
8551+    def get_readonly(self):
8552+        return self
8553+
8554+
8555 class DirectoryURIVerifier(_DirectoryBaseURI):
8556     implements(IVerifierURI)
8557 
8558hunk ./src/allmydata/uri.py 855
8559     def is_mutable(self):
8560         return False
8561 
8562+    def is_readonly(self):
8563+        return True
8564+
8565+    def get_readonly(self):
8566+        return self
8567+
8568 
8569 class ImmutableDirectoryURIVerifier(DirectoryURIVerifier):
8570     implements(IVerifierURI)
8571}
8572[Fix some more test failures. refs #999
8573david-sarah@jacaranda.org**20110922045451
8574 Ignore-this: b726193cbd03a7c3d343f6e4a0f33ee7
8575] {
8576hunk ./src/allmydata/scripts/debug.py 42
8577     from allmydata.util.encodingutil import quote_output
8578 
8579     out = options.stdout
8580+    filename = options['filename']
8581 
8582     # check the version, to see if we have a mutable or immutable share
8583hunk ./src/allmydata/scripts/debug.py 45
8584-    print >>out, "share filename: %s" % quote_output(options['filename'])
8585+    print >>out, "share filename: %s" % quote_output(filename)
8586 
8587hunk ./src/allmydata/scripts/debug.py 47
8588-    share = get_share("", 0, fp)
8589+    share = get_share("", 0, FilePath(filename))
8590     if share.sharetype == "mutable":
8591         return dump_mutable_share(options, share)
8592     else:
8593hunk ./src/allmydata/storage/backends/disk/mutable.py 85
8594         self.parent = parent # for logging
8595 
8596     def log(self, *args, **kwargs):
8597-        return self.parent.log(*args, **kwargs)
8598+        if self.parent:
8599+            return self.parent.log(*args, **kwargs)
8600 
8601     def create(self, serverid, write_enabler):
8602         assert not self._home.exists()
8603hunk ./src/allmydata/storage/common.py 6
8604 class DataTooLargeError(Exception):
8605     pass
8606 
8607-class UnknownMutableContainerVersionError(Exception):
8608+class UnknownContainerVersionError(Exception):
8609     pass
8610 
8611hunk ./src/allmydata/storage/common.py 9
8612-class UnknownImmutableContainerVersionError(Exception):
8613+class UnknownMutableContainerVersionError(UnknownContainerVersionError):
8614+    pass
8615+
8616+class UnknownImmutableContainerVersionError(UnknownContainerVersionError):
8617     pass
8618 
8619 
8620hunk ./src/allmydata/storage/crawler.py 208
8621         try:
8622             state = pickle.loads(self.statefp.getContent())
8623         except EnvironmentError:
8624+            if self.statefp.exists():
8625+                raise
8626             state = {"version": 1,
8627                      "last-cycle-finished": None,
8628                      "current-cycle": None,
8629hunk ./src/allmydata/storage/server.py 24
8630 
8631     name = 'storage'
8632     LeaseCheckerClass = LeaseCheckingCrawler
8633+    BucketCounterClass = BucketCountingCrawler
8634     DEFAULT_EXPIRATION_POLICY = {
8635         'enabled': False,
8636         'mode': 'age',
8637hunk ./src/allmydata/storage/server.py 70
8638 
8639     def _setup_bucket_counter(self):
8640         statefp = self._statedir.child("bucket_counter.state")
8641-        self.bucket_counter = BucketCountingCrawler(self.backend, statefp)
8642+        self.bucket_counter = self.BucketCounterClass(self.backend, statefp)
8643         self.bucket_counter.setServiceParent(self)
8644 
8645     def _setup_lease_checker(self, expiration_policy):
8646hunk ./src/allmydata/storage/server.py 224
8647             share.add_or_renew_lease(lease_info)
8648             alreadygot.add(share.get_shnum())
8649 
8650-        for shnum in sharenums - alreadygot:
8651+        for shnum in set(sharenums) - alreadygot:
8652             if shareset.has_incoming(shnum):
8653                 # Note that we don't create BucketWriters for shnums that
8654                 # have a partial share (in incoming/), so if a second upload
8655hunk ./src/allmydata/storage/server.py 247
8656 
8657     def remote_add_lease(self, storageindex, renew_secret, cancel_secret,
8658                          owner_num=1):
8659-        # cancel_secret is no longer used.
8660         start = time.time()
8661         self.count("add-lease")
8662         new_expire_time = time.time() + 31*24*60*60
8663hunk ./src/allmydata/storage/server.py 250
8664-        lease_info = LeaseInfo(owner_num, renew_secret,
8665+        lease_info = LeaseInfo(owner_num, renew_secret, cancel_secret,
8666                                new_expire_time, self._serverid)
8667 
8668         try:
8669hunk ./src/allmydata/storage/server.py 254
8670-            self.backend.add_or_renew_lease(lease_info)
8671+            shareset = self.backend.get_shareset(storageindex)
8672+            shareset.add_or_renew_lease(lease_info)
8673         finally:
8674             self.add_latency("add-lease", time.time() - start)
8675 
8676hunk ./src/allmydata/test/test_crawler.py 3
8677 
8678 import time
8679-import os.path
8680+
8681 from twisted.trial import unittest
8682 from twisted.application import service
8683 from twisted.internet import defer
8684hunk ./src/allmydata/test/test_crawler.py 10
8685 from twisted.python.filepath import FilePath
8686 from foolscap.api import eventually, fireEventually
8687 
8688-from allmydata.util import fileutil, hashutil, pollmixin
8689+from allmydata.util import hashutil, pollmixin
8690 from allmydata.storage.server import StorageServer, si_b2a
8691 from allmydata.storage.crawler import ShareCrawler, TimeSliceExceeded
8692 from allmydata.storage.backends.disk.disk_backend import DiskBackend
8693hunk ./src/allmydata/test/test_mutable.py 3025
8694             cso.stderr = StringIO()
8695             debug.catalog_shares(cso)
8696             shares = cso.stdout.getvalue().splitlines()
8697+            self.failIf(len(shares) < 1, shares)
8698             oneshare = shares[0] # all shares should be MDMF
8699             self.failIf(oneshare.startswith("UNKNOWN"), oneshare)
8700             self.failUnless(oneshare.startswith("MDMF"), oneshare)
8701hunk ./src/allmydata/test/test_storage.py 1
8702-import time, os.path, platform, stat, re, simplejson, struct, shutil, itertools
8703+import time, os.path, platform, re, simplejson, struct, itertools
8704 
8705 import mock
8706 
8707hunk ./src/allmydata/test/test_storage.py 15
8708 from allmydata.util import fileutil, hashutil, base32, pollmixin, time_format
8709 from allmydata.storage.server import StorageServer
8710 from allmydata.storage.backends.disk.disk_backend import DiskBackend
8711+from allmydata.storage.backends.disk.immutable import ImmutableDiskShare
8712 from allmydata.storage.backends.disk.mutable import MutableDiskShare
8713 from allmydata.storage.bucket import BucketWriter, BucketReader
8714hunk ./src/allmydata/test/test_storage.py 18
8715-from allmydata.storage.common import DataTooLargeError, \
8716+from allmydata.storage.common import DataTooLargeError, UnknownContainerVersionError, \
8717      UnknownMutableContainerVersionError, UnknownImmutableContainerVersionError
8718 from allmydata.storage.lease import LeaseInfo
8719 from allmydata.storage.crawler import BucketCountingCrawler
8720hunk ./src/allmydata/test/test_storage.py 88
8721 
8722     def test_create(self):
8723         incoming, final = self.make_workdir("test_create")
8724-        share = ImmutableDiskShare("", 0, incoming, final, 200)
8725+        share = ImmutableDiskShare("", 0, incoming, final, max_size=200)
8726         bw = BucketWriter(self, share, self.make_lease(), FakeCanary())
8727         bw.remote_write(0, "a"*25)
8728         bw.remote_write(25, "b"*25)
8729hunk ./src/allmydata/test/test_storage.py 98
8730 
8731     def test_readwrite(self):
8732         incoming, final = self.make_workdir("test_readwrite")
8733-        share = ImmutableDiskShare("", 0, incoming, 200)
8734+        share = ImmutableDiskShare("", 0, incoming, final, max_size=200)
8735         bw = BucketWriter(self, share, self.make_lease(), FakeCanary())
8736         bw.remote_write(0, "a"*25)
8737         bw.remote_write(25, "b"*25)
8738hunk ./src/allmydata/test/test_storage.py 106
8739         bw.remote_close()
8740 
8741         # now read from it
8742-        br = BucketReader(self, bw.finalhome)
8743+        br = BucketReader(self, share)
8744         self.failUnlessEqual(br.remote_read(0, 25), "a"*25)
8745         self.failUnlessEqual(br.remote_read(25, 25), "b"*25)
8746         self.failUnlessEqual(br.remote_read(50, 7), "c"*7)
8747hunk ./src/allmydata/test/test_storage.py 131
8748         ownernumber = struct.pack('>L', 0)
8749         renewsecret  = 'THIS LETS ME RENEW YOUR FILE....'
8750         assert len(renewsecret) == 32
8751-        cancelsecret = 'THIS LETS ME KILL YOUR FILE HAHA'
8752+        cancelsecret = 'THIS USED TO LET ME KILL YR FILE'
8753         assert len(cancelsecret) == 32
8754         expirationtime = struct.pack('>L', 60*60*24*31) # 31 days in seconds
8755 
8756hunk ./src/allmydata/test/test_storage.py 142
8757         incoming, final = self.make_workdir("test_read_past_end_of_share_data")
8758 
8759         final.setContent(share_file_data)
8760+        share = ImmutableDiskShare("", 0, final)
8761 
8762         mockstorageserver = mock.Mock()
8763 
8764hunk ./src/allmydata/test/test_storage.py 147
8765         # Now read from it.
8766-        br = BucketReader(mockstorageserver, final)
8767+        br = BucketReader(mockstorageserver, share)
8768 
8769         self.failUnlessEqual(br.remote_read(0, len(share_data)), share_data)
8770 
8771hunk ./src/allmydata/test/test_storage.py 260
8772 
8773         # now read everything back
8774         def _start_reading(res):
8775-            br = BucketReader(self, sharefp)
8776+            share = ImmutableDiskShare("", 0, sharefp)
8777+            br = BucketReader(self, share)
8778             rb = RemoteBucket()
8779             rb.target = br
8780             server = NoNetworkServer("abc", None)
8781hunk ./src/allmydata/test/test_storage.py 346
8782         if 'cygwin' in syslow or 'windows' in syslow or 'darwin' in syslow:
8783             raise unittest.SkipTest("If your filesystem doesn't support efficient sparse files then it is very expensive (Mac OS X and Windows don't support efficient sparse files).")
8784 
8785-        avail = fileutil.get_available_space('.', 512*2**20)
8786+        avail = fileutil.get_available_space(FilePath('.'), 512*2**20)
8787         if avail <= 4*2**30:
8788             raise unittest.SkipTest("This test will spuriously fail if you have less than 4 GiB free on your filesystem.")
8789 
8790hunk ./src/allmydata/test/test_storage.py 476
8791         w[0].remote_write(0, "\xff"*10)
8792         w[0].remote_close()
8793 
8794-        fp = ss.backend.get_shareset("si1").sharehomedir.child("0")
8795+        fp = ss.backend.get_shareset("si1")._sharehomedir.child("0")
8796         f = fp.open("rb+")
8797hunk ./src/allmydata/test/test_storage.py 478
8798-        f.seek(0)
8799-        f.write(struct.pack(">L", 0)) # this is invalid: minimum used is v1
8800-        f.close()
8801+        try:
8802+            f.seek(0)
8803+            f.write(struct.pack(">L", 0)) # this is invalid: minimum used is v1
8804+        finally:
8805+            f.close()
8806 
8807         ss.remote_get_buckets("allocate")
8808 
8809hunk ./src/allmydata/test/test_storage.py 575
8810 
8811     def test_seek(self):
8812         basedir = self.workdir("test_seek_behavior")
8813-        fileutil.make_dirs(basedir)
8814-        filename = os.path.join(basedir, "testfile")
8815-        f = open(filename, "wb")
8816-        f.write("start")
8817-        f.close()
8818+        basedir.makedirs()
8819+        fp = basedir.child("testfile")
8820+        fp.setContent("start")
8821+
8822         # mode="w" allows seeking-to-create-holes, but truncates pre-existing
8823         # files. mode="a" preserves previous contents but does not allow
8824         # seeking-to-create-holes. mode="r+" allows both.
8825hunk ./src/allmydata/test/test_storage.py 582
8826-        f = open(filename, "rb+")
8827-        f.seek(100)
8828-        f.write("100")
8829-        f.close()
8830-        filelen = os.stat(filename)[stat.ST_SIZE]
8831+        f = fp.open("rb+")
8832+        try:
8833+            f.seek(100)
8834+            f.write("100")
8835+        finally:
8836+            f.close()
8837+        fp.restat()
8838+        filelen = fp.getsize()
8839         self.failUnlessEqual(filelen, 100+3)
8840hunk ./src/allmydata/test/test_storage.py 591
8841-        f2 = open(filename, "rb")
8842-        self.failUnlessEqual(f2.read(5), "start")
8843-
8844+        f2 = fp.open("rb")
8845+        try:
8846+            self.failUnlessEqual(f2.read(5), "start")
8847+        finally:
8848+            f2.close()
8849 
8850     def test_leases(self):
8851         ss = self.create("test_leases")
8852hunk ./src/allmydata/test/test_storage.py 693
8853 
8854     def test_readonly(self):
8855         workdir = self.workdir("test_readonly")
8856-        ss = StorageServer(workdir, "\x00" * 20, readonly_storage=True)
8857+        backend = DiskBackend(workdir, readonly=True)
8858+        ss = StorageServer("\x00" * 20, backend, workdir)
8859         ss.setServiceParent(self.sparent)
8860 
8861         already,writers = self.allocate(ss, "vid", [0,1,2], 75)
8862hunk ./src/allmydata/test/test_storage.py 710
8863 
8864     def test_discard(self):
8865         # discard is really only used for other tests, but we test it anyways
8866+        # XXX replace this with a null backend test
8867         workdir = self.workdir("test_discard")
8868hunk ./src/allmydata/test/test_storage.py 712
8869-        ss = StorageServer(workdir, "\x00" * 20, discard_storage=True)
8870+        backend = DiskBackend(workdir, readonly=False, discard_storage=True)
8871+        ss = StorageServer("\x00" * 20, backend, workdir)
8872         ss.setServiceParent(self.sparent)
8873 
8874         already,writers = self.allocate(ss, "vid", [0,1,2], 75)
8875hunk ./src/allmydata/test/test_storage.py 731
8876 
8877     def test_advise_corruption(self):
8878         workdir = self.workdir("test_advise_corruption")
8879-        ss = StorageServer(workdir, "\x00" * 20, discard_storage=True)
8880+        backend = DiskBackend(workdir, readonly=False, discard_storage=True)
8881+        ss = StorageServer("\x00" * 20, backend, workdir)
8882         ss.setServiceParent(self.sparent)
8883 
8884         si0_s = base32.b2a("si0")
8885hunk ./src/allmydata/test/test_storage.py 738
8886         ss.remote_advise_corrupt_share("immutable", "si0", 0,
8887                                        "This share smells funny.\n")
8888-        reportdir = os.path.join(workdir, "corruption-advisories")
8889-        reports = os.listdir(reportdir)
8890+        reportdir = workdir.child("corruption-advisories")
8891+        reports = [child.basename() for child in reportdir.children()]
8892         self.failUnlessEqual(len(reports), 1)
8893         report_si0 = reports[0]
8894hunk ./src/allmydata/test/test_storage.py 742
8895-        self.failUnlessIn(si0_s, report_si0)
8896-        f = open(os.path.join(reportdir, report_si0), "r")
8897-        report = f.read()
8898-        f.close()
8899+        self.failUnlessIn(si0_s, str(report_si0))
8900+        report = reportdir.child(report_si0).getContent()
8901+
8902         self.failUnlessIn("type: immutable", report)
8903         self.failUnlessIn("storage_index: %s" % si0_s, report)
8904         self.failUnlessIn("share_number: 0", report)
8905hunk ./src/allmydata/test/test_storage.py 762
8906         self.failUnlessEqual(set(b.keys()), set([1]))
8907         b[1].remote_advise_corrupt_share("This share tastes like dust.\n")
8908 
8909-        reports = os.listdir(reportdir)
8910+        reports = [child.basename() for child in reportdir.children()]
8911         self.failUnlessEqual(len(reports), 2)
8912hunk ./src/allmydata/test/test_storage.py 764
8913-        report_si1 = [r for r in reports if si1_s in r][0]
8914-        f = open(os.path.join(reportdir, report_si1), "r")
8915-        report = f.read()
8916-        f.close()
8917+        report_si1 = [r for r in reports if si1_s in str(r)][0]
8918+        report = reportdir.child(report_si1).getContent()
8919+
8920         self.failUnlessIn("type: immutable", report)
8921         self.failUnlessIn("storage_index: %s" % si1_s, report)
8922         self.failUnlessIn("share_number: 1", report)
8923hunk ./src/allmydata/test/test_storage.py 783
8924         return self.sparent.stopService()
8925 
8926     def workdir(self, name):
8927-        basedir = os.path.join("storage", "MutableServer", name)
8928-        return basedir
8929+        return FilePath("storage").child("MutableServer").child(name)
8930 
8931     def create(self, name):
8932         workdir = self.workdir(name)
8933hunk ./src/allmydata/test/test_storage.py 787
8934-        ss = StorageServer(workdir, "\x00" * 20)
8935+        backend = DiskBackend(workdir)
8936+        ss = StorageServer("\x00" * 20, backend, workdir)
8937         ss.setServiceParent(self.sparent)
8938         return ss
8939 
8940hunk ./src/allmydata/test/test_storage.py 810
8941         cancel_secret = self.cancel_secret(lease_tag)
8942         rstaraw = ss.remote_slot_testv_and_readv_and_writev
8943         testandwritev = dict( [ (shnum, ([], [], None) )
8944-                         for shnum in sharenums ] )
8945+                                for shnum in sharenums ] )
8946         readv = []
8947         rc = rstaraw(storage_index,
8948                      (write_enabler, renew_secret, cancel_secret),
8949hunk ./src/allmydata/test/test_storage.py 824
8950     def test_bad_magic(self):
8951         ss = self.create("test_bad_magic")
8952         self.allocate(ss, "si1", "we1", self._lease_secret.next(), set([0]), 10)
8953-        fp = ss.backend.get_shareset("si1").sharehomedir.child("0")
8954+        fp = ss.backend.get_shareset("si1")._sharehomedir.child("0")
8955         f = fp.open("rb+")
8956hunk ./src/allmydata/test/test_storage.py 826
8957-        f.seek(0)
8958-        f.write("BAD MAGIC")
8959-        f.close()
8960+        try:
8961+            f.seek(0)
8962+            f.write("BAD MAGIC")
8963+        finally:
8964+            f.close()
8965         read = ss.remote_slot_readv
8966hunk ./src/allmydata/test/test_storage.py 832
8967-        e = self.failUnlessRaises(UnknownMutableContainerVersionError,
8968+
8969+        # This used to test for UnknownMutableContainerVersionError,
8970+        # but the current code raises UnknownImmutableContainerVersionError.
8971+        # (It changed because remote_slot_readv now works with either
8972+        # mutable or immutable shares.) Since the share file doesn't have
8973+        # the mutable magic, it's not clear that this is wrong.
8974+        # For now, accept either exception.
8975+        e = self.failUnlessRaises(UnknownContainerVersionError,
8976                                   read, "si1", [0], [(0,10)])
8977hunk ./src/allmydata/test/test_storage.py 841
8978-        self.failUnlessIn(" had magic ", str(e))
8979+        self.failUnlessIn(" had ", str(e))
8980         self.failUnlessIn(" but we wanted ", str(e))
8981 
8982     def test_container_size(self):
8983hunk ./src/allmydata/test/test_storage.py 1248
8984 
8985         # create a random non-numeric file in the bucket directory, to
8986         # exercise the code that's supposed to ignore those.
8987-        bucket_dir = ss.backend.get_shareset("si1").sharehomedir
8988+        bucket_dir = ss.backend.get_shareset("si1")._sharehomedir
8989         bucket_dir.child("ignore_me.txt").setContent("you ought to be ignoring me\n")
8990 
8991hunk ./src/allmydata/test/test_storage.py 1251
8992-        s0 = MutableDiskShare(os.path.join(bucket_dir, "0"))
8993+        s0 = MutableDiskShare("", 0, bucket_dir.child("0"))
8994         self.failUnlessEqual(len(list(s0.get_leases())), 1)
8995 
8996         # add-lease on a missing storage index is silently ignored
8997hunk ./src/allmydata/test/test_storage.py 1365
8998         # note: this is a detail of the storage server implementation, and
8999         # may change in the future
9000         prefix = si[:2]
9001-        prefixdir = os.path.join(self.workdir("test_remove"), "shares", prefix)
9002-        bucketdir = os.path.join(prefixdir, si)
9003-        self.failUnless(os.path.exists(prefixdir), prefixdir)
9004-        self.failIf(os.path.exists(bucketdir), bucketdir)
9005+        prefixdir = self.workdir("test_remove").child("shares").child(prefix)
9006+        bucketdir = prefixdir.child(si)
9007+        self.failUnless(prefixdir.exists(), prefixdir)
9008+        self.failIf(bucketdir.exists(), bucketdir)
9009 
9010 
9011 class MDMFProxies(unittest.TestCase, ShouldFailMixin):
9012hunk ./src/allmydata/test/test_storage.py 1420
9013 
9014 
9015     def workdir(self, name):
9016-        basedir = os.path.join("storage", "MutableServer", name)
9017-        return basedir
9018-
9019+        return FilePath("storage").child("MDMFProxies").child(name)
9020 
9021     def create(self, name):
9022         workdir = self.workdir(name)
9023hunk ./src/allmydata/test/test_storage.py 1424
9024-        ss = StorageServer(workdir, "\x00" * 20)
9025+        backend = DiskBackend(workdir)
9026+        ss = StorageServer("\x00" * 20, backend, workdir)
9027         ss.setServiceParent(self.sparent)
9028         return ss
9029 
9030hunk ./src/allmydata/test/test_storage.py 2798
9031         return self.sparent.stopService()
9032 
9033     def workdir(self, name):
9034-        return FilePath("storage").child("Server").child(name)
9035+        return FilePath("storage").child("Stats").child(name)
9036 
9037     def create(self, name):
9038         workdir = self.workdir(name)
9039hunk ./src/allmydata/test/test_storage.py 2886
9040             d.callback(None)
9041 
9042 class MyStorageServer(StorageServer):
9043-    def add_bucket_counter(self):
9044-        statefile = os.path.join(self.storedir, "bucket_counter.state")
9045-        self.bucket_counter = MyBucketCountingCrawler(self, statefile)
9046-        self.bucket_counter.setServiceParent(self)
9047+    BucketCounterClass = MyBucketCountingCrawler
9048+
9049 
9050 class BucketCounter(unittest.TestCase, pollmixin.PollMixin):
9051 
9052hunk ./src/allmydata/test/test_storage.py 2899
9053 
9054     def test_bucket_counter(self):
9055         basedir = "storage/BucketCounter/bucket_counter"
9056-        fileutil.make_dirs(basedir)
9057-        ss = StorageServer(basedir, "\x00" * 20)
9058+        fp = FilePath(basedir)
9059+        backend = DiskBackend(fp)
9060+        ss = StorageServer("\x00" * 20, backend, fp)
9061+
9062         # to make sure we capture the bucket-counting-crawler in the middle
9063         # of a cycle, we reach in and reduce its maximum slice time to 0. We
9064         # also make it start sooner than usual.
9065hunk ./src/allmydata/test/test_storage.py 2958
9066 
9067     def test_bucket_counter_cleanup(self):
9068         basedir = "storage/BucketCounter/bucket_counter_cleanup"
9069-        fileutil.make_dirs(basedir)
9070-        ss = StorageServer(basedir, "\x00" * 20)
9071+        fp = FilePath(basedir)
9072+        backend = DiskBackend(fp)
9073+        ss = StorageServer("\x00" * 20, backend, fp)
9074+
9075         # to make sure we capture the bucket-counting-crawler in the middle
9076         # of a cycle, we reach in and reduce its maximum slice time to 0.
9077         ss.bucket_counter.slow_start = 0
9078hunk ./src/allmydata/test/test_storage.py 3002
9079 
9080     def test_bucket_counter_eta(self):
9081         basedir = "storage/BucketCounter/bucket_counter_eta"
9082-        fileutil.make_dirs(basedir)
9083-        ss = MyStorageServer(basedir, "\x00" * 20)
9084+        fp = FilePath(basedir)
9085+        backend = DiskBackend(fp)
9086+        ss = MyStorageServer("\x00" * 20, backend, fp)
9087         ss.bucket_counter.slow_start = 0
9088         # these will be fired inside finished_prefix()
9089         hooks = ss.bucket_counter.hook_ds = [defer.Deferred() for i in range(3)]
9090hunk ./src/allmydata/test/test_storage.py 3125
9091 
9092     def test_basic(self):
9093         basedir = "storage/LeaseCrawler/basic"
9094-        fileutil.make_dirs(basedir)
9095-        ss = InstrumentedStorageServer(basedir, "\x00" * 20)
9096+        fp = FilePath(basedir)
9097+        backend = DiskBackend(fp)
9098+        ss = InstrumentedStorageServer("\x00" * 20, backend, fp)
9099+
9100         # make it start sooner than usual.
9101         lc = ss.lease_checker
9102         lc.slow_start = 0
9103hunk ./src/allmydata/test/test_storage.py 3141
9104         [immutable_si_0, immutable_si_1, mutable_si_2, mutable_si_3] = self.sis
9105 
9106         # add a non-sharefile to exercise another code path
9107-        fp = ss.backend.get_shareset(immutable_si_0).sharehomedir.child("not-a-share")
9108+        fp = ss.backend.get_shareset(immutable_si_0)._sharehomedir.child("not-a-share")
9109         fp.setContent("I am not a share.\n")
9110 
9111         # this is before the crawl has started, so we're not in a cycle yet
9112hunk ./src/allmydata/test/test_storage.py 3264
9113             self.failUnlessEqual(rec["configured-sharebytes"], 0)
9114 
9115             def _get_sharefile(si):
9116-                return list(ss._iter_share_files(si))[0]
9117+                return list(ss.backend.get_shareset(si).get_shares())[0]
9118             def count_leases(si):
9119                 return len(list(_get_sharefile(si).get_leases()))
9120             self.failUnlessEqual(count_leases(immutable_si_0), 1)
9121hunk ./src/allmydata/test/test_storage.py 3296
9122         for i,lease in enumerate(sf.get_leases()):
9123             if lease.renew_secret == renew_secret:
9124                 lease.expiration_time = new_expire_time
9125-                f = open(sf.home, 'rb+')
9126-                sf._write_lease_record(f, i, lease)
9127-                f.close()
9128+                f = sf._home.open('rb+')
9129+                try:
9130+                    sf._write_lease_record(f, i, lease)
9131+                finally:
9132+                    f.close()
9133                 return
9134         raise IndexError("unable to renew non-existent lease")
9135 
9136hunk ./src/allmydata/test/test_storage.py 3306
9137     def test_expire_age(self):
9138         basedir = "storage/LeaseCrawler/expire_age"
9139-        fileutil.make_dirs(basedir)
9140+        fp = FilePath(basedir)
9141+        backend = DiskBackend(fp)
9142+
9143         # setting 'override_lease_duration' to 2000 means that any lease that
9144         # is more than 2000 seconds old will be expired.
9145         expiration_policy = {
9146hunk ./src/allmydata/test/test_storage.py 3317
9147             'override_lease_duration': 2000,
9148             'sharetypes': ('mutable', 'immutable'),
9149         }
9150-        ss = InstrumentedStorageServer(basedir, "\x00" * 20, expiration_policy)
9151+        ss = InstrumentedStorageServer("\x00" * 20, backend, fp, expiration_policy=expiration_policy)
9152+
9153         # make it start sooner than usual.
9154         lc = ss.lease_checker
9155         lc.slow_start = 0
9156hunk ./src/allmydata/test/test_storage.py 3330
9157         [immutable_si_0, immutable_si_1, mutable_si_2, mutable_si_3] = self.sis
9158 
9159         def count_shares(si):
9160-            return len(list(ss._iter_share_files(si)))
9161+            return len(list(ss.backend.get_shareset(si).get_shares()))
9162         def _get_sharefile(si):
9163hunk ./src/allmydata/test/test_storage.py 3332
9164-            return list(ss._iter_share_files(si))[0]
9165+            return list(ss.backend.get_shareset(si).get_shares())[0]
9166         def count_leases(si):
9167             return len(list(_get_sharefile(si).get_leases()))
9168 
9169hunk ./src/allmydata/test/test_storage.py 3355
9170 
9171         sf0 = _get_sharefile(immutable_si_0)
9172         self.backdate_lease(sf0, self.renew_secrets[0], now - 1000)
9173-        sf0_size = os.stat(sf0.home).st_size
9174+        sf0_size = sf0.get_size()
9175 
9176         # immutable_si_1 gets an extra lease
9177         sf1 = _get_sharefile(immutable_si_1)
9178hunk ./src/allmydata/test/test_storage.py 3363
9179 
9180         sf2 = _get_sharefile(mutable_si_2)
9181         self.backdate_lease(sf2, self.renew_secrets[3], now - 1000)
9182-        sf2_size = os.stat(sf2.home).st_size
9183+        sf2_size = sf2.get_size()
9184 
9185         # mutable_si_3 gets an extra lease
9186         sf3 = _get_sharefile(mutable_si_3)
9187hunk ./src/allmydata/test/test_storage.py 3450
9188 
9189     def test_expire_cutoff_date(self):
9190         basedir = "storage/LeaseCrawler/expire_cutoff_date"
9191-        fileutil.make_dirs(basedir)
9192+        fp = FilePath(basedir)
9193+        backend = DiskBackend(fp)
9194+
9195         # setting 'cutoff_date' to 2000 seconds ago means that any lease that
9196         # is more than 2000 seconds old will be expired.
9197         now = time.time()
9198hunk ./src/allmydata/test/test_storage.py 3463
9199             'cutoff_date': then,
9200             'sharetypes': ('mutable', 'immutable'),
9201         }
9202-        ss = InstrumentedStorageServer(basedir, "\x00" * 20, expiration_policy)
9203+        ss = InstrumentedStorageServer("\x00" * 20, backend, fp, expiration_policy=expiration_policy)
9204+
9205         # make it start sooner than usual.
9206         lc = ss.lease_checker
9207         lc.slow_start = 0
9208hunk ./src/allmydata/test/test_storage.py 3476
9209         [immutable_si_0, immutable_si_1, mutable_si_2, mutable_si_3] = self.sis
9210 
9211         def count_shares(si):
9212-            return len(list(ss._iter_share_files(si)))
9213+            return len(list(ss.backend.get_shareset(si).get_shares()))
9214         def _get_sharefile(si):
9215hunk ./src/allmydata/test/test_storage.py 3478
9216-            return list(ss._iter_share_files(si))[0]
9217+            return list(ss.backend.get_shareset(si).get_shares())[0]
9218         def count_leases(si):
9219             return len(list(_get_sharefile(si).get_leases()))
9220 
9221hunk ./src/allmydata/test/test_storage.py 3505
9222 
9223         sf0 = _get_sharefile(immutable_si_0)
9224         self.backdate_lease(sf0, self.renew_secrets[0], new_expiration_time)
9225-        sf0_size = os.stat(sf0.home).st_size
9226+        sf0_size = sf0.get_size()
9227 
9228         # immutable_si_1 gets an extra lease
9229         sf1 = _get_sharefile(immutable_si_1)
9230hunk ./src/allmydata/test/test_storage.py 3513
9231 
9232         sf2 = _get_sharefile(mutable_si_2)
9233         self.backdate_lease(sf2, self.renew_secrets[3], new_expiration_time)
9234-        sf2_size = os.stat(sf2.home).st_size
9235+        sf2_size = sf2.get_size()
9236 
9237         # mutable_si_3 gets an extra lease
9238         sf3 = _get_sharefile(mutable_si_3)
9239hunk ./src/allmydata/test/test_storage.py 3605
9240 
9241     def test_only_immutable(self):
9242         basedir = "storage/LeaseCrawler/only_immutable"
9243-        fileutil.make_dirs(basedir)
9244+        fp = FilePath(basedir)
9245+        backend = DiskBackend(fp)
9246+
9247         # setting 'cutoff_date' to 2000 seconds ago means that any lease that
9248         # is more than 2000 seconds old will be expired.
9249         now = time.time()
9250hunk ./src/allmydata/test/test_storage.py 3618
9251             'cutoff_date': then,
9252             'sharetypes': ('immutable',),
9253         }
9254-        ss = StorageServer(basedir, "\x00" * 20, expiration_policy)
9255+        ss = StorageServer("\x00" * 20, backend, fp, expiration_policy=expiration_policy)
9256         lc = ss.lease_checker
9257         lc.slow_start = 0
9258         webstatus = StorageStatus(ss)
9259hunk ./src/allmydata/test/test_storage.py 3629
9260         new_expiration_time = now - 3000 + 31*24*60*60
9261 
9262         def count_shares(si):
9263-            return len(list(ss._iter_share_files(si)))
9264+            return len(list(ss.backend.get_shareset(si).get_shares()))
9265         def _get_sharefile(si):
9266hunk ./src/allmydata/test/test_storage.py 3631
9267-            return list(ss._iter_share_files(si))[0]
9268+            return list(ss.backend.get_shareset(si).get_shares())[0]
9269         def count_leases(si):
9270             return len(list(_get_sharefile(si).get_leases()))
9271 
9272hunk ./src/allmydata/test/test_storage.py 3668
9273 
9274     def test_only_mutable(self):
9275         basedir = "storage/LeaseCrawler/only_mutable"
9276-        fileutil.make_dirs(basedir)
9277+        fp = FilePath(basedir)
9278+        backend = DiskBackend(fp)
9279+
9280         # setting 'cutoff_date' to 2000 seconds ago means that any lease that
9281         # is more than 2000 seconds old will be expired.
9282         now = time.time()
9283hunk ./src/allmydata/test/test_storage.py 3681
9284             'cutoff_date': then,
9285             'sharetypes': ('mutable',),
9286         }
9287-        ss = StorageServer(basedir, "\x00" * 20, expiration_policy)
9288+        ss = StorageServer("\x00" * 20, backend, fp, expiration_policy=expiration_policy)
9289         lc = ss.lease_checker
9290         lc.slow_start = 0
9291         webstatus = StorageStatus(ss)
9292hunk ./src/allmydata/test/test_storage.py 3692
9293         new_expiration_time = now - 3000 + 31*24*60*60
9294 
9295         def count_shares(si):
9296-            return len(list(ss._iter_share_files(si)))
9297+            return len(list(ss.backend.get_shareset(si).get_shares()))
9298         def _get_sharefile(si):
9299hunk ./src/allmydata/test/test_storage.py 3694
9300-            return list(ss._iter_share_files(si))[0]
9301+            return list(ss.backend.get_shareset(si).get_shares())[0]
9302         def count_leases(si):
9303             return len(list(_get_sharefile(si).get_leases()))
9304 
9305hunk ./src/allmydata/test/test_storage.py 3731
9306 
9307     def test_bad_mode(self):
9308         basedir = "storage/LeaseCrawler/bad_mode"
9309-        fileutil.make_dirs(basedir)
9310+        fp = FilePath(basedir)
9311+        backend = DiskBackend(fp)
9312+
9313+        expiration_policy = {
9314+            'enabled': True,
9315+            'mode': 'bogus',
9316+            'override_lease_duration': None,
9317+            'cutoff_date': None,
9318+            'sharetypes': ('mutable', 'immutable'),
9319+        }
9320         e = self.failUnlessRaises(ValueError,
9321hunk ./src/allmydata/test/test_storage.py 3742
9322-                                  StorageServer, basedir, "\x00" * 20,
9323-                                  expiration_mode="bogus")
9324+                                  StorageServer, "\x00" * 20, backend, fp,
9325+                                  expiration_policy=expiration_policy)
9326         self.failUnlessIn("GC mode 'bogus' must be 'age' or 'cutoff-date'", str(e))
9327 
9328     def test_parse_duration(self):
9329hunk ./src/allmydata/test/test_storage.py 3767
9330 
9331     def test_limited_history(self):
9332         basedir = "storage/LeaseCrawler/limited_history"
9333-        fileutil.make_dirs(basedir)
9334-        ss = StorageServer(basedir, "\x00" * 20)
9335+        fp = FilePath(basedir)
9336+        backend = DiskBackend(fp)
9337+        ss = StorageServer("\x00" * 20, backend, fp)
9338+
9339         # make it start sooner than usual.
9340         lc = ss.lease_checker
9341         lc.slow_start = 0
9342hunk ./src/allmydata/test/test_storage.py 3801
9343 
9344     def test_unpredictable_future(self):
9345         basedir = "storage/LeaseCrawler/unpredictable_future"
9346-        fileutil.make_dirs(basedir)
9347-        ss = StorageServer(basedir, "\x00" * 20)
9348+        fp = FilePath(basedir)
9349+        backend = DiskBackend(fp)
9350+        ss = StorageServer("\x00" * 20, backend, fp)
9351+
9352         # make it start sooner than usual.
9353         lc = ss.lease_checker
9354         lc.slow_start = 0
9355hunk ./src/allmydata/test/test_storage.py 3866
9356 
9357     def test_no_st_blocks(self):
9358         basedir = "storage/LeaseCrawler/no_st_blocks"
9359-        fileutil.make_dirs(basedir)
9360+        fp = FilePath(basedir)
9361+        backend = DiskBackend(fp)
9362+
9363         # A negative 'override_lease_duration' means that the "configured-"
9364         # space-recovered counts will be non-zero, since all shares will have
9365         # expired by then.
9366hunk ./src/allmydata/test/test_storage.py 3878
9367             'override_lease_duration': -1000,
9368             'sharetypes': ('mutable', 'immutable'),
9369         }
9370-        ss = No_ST_BLOCKS_StorageServer(basedir, "\x00" * 20, expiration_policy)
9371+        ss = No_ST_BLOCKS_StorageServer("\x00" * 20, backend, fp, expiration_policy=expiration_policy)
9372 
9373         # make it start sooner than usual.
9374         lc = ss.lease_checker
9375hunk ./src/allmydata/test/test_storage.py 3911
9376             UnknownImmutableContainerVersionError,
9377             ]
9378         basedir = "storage/LeaseCrawler/share_corruption"
9379-        fileutil.make_dirs(basedir)
9380-        ss = InstrumentedStorageServer(basedir, "\x00" * 20)
9381+        fp = FilePath(basedir)
9382+        backend = DiskBackend(fp)
9383+        ss = InstrumentedStorageServer("\x00" * 20, backend, fp)
9384         w = StorageStatus(ss)
9385         # make it start sooner than usual.
9386         lc = ss.lease_checker
9387hunk ./src/allmydata/test/test_storage.py 3928
9388         [immutable_si_0, immutable_si_1, mutable_si_2, mutable_si_3] = self.sis
9389         first = min(self.sis)
9390         first_b32 = base32.b2a(first)
9391-        fp = ss.backend.get_shareset(first).sharehomedir.child("0")
9392+        fp = ss.backend.get_shareset(first)._sharehomedir.child("0")
9393         f = fp.open("rb+")
9394hunk ./src/allmydata/test/test_storage.py 3930
9395-        f.seek(0)
9396-        f.write("BAD MAGIC")
9397-        f.close()
9398+        try:
9399+            f.seek(0)
9400+            f.write("BAD MAGIC")
9401+        finally:
9402+            f.close()
9403         # if get_share_file() doesn't see the correct mutable magic, it
9404         # assumes the file is an immutable share, and then
9405         # immutable.ShareFile sees a bad version. So regardless of which kind
9406hunk ./src/allmydata/test/test_storage.py 3943
9407 
9408         # also create an empty bucket
9409         empty_si = base32.b2a("\x04"*16)
9410-        empty_bucket_dir = ss.backend.get_shareset(empty_si).sharehomedir
9411+        empty_bucket_dir = ss.backend.get_shareset(empty_si)._sharehomedir
9412         fileutil.fp_make_dirs(empty_bucket_dir)
9413 
9414         ss.setServiceParent(self.s)
9415hunk ./src/allmydata/test/test_storage.py 4031
9416 
9417     def test_status(self):
9418         basedir = "storage/WebStatus/status"
9419-        fileutil.make_dirs(basedir)
9420-        ss = StorageServer(basedir, "\x00" * 20)
9421+        fp = FilePath(basedir)
9422+        backend = DiskBackend(fp)
9423+        ss = StorageServer("\x00" * 20, backend, fp)
9424         ss.setServiceParent(self.s)
9425         w = StorageStatus(ss)
9426         d = self.render1(w)
9427hunk ./src/allmydata/test/test_storage.py 4065
9428         # Some platforms may have no disk stats API. Make sure the code can handle that
9429         # (test runs on all platforms).
9430         basedir = "storage/WebStatus/status_no_disk_stats"
9431-        fileutil.make_dirs(basedir)
9432-        ss = StorageServer(basedir, "\x00" * 20)
9433+        fp = FilePath(basedir)
9434+        backend = DiskBackend(fp)
9435+        ss = StorageServer("\x00" * 20, backend, fp)
9436         ss.setServiceParent(self.s)
9437         w = StorageStatus(ss)
9438         html = w.renderSynchronously()
9439hunk ./src/allmydata/test/test_storage.py 4085
9440         # If the API to get disk stats exists but a call to it fails, then the status should
9441         # show that no shares will be accepted, and get_available_space() should be 0.
9442         basedir = "storage/WebStatus/status_bad_disk_stats"
9443-        fileutil.make_dirs(basedir)
9444-        ss = StorageServer(basedir, "\x00" * 20)
9445+        fp = FilePath(basedir)
9446+        backend = DiskBackend(fp)
9447+        ss = StorageServer("\x00" * 20, backend, fp)
9448         ss.setServiceParent(self.s)
9449         w = StorageStatus(ss)
9450         html = w.renderSynchronously()
9451}
9452[Fix most of the crawler tests. refs #999
9453david-sarah@jacaranda.org**20110922183008
9454 Ignore-this: 116c0848008f3989ba78d87c07ec783c
9455] {
9456hunk ./src/allmydata/storage/backends/disk/disk_backend.py 160
9457         self._discard_storage = discard_storage
9458 
9459     def get_overhead(self):
9460-        return (fileutil.get_disk_usage(self._sharehomedir) +
9461-                fileutil.get_disk_usage(self._incominghomedir))
9462+        return (fileutil.get_used_space(self._sharehomedir) +
9463+                fileutil.get_used_space(self._incominghomedir))
9464 
9465     def get_shares(self):
9466         """
9467hunk ./src/allmydata/storage/crawler.py 2
9468 
9469-import time, struct
9470-import cPickle as pickle
9471+import time, pickle, struct
9472 from twisted.internet import reactor
9473 from twisted.application import service
9474 
9475hunk ./src/allmydata/storage/crawler.py 205
9476         #                            shareset to be processed, or None if we
9477         #                            are sleeping between cycles
9478         try:
9479-            state = pickle.loads(self.statefp.getContent())
9480+            pickled = self.statefp.getContent()
9481         except EnvironmentError:
9482             if self.statefp.exists():
9483                 raise
9484hunk ./src/allmydata/storage/crawler.py 215
9485                      "last-complete-prefix": None,
9486                      "last-complete-bucket": None,
9487                      }
9488+        else:
9489+            state = pickle.loads(pickled)
9490+
9491         state.setdefault("current-cycle-start-time", time.time()) # approximate
9492         self.state = state
9493         lcp = state["last-complete-prefix"]
9494hunk ./src/allmydata/storage/crawler.py 246
9495         else:
9496             last_complete_prefix = self.prefixes[lcpi]
9497         self.state["last-complete-prefix"] = last_complete_prefix
9498-        self.statefp.setContent(pickle.dumps(self.state))
9499+        pickled = pickle.dumps(self.state)
9500+        self.statefp.setContent(pickled)
9501 
9502     def startService(self):
9503         # arrange things to look like we were just sleeping, so
9504hunk ./src/allmydata/storage/expirer.py 86
9505         # initialize history
9506         if not self.historyfp.exists():
9507             history = {} # cyclenum -> dict
9508-            self.historyfp.setContent(pickle.dumps(history))
9509+            pickled = pickle.dumps(history)
9510+            self.historyfp.setContent(pickled)
9511 
9512     def create_empty_cycle_dict(self):
9513         recovered = self.create_empty_recovered_dict()
9514hunk ./src/allmydata/storage/expirer.py 111
9515     def started_cycle(self, cycle):
9516         self.state["cycle-to-date"] = self.create_empty_cycle_dict()
9517 
9518-    def process_storage_index(self, cycle, prefix, container):
9519+    def process_shareset(self, cycle, prefix, shareset):
9520         would_keep_shares = []
9521         wks = None
9522hunk ./src/allmydata/storage/expirer.py 114
9523-        sharetype = None
9524 
9525hunk ./src/allmydata/storage/expirer.py 115
9526-        for share in container.get_shares():
9527-            sharetype = share.sharetype
9528+        for share in shareset.get_shares():
9529             try:
9530                 wks = self.process_share(share)
9531             except (UnknownMutableContainerVersionError,
9532hunk ./src/allmydata/storage/expirer.py 128
9533                 wks = (1, 1, 1, "unknown")
9534             would_keep_shares.append(wks)
9535 
9536-        container_type = None
9537+        shareset_type = None
9538         if wks:
9539hunk ./src/allmydata/storage/expirer.py 130
9540-            # use the last share's sharetype as the container type
9541-            container_type = wks[3]
9542+            # use the last share's type as the shareset type
9543+            shareset_type = wks[3]
9544         rec = self.state["cycle-to-date"]["space-recovered"]
9545         self.increment(rec, "examined-buckets", 1)
9546hunk ./src/allmydata/storage/expirer.py 134
9547-        if sharetype:
9548-            self.increment(rec, "examined-buckets-"+container_type, 1)
9549+        if shareset_type:
9550+            self.increment(rec, "examined-buckets-"+shareset_type, 1)
9551 
9552hunk ./src/allmydata/storage/expirer.py 137
9553-        container_diskbytes = container.get_overhead()
9554+        shareset_diskbytes = shareset.get_overhead()
9555 
9556         if sum([wks[0] for wks in would_keep_shares]) == 0:
9557hunk ./src/allmydata/storage/expirer.py 140
9558-            self.increment_container_space("original", container_diskbytes, sharetype)
9559+            self.increment_shareset_space("original", shareset_diskbytes, shareset_type)
9560         if sum([wks[1] for wks in would_keep_shares]) == 0:
9561hunk ./src/allmydata/storage/expirer.py 142
9562-            self.increment_container_space("configured", container_diskbytes, sharetype)
9563+            self.increment_shareset_space("configured", shareset_diskbytes, shareset_type)
9564         if sum([wks[2] for wks in would_keep_shares]) == 0:
9565hunk ./src/allmydata/storage/expirer.py 144
9566-            self.increment_container_space("actual", container_diskbytes, sharetype)
9567+            self.increment_shareset_space("actual", shareset_diskbytes, shareset_type)
9568 
9569     def process_share(self, share):
9570         sharetype = share.sharetype
9571hunk ./src/allmydata/storage/expirer.py 189
9572 
9573         so_far = self.state["cycle-to-date"]
9574         self.increment(so_far["leases-per-share-histogram"], num_leases, 1)
9575-        self.increment_space("examined", diskbytes, sharetype)
9576+        self.increment_space("examined", sharebytes, diskbytes, sharetype)
9577 
9578         would_keep_share = [1, 1, 1, sharetype]
9579 
9580hunk ./src/allmydata/storage/expirer.py 220
9581             self.increment(so_far_sr, a+"-sharebytes-"+sharetype, sharebytes)
9582             self.increment(so_far_sr, a+"-diskbytes-"+sharetype, diskbytes)
9583 
9584-    def increment_container_space(self, a, container_diskbytes, container_type):
9585+    def increment_shareset_space(self, a, shareset_diskbytes, shareset_type):
9586         rec = self.state["cycle-to-date"]["space-recovered"]
9587hunk ./src/allmydata/storage/expirer.py 222
9588-        self.increment(rec, a+"-diskbytes", container_diskbytes)
9589+        self.increment(rec, a+"-diskbytes", shareset_diskbytes)
9590         self.increment(rec, a+"-buckets", 1)
9591hunk ./src/allmydata/storage/expirer.py 224
9592-        if container_type:
9593-            self.increment(rec, a+"-diskbytes-"+container_type, container_diskbytes)
9594-            self.increment(rec, a+"-buckets-"+container_type, 1)
9595+        if shareset_type:
9596+            self.increment(rec, a+"-diskbytes-"+shareset_type, shareset_diskbytes)
9597+            self.increment(rec, a+"-buckets-"+shareset_type, 1)
9598 
9599     def increment(self, d, k, delta=1):
9600         if k not in d:
9601hunk ./src/allmydata/storage/expirer.py 280
9602         # copy() needs to become a deepcopy
9603         h["space-recovered"] = s["space-recovered"].copy()
9604 
9605-        history = pickle.loads(self.historyfp.getContent())
9606+        pickled = self.historyfp.getContent()
9607+        history = pickle.loads(pickled)
9608         history[cycle] = h
9609         while len(history) > 10:
9610             oldcycles = sorted(history.keys())
9611hunk ./src/allmydata/storage/expirer.py 286
9612             del history[oldcycles[0]]
9613-        self.historyfp.setContent(pickle.dumps(history))
9614+        repickled = pickle.dumps(history)
9615+        self.historyfp.setContent(repickled)
9616 
9617     def get_state(self):
9618         """In addition to the crawler state described in
9619hunk ./src/allmydata/storage/expirer.py 356
9620         progress = self.get_progress()
9621 
9622         state = ShareCrawler.get_state(self) # does a shallow copy
9623-        history = pickle.loads(self.historyfp.getContent())
9624+        pickled = self.historyfp.getContent()
9625+        history = pickle.loads(pickled)
9626         state["history"] = history
9627 
9628         if not progress["cycle-in-progress"]:
9629hunk ./src/allmydata/test/test_crawler.py 25
9630         ShareCrawler.__init__(self, *args, **kwargs)
9631         self.all_buckets = []
9632         self.finished_d = defer.Deferred()
9633-    def process_bucket(self, cycle, prefix, prefixdir, storage_index_b32):
9634-        self.all_buckets.append(storage_index_b32)
9635+
9636+    def process_shareset(self, cycle, prefix, shareset):
9637+        self.all_buckets.append(shareset.get_storage_index_string())
9638+
9639     def finished_cycle(self, cycle):
9640         eventually(self.finished_d.callback, None)
9641 
9642hunk ./src/allmydata/test/test_crawler.py 41
9643         self.all_buckets = []
9644         self.finished_d = defer.Deferred()
9645         self.yield_cb = None
9646-    def process_bucket(self, cycle, prefix, prefixdir, storage_index_b32):
9647-        self.all_buckets.append(storage_index_b32)
9648+
9649+    def process_shareset(self, cycle, prefix, shareset):
9650+        self.all_buckets.append(shareset.get_storage_index_string())
9651         self.countdown -= 1
9652         if self.countdown == 0:
9653             # force a timeout. We restore it in yielding()
9654hunk ./src/allmydata/test/test_crawler.py 66
9655         self.accumulated = 0.0
9656         self.cycles = 0
9657         self.last_yield = 0.0
9658-    def process_bucket(self, cycle, prefix, prefixdir, storage_index_b32):
9659+
9660+    def process_shareset(self, cycle, prefix, shareset):
9661         start = time.time()
9662         time.sleep(0.05)
9663         elapsed = time.time() - start
9664hunk ./src/allmydata/test/test_crawler.py 85
9665         ShareCrawler.__init__(self, *args, **kwargs)
9666         self.counter = 0
9667         self.finished_d = defer.Deferred()
9668-    def process_bucket(self, cycle, prefix, prefixdir, storage_index_b32):
9669+
9670+    def process_shareset(self, cycle, prefix, shareset):
9671         self.counter += 1
9672     def finished_cycle(self, cycle):
9673         self.finished_d.callback(None)
9674hunk ./src/allmydata/test/test_storage.py 3041
9675 
9676 class InstrumentedLeaseCheckingCrawler(LeaseCheckingCrawler):
9677     stop_after_first_bucket = False
9678-    def process_bucket(self, *args, **kwargs):
9679-        LeaseCheckingCrawler.process_bucket(self, *args, **kwargs)
9680+
9681+    def process_shareset(self, cycle, prefix, shareset):
9682+        LeaseCheckingCrawler.process_shareset(self, cycle, prefix, shareset)
9683         if self.stop_after_first_bucket:
9684             self.stop_after_first_bucket = False
9685             self.cpu_slice = -1.0
9686hunk ./src/allmydata/test/test_storage.py 3051
9687         if not self.stop_after_first_bucket:
9688             self.cpu_slice = 500
9689 
9690+class InstrumentedStorageServer(StorageServer):
9691+    LeaseCheckerClass = InstrumentedLeaseCheckingCrawler
9692+
9693+
9694 class BrokenStatResults:
9695     pass
9696 class No_ST_BLOCKS_LeaseCheckingCrawler(LeaseCheckingCrawler):
9697hunk ./src/allmydata/test/test_storage.py 3069
9698             setattr(bsr, attrname, getattr(s, attrname))
9699         return bsr
9700 
9701-class InstrumentedStorageServer(StorageServer):
9702-    LeaseCheckerClass = InstrumentedLeaseCheckingCrawler
9703 class No_ST_BLOCKS_StorageServer(StorageServer):
9704     LeaseCheckerClass = No_ST_BLOCKS_LeaseCheckingCrawler
9705 
9706}
9707[Reinstate the cancel_lease methods of ImmutableDiskShare and MutableDiskShare, since they are needed for lease expiry. refs #999
9708david-sarah@jacaranda.org**20110922183323
9709 Ignore-this: a11fb0dd0078ff627cb727fc769ec848
9710] {
9711hunk ./src/allmydata/storage/backends/disk/immutable.py 260
9712         except IndexError:
9713             self.add_lease(lease_info)
9714 
9715+    def cancel_lease(self, cancel_secret):
9716+        """Remove a lease with the given cancel_secret. If the last lease is
9717+        cancelled, the file will be removed. Return the number of bytes that
9718+        were freed (by truncating the list of leases, and possibly by
9719+        deleting the file). Raise IndexError if there was no lease with the
9720+        given cancel_secret.
9721+        """
9722+
9723+        leases = list(self.get_leases())
9724+        num_leases_removed = 0
9725+        for i, lease in enumerate(leases):
9726+            if constant_time_compare(lease.cancel_secret, cancel_secret):
9727+                leases[i] = None
9728+                num_leases_removed += 1
9729+        if not num_leases_removed:
9730+            raise IndexError("unable to find matching lease to cancel")
9731+
9732+        space_freed = 0
9733+        if num_leases_removed:
9734+            # pack and write out the remaining leases. We write these out in
9735+            # the same order as they were added, so that if we crash while
9736+            # doing this, we won't lose any non-cancelled leases.
9737+            leases = [l for l in leases if l] # remove the cancelled leases
9738+            if len(leases) > 0:
9739+                f = self._home.open('rb+')
9740+                try:
9741+                    for i, lease in enumerate(leases):
9742+                        self._write_lease_record(f, i, lease)
9743+                    self._write_num_leases(f, len(leases))
9744+                    self._truncate_leases(f, len(leases))
9745+                finally:
9746+                    f.close()
9747+                space_freed = self.LEASE_SIZE * num_leases_removed
9748+            else:
9749+                space_freed = fileutil.get_used_space(self._home)
9750+                self.unlink()
9751+        return space_freed
9752+
9753hunk ./src/allmydata/storage/backends/disk/mutable.py 361
9754         except IndexError:
9755             self.add_lease(lease_info)
9756 
9757+    def cancel_lease(self, cancel_secret):
9758+        """Remove any leases with the given cancel_secret. If the last lease
9759+        is cancelled, the file will be removed. Return the number of bytes
9760+        that were freed (by truncating the list of leases, and possibly by
9761+        deleting the file). Raise IndexError if there was no lease with the
9762+        given cancel_secret."""
9763+
9764+        # XXX can this be more like ImmutableDiskShare.cancel_lease?
9765+
9766+        accepting_nodeids = set()
9767+        modified = 0
9768+        remaining = 0
9769+        blank_lease = LeaseInfo(owner_num=0,
9770+                                renew_secret="\x00"*32,
9771+                                cancel_secret="\x00"*32,
9772+                                expiration_time=0,
9773+                                nodeid="\x00"*20)
9774+        f = self._home.open('rb+')
9775+        try:
9776+            for (leasenum, lease) in self._enumerate_leases(f):
9777+                accepting_nodeids.add(lease.nodeid)
9778+                if constant_time_compare(lease.cancel_secret, cancel_secret):
9779+                    self._write_lease_record(f, leasenum, blank_lease)
9780+                    modified += 1
9781+                else:
9782+                    remaining += 1
9783+            if modified:
9784+                freed_space = self._pack_leases(f)
9785+        finally:
9786+            f.close()
9787+
9788+        if modified > 0:
9789+            if remaining == 0:
9790+                freed_space = fileutil.get_used_space(self._home)
9791+                self.unlink()
9792+            return freed_space
9793+
9794+        msg = ("Unable to cancel non-existent lease. I have leases "
9795+               "accepted by nodeids: ")
9796+        msg += ",".join([("'%s'" % idlib.nodeid_b2a(anid))
9797+                         for anid in accepting_nodeids])
9798+        msg += " ."
9799+        raise IndexError(msg)
9800+
9801+    def _pack_leases(self, f):
9802+        # TODO: reclaim space from cancelled leases
9803+        return 0
9804+
9805     def _read_write_enabler_and_nodeid(self, f):
9806         f.seek(0)
9807         data = f.read(self.HEADER_SIZE)
9808}
9809[Blank line cleanups.
9810david-sarah@jacaranda.org**20110923012044
9811 Ignore-this: 8e1c4ecb5b0c65673af35872876a8591
9812] {
9813hunk ./src/allmydata/interfaces.py 33
9814 LeaseRenewSecret = Hash # used to protect lease renewal requests
9815 LeaseCancelSecret = Hash # used to protect lease cancellation requests
9816 
9817+
9818 class RIStubClient(RemoteInterface):
9819     """Each client publishes a service announcement for a dummy object called
9820     the StubClient. This object doesn't actually offer any services, but the
9821hunk ./src/allmydata/interfaces.py 42
9822     the grid and the client versions in use). This is the (empty)
9823     RemoteInterface for the StubClient."""
9824 
9825+
9826 class RIBucketWriter(RemoteInterface):
9827     """ Objects of this kind live on the server side. """
9828     def write(offset=Offset, data=ShareData):
9829hunk ./src/allmydata/interfaces.py 61
9830         """
9831         return None
9832 
9833+
9834 class RIBucketReader(RemoteInterface):
9835     def read(offset=Offset, length=ReadSize):
9836         return ShareData
9837hunk ./src/allmydata/interfaces.py 78
9838         documentation.
9839         """
9840 
9841+
9842 TestVector = ListOf(TupleOf(Offset, ReadSize, str, str))
9843 # elements are (offset, length, operator, specimen)
9844 # operator is one of "lt, le, eq, ne, ge, gt"
9845hunk ./src/allmydata/interfaces.py 95
9846 ReadData = ListOf(ShareData)
9847 # returns data[offset:offset+length] for each element of TestVector
9848 
9849+
9850 class RIStorageServer(RemoteInterface):
9851     __remote_name__ = "RIStorageServer.tahoe.allmydata.com"
9852 
9853hunk ./src/allmydata/interfaces.py 2255
9854 
9855     def get_storage_index():
9856         """Return a string with the (binary) storage index."""
9857+
9858     def get_storage_index_string():
9859         """Return a string with the (printable) abbreviated storage index."""
9860hunk ./src/allmydata/interfaces.py 2258
9861+
9862     def get_uri():
9863         """Return the (string) URI of the object that was checked."""
9864 
9865hunk ./src/allmydata/interfaces.py 2353
9866     def get_report():
9867         """Return a list of strings with more detailed results."""
9868 
9869+
9870 class ICheckAndRepairResults(Interface):
9871     """I contain the detailed results of a check/verify/repair operation.
9872 
9873hunk ./src/allmydata/interfaces.py 2363
9874 
9875     def get_storage_index():
9876         """Return a string with the (binary) storage index."""
9877+
9878     def get_storage_index_string():
9879         """Return a string with the (printable) abbreviated storage index."""
9880hunk ./src/allmydata/interfaces.py 2366
9881+
9882     def get_repair_attempted():
9883         """Return a boolean, True if a repair was attempted. We might not
9884         attempt to repair the file because it was healthy, or healthy enough
9885hunk ./src/allmydata/interfaces.py 2372
9886         (i.e. some shares were missing but not enough to exceed some
9887         threshold), or because we don't know how to repair this object."""
9888+
9889     def get_repair_successful():
9890         """Return a boolean, True if repair was attempted and the file/dir
9891         was fully healthy afterwards. False if no repair was attempted or if
9892hunk ./src/allmydata/interfaces.py 2377
9893         a repair attempt failed."""
9894+
9895     def get_pre_repair_results():
9896         """Return an ICheckResults instance that describes the state of the
9897         file/dir before any repair was attempted."""
9898hunk ./src/allmydata/interfaces.py 2381
9899+
9900     def get_post_repair_results():
9901         """Return an ICheckResults instance that describes the state of the
9902         file/dir after any repair was attempted. If no repair was attempted,
9903hunk ./src/allmydata/interfaces.py 2615
9904         (childnode, metadata_dict) tuples), the directory will be populated
9905         with those children, otherwise it will be empty."""
9906 
9907+
9908 class IClientStatus(Interface):
9909     def list_all_uploads():
9910         """Return a list of uploader objects, one for each upload that
9911hunk ./src/allmydata/interfaces.py 2621
9912         currently has an object available (tracked with weakrefs). This is
9913         intended for debugging purposes."""
9914+
9915     def list_active_uploads():
9916         """Return a list of active IUploadStatus objects."""
9917hunk ./src/allmydata/interfaces.py 2624
9918+
9919     def list_recent_uploads():
9920         """Return a list of IUploadStatus objects for the most recently
9921         started uploads."""
9922hunk ./src/allmydata/interfaces.py 2633
9923         """Return a list of downloader objects, one for each download that
9924         currently has an object available (tracked with weakrefs). This is
9925         intended for debugging purposes."""
9926+
9927     def list_active_downloads():
9928         """Return a list of active IDownloadStatus objects."""
9929hunk ./src/allmydata/interfaces.py 2636
9930+
9931     def list_recent_downloads():
9932         """Return a list of IDownloadStatus objects for the most recently
9933         started downloads."""
9934hunk ./src/allmydata/interfaces.py 2641
9935 
9936+
9937 class IUploadStatus(Interface):
9938     def get_started():
9939         """Return a timestamp (float with seconds since epoch) indicating
9940hunk ./src/allmydata/interfaces.py 2646
9941         when the operation was started."""
9942+
9943     def get_storage_index():
9944         """Return a string with the (binary) storage index in use on this
9945         upload. Returns None if the storage index has not yet been
9946hunk ./src/allmydata/interfaces.py 2651
9947         calculated."""
9948+
9949     def get_size():
9950         """Return an integer with the number of bytes that will eventually
9951         be uploaded for this file. Returns None if the size is not yet known.
9952hunk ./src/allmydata/interfaces.py 2656
9953         """
9954+
9955     def using_helper():
9956         """Return True if this upload is using a Helper, False if not."""
9957hunk ./src/allmydata/interfaces.py 2659
9958+
9959     def get_status():
9960         """Return a string describing the current state of the upload
9961         process."""
9962hunk ./src/allmydata/interfaces.py 2663
9963+
9964     def get_progress():
9965         """Returns a tuple of floats, (chk, ciphertext, encode_and_push),
9966         each from 0.0 to 1.0 . 'chk' describes how much progress has been
9967hunk ./src/allmydata/interfaces.py 2675
9968         process has finished: for helper uploads this is dependent upon the
9969         helper providing progress reports. It might be reasonable to add all
9970         three numbers and report the sum to the user."""
9971+
9972     def get_active():
9973         """Return True if the upload is currently active, False if not."""
9974hunk ./src/allmydata/interfaces.py 2678
9975+
9976     def get_results():
9977         """Return an instance of UploadResults (which contains timing and
9978         sharemap information). Might return None if the upload is not yet
9979hunk ./src/allmydata/interfaces.py 2683
9980         finished."""
9981+
9982     def get_counter():
9983         """Each upload status gets a unique number: this method returns that
9984         number. This provides a handle to this particular upload, so a web
9985hunk ./src/allmydata/interfaces.py 2689
9986         page can generate a suitable hyperlink."""
9987 
9988+
9989 class IDownloadStatus(Interface):
9990     def get_started():
9991         """Return a timestamp (float with seconds since epoch) indicating
9992hunk ./src/allmydata/interfaces.py 2694
9993         when the operation was started."""
9994+
9995     def get_storage_index():
9996         """Return a string with the (binary) storage index in use on this
9997         download. This may be None if there is no storage index (i.e. LIT
9998hunk ./src/allmydata/interfaces.py 2699
9999         files)."""
10000+
10001     def get_size():
10002         """Return an integer with the number of bytes that will eventually be
10003         retrieved for this file. Returns None if the size is not yet known.
10004hunk ./src/allmydata/interfaces.py 2704
10005         """
10006+
10007     def using_helper():
10008         """Return True if this download is using a Helper, False if not."""
10009hunk ./src/allmydata/interfaces.py 2707
10010+
10011     def get_status():
10012         """Return a string describing the current state of the download
10013         process."""
10014hunk ./src/allmydata/interfaces.py 2711
10015+
10016     def get_progress():
10017         """Returns a float (from 0.0 to 1.0) describing the amount of the
10018         download that has completed. This value will remain at 0.0 until the
10019hunk ./src/allmydata/interfaces.py 2716
10020         first byte of plaintext is pushed to the download target."""
10021+
10022     def get_active():
10023         """Return True if the download is currently active, False if not."""
10024hunk ./src/allmydata/interfaces.py 2719
10025+
10026     def get_counter():
10027         """Each download status gets a unique number: this method returns
10028         that number. This provides a handle to this particular download, so a
10029hunk ./src/allmydata/interfaces.py 2725
10030         web page can generate a suitable hyperlink."""
10031 
10032+
10033 class IServermapUpdaterStatus(Interface):
10034     pass
10035hunk ./src/allmydata/interfaces.py 2728
10036+
10037+
10038 class IPublishStatus(Interface):
10039     pass
10040hunk ./src/allmydata/interfaces.py 2732
10041+
10042+
10043 class IRetrieveStatus(Interface):
10044     pass
10045 
10046hunk ./src/allmydata/interfaces.py 2737
10047+
10048 class NotCapableError(Exception):
10049     """You have tried to write to a read-only node."""
10050 
10051hunk ./src/allmydata/interfaces.py 2741
10052+
10053 class BadWriteEnablerError(Exception):
10054     pass
10055 
10056hunk ./src/allmydata/interfaces.py 2745
10057-class RIControlClient(RemoteInterface):
10058 
10059hunk ./src/allmydata/interfaces.py 2746
10060+class RIControlClient(RemoteInterface):
10061     def wait_for_client_connections(num_clients=int):
10062         """Do not return until we have connections to at least NUM_CLIENTS
10063         storage servers.
10064hunk ./src/allmydata/interfaces.py 2801
10065 
10066         return DictOf(str, float)
10067 
10068+
10069 UploadResults = Any() #DictOf(str, str)
10070 
10071hunk ./src/allmydata/interfaces.py 2804
10072+
10073 class RIEncryptedUploadable(RemoteInterface):
10074     __remote_name__ = "RIEncryptedUploadable.tahoe.allmydata.com"
10075 
10076hunk ./src/allmydata/interfaces.py 2877
10077         """
10078         return DictOf(str, DictOf(str, ChoiceOf(float, int, long, None)))
10079 
10080+
10081 class RIStatsGatherer(RemoteInterface):
10082     __remote_name__ = "RIStatsGatherer.tahoe.allmydata.com"
10083     """
10084hunk ./src/allmydata/interfaces.py 2917
10085 class FileTooLargeError(Exception):
10086     pass
10087 
10088+
10089 class IValidatedThingProxy(Interface):
10090     def start():
10091         """ Acquire a thing and validate it. Return a deferred that is
10092hunk ./src/allmydata/interfaces.py 2924
10093         eventually fired with self if the thing is valid or errbacked if it
10094         can't be acquired or validated."""
10095 
10096+
10097 class InsufficientVersionError(Exception):
10098     def __init__(self, needed, got):
10099         self.needed = needed
10100hunk ./src/allmydata/interfaces.py 2933
10101         return "InsufficientVersionError(need '%s', got %s)" % (self.needed,
10102                                                                 self.got)
10103 
10104+
10105 class EmptyPathnameComponentError(Exception):
10106     """The webapi disallows empty pathname components."""
10107hunk ./src/allmydata/test/test_crawler.py 21
10108 class BucketEnumeratingCrawler(ShareCrawler):
10109     cpu_slice = 500 # make sure it can complete in a single slice
10110     slow_start = 0
10111+
10112     def __init__(self, *args, **kwargs):
10113         ShareCrawler.__init__(self, *args, **kwargs)
10114         self.all_buckets = []
10115hunk ./src/allmydata/test/test_crawler.py 33
10116     def finished_cycle(self, cycle):
10117         eventually(self.finished_d.callback, None)
10118 
10119+
10120 class PacedCrawler(ShareCrawler):
10121     cpu_slice = 500 # make sure it can complete in a single slice
10122     slow_start = 0
10123hunk ./src/allmydata/test/test_crawler.py 37
10124+
10125     def __init__(self, *args, **kwargs):
10126         ShareCrawler.__init__(self, *args, **kwargs)
10127         self.countdown = 6
10128hunk ./src/allmydata/test/test_crawler.py 51
10129         if self.countdown == 0:
10130             # force a timeout. We restore it in yielding()
10131             self.cpu_slice = -1.0
10132+
10133     def yielding(self, sleep_time):
10134         self.cpu_slice = 500
10135         if self.yield_cb:
10136hunk ./src/allmydata/test/test_crawler.py 56
10137             self.yield_cb()
10138+
10139     def finished_cycle(self, cycle):
10140         eventually(self.finished_d.callback, None)
10141 
10142hunk ./src/allmydata/test/test_crawler.py 60
10143+
10144 class ConsumingCrawler(ShareCrawler):
10145     cpu_slice = 0.5
10146     allowed_cpu_percentage = 0.5
10147hunk ./src/allmydata/test/test_crawler.py 79
10148         elapsed = time.time() - start
10149         self.accumulated += elapsed
10150         self.last_yield += elapsed
10151+
10152     def finished_cycle(self, cycle):
10153         self.cycles += 1
10154hunk ./src/allmydata/test/test_crawler.py 82
10155+
10156     def yielding(self, sleep_time):
10157         self.last_yield = 0.0
10158 
10159hunk ./src/allmydata/test/test_crawler.py 86
10160+
10161 class OneShotCrawler(ShareCrawler):
10162     cpu_slice = 500 # make sure it can complete in a single slice
10163     slow_start = 0
10164hunk ./src/allmydata/test/test_crawler.py 90
10165+
10166     def __init__(self, *args, **kwargs):
10167         ShareCrawler.__init__(self, *args, **kwargs)
10168         self.counter = 0
10169hunk ./src/allmydata/test/test_crawler.py 98
10170 
10171     def process_shareset(self, cycle, prefix, shareset):
10172         self.counter += 1
10173+
10174     def finished_cycle(self, cycle):
10175         self.finished_d.callback(None)
10176         self.disownServiceParent()
10177hunk ./src/allmydata/test/test_crawler.py 103
10178 
10179+
10180 class Basic(unittest.TestCase, StallMixin, pollmixin.PollMixin):
10181     def setUp(self):
10182         self.s = service.MultiService()
10183hunk ./src/allmydata/test/test_crawler.py 114
10184 
10185     def si(self, i):
10186         return hashutil.storage_index_hash(str(i))
10187+
10188     def rs(self, i, serverid):
10189         return hashutil.bucket_renewal_secret_hash(str(i), serverid)
10190hunk ./src/allmydata/test/test_crawler.py 117
10191+
10192     def cs(self, i, serverid):
10193         return hashutil.bucket_cancel_secret_hash(str(i), serverid)
10194 
10195hunk ./src/allmydata/test/test_storage.py 39
10196 from allmydata.test.no_network import NoNetworkServer
10197 from allmydata.web.storage import StorageStatus, remove_prefix
10198 
10199+
10200 class Marker:
10201     pass
10202hunk ./src/allmydata/test/test_storage.py 42
10203+
10204+
10205 class FakeCanary:
10206     def __init__(self, ignore_disconnectors=False):
10207         self.ignore = ignore_disconnectors
10208hunk ./src/allmydata/test/test_storage.py 59
10209             return
10210         del self.disconnectors[marker]
10211 
10212+
10213 class FakeStatsProvider:
10214     def count(self, name, delta=1):
10215         pass
10216hunk ./src/allmydata/test/test_storage.py 66
10217     def register_producer(self, producer):
10218         pass
10219 
10220+
10221 class Bucket(unittest.TestCase):
10222     def make_workdir(self, name):
10223         basedir = FilePath("storage").child("Bucket").child(name)
10224hunk ./src/allmydata/test/test_storage.py 165
10225         result_of_read = br.remote_read(0, len(share_data)+1)
10226         self.failUnlessEqual(result_of_read, share_data)
10227 
10228+
10229 class RemoteBucket:
10230 
10231     def __init__(self):
10232hunk ./src/allmydata/test/test_storage.py 309
10233         return self._do_test_readwrite("test_readwrite_v2",
10234                                        0x44, WriteBucketProxy_v2, ReadBucketProxy)
10235 
10236+
10237 class Server(unittest.TestCase):
10238 
10239     def setUp(self):
10240hunk ./src/allmydata/test/test_storage.py 780
10241         self.failUnlessIn("This share tastes like dust.", report)
10242 
10243 
10244-
10245 class MutableServer(unittest.TestCase):
10246 
10247     def setUp(self):
10248hunk ./src/allmydata/test/test_storage.py 1407
10249         # header.
10250         self.salt_hash_tree_s = self.serialize_blockhashes(self.salt_hash_tree[1:])
10251 
10252-
10253     def tearDown(self):
10254         self.sparent.stopService()
10255         fileutil.fp_remove(self.workdir("MDMFProxies storage test server"))
10256hunk ./src/allmydata/test/test_storage.py 1411
10257 
10258-
10259     def write_enabler(self, we_tag):
10260         return hashutil.tagged_hash("we_blah", we_tag)
10261 
10262hunk ./src/allmydata/test/test_storage.py 1414
10263-
10264     def renew_secret(self, tag):
10265         return hashutil.tagged_hash("renew_blah", str(tag))
10266 
10267hunk ./src/allmydata/test/test_storage.py 1417
10268-
10269     def cancel_secret(self, tag):
10270         return hashutil.tagged_hash("cancel_blah", str(tag))
10271 
10272hunk ./src/allmydata/test/test_storage.py 1420
10273-
10274     def workdir(self, name):
10275         return FilePath("storage").child("MDMFProxies").child(name)
10276 
10277hunk ./src/allmydata/test/test_storage.py 1430
10278         ss.setServiceParent(self.sparent)
10279         return ss
10280 
10281-
10282     def build_test_mdmf_share(self, tail_segment=False, empty=False):
10283         # Start with the checkstring
10284         data = struct.pack(">BQ32s",
10285hunk ./src/allmydata/test/test_storage.py 1527
10286         data += self.block_hash_tree_s
10287         return data
10288 
10289-
10290     def write_test_share_to_server(self,
10291                                    storage_index,
10292                                    tail_segment=False,
10293hunk ./src/allmydata/test/test_storage.py 1548
10294         results = write(storage_index, self.secrets, tws, readv)
10295         self.failUnless(results[0])
10296 
10297-
10298     def build_test_sdmf_share(self, empty=False):
10299         if empty:
10300             sharedata = ""
10301hunk ./src/allmydata/test/test_storage.py 1598
10302         self.offsets['EOF'] = eof_offset
10303         return final_share
10304 
10305-
10306     def write_sdmf_share_to_server(self,
10307                                    storage_index,
10308                                    empty=False):
10309hunk ./src/allmydata/test/test_storage.py 1613
10310         results = write(storage_index, self.secrets, tws, readv)
10311         self.failUnless(results[0])
10312 
10313-
10314     def test_read(self):
10315         self.write_test_share_to_server("si1")
10316         mr = MDMFSlotReadProxy(self.rref, "si1", 0)
10317hunk ./src/allmydata/test/test_storage.py 1682
10318             self.failUnlessEqual(checkstring, checkstring))
10319         return d
10320 
10321-
10322     def test_read_with_different_tail_segment_size(self):
10323         self.write_test_share_to_server("si1", tail_segment=True)
10324         mr = MDMFSlotReadProxy(self.rref, "si1", 0)
10325hunk ./src/allmydata/test/test_storage.py 1693
10326         d.addCallback(_check_tail_segment)
10327         return d
10328 
10329-
10330     def test_get_block_with_invalid_segnum(self):
10331         self.write_test_share_to_server("si1")
10332         mr = MDMFSlotReadProxy(self.rref, "si1", 0)
10333hunk ./src/allmydata/test/test_storage.py 1703
10334                             mr.get_block_and_salt, 7))
10335         return d
10336 
10337-
10338     def test_get_encoding_parameters_first(self):
10339         self.write_test_share_to_server("si1")
10340         mr = MDMFSlotReadProxy(self.rref, "si1", 0)
10341hunk ./src/allmydata/test/test_storage.py 1715
10342         d.addCallback(_check_encoding_parameters)
10343         return d
10344 
10345-
10346     def test_get_seqnum_first(self):
10347         self.write_test_share_to_server("si1")
10348         mr = MDMFSlotReadProxy(self.rref, "si1", 0)
10349hunk ./src/allmydata/test/test_storage.py 1723
10350             self.failUnlessEqual(seqnum, 0))
10351         return d
10352 
10353-
10354     def test_get_root_hash_first(self):
10355         self.write_test_share_to_server("si1")
10356         mr = MDMFSlotReadProxy(self.rref, "si1", 0)
10357hunk ./src/allmydata/test/test_storage.py 1731
10358             self.failUnlessEqual(root_hash, self.root_hash))
10359         return d
10360 
10361-
10362     def test_get_checkstring_first(self):
10363         self.write_test_share_to_server("si1")
10364         mr = MDMFSlotReadProxy(self.rref, "si1", 0)
10365hunk ./src/allmydata/test/test_storage.py 1739
10366             self.failUnlessEqual(checkstring, self.checkstring))
10367         return d
10368 
10369-
10370     def test_write_read_vectors(self):
10371         # When writing for us, the storage server will return to us a
10372         # read vector, along with its result. If a write fails because
10373hunk ./src/allmydata/test/test_storage.py 1777
10374         # The checkstring remains the same for the rest of the process.
10375         return d
10376 
10377-
10378     def test_private_key_after_share_hash_chain(self):
10379         mw = self._make_new_mw("si1", 0)
10380         d = defer.succeed(None)
10381hunk ./src/allmydata/test/test_storage.py 1795
10382                             mw.put_encprivkey, self.encprivkey))
10383         return d
10384 
10385-
10386     def test_signature_after_verification_key(self):
10387         mw = self._make_new_mw("si1", 0)
10388         d = defer.succeed(None)
10389hunk ./src/allmydata/test/test_storage.py 1821
10390                             mw.put_signature, self.signature))
10391         return d
10392 
10393-
10394     def test_uncoordinated_write(self):
10395         # Make two mutable writers, both pointing to the same storage
10396         # server, both at the same storage index, and try writing to the
10397hunk ./src/allmydata/test/test_storage.py 1853
10398         d.addCallback(_check_failure)
10399         return d
10400 
10401-
10402     def test_invalid_salt_size(self):
10403         # Salts need to be 16 bytes in size. Writes that attempt to
10404         # write more or less than this should be rejected.
10405hunk ./src/allmydata/test/test_storage.py 1871
10406                             another_invalid_salt))
10407         return d
10408 
10409-
10410     def test_write_test_vectors(self):
10411         # If we give the write proxy a bogus test vector at
10412         # any point during the process, it should fail to write when we
10413hunk ./src/allmydata/test/test_storage.py 1904
10414         d.addCallback(_check_success)
10415         return d
10416 
10417-
10418     def serialize_blockhashes(self, blockhashes):
10419         return "".join(blockhashes)
10420 
10421hunk ./src/allmydata/test/test_storage.py 1907
10422-
10423     def serialize_sharehashes(self, sharehashes):
10424         ret = "".join([struct.pack(">H32s", i, sharehashes[i])
10425                         for i in sorted(sharehashes.keys())])
10426hunk ./src/allmydata/test/test_storage.py 1912
10427         return ret
10428 
10429-
10430     def test_write(self):
10431         # This translates to a file with 6 6-byte segments, and with 2-byte
10432         # blocks.
10433hunk ./src/allmydata/test/test_storage.py 2043
10434                                 6, datalength)
10435         return mw
10436 
10437-
10438     def test_write_rejected_with_too_many_blocks(self):
10439         mw = self._make_new_mw("si0", 0)
10440 
10441hunk ./src/allmydata/test/test_storage.py 2059
10442                             mw.put_block, self.block, 7, self.salt))
10443         return d
10444 
10445-
10446     def test_write_rejected_with_invalid_salt(self):
10447         # Try writing an invalid salt. Salts are 16 bytes -- any more or
10448         # less should cause an error.
10449hunk ./src/allmydata/test/test_storage.py 2070
10450                             None, mw.put_block, self.block, 7, bad_salt))
10451         return d
10452 
10453-
10454     def test_write_rejected_with_invalid_root_hash(self):
10455         # Try writing an invalid root hash. This should be SHA256d, and
10456         # 32 bytes long as a result.
10457hunk ./src/allmydata/test/test_storage.py 2095
10458                             None, mw.put_root_hash, invalid_root_hash))
10459         return d
10460 
10461-
10462     def test_write_rejected_with_invalid_blocksize(self):
10463         # The blocksize implied by the writer that we get from
10464         # _make_new_mw is 2bytes -- any more or any less than this
10465hunk ./src/allmydata/test/test_storage.py 2128
10466             mw.put_block(valid_block, 5, self.salt))
10467         return d
10468 
10469-
10470     def test_write_enforces_order_constraints(self):
10471         # We require that the MDMFSlotWriteProxy be interacted with in a
10472         # specific way.
10473hunk ./src/allmydata/test/test_storage.py 2213
10474             mw0.put_verification_key(self.verification_key))
10475         return d
10476 
10477-
10478     def test_end_to_end(self):
10479         mw = self._make_new_mw("si1", 0)
10480         # Write a share using the mutable writer, and make sure that the
10481hunk ./src/allmydata/test/test_storage.py 2378
10482             self.failUnlessEqual(root_hash, self.root_hash, root_hash))
10483         return d
10484 
10485-
10486     def test_only_reads_one_segment_sdmf(self):
10487         # SDMF shares have only one segment, so it doesn't make sense to
10488         # read more segments than that. The reader should know this and
10489hunk ./src/allmydata/test/test_storage.py 2395
10490                             mr.get_block_and_salt, 1))
10491         return d
10492 
10493-
10494     def test_read_with_prefetched_mdmf_data(self):
10495         # The MDMFSlotReadProxy will prefill certain fields if you pass
10496         # it data that you have already fetched. This is useful for
10497hunk ./src/allmydata/test/test_storage.py 2459
10498         d.addCallback(_check_block_and_salt)
10499         return d
10500 
10501-
10502     def test_read_with_prefetched_sdmf_data(self):
10503         sdmf_data = self.build_test_sdmf_share()
10504         self.write_sdmf_share_to_server("si1")
10505hunk ./src/allmydata/test/test_storage.py 2522
10506         d.addCallback(_check_block_and_salt)
10507         return d
10508 
10509-
10510     def test_read_with_empty_mdmf_file(self):
10511         # Some tests upload a file with no contents to test things
10512         # unrelated to the actual handling of the content of the file.
10513hunk ./src/allmydata/test/test_storage.py 2550
10514                             mr.get_block_and_salt, 0))
10515         return d
10516 
10517-
10518     def test_read_with_empty_sdmf_file(self):
10519         self.write_sdmf_share_to_server("si1", empty=True)
10520         mr = MDMFSlotReadProxy(self.rref, "si1", 0)
10521hunk ./src/allmydata/test/test_storage.py 2575
10522                             mr.get_block_and_salt, 0))
10523         return d
10524 
10525-
10526     def test_verinfo_with_sdmf_file(self):
10527         self.write_sdmf_share_to_server("si1")
10528         mr = MDMFSlotReadProxy(self.rref, "si1", 0)
10529hunk ./src/allmydata/test/test_storage.py 2615
10530         d.addCallback(_check_verinfo)
10531         return d
10532 
10533-
10534     def test_verinfo_with_mdmf_file(self):
10535         self.write_test_share_to_server("si1")
10536         mr = MDMFSlotReadProxy(self.rref, "si1", 0)
10537hunk ./src/allmydata/test/test_storage.py 2653
10538         d.addCallback(_check_verinfo)
10539         return d
10540 
10541-
10542     def test_sdmf_writer(self):
10543         # Go through the motions of writing an SDMF share to the storage
10544         # server. Then read the storage server to see that the share got
10545hunk ./src/allmydata/test/test_storage.py 2696
10546         d.addCallback(_then)
10547         return d
10548 
10549-
10550     def test_sdmf_writer_preexisting_share(self):
10551         data = self.build_test_sdmf_share()
10552         self.write_sdmf_share_to_server("si1")
10553hunk ./src/allmydata/test/test_storage.py 2839
10554         self.failUnless(output["get"]["99_0_percentile"] is None, output)
10555         self.failUnless(output["get"]["99_9_percentile"] is None, output)
10556 
10557+
10558 def remove_tags(s):
10559     s = re.sub(r'<[^>]*>', ' ', s)
10560     s = re.sub(r'\s+', ' ', s)
10561hunk ./src/allmydata/test/test_storage.py 2845
10562     return s
10563 
10564+
10565 class MyBucketCountingCrawler(BucketCountingCrawler):
10566     def finished_prefix(self, cycle, prefix):
10567         BucketCountingCrawler.finished_prefix(self, cycle, prefix)
10568hunk ./src/allmydata/test/test_storage.py 2974
10569         backend = DiskBackend(fp)
10570         ss = MyStorageServer("\x00" * 20, backend, fp)
10571         ss.bucket_counter.slow_start = 0
10572+
10573         # these will be fired inside finished_prefix()
10574         hooks = ss.bucket_counter.hook_ds = [defer.Deferred() for i in range(3)]
10575         w = StorageStatus(ss)
10576hunk ./src/allmydata/test/test_storage.py 3008
10577         ss.setServiceParent(self.s)
10578         return d
10579 
10580+
10581 class InstrumentedLeaseCheckingCrawler(LeaseCheckingCrawler):
10582     stop_after_first_bucket = False
10583 
10584hunk ./src/allmydata/test/test_storage.py 3017
10585         if self.stop_after_first_bucket:
10586             self.stop_after_first_bucket = False
10587             self.cpu_slice = -1.0
10588+
10589     def yielding(self, sleep_time):
10590         if not self.stop_after_first_bucket:
10591             self.cpu_slice = 500
10592hunk ./src/allmydata/test/test_storage.py 3028
10593 
10594 class BrokenStatResults:
10595     pass
10596+
10597 class No_ST_BLOCKS_LeaseCheckingCrawler(LeaseCheckingCrawler):
10598     def stat(self, fn):
10599         s = os.stat(fn)
10600hunk ./src/allmydata/test/test_storage.py 3044
10601 class No_ST_BLOCKS_StorageServer(StorageServer):
10602     LeaseCheckerClass = No_ST_BLOCKS_LeaseCheckingCrawler
10603 
10604+
10605 class LeaseCrawler(unittest.TestCase, pollmixin.PollMixin, WebRenderingMixin):
10606 
10607     def setUp(self):
10608hunk ./src/allmydata/test/test_storage.py 3891
10609         backend = DiskBackend(fp)
10610         ss = InstrumentedStorageServer("\x00" * 20, backend, fp)
10611         w = StorageStatus(ss)
10612+
10613         # make it start sooner than usual.
10614         lc = ss.lease_checker
10615         lc.stop_after_first_bucket = True
10616hunk ./src/allmydata/util/fileutil.py 460
10617              'avail': avail,
10618            }
10619 
10620+
10621 def get_available_space(whichdirfp, reserved_space):
10622     """Returns available space for share storage in bytes, or None if no
10623     API to get this information is available.
10624}
10625[mutable/publish.py: elements should not be removed from a dictionary while it is being iterated over. refs #393
10626david-sarah@jacaranda.org**20110923040825
10627 Ignore-this: 135da94bd344db6ccd59a576b54901c1
10628] {
10629hunk ./src/allmydata/mutable/publish.py 6
10630 import os, time
10631 from StringIO import StringIO
10632 from itertools import count
10633+from copy import copy
10634 from zope.interface import implements
10635 from twisted.internet import defer
10636 from twisted.python import failure
10637hunk ./src/allmydata/mutable/publish.py 865
10638         ds = []
10639         verification_key = self._pubkey.serialize()
10640 
10641-
10642-        # TODO: Bad, since we remove from this same dict. We need to
10643-        # make a copy, or just use a non-iterated value.
10644-        for (shnum, writer) in self.writers.iteritems():
10645+        for (shnum, writer) in copy(self.writers).iteritems():
10646             writer.put_verification_key(verification_key)
10647             d = writer.finish_publishing()
10648             d.addErrback(self._connection_problem, writer)
10649}
10650[A few comment cleanups. refs #999
10651david-sarah@jacaranda.org**20110923041003
10652 Ignore-this: f574b4a3954b6946016646011ad15edf
10653] {
10654hunk ./src/allmydata/storage/backends/disk/disk_backend.py 17
10655 
10656 # storage/
10657 # storage/shares/incoming
10658-#   incoming/ holds temp dirs named $START/$STORAGEINDEX/$SHARENUM which will
10659-#   be moved to storage/shares/$START/$STORAGEINDEX/$SHARENUM upon success
10660-# storage/shares/$START/$STORAGEINDEX
10661-# storage/shares/$START/$STORAGEINDEX/$SHARENUM
10662+#   incoming/ holds temp dirs named $PREFIX/$STORAGEINDEX/$SHNUM which will
10663+#   be moved to storage/shares/$PREFIX/$STORAGEINDEX/$SHNUM upon success
10664+# storage/shares/$PREFIX/$STORAGEINDEX
10665+# storage/shares/$PREFIX/$STORAGEINDEX/$SHNUM
10666 
10667hunk ./src/allmydata/storage/backends/disk/disk_backend.py 22
10668-# Where "$START" denotes the first 10 bits worth of $STORAGEINDEX (that's 2
10669+# Where "$PREFIX" denotes the first 10 bits worth of $STORAGEINDEX (that's 2
10670 # base-32 chars).
10671 # $SHARENUM matches this regex:
10672 NUM_RE=re.compile("^[0-9]+$")
10673hunk ./src/allmydata/storage/backends/disk/immutable.py 16
10674 from allmydata.storage.lease import LeaseInfo
10675 
10676 
10677-# each share file (in storage/shares/$SI/$SHNUM) contains lease information
10678-# and share data. The share data is accessed by RIBucketWriter.write and
10679-# RIBucketReader.read . The lease information is not accessible through these
10680-# interfaces.
10681+# Each share file (in storage/shares/$PREFIX/$STORAGEINDEX/$SHNUM) contains
10682+# lease information and share data. The share data is accessed by
10683+# RIBucketWriter.write and RIBucketReader.read . The lease information is not
10684+# accessible through these remote interfaces.
10685 
10686 # The share file has the following layout:
10687 #  0x00: share file version number, four bytes, current version is 1
10688hunk ./src/allmydata/storage/backends/disk/immutable.py 211
10689 
10690     # These lease operations are intended for use by disk_backend.py.
10691     # Other clients should not depend on the fact that the disk backend
10692-    # stores leases in share files. XXX bucket.py also relies on this.
10693+    # stores leases in share files.
10694+    # XXX BucketWriter in bucket.py also relies on add_lease.
10695 
10696     def get_leases(self):
10697         """Yields a LeaseInfo instance for all leases."""
10698}
10699[Move advise_corrupt_share to allmydata/storage/backends/base.py, since it will be common to the disk and S3 backends. refs #999
10700david-sarah@jacaranda.org**20110923041115
10701 Ignore-this: 782b49f243bd98fcb6c249f8e40fd9f
10702] {
10703hunk ./src/allmydata/storage/backends/base.py 4
10704 
10705 from twisted.application import service
10706 
10707+from allmydata.util import fileutil, log, time_format
10708 from allmydata.storage.common import si_b2a
10709 from allmydata.storage.lease import LeaseInfo
10710 from allmydata.storage.bucket import BucketReader
10711hunk ./src/allmydata/storage/backends/base.py 13
10712 class Backend(service.MultiService):
10713     def __init__(self):
10714         service.MultiService.__init__(self)
10715+        self._corruption_advisory_dir = None
10716+
10717+    def advise_corrupt_share(self, sharetype, storageindex, shnum, reason):
10718+        if self._corruption_advisory_dir is not None:
10719+            fileutil.fp_make_dirs(self._corruption_advisory_dir)
10720+            now = time_format.iso_utc(sep="T")
10721+            si_s = si_b2a(storageindex)
10722+
10723+            # Windows can't handle colons in the filename.
10724+            name = ("%s--%s-%d" % (now, si_s, shnum)).replace(":", "")
10725+            f = self._corruption_advisory_dir.child(name).open("w")
10726+            try:
10727+                f.write("report: Share Corruption\n")
10728+                f.write("type: %s\n" % sharetype)
10729+                f.write("storage_index: %s\n" % si_s)
10730+                f.write("share_number: %d\n" % shnum)
10731+                f.write("\n")
10732+                f.write(reason)
10733+                f.write("\n")
10734+            finally:
10735+                f.close()
10736+
10737+        log.msg(format=("client claims corruption in (%(share_type)s) " +
10738+                        "%(si)s-%(shnum)d: %(reason)s"),
10739+                share_type=sharetype, si=si_s, shnum=shnum, reason=reason,
10740+                level=log.SCARY, umid="2fASGx")
10741 
10742 
10743 class ShareSet(object):
10744hunk ./src/allmydata/storage/backends/disk/disk_backend.py 8
10745 
10746 from zope.interface import implements
10747 from allmydata.interfaces import IStorageBackend, IShareSet
10748-from allmydata.util import fileutil, log, time_format
10749+from allmydata.util import fileutil, log
10750 from allmydata.storage.common import si_b2a, si_a2b
10751 from allmydata.storage.bucket import BucketWriter
10752 from allmydata.storage.backends.base import Backend, ShareSet
10753hunk ./src/allmydata/storage/backends/disk/disk_backend.py 125
10754             return 0
10755         return fileutil.get_available_space(self._sharedir, self._reserved_space)
10756 
10757-    def advise_corrupt_share(self, sharetype, storageindex, shnum, reason):
10758-        fileutil.fp_make_dirs(self._corruption_advisory_dir)
10759-        now = time_format.iso_utc(sep="T")
10760-        si_s = si_b2a(storageindex)
10761-
10762-        # Windows can't handle colons in the filename.
10763-        name = ("%s--%s-%d" % (now, si_s, shnum)).replace(":", "")
10764-        f = self._corruption_advisory_dir.child(name).open("w")
10765-        try:
10766-            f.write("report: Share Corruption\n")
10767-            f.write("type: %s\n" % sharetype)
10768-            f.write("storage_index: %s\n" % si_s)
10769-            f.write("share_number: %d\n" % shnum)
10770-            f.write("\n")
10771-            f.write(reason)
10772-            f.write("\n")
10773-        finally:
10774-            f.close()
10775-
10776-        log.msg(format=("client claims corruption in (%(share_type)s) " +
10777-                        "%(si)s-%(shnum)d: %(reason)s"),
10778-                share_type=sharetype, si=si_s, shnum=shnum, reason=reason,
10779-                level=log.SCARY, umid="SGx2fA")
10780-
10781 
10782 class DiskShareSet(ShareSet):
10783     implements(IShareSet)
10784}
10785[Add incomplete S3 backend. refs #999
10786david-sarah@jacaranda.org**20110923041314
10787 Ignore-this: b48df65699e3926dcbb87b5f755cdbf1
10788] {
10789adddir ./src/allmydata/storage/backends/s3
10790addfile ./src/allmydata/storage/backends/s3/__init__.py
10791addfile ./src/allmydata/storage/backends/s3/immutable.py
10792hunk ./src/allmydata/storage/backends/s3/immutable.py 1
10793+
10794+import struct
10795+
10796+from zope.interface import implements
10797+
10798+from allmydata.interfaces import IStoredShare
10799+from allmydata.util.assertutil import precondition
10800+from allmydata.storage.common import si_b2a, UnknownImmutableContainerVersionError, DataTooLargeError
10801+
10802+
10803+# Each share file (in storage/shares/$PREFIX/$STORAGEINDEX/$SHNUM) contains
10804+# lease information [currently inaccessible] and share data. The share data is
10805+# accessed by RIBucketWriter.write and RIBucketReader.read .
10806+
10807+# The share file has the following layout:
10808+#  0x00: share file version number, four bytes, current version is 1
10809+#  0x04: always zero (was share data length prior to Tahoe-LAFS v1.3.0)
10810+#  0x08: number of leases, four bytes big-endian
10811+#  0x0c: beginning of share data (see immutable.layout.WriteBucketProxy)
10812+#  data_length+0x0c: first lease. Each lease record is 72 bytes.
10813+
10814+
10815+class ImmutableS3Share(object):
10816+    implements(IStoredShare)
10817+
10818+    sharetype = "immutable"
10819+    LEASE_SIZE = struct.calcsize(">L32s32sL")  # for compatibility
10820+
10821+
10822+    def __init__(self, storageindex, shnum, s3bucket, create=False, max_size=None):
10823+        """
10824+        If max_size is not None then I won't allow more than max_size to be written to me.
10825+        """
10826+        precondition((max_size is not None) or not create, max_size, create)
10827+        self._storageindex = storageindex
10828+        self._max_size = max_size
10829+
10830+        self._s3bucket = s3bucket
10831+        si_s = si_b2a(storageindex)
10832+        self._key = "storage/shares/%s/%s/%d" % (si_s[:2], si_s, shnum)
10833+        self._shnum = shnum
10834+
10835+        if create:
10836+            # The second field, which was the four-byte share data length in
10837+            # Tahoe-LAFS versions prior to 1.3.0, is not used; we always write 0.
10838+            # We also write 0 for the number of leases.
10839+            self._home.setContent(struct.pack(">LLL", 1, 0, 0) )
10840+            self._end_offset = max_size + 0x0c
10841+
10842+            # TODO: start write to S3.
10843+        else:
10844+            # TODO: get header
10845+            header = "\x00"*12
10846+            (version, unused, num_leases) = struct.unpack(">LLL", header)
10847+
10848+            if version != 1:
10849+                msg = "sharefile %s had version %d but we wanted 1" % \
10850+                      (self._home, version)
10851+                raise UnknownImmutableContainerVersionError(msg)
10852+
10853+            # We cannot write leases in share files, but allow them to be present
10854+            # in case a share file is copied from a disk backend, or in case we
10855+            # need them in future.
10856+            # TODO: filesize = size of S3 object
10857+            self._end_offset = filesize - (num_leases * self.LEASE_SIZE)
10858+        self._data_offset = 0xc
10859+
10860+    def __repr__(self):
10861+        return ("<ImmutableS3Share %s:%r at %r>"
10862+                % (si_b2a(self._storageindex), self._shnum, self._key))
10863+
10864+    def close(self):
10865+        # TODO: finalize write to S3.
10866+        pass
10867+
10868+    def get_used_space(self):
10869+        return self._size
10870+
10871+    def get_storage_index(self):
10872+        return self._storageindex
10873+
10874+    def get_storage_index_string(self):
10875+        return si_b2a(self._storageindex)
10876+
10877+    def get_shnum(self):
10878+        return self._shnum
10879+
10880+    def unlink(self):
10881+        # TODO: remove the S3 object.
10882+        pass
10883+
10884+    def get_allocated_size(self):
10885+        return self._max_size
10886+
10887+    def get_size(self):
10888+        return self._size
10889+
10890+    def get_data_length(self):
10891+        return self._end_offset - self._data_offset
10892+
10893+    def read_share_data(self, offset, length):
10894+        precondition(offset >= 0)
10895+
10896+        # Reads beyond the end of the data are truncated. Reads that start
10897+        # beyond the end of the data return an empty string.
10898+        seekpos = self._data_offset+offset
10899+        actuallength = max(0, min(length, self._end_offset-seekpos))
10900+        if actuallength == 0:
10901+            return ""
10902+
10903+        # TODO: perform an S3 GET request, possibly with a Content-Range header.
10904+        return "\x00"*actuallength
10905+
10906+    def write_share_data(self, offset, data):
10907+        assert offset >= self._size, "offset = %r, size = %r" % (offset, self._size)
10908+
10909+        # TODO: write data to S3. If offset > self._size, fill the space
10910+        # between with zeroes.
10911+
10912+        self._size = offset + len(data)
10913+
10914+    def add_lease(self, lease_info):
10915+        pass
10916addfile ./src/allmydata/storage/backends/s3/mutable.py
10917hunk ./src/allmydata/storage/backends/s3/mutable.py 1
10918+
10919+import struct
10920+
10921+from zope.interface import implements
10922+
10923+from allmydata.interfaces import IStoredMutableShare, BadWriteEnablerError
10924+from allmydata.util import fileutil, idlib, log
10925+from allmydata.util.assertutil import precondition
10926+from allmydata.util.hashutil import constant_time_compare
10927+from allmydata.util.encodingutil import quote_filepath
10928+from allmydata.storage.common import si_b2a, UnknownMutableContainerVersionError, \
10929+     DataTooLargeError
10930+from allmydata.storage.lease import LeaseInfo
10931+from allmydata.storage.backends.base import testv_compare
10932+
10933+
10934+# The MutableDiskShare is like the ImmutableDiskShare, but used for mutable data.
10935+# It has a different layout. See docs/mutable.rst for more details.
10936+
10937+# #   offset    size    name
10938+# 1   0         32      magic verstr "tahoe mutable container v1" plus binary
10939+# 2   32        20      write enabler's nodeid
10940+# 3   52        32      write enabler
10941+# 4   84        8       data size (actual share data present) (a)
10942+# 5   92        8       offset of (8) count of extra leases (after data)
10943+# 6   100       368     four leases, 92 bytes each
10944+#                        0    4   ownerid (0 means "no lease here")
10945+#                        4    4   expiration timestamp
10946+#                        8   32   renewal token
10947+#                        40  32   cancel token
10948+#                        72  20   nodeid that accepted the tokens
10949+# 7   468       (a)     data
10950+# 8   ??        4       count of extra leases
10951+# 9   ??        n*92    extra leases
10952+
10953+
10954+# The struct module doc says that L's are 4 bytes in size, and that Q's are
10955+# 8 bytes in size. Since compatibility depends upon this, double-check it.
10956+assert struct.calcsize(">L") == 4, struct.calcsize(">L")
10957+assert struct.calcsize(">Q") == 8, struct.calcsize(">Q")
10958+
10959+
10960+class MutableDiskShare(object):
10961+    implements(IStoredMutableShare)
10962+
10963+    sharetype = "mutable"
10964+    DATA_LENGTH_OFFSET = struct.calcsize(">32s20s32s")
10965+    EXTRA_LEASE_OFFSET = DATA_LENGTH_OFFSET + 8
10966+    HEADER_SIZE = struct.calcsize(">32s20s32sQQ") # doesn't include leases
10967+    LEASE_SIZE = struct.calcsize(">LL32s32s20s")
10968+    assert LEASE_SIZE == 92
10969+    DATA_OFFSET = HEADER_SIZE + 4*LEASE_SIZE
10970+    assert DATA_OFFSET == 468, DATA_OFFSET
10971+
10972+    # our sharefiles share with a recognizable string, plus some random
10973+    # binary data to reduce the chance that a regular text file will look
10974+    # like a sharefile.
10975+    MAGIC = "Tahoe mutable container v1\n" + "\x75\x09\x44\x03\x8e"
10976+    assert len(MAGIC) == 32
10977+    MAX_SIZE = 2*1000*1000*1000 # 2GB, kind of arbitrary
10978+    # TODO: decide upon a policy for max share size
10979+
10980+    def __init__(self, storageindex, shnum, home, parent=None):
10981+        self._storageindex = storageindex
10982+        self._shnum = shnum
10983+        self._home = home
10984+        if self._home.exists():
10985+            # we don't cache anything, just check the magic
10986+            f = self._home.open('rb')
10987+            try:
10988+                data = f.read(self.HEADER_SIZE)
10989+                (magic,
10990+                 write_enabler_nodeid, write_enabler,
10991+                 data_length, extra_least_offset) = \
10992+                 struct.unpack(">32s20s32sQQ", data)
10993+                if magic != self.MAGIC:
10994+                    msg = "sharefile %s had magic '%r' but we wanted '%r'" % \
10995+                          (quote_filepath(self._home), magic, self.MAGIC)
10996+                    raise UnknownMutableContainerVersionError(msg)
10997+            finally:
10998+                f.close()
10999+        self.parent = parent # for logging
11000+
11001+    def log(self, *args, **kwargs):
11002+        if self.parent:
11003+            return self.parent.log(*args, **kwargs)
11004+
11005+    def create(self, serverid, write_enabler):
11006+        assert not self._home.exists()
11007+        data_length = 0
11008+        extra_lease_offset = (self.HEADER_SIZE
11009+                              + 4 * self.LEASE_SIZE
11010+                              + data_length)
11011+        assert extra_lease_offset == self.DATA_OFFSET # true at creation
11012+        num_extra_leases = 0
11013+        f = self._home.open('wb')
11014+        try:
11015+            header = struct.pack(">32s20s32sQQ",
11016+                                 self.MAGIC, serverid, write_enabler,
11017+                                 data_length, extra_lease_offset,
11018+                                 )
11019+            leases = ("\x00"*self.LEASE_SIZE) * 4
11020+            f.write(header + leases)
11021+            # data goes here, empty after creation
11022+            f.write(struct.pack(">L", num_extra_leases))
11023+            # extra leases go here, none at creation
11024+        finally:
11025+            f.close()
11026+
11027+    def __repr__(self):
11028+        return ("<MutableDiskShare %s:%r at %s>"
11029+                % (si_b2a(self._storageindex), self._shnum, quote_filepath(self._home)))
11030+
11031+    def get_used_space(self):
11032+        return fileutil.get_used_space(self._home)
11033+
11034+    def get_storage_index(self):
11035+        return self._storageindex
11036+
11037+    def get_storage_index_string(self):
11038+        return si_b2a(self._storageindex)
11039+
11040+    def get_shnum(self):
11041+        return self._shnum
11042+
11043+    def unlink(self):
11044+        self._home.remove()
11045+
11046+    def _read_data_length(self, f):
11047+        f.seek(self.DATA_LENGTH_OFFSET)
11048+        (data_length,) = struct.unpack(">Q", f.read(8))
11049+        return data_length
11050+
11051+    def _write_data_length(self, f, data_length):
11052+        f.seek(self.DATA_LENGTH_OFFSET)
11053+        f.write(struct.pack(">Q", data_length))
11054+
11055+    def _read_share_data(self, f, offset, length):
11056+        precondition(offset >= 0)
11057+        data_length = self._read_data_length(f)
11058+        if offset+length > data_length:
11059+            # reads beyond the end of the data are truncated. Reads that
11060+            # start beyond the end of the data return an empty string.
11061+            length = max(0, data_length-offset)
11062+        if length == 0:
11063+            return ""
11064+        precondition(offset+length <= data_length)
11065+        f.seek(self.DATA_OFFSET+offset)
11066+        data = f.read(length)
11067+        return data
11068+
11069+    def _read_extra_lease_offset(self, f):
11070+        f.seek(self.EXTRA_LEASE_OFFSET)
11071+        (extra_lease_offset,) = struct.unpack(">Q", f.read(8))
11072+        return extra_lease_offset
11073+
11074+    def _write_extra_lease_offset(self, f, offset):
11075+        f.seek(self.EXTRA_LEASE_OFFSET)
11076+        f.write(struct.pack(">Q", offset))
11077+
11078+    def _read_num_extra_leases(self, f):
11079+        offset = self._read_extra_lease_offset(f)
11080+        f.seek(offset)
11081+        (num_extra_leases,) = struct.unpack(">L", f.read(4))
11082+        return num_extra_leases
11083+
11084+    def _write_num_extra_leases(self, f, num_leases):
11085+        extra_lease_offset = self._read_extra_lease_offset(f)
11086+        f.seek(extra_lease_offset)
11087+        f.write(struct.pack(">L", num_leases))
11088+
11089+    def _change_container_size(self, f, new_container_size):
11090+        if new_container_size > self.MAX_SIZE:
11091+            raise DataTooLargeError()
11092+        old_extra_lease_offset = self._read_extra_lease_offset(f)
11093+        new_extra_lease_offset = self.DATA_OFFSET + new_container_size
11094+        if new_extra_lease_offset < old_extra_lease_offset:
11095+            # TODO: allow containers to shrink. For now they remain large.
11096+            return
11097+        num_extra_leases = self._read_num_extra_leases(f)
11098+        f.seek(old_extra_lease_offset)
11099+        leases_size = 4 + num_extra_leases * self.LEASE_SIZE
11100+        extra_lease_data = f.read(leases_size)
11101+
11102+        # Zero out the old lease info (in order to minimize the chance that
11103+        # it could accidentally be exposed to a reader later, re #1528).
11104+        f.seek(old_extra_lease_offset)
11105+        f.write('\x00' * leases_size)
11106+        f.flush()
11107+
11108+        # An interrupt here will corrupt the leases.
11109+
11110+        f.seek(new_extra_lease_offset)
11111+        f.write(extra_lease_data)
11112+        self._write_extra_lease_offset(f, new_extra_lease_offset)
11113+
11114+    def _write_share_data(self, f, offset, data):
11115+        length = len(data)
11116+        precondition(offset >= 0)
11117+        data_length = self._read_data_length(f)
11118+        extra_lease_offset = self._read_extra_lease_offset(f)
11119+
11120+        if offset+length >= data_length:
11121+            # They are expanding their data size.
11122+
11123+            if self.DATA_OFFSET+offset+length > extra_lease_offset:
11124+                # TODO: allow containers to shrink. For now, they remain
11125+                # large.
11126+
11127+                # Their new data won't fit in the current container, so we
11128+                # have to move the leases. With luck, they're expanding it
11129+                # more than the size of the extra lease block, which will
11130+                # minimize the corrupt-the-share window
11131+                self._change_container_size(f, offset+length)
11132+                extra_lease_offset = self._read_extra_lease_offset(f)
11133+
11134+                # an interrupt here is ok.. the container has been enlarged
11135+                # but the data remains untouched
11136+
11137+            assert self.DATA_OFFSET+offset+length <= extra_lease_offset
11138+            # Their data now fits in the current container. We must write
11139+            # their new data and modify the recorded data size.
11140+
11141+            # Fill any newly exposed empty space with 0's.
11142+            if offset > data_length:
11143+                f.seek(self.DATA_OFFSET+data_length)
11144+                f.write('\x00'*(offset - data_length))
11145+                f.flush()
11146+
11147+            new_data_length = offset+length
11148+            self._write_data_length(f, new_data_length)
11149+            # an interrupt here will result in a corrupted share
11150+
11151+        # now all that's left to do is write out their data
11152+        f.seek(self.DATA_OFFSET+offset)
11153+        f.write(data)
11154+        return
11155+
11156+    def _write_lease_record(self, f, lease_number, lease_info):
11157+        extra_lease_offset = self._read_extra_lease_offset(f)
11158+        num_extra_leases = self._read_num_extra_leases(f)
11159+        if lease_number < 4:
11160+            offset = self.HEADER_SIZE + lease_number * self.LEASE_SIZE
11161+        elif (lease_number-4) < num_extra_leases:
11162+            offset = (extra_lease_offset
11163+                      + 4
11164+                      + (lease_number-4)*self.LEASE_SIZE)
11165+        else:
11166+            # must add an extra lease record
11167+            self._write_num_extra_leases(f, num_extra_leases+1)
11168+            offset = (extra_lease_offset
11169+                      + 4
11170+                      + (lease_number-4)*self.LEASE_SIZE)
11171+        f.seek(offset)
11172+        assert f.tell() == offset
11173+        f.write(lease_info.to_mutable_data())
11174+
11175+    def _read_lease_record(self, f, lease_number):
11176+        # returns a LeaseInfo instance, or None
11177+        extra_lease_offset = self._read_extra_lease_offset(f)
11178+        num_extra_leases = self._read_num_extra_leases(f)
11179+        if lease_number < 4:
11180+            offset = self.HEADER_SIZE + lease_number * self.LEASE_SIZE
11181+        elif (lease_number-4) < num_extra_leases:
11182+            offset = (extra_lease_offset
11183+                      + 4
11184+                      + (lease_number-4)*self.LEASE_SIZE)
11185+        else:
11186+            raise IndexError("No such lease number %d" % lease_number)
11187+        f.seek(offset)
11188+        assert f.tell() == offset
11189+        data = f.read(self.LEASE_SIZE)
11190+        lease_info = LeaseInfo().from_mutable_data(data)
11191+        if lease_info.owner_num == 0:
11192+            return None
11193+        return lease_info
11194+
11195+    def _get_num_lease_slots(self, f):
11196+        # how many places do we have allocated for leases? Not all of them
11197+        # are filled.
11198+        num_extra_leases = self._read_num_extra_leases(f)
11199+        return 4+num_extra_leases
11200+
11201+    def _get_first_empty_lease_slot(self, f):
11202+        # return an int with the index of an empty slot, or None if we do not
11203+        # currently have an empty slot
11204+
11205+        for i in range(self._get_num_lease_slots(f)):
11206+            if self._read_lease_record(f, i) is None:
11207+                return i
11208+        return None
11209+
11210+    def get_leases(self):
11211+        """Yields a LeaseInfo instance for all leases."""
11212+        f = self._home.open('rb')
11213+        try:
11214+            for i, lease in self._enumerate_leases(f):
11215+                yield lease
11216+        finally:
11217+            f.close()
11218+
11219+    def _enumerate_leases(self, f):
11220+        for i in range(self._get_num_lease_slots(f)):
11221+            try:
11222+                data = self._read_lease_record(f, i)
11223+                if data is not None:
11224+                    yield i, data
11225+            except IndexError:
11226+                return
11227+
11228+    # These lease operations are intended for use by disk_backend.py.
11229+    # Other non-test clients should not depend on the fact that the disk
11230+    # backend stores leases in share files.
11231+
11232+    def add_lease(self, lease_info):
11233+        precondition(lease_info.owner_num != 0) # 0 means "no lease here"
11234+        f = self._home.open('rb+')
11235+        try:
11236+            num_lease_slots = self._get_num_lease_slots(f)
11237+            empty_slot = self._get_first_empty_lease_slot(f)
11238+            if empty_slot is not None:
11239+                self._write_lease_record(f, empty_slot, lease_info)
11240+            else:
11241+                self._write_lease_record(f, num_lease_slots, lease_info)
11242+        finally:
11243+            f.close()
11244+
11245+    def renew_lease(self, renew_secret, new_expire_time):
11246+        accepting_nodeids = set()
11247+        f = self._home.open('rb+')
11248+        try:
11249+            for (leasenum, lease) in self._enumerate_leases(f):
11250+                if constant_time_compare(lease.renew_secret, renew_secret):
11251+                    # yup. See if we need to update the owner time.
11252+                    if new_expire_time > lease.expiration_time:
11253+                        # yes
11254+                        lease.expiration_time = new_expire_time
11255+                        self._write_lease_record(f, leasenum, lease)
11256+                    return
11257+                accepting_nodeids.add(lease.nodeid)
11258+        finally:
11259+            f.close()
11260+        # Return the accepting_nodeids set, to give the client a chance to
11261+        # update the leases on a share that has been migrated from its
11262+        # original server to a new one.
11263+        msg = ("Unable to renew non-existent lease. I have leases accepted by"
11264+               " nodeids: ")
11265+        msg += ",".join([("'%s'" % idlib.nodeid_b2a(anid))
11266+                         for anid in accepting_nodeids])
11267+        msg += " ."
11268+        raise IndexError(msg)
11269+
11270+    def add_or_renew_lease(self, lease_info):
11271+        precondition(lease_info.owner_num != 0) # 0 means "no lease here"
11272+        try:
11273+            self.renew_lease(lease_info.renew_secret,
11274+                             lease_info.expiration_time)
11275+        except IndexError:
11276+            self.add_lease(lease_info)
11277+
11278+    def cancel_lease(self, cancel_secret):
11279+        """Remove any leases with the given cancel_secret. If the last lease
11280+        is cancelled, the file will be removed. Return the number of bytes
11281+        that were freed (by truncating the list of leases, and possibly by
11282+        deleting the file). Raise IndexError if there was no lease with the
11283+        given cancel_secret."""
11284+
11285+        # XXX can this be more like ImmutableDiskShare.cancel_lease?
11286+
11287+        accepting_nodeids = set()
11288+        modified = 0
11289+        remaining = 0
11290+        blank_lease = LeaseInfo(owner_num=0,
11291+                                renew_secret="\x00"*32,
11292+                                cancel_secret="\x00"*32,
11293+                                expiration_time=0,
11294+                                nodeid="\x00"*20)
11295+        f = self._home.open('rb+')
11296+        try:
11297+            for (leasenum, lease) in self._enumerate_leases(f):
11298+                accepting_nodeids.add(lease.nodeid)
11299+                if constant_time_compare(lease.cancel_secret, cancel_secret):
11300+                    self._write_lease_record(f, leasenum, blank_lease)
11301+                    modified += 1
11302+                else:
11303+                    remaining += 1
11304+            if modified:
11305+                freed_space = self._pack_leases(f)
11306+        finally:
11307+            f.close()
11308+
11309+        if modified > 0:
11310+            if remaining == 0:
11311+                freed_space = fileutil.get_used_space(self._home)
11312+                self.unlink()
11313+            return freed_space
11314+
11315+        msg = ("Unable to cancel non-existent lease. I have leases "
11316+               "accepted by nodeids: ")
11317+        msg += ",".join([("'%s'" % idlib.nodeid_b2a(anid))
11318+                         for anid in accepting_nodeids])
11319+        msg += " ."
11320+        raise IndexError(msg)
11321+
11322+    def _pack_leases(self, f):
11323+        # TODO: reclaim space from cancelled leases
11324+        return 0
11325+
11326+    def _read_write_enabler_and_nodeid(self, f):
11327+        f.seek(0)
11328+        data = f.read(self.HEADER_SIZE)
11329+        (magic,
11330+         write_enabler_nodeid, write_enabler,
11331+         data_length, extra_least_offset) = \
11332+         struct.unpack(">32s20s32sQQ", data)
11333+        assert magic == self.MAGIC
11334+        return (write_enabler, write_enabler_nodeid)
11335+
11336+    def readv(self, readv):
11337+        datav = []
11338+        f = self._home.open('rb')
11339+        try:
11340+            for (offset, length) in readv:
11341+                datav.append(self._read_share_data(f, offset, length))
11342+        finally:
11343+            f.close()
11344+        return datav
11345+
11346+    def get_size(self):
11347+        return self._home.getsize()
11348+
11349+    def get_data_length(self):
11350+        f = self._home.open('rb')
11351+        try:
11352+            data_length = self._read_data_length(f)
11353+        finally:
11354+            f.close()
11355+        return data_length
11356+
11357+    def check_write_enabler(self, write_enabler, si_s):
11358+        f = self._home.open('rb+')
11359+        try:
11360+            (real_write_enabler, write_enabler_nodeid) = self._read_write_enabler_and_nodeid(f)
11361+        finally:
11362+            f.close()
11363+        # avoid a timing attack
11364+        #if write_enabler != real_write_enabler:
11365+        if not constant_time_compare(write_enabler, real_write_enabler):
11366+            # accomodate share migration by reporting the nodeid used for the
11367+            # old write enabler.
11368+            self.log(format="bad write enabler on SI %(si)s,"
11369+                     " recorded by nodeid %(nodeid)s",
11370+                     facility="tahoe.storage",
11371+                     level=log.WEIRD, umid="cE1eBQ",
11372+                     si=si_s, nodeid=idlib.nodeid_b2a(write_enabler_nodeid))
11373+            msg = "The write enabler was recorded by nodeid '%s'." % \
11374+                  (idlib.nodeid_b2a(write_enabler_nodeid),)
11375+            raise BadWriteEnablerError(msg)
11376+
11377+    def check_testv(self, testv):
11378+        test_good = True
11379+        f = self._home.open('rb+')
11380+        try:
11381+            for (offset, length, operator, specimen) in testv:
11382+                data = self._read_share_data(f, offset, length)
11383+                if not testv_compare(data, operator, specimen):
11384+                    test_good = False
11385+                    break
11386+        finally:
11387+            f.close()
11388+        return test_good
11389+
11390+    def writev(self, datav, new_length):
11391+        f = self._home.open('rb+')
11392+        try:
11393+            for (offset, data) in datav:
11394+                self._write_share_data(f, offset, data)
11395+            if new_length is not None:
11396+                cur_length = self._read_data_length(f)
11397+                if new_length < cur_length:
11398+                    self._write_data_length(f, new_length)
11399+                    # TODO: if we're going to shrink the share file when the
11400+                    # share data has shrunk, then call
11401+                    # self._change_container_size() here.
11402+        finally:
11403+            f.close()
11404+
11405+    def close(self):
11406+        pass
11407+
11408+
11409+def create_mutable_disk_share(storageindex, shnum, fp, serverid, write_enabler, parent):
11410+    ms = MutableDiskShare(storageindex, shnum, fp, parent)
11411+    ms.create(serverid, write_enabler)
11412+    del ms
11413+    return MutableDiskShare(storageindex, shnum, fp, parent)
11414addfile ./src/allmydata/storage/backends/s3/s3_backend.py
11415hunk ./src/allmydata/storage/backends/s3/s3_backend.py 1
11416+
11417+from zope.interface import implements
11418+from allmydata.interfaces import IStorageBackend, IShareSet
11419+from allmydata.storage.common import si_b2a, si_a2b
11420+from allmydata.storage.bucket import BucketWriter
11421+from allmydata.storage.backends.base import Backend, ShareSet
11422+from allmydata.storage.backends.s3.immutable import ImmutableS3Share
11423+from allmydata.storage.backends.s3.mutable import MutableS3Share
11424+
11425+# The S3 bucket has keys of the form shares/$STORAGEINDEX/$SHARENUM
11426+
11427+
11428+class S3Backend(Backend):
11429+    implements(IStorageBackend)
11430+
11431+    def __init__(self, s3bucket, readonly=False, max_space=None, corruption_advisory_dir=None):
11432+        Backend.__init__(self)
11433+        self._s3bucket = s3bucket
11434+        self._readonly = readonly
11435+        if max_space is None:
11436+            self._max_space = 2**64
11437+        else:
11438+            self._max_space = int(max_space)
11439+
11440+        # TODO: any set-up for S3?
11441+
11442+        # we don't actually create the corruption-advisory dir until necessary
11443+        self._corruption_advisory_dir = corruption_advisory_dir
11444+
11445+    def get_sharesets_for_prefix(self, prefix):
11446+        # TODO: query S3 for keys matching prefix
11447+        return []
11448+
11449+    def get_shareset(self, storageindex):
11450+        return S3ShareSet(storageindex, self._s3bucket)
11451+
11452+    def fill_in_space_stats(self, stats):
11453+        stats['storage_server.max_space'] = self._max_space
11454+
11455+        # TODO: query space usage of S3 bucket
11456+        stats['storage_server.accepting_immutable_shares'] = int(not self._readonly)
11457+
11458+    def get_available_space(self):
11459+        if self._readonly:
11460+            return 0
11461+        # TODO: query space usage of S3 bucket
11462+        return self._max_space
11463+
11464+
11465+class S3ShareSet(ShareSet):
11466+    implements(IShareSet)
11467+
11468+    def __init__(self, storageindex, s3bucket):
11469+        ShareSet.__init__(self, storageindex)
11470+        self._s3bucket = s3bucket
11471+
11472+    def get_overhead(self):
11473+        return 0
11474+
11475+    def get_shares(self):
11476+        """
11477+        Generate IStorageBackendShare objects for shares we have for this storage index.
11478+        ("Shares we have" means completed ones, excluding incoming ones.)
11479+        """
11480+        pass
11481+
11482+    def has_incoming(self, shnum):
11483+        # TODO: this might need to be more like the disk backend; review callers
11484+        return False
11485+
11486+    def make_bucket_writer(self, storageserver, shnum, max_space_per_bucket, lease_info, canary):
11487+        immsh = ImmutableS3Share(self.get_storage_index(), shnum, self._s3bucket,
11488+                                 max_size=max_space_per_bucket)
11489+        bw = BucketWriter(storageserver, immsh, lease_info, canary)
11490+        return bw
11491+
11492+    def _create_mutable_share(self, storageserver, shnum, write_enabler):
11493+        # TODO
11494+        serverid = storageserver.get_serverid()
11495+        return MutableS3Share(self.get_storage_index(), shnum, self._s3bucket, serverid, write_enabler, storageserver)
11496+
11497+    def _clean_up_after_unlink(self):
11498+        pass
11499+
11500}
11501[interfaces.py: add fill_in_space_stats method to IStorageBackend. refs #999
11502david-sarah@jacaranda.org**20110923203723
11503 Ignore-this: 59371c150532055939794fed6c77dcb6
11504] {
11505hunk ./src/allmydata/interfaces.py 304
11506     def get_sharesets_for_prefix(prefix):
11507         """
11508         Generates IShareSet objects for all storage indices matching the
11509-        given prefix for which this backend holds shares.
11510+        given base-32 prefix for which this backend holds shares.
11511         """
11512 
11513     def get_shareset(storageindex):
11514hunk ./src/allmydata/interfaces.py 312
11515         Get an IShareSet object for the given storage index.
11516         """
11517 
11518+    def fill_in_space_stats(stats):
11519+        """
11520+        Fill in the 'stats' dict with space statistics for this backend, in
11521+        'storage_server.*' keys.
11522+        """
11523+
11524     def advise_corrupt_share(storageindex, sharetype, shnum, reason):
11525         """
11526         Clients who discover hash failures in shares that they have
11527}
11528[Remove redundant si_s argument from check_write_enabler. refs #999
11529david-sarah@jacaranda.org**20110923204425
11530 Ignore-this: 25be760118dbce2eb661137f7d46dd20
11531] {
11532hunk ./src/allmydata/interfaces.py 500
11533 
11534 
11535 class IStoredMutableShare(IStoredShare):
11536-    def check_write_enabler(write_enabler, si_s):
11537+    def check_write_enabler(write_enabler):
11538         """
11539         XXX
11540         """
11541hunk ./src/allmydata/storage/backends/base.py 102
11542         if len(secrets) > 2:
11543             cancel_secret = secrets[2]
11544 
11545-        si_s = self.get_storage_index_string()
11546         shares = {}
11547         for share in self.get_shares():
11548             # XXX is it correct to ignore immutable shares? Maybe get_shares should
11549hunk ./src/allmydata/storage/backends/base.py 107
11550             # have a parameter saying what type it's expecting.
11551             if share.sharetype == "mutable":
11552-                share.check_write_enabler(write_enabler, si_s)
11553+                share.check_write_enabler(write_enabler)
11554                 shares[share.get_shnum()] = share
11555 
11556         # write_enabler is good for all existing shares
11557hunk ./src/allmydata/storage/backends/disk/mutable.py 440
11558             f.close()
11559         return data_length
11560 
11561-    def check_write_enabler(self, write_enabler, si_s):
11562+    def check_write_enabler(self, write_enabler):
11563         f = self._home.open('rb+')
11564         try:
11565             (real_write_enabler, write_enabler_nodeid) = self._read_write_enabler_and_nodeid(f)
11566hunk ./src/allmydata/storage/backends/disk/mutable.py 447
11567         finally:
11568             f.close()
11569         # avoid a timing attack
11570-        #if write_enabler != real_write_enabler:
11571         if not constant_time_compare(write_enabler, real_write_enabler):
11572             # accomodate share migration by reporting the nodeid used for the
11573             # old write enabler.
11574hunk ./src/allmydata/storage/backends/disk/mutable.py 454
11575                      " recorded by nodeid %(nodeid)s",
11576                      facility="tahoe.storage",
11577                      level=log.WEIRD, umid="cE1eBQ",
11578-                     si=si_s, nodeid=idlib.nodeid_b2a(write_enabler_nodeid))
11579+                     si=self.get_storage_index_string(),
11580+                     nodeid=idlib.nodeid_b2a(write_enabler_nodeid))
11581             msg = "The write enabler was recorded by nodeid '%s'." % \
11582                   (idlib.nodeid_b2a(write_enabler_nodeid),)
11583             raise BadWriteEnablerError(msg)
11584hunk ./src/allmydata/storage/backends/s3/mutable.py 440
11585             f.close()
11586         return data_length
11587 
11588-    def check_write_enabler(self, write_enabler, si_s):
11589+    def check_write_enabler(self, write_enabler):
11590         f = self._home.open('rb+')
11591         try:
11592             (real_write_enabler, write_enabler_nodeid) = self._read_write_enabler_and_nodeid(f)
11593hunk ./src/allmydata/storage/backends/s3/mutable.py 447
11594         finally:
11595             f.close()
11596         # avoid a timing attack
11597-        #if write_enabler != real_write_enabler:
11598         if not constant_time_compare(write_enabler, real_write_enabler):
11599             # accomodate share migration by reporting the nodeid used for the
11600             # old write enabler.
11601hunk ./src/allmydata/storage/backends/s3/mutable.py 454
11602                      " recorded by nodeid %(nodeid)s",
11603                      facility="tahoe.storage",
11604                      level=log.WEIRD, umid="cE1eBQ",
11605-                     si=si_s, nodeid=idlib.nodeid_b2a(write_enabler_nodeid))
11606+                     si=self.get_storage_index_string(),
11607+                     nodeid=idlib.nodeid_b2a(write_enabler_nodeid))
11608             msg = "The write enabler was recorded by nodeid '%s'." % \
11609                   (idlib.nodeid_b2a(write_enabler_nodeid),)
11610             raise BadWriteEnablerError(msg)
11611}
11612[Implement readv for immutable shares. refs #999
11613david-sarah@jacaranda.org**20110923204611
11614 Ignore-this: 24f14b663051169d66293020e40c5a05
11615] {
11616hunk ./src/allmydata/storage/backends/disk/immutable.py 156
11617     def get_data_length(self):
11618         return self._lease_offset - self._data_offset
11619 
11620-    #def readv(self, read_vector):
11621-    #    ...
11622+    def readv(self, readv):
11623+        datav = []
11624+        f = self._home.open('rb')
11625+        try:
11626+            for (offset, length) in readv:
11627+                datav.append(self._read_share_data(f, offset, length))
11628+        finally:
11629+            f.close()
11630+        return datav
11631 
11632hunk ./src/allmydata/storage/backends/disk/immutable.py 166
11633-    def read_share_data(self, offset, length):
11634+    def _read_share_data(self, f, offset, length):
11635         precondition(offset >= 0)
11636 
11637         # Reads beyond the end of the data are truncated. Reads that start
11638hunk ./src/allmydata/storage/backends/disk/immutable.py 175
11639         actuallength = max(0, min(length, self._lease_offset-seekpos))
11640         if actuallength == 0:
11641             return ""
11642+        f.seek(seekpos)
11643+        return f.read(actuallength)
11644+
11645+    def read_share_data(self, offset, length):
11646         f = self._home.open(mode='rb')
11647         try:
11648hunk ./src/allmydata/storage/backends/disk/immutable.py 181
11649-            f.seek(seekpos)
11650-            sharedata = f.read(actuallength)
11651+            return self._read_share_data(f, offset, length)
11652         finally:
11653             f.close()
11654hunk ./src/allmydata/storage/backends/disk/immutable.py 184
11655-        return sharedata
11656 
11657     def write_share_data(self, offset, data):
11658         length = len(data)
11659hunk ./src/allmydata/storage/backends/null/null_backend.py 89
11660         return self.shnum
11661 
11662     def unlink(self):
11663-        os.unlink(self.fname)
11664+        pass
11665+
11666+    def readv(self, readv):
11667+        datav = []
11668+        for (offset, length) in readv:
11669+            datav.append("")
11670+        return datav
11671 
11672     def read_share_data(self, offset, length):
11673         precondition(offset >= 0)
11674hunk ./src/allmydata/storage/backends/s3/immutable.py 101
11675     def get_data_length(self):
11676         return self._end_offset - self._data_offset
11677 
11678+    def readv(self, readv):
11679+        datav = []
11680+        for (offset, length) in readv:
11681+            datav.append(self.read_share_data(offset, length))
11682+        return datav
11683+
11684     def read_share_data(self, offset, length):
11685         precondition(offset >= 0)
11686 
11687}
11688[The cancel secret needs to be unique, even if it isn't explicitly provided. refs #999
11689david-sarah@jacaranda.org**20110923204914
11690 Ignore-this: 6c44bb908dd4c0cdc59506b2d87a47b0
11691] {
11692hunk ./src/allmydata/storage/backends/base.py 98
11693 
11694         write_enabler = secrets[0]
11695         renew_secret = secrets[1]
11696-        cancel_secret = '\x00'*32
11697         if len(secrets) > 2:
11698             cancel_secret = secrets[2]
11699hunk ./src/allmydata/storage/backends/base.py 100
11700+        else:
11701+            cancel_secret = renew_secret
11702 
11703         shares = {}
11704         for share in self.get_shares():
11705}
11706[Make EmptyShare.check_testv a simple function. refs #999
11707david-sarah@jacaranda.org**20110923204945
11708 Ignore-this: d0132c085f40c39815fa920b77fc39ab
11709] {
11710hunk ./src/allmydata/storage/backends/base.py 125
11711             else:
11712                 # compare the vectors against an empty share, in which all
11713                 # reads return empty strings
11714-                if not EmptyShare().check_testv(testv):
11715+                if not empty_check_testv(testv):
11716                     storageserver.log("testv failed (empty): [%d] %r" % (sharenum, testv))
11717                     testv_is_good = False
11718                     break
11719hunk ./src/allmydata/storage/backends/base.py 195
11720     # never reached
11721 
11722 
11723-class EmptyShare:
11724-    def check_testv(self, testv):
11725-        test_good = True
11726-        for (offset, length, operator, specimen) in testv:
11727-            data = ""
11728-            if not testv_compare(data, operator, specimen):
11729-                test_good = False
11730-                break
11731-        return test_good
11732+def empty_check_testv(testv):
11733+    test_good = True
11734+    for (offset, length, operator, specimen) in testv:
11735+        data = ""
11736+        if not testv_compare(data, operator, specimen):
11737+            test_good = False
11738+            break
11739+    return test_good
11740 
11741}
11742[Update the null backend to take into account interface changes. Also, it now records which shares are present, but not their contents. refs #999
11743david-sarah@jacaranda.org**20110923205219
11744 Ignore-this: 42a23d7e253255003dc63facea783251
11745] {
11746hunk ./src/allmydata/storage/backends/null/null_backend.py 2
11747 
11748-import os, struct
11749-
11750 from zope.interface import implements
11751 
11752 from allmydata.interfaces import IStorageBackend, IShareSet, IStoredShare, IStoredMutableShare
11753hunk ./src/allmydata/storage/backends/null/null_backend.py 6
11754 from allmydata.util.assertutil import precondition
11755-from allmydata.util.hashutil import constant_time_compare
11756-from allmydata.storage.backends.base import Backend, ShareSet
11757-from allmydata.storage.bucket import BucketWriter
11758+from allmydata.storage.backends.base import Backend, empty_check_testv
11759+from allmydata.storage.bucket import BucketWriter, BucketReader
11760 from allmydata.storage.common import si_b2a
11761hunk ./src/allmydata/storage/backends/null/null_backend.py 9
11762-from allmydata.storage.lease import LeaseInfo
11763 
11764 
11765 class NullBackend(Backend):
11766hunk ./src/allmydata/storage/backends/null/null_backend.py 13
11767     implements(IStorageBackend)
11768+    """
11769+    I am a test backend that records (in memory) which shares exist, but not their contents, leases,
11770+    or write-enablers.
11771+    """
11772 
11773     def __init__(self):
11774         Backend.__init__(self)
11775hunk ./src/allmydata/storage/backends/null/null_backend.py 20
11776+        # mapping from storageindex to NullShareSet
11777+        self._sharesets = {}
11778 
11779hunk ./src/allmydata/storage/backends/null/null_backend.py 23
11780-    def get_available_space(self, reserved_space):
11781+    def get_available_space(self):
11782         return None
11783 
11784     def get_sharesets_for_prefix(self, prefix):
11785hunk ./src/allmydata/storage/backends/null/null_backend.py 27
11786-        pass
11787+        sharesets = []
11788+        for (si, shareset) in self._sharesets.iteritems():
11789+            if si_b2a(si).startswith(prefix):
11790+                sharesets.append(shareset)
11791+
11792+        def _by_base32si(b):
11793+            return b.get_storage_index_string()
11794+        sharesets.sort(key=_by_base32si)
11795+        return sharesets
11796 
11797     def get_shareset(self, storageindex):
11798hunk ./src/allmydata/storage/backends/null/null_backend.py 38
11799-        return NullShareSet(storageindex)
11800+        shareset = self._sharesets.get(storageindex, None)
11801+        if shareset is None:
11802+            shareset = NullShareSet(storageindex)
11803+            self._sharesets[storageindex] = shareset
11804+        return shareset
11805 
11806     def fill_in_space_stats(self, stats):
11807         pass
11808hunk ./src/allmydata/storage/backends/null/null_backend.py 47
11809 
11810-    def set_storage_server(self, ss):
11811-        self.ss = ss
11812 
11813hunk ./src/allmydata/storage/backends/null/null_backend.py 48
11814-    def advise_corrupt_share(self, sharetype, storageindex, shnum, reason):
11815-        pass
11816-
11817-
11818-class NullShareSet(ShareSet):
11819+class NullShareSet(object):
11820     implements(IShareSet)
11821 
11822     def __init__(self, storageindex):
11823hunk ./src/allmydata/storage/backends/null/null_backend.py 53
11824         self.storageindex = storageindex
11825+        self._incoming_shnums = set()
11826+        self._immutable_shnums = set()
11827+        self._mutable_shnums = set()
11828+
11829+    def close_shnum(self, shnum):
11830+        self._incoming_shnums.remove(shnum)
11831+        self._immutable_shnums.add(shnum)
11832 
11833     def get_overhead(self):
11834         return 0
11835hunk ./src/allmydata/storage/backends/null/null_backend.py 64
11836 
11837-    def get_incoming_shnums(self):
11838-        return frozenset()
11839-
11840     def get_shares(self):
11841hunk ./src/allmydata/storage/backends/null/null_backend.py 65
11842+        for shnum in self._immutable_shnums:
11843+            yield ImmutableNullShare(self, shnum)
11844+        for shnum in self._mutable_shnums:
11845+            yield MutableNullShare(self, shnum)
11846+
11847+    def renew_lease(self, renew_secret, new_expiration_time):
11848+        raise IndexError("no such lease to renew")
11849+
11850+    def get_leases(self):
11851         pass
11852 
11853hunk ./src/allmydata/storage/backends/null/null_backend.py 76
11854-    def get_share(self, shnum):
11855-        return None
11856+    def add_or_renew_lease(self, lease_info):
11857+        pass
11858+
11859+    def has_incoming(self, shnum):
11860+        return shnum in self._incoming_shnums
11861 
11862     def get_storage_index(self):
11863         return self.storageindex
11864hunk ./src/allmydata/storage/backends/null/null_backend.py 89
11865         return si_b2a(self.storageindex)
11866 
11867     def make_bucket_writer(self, storageserver, shnum, max_space_per_bucket, lease_info, canary):
11868-        immutableshare = ImmutableNullShare()
11869-        return BucketWriter(self.ss, immutableshare, max_space_per_bucket, lease_info, canary)
11870+        self._incoming_shnums.add(shnum)
11871+        immutableshare = ImmutableNullShare(self, shnum)
11872+        bw = BucketWriter(storageserver, immutableshare, lease_info, canary)
11873+        bw.throw_out_all_data = True
11874+        return bw
11875 
11876hunk ./src/allmydata/storage/backends/null/null_backend.py 95
11877-    def _create_mutable_share(self, storageserver, shnum, write_enabler):
11878-        return MutableNullShare()
11879+    def make_bucket_reader(self, storageserver, share):
11880+        return BucketReader(storageserver, share)
11881 
11882hunk ./src/allmydata/storage/backends/null/null_backend.py 98
11883-    def _clean_up_after_unlink(self):
11884-        pass
11885+    def testv_and_readv_and_writev(self, storageserver, secrets,
11886+                                   test_and_write_vectors, read_vector,
11887+                                   expiration_time):
11888+        # evaluate test vectors
11889+        testv_is_good = True
11890+        for sharenum in test_and_write_vectors:
11891+            # compare the vectors against an empty share, in which all
11892+            # reads return empty strings
11893+            (testv, datav, new_length) = test_and_write_vectors[sharenum]
11894+            if not empty_check_testv(testv):
11895+                storageserver.log("testv failed (empty): [%d] %r" % (sharenum, testv))
11896+                testv_is_good = False
11897+                break
11898 
11899hunk ./src/allmydata/storage/backends/null/null_backend.py 112
11900+        # gather the read vectors
11901+        read_data = {}
11902+        for shnum in self._mutable_shnums:
11903+            read_data[shnum] = ""
11904 
11905hunk ./src/allmydata/storage/backends/null/null_backend.py 117
11906-class ImmutableNullShare:
11907-    implements(IStoredShare)
11908-    sharetype = "immutable"
11909+        if testv_is_good:
11910+            # now apply the write vectors
11911+            for shnum in test_and_write_vectors:
11912+                (testv, datav, new_length) = test_and_write_vectors[shnum]
11913+                if new_length == 0:
11914+                    self._mutable_shnums.remove(shnum)
11915+                else:
11916+                    self._mutable_shnums.add(shnum)
11917 
11918hunk ./src/allmydata/storage/backends/null/null_backend.py 126
11919-    def __init__(self):
11920-        """ If max_size is not None then I won't allow more than
11921-        max_size to be written to me. If create=True then max_size
11922-        must not be None. """
11923-        pass
11924+        return (testv_is_good, read_data)
11925+
11926+    def readv(self, wanted_shnums, read_vector):
11927+        return {}
11928+
11929+
11930+class NullShareBase(object):
11931+    def __init__(self, shareset, shnum):
11932+        self.shareset = shareset
11933+        self.shnum = shnum
11934+
11935+    def get_storage_index(self):
11936+        return self.shareset.get_storage_index()
11937+
11938+    def get_storage_index_string(self):
11939+        return self.shareset.get_storage_index_string()
11940 
11941     def get_shnum(self):
11942         return self.shnum
11943hunk ./src/allmydata/storage/backends/null/null_backend.py 146
11944 
11945+    def get_data_length(self):
11946+        return 0
11947+
11948+    def get_size(self):
11949+        return 0
11950+
11951+    def get_used_space(self):
11952+        return 0
11953+
11954     def unlink(self):
11955         pass
11956 
11957hunk ./src/allmydata/storage/backends/null/null_backend.py 166
11958 
11959     def read_share_data(self, offset, length):
11960         precondition(offset >= 0)
11961-        # Reads beyond the end of the data are truncated. Reads that start
11962-        # beyond the end of the data return an empty string.
11963-        seekpos = self._data_offset+offset
11964-        fsize = os.path.getsize(self.fname)
11965-        actuallength = max(0, min(length, fsize-seekpos)) # XXX #1528
11966-        if actuallength == 0:
11967-            return ""
11968-        f = open(self.fname, 'rb')
11969-        f.seek(seekpos)
11970-        return f.read(actuallength)
11971+        return ""
11972 
11973     def write_share_data(self, offset, data):
11974         pass
11975hunk ./src/allmydata/storage/backends/null/null_backend.py 171
11976 
11977-    def _write_lease_record(self, f, lease_number, lease_info):
11978-        offset = self._lease_offset + lease_number * self.LEASE_SIZE
11979-        f.seek(offset)
11980-        assert f.tell() == offset
11981-        f.write(lease_info.to_immutable_data())
11982-
11983-    def _read_num_leases(self, f):
11984-        f.seek(0x08)
11985-        (num_leases,) = struct.unpack(">L", f.read(4))
11986-        return num_leases
11987-
11988-    def _write_num_leases(self, f, num_leases):
11989-        f.seek(0x08)
11990-        f.write(struct.pack(">L", num_leases))
11991-
11992-    def _truncate_leases(self, f, num_leases):
11993-        f.truncate(self._lease_offset + num_leases * self.LEASE_SIZE)
11994-
11995     def get_leases(self):
11996hunk ./src/allmydata/storage/backends/null/null_backend.py 172
11997-        """Yields a LeaseInfo instance for all leases."""
11998-        f = open(self.fname, 'rb')
11999-        (version, unused, num_leases) = struct.unpack(">LLL", f.read(0xc))
12000-        f.seek(self._lease_offset)
12001-        for i in range(num_leases):
12002-            data = f.read(self.LEASE_SIZE)
12003-            if data:
12004-                yield LeaseInfo().from_immutable_data(data)
12005+        pass
12006 
12007     def add_lease(self, lease):
12008         pass
12009hunk ./src/allmydata/storage/backends/null/null_backend.py 178
12010 
12011     def renew_lease(self, renew_secret, new_expire_time):
12012-        for i,lease in enumerate(self.get_leases()):
12013-            if constant_time_compare(lease.renew_secret, renew_secret):
12014-                # yup. See if we need to update the owner time.
12015-                if new_expire_time > lease.expiration_time:
12016-                    # yes
12017-                    lease.expiration_time = new_expire_time
12018-                    f = open(self.fname, 'rb+')
12019-                    self._write_lease_record(f, i, lease)
12020-                    f.close()
12021-                return
12022         raise IndexError("unable to renew non-existent lease")
12023 
12024     def add_or_renew_lease(self, lease_info):
12025hunk ./src/allmydata/storage/backends/null/null_backend.py 181
12026-        try:
12027-            self.renew_lease(lease_info.renew_secret,
12028-                             lease_info.expiration_time)
12029-        except IndexError:
12030-            self.add_lease(lease_info)
12031+        pass
12032 
12033 
12034hunk ./src/allmydata/storage/backends/null/null_backend.py 184
12035-class MutableNullShare:
12036+class ImmutableNullShare(NullShareBase):
12037+    implements(IStoredShare)
12038+    sharetype = "immutable"
12039+
12040+    def close(self):
12041+        self.shareset.close_shnum(self.shnum)
12042+
12043+
12044+class MutableNullShare(NullShareBase):
12045     implements(IStoredMutableShare)
12046     sharetype = "mutable"
12047hunk ./src/allmydata/storage/backends/null/null_backend.py 195
12048+
12049+    def check_write_enabler(self, write_enabler):
12050+        # Null backend doesn't check write enablers.
12051+        pass
12052+
12053+    def check_testv(self, testv):
12054+        return empty_check_testv(testv)
12055+
12056+    def writev(self, datav, new_length):
12057+        pass
12058+
12059+    def close(self):
12060+        pass
12061 
12062hunk ./src/allmydata/storage/backends/null/null_backend.py 209
12063-    """ XXX: TODO """
12064}
12065[Update the S3 backend. refs #999
12066david-sarah@jacaranda.org**20110923205345
12067 Ignore-this: 5ca623a17e09ddad4cab2f51b49aec0a
12068] {
12069hunk ./src/allmydata/storage/backends/s3/immutable.py 11
12070 from allmydata.storage.common import si_b2a, UnknownImmutableContainerVersionError, DataTooLargeError
12071 
12072 
12073-# Each share file (in storage/shares/$PREFIX/$STORAGEINDEX/$SHNUM) contains
12074+# Each share file (with key 'shares/$PREFIX/$STORAGEINDEX/$SHNUM') contains
12075 # lease information [currently inaccessible] and share data. The share data is
12076 # accessed by RIBucketWriter.write and RIBucketReader.read .
12077 
12078hunk ./src/allmydata/storage/backends/s3/immutable.py 65
12079             # in case a share file is copied from a disk backend, or in case we
12080             # need them in future.
12081             # TODO: filesize = size of S3 object
12082+            filesize = 0
12083             self._end_offset = filesize - (num_leases * self.LEASE_SIZE)
12084         self._data_offset = 0xc
12085 
12086hunk ./src/allmydata/storage/backends/s3/immutable.py 122
12087         return "\x00"*actuallength
12088 
12089     def write_share_data(self, offset, data):
12090-        assert offset >= self._size, "offset = %r, size = %r" % (offset, self._size)
12091+        length = len(data)
12092+        precondition(offset >= self._size, "offset = %r, size = %r" % (offset, self._size))
12093+        if self._max_size is not None and offset+length > self._max_size:
12094+            raise DataTooLargeError(self._max_size, offset, length)
12095 
12096         # TODO: write data to S3. If offset > self._size, fill the space
12097         # between with zeroes.
12098hunk ./src/allmydata/storage/backends/s3/mutable.py 17
12099 from allmydata.storage.backends.base import testv_compare
12100 
12101 
12102-# The MutableDiskShare is like the ImmutableDiskShare, but used for mutable data.
12103+# The MutableS3Share is like the ImmutableS3Share, but used for mutable data.
12104 # It has a different layout. See docs/mutable.rst for more details.
12105 
12106 # #   offset    size    name
12107hunk ./src/allmydata/storage/backends/s3/mutable.py 43
12108 assert struct.calcsize(">Q") == 8, struct.calcsize(">Q")
12109 
12110 
12111-class MutableDiskShare(object):
12112+class MutableS3Share(object):
12113     implements(IStoredMutableShare)
12114 
12115     sharetype = "mutable"
12116hunk ./src/allmydata/storage/backends/s3/mutable.py 111
12117             f.close()
12118 
12119     def __repr__(self):
12120-        return ("<MutableDiskShare %s:%r at %s>"
12121+        return ("<MutableS3Share %s:%r at %s>"
12122                 % (si_b2a(self._storageindex), self._shnum, quote_filepath(self._home)))
12123 
12124     def get_used_space(self):
12125hunk ./src/allmydata/storage/backends/s3/mutable.py 311
12126             except IndexError:
12127                 return
12128 
12129-    # These lease operations are intended for use by disk_backend.py.
12130-    # Other non-test clients should not depend on the fact that the disk
12131-    # backend stores leases in share files.
12132-
12133-    def add_lease(self, lease_info):
12134-        precondition(lease_info.owner_num != 0) # 0 means "no lease here"
12135-        f = self._home.open('rb+')
12136-        try:
12137-            num_lease_slots = self._get_num_lease_slots(f)
12138-            empty_slot = self._get_first_empty_lease_slot(f)
12139-            if empty_slot is not None:
12140-                self._write_lease_record(f, empty_slot, lease_info)
12141-            else:
12142-                self._write_lease_record(f, num_lease_slots, lease_info)
12143-        finally:
12144-            f.close()
12145-
12146-    def renew_lease(self, renew_secret, new_expire_time):
12147-        accepting_nodeids = set()
12148-        f = self._home.open('rb+')
12149-        try:
12150-            for (leasenum, lease) in self._enumerate_leases(f):
12151-                if constant_time_compare(lease.renew_secret, renew_secret):
12152-                    # yup. See if we need to update the owner time.
12153-                    if new_expire_time > lease.expiration_time:
12154-                        # yes
12155-                        lease.expiration_time = new_expire_time
12156-                        self._write_lease_record(f, leasenum, lease)
12157-                    return
12158-                accepting_nodeids.add(lease.nodeid)
12159-        finally:
12160-            f.close()
12161-        # Return the accepting_nodeids set, to give the client a chance to
12162-        # update the leases on a share that has been migrated from its
12163-        # original server to a new one.
12164-        msg = ("Unable to renew non-existent lease. I have leases accepted by"
12165-               " nodeids: ")
12166-        msg += ",".join([("'%s'" % idlib.nodeid_b2a(anid))
12167-                         for anid in accepting_nodeids])
12168-        msg += " ."
12169-        raise IndexError(msg)
12170-
12171-    def add_or_renew_lease(self, lease_info):
12172-        precondition(lease_info.owner_num != 0) # 0 means "no lease here"
12173-        try:
12174-            self.renew_lease(lease_info.renew_secret,
12175-                             lease_info.expiration_time)
12176-        except IndexError:
12177-            self.add_lease(lease_info)
12178-
12179-    def cancel_lease(self, cancel_secret):
12180-        """Remove any leases with the given cancel_secret. If the last lease
12181-        is cancelled, the file will be removed. Return the number of bytes
12182-        that were freed (by truncating the list of leases, and possibly by
12183-        deleting the file). Raise IndexError if there was no lease with the
12184-        given cancel_secret."""
12185-
12186-        # XXX can this be more like ImmutableDiskShare.cancel_lease?
12187-
12188-        accepting_nodeids = set()
12189-        modified = 0
12190-        remaining = 0
12191-        blank_lease = LeaseInfo(owner_num=0,
12192-                                renew_secret="\x00"*32,
12193-                                cancel_secret="\x00"*32,
12194-                                expiration_time=0,
12195-                                nodeid="\x00"*20)
12196-        f = self._home.open('rb+')
12197-        try:
12198-            for (leasenum, lease) in self._enumerate_leases(f):
12199-                accepting_nodeids.add(lease.nodeid)
12200-                if constant_time_compare(lease.cancel_secret, cancel_secret):
12201-                    self._write_lease_record(f, leasenum, blank_lease)
12202-                    modified += 1
12203-                else:
12204-                    remaining += 1
12205-            if modified:
12206-                freed_space = self._pack_leases(f)
12207-        finally:
12208-            f.close()
12209-
12210-        if modified > 0:
12211-            if remaining == 0:
12212-                freed_space = fileutil.get_used_space(self._home)
12213-                self.unlink()
12214-            return freed_space
12215-
12216-        msg = ("Unable to cancel non-existent lease. I have leases "
12217-               "accepted by nodeids: ")
12218-        msg += ",".join([("'%s'" % idlib.nodeid_b2a(anid))
12219-                         for anid in accepting_nodeids])
12220-        msg += " ."
12221-        raise IndexError(msg)
12222-
12223-    def _pack_leases(self, f):
12224-        # TODO: reclaim space from cancelled leases
12225-        return 0
12226-
12227     def _read_write_enabler_and_nodeid(self, f):
12228         f.seek(0)
12229         data = f.read(self.HEADER_SIZE)
12230hunk ./src/allmydata/storage/backends/s3/mutable.py 394
12231         pass
12232 
12233 
12234-def create_mutable_disk_share(storageindex, shnum, fp, serverid, write_enabler, parent):
12235-    ms = MutableDiskShare(storageindex, shnum, fp, parent)
12236+def create_mutable_s3_share(storageindex, shnum, fp, serverid, write_enabler, parent):
12237+    ms = MutableS3Share(storageindex, shnum, fp, parent)
12238     ms.create(serverid, write_enabler)
12239     del ms
12240hunk ./src/allmydata/storage/backends/s3/mutable.py 398
12241-    return MutableDiskShare(storageindex, shnum, fp, parent)
12242+    return MutableS3Share(storageindex, shnum, fp, parent)
12243hunk ./src/allmydata/storage/backends/s3/s3_backend.py 10
12244 from allmydata.storage.backends.s3.immutable import ImmutableS3Share
12245 from allmydata.storage.backends.s3.mutable import MutableS3Share
12246 
12247-# The S3 bucket has keys of the form shares/$STORAGEINDEX/$SHARENUM
12248-
12249+# The S3 bucket has keys of the form shares/$PREFIX/$STORAGEINDEX/$SHNUM .
12250 
12251 class S3Backend(Backend):
12252     implements(IStorageBackend)
12253}
12254[Minor cleanup to disk backend. refs #999
12255david-sarah@jacaranda.org**20110923205510
12256 Ignore-this: 79f92d7c2edb14cfedb167247c3f0d08
12257] {
12258hunk ./src/allmydata/storage/backends/disk/immutable.py 87
12259                 (version, unused, num_leases) = struct.unpack(">LLL", f.read(0xc))
12260             finally:
12261                 f.close()
12262-            filesize = self._home.getsize()
12263             if version != 1:
12264                 msg = "sharefile %s had version %d but we wanted 1" % \
12265                       (self._home, version)
12266hunk ./src/allmydata/storage/backends/disk/immutable.py 91
12267                 raise UnknownImmutableContainerVersionError(msg)
12268+
12269+            filesize = self._home.getsize()
12270             self._num_leases = num_leases
12271             self._lease_offset = filesize - (num_leases * self.LEASE_SIZE)
12272         self._data_offset = 0xc
12273}
12274
12275Context:
12276
12277[test_mutable.py: skip test_publish_surprise_mdmf, which is causing an error. refs #1534, #393
12278david-sarah@jacaranda.org**20110920183319
12279 Ignore-this: 6fb020e09e8de437cbcc2c9f57835b31
12280]
12281[test/test_mutable: write publish surprise test for MDMF, rename existing test_publish_surprise to clarify that it is for SDMF
12282kevan@isnotajoke.com**20110918003657
12283 Ignore-this: 722c507e8f5b537ff920e0555951059a
12284]
12285[test/test_mutable: refactor publish surprise test into common test fixture, rewrite test_publish_surprise to use test fixture
12286kevan@isnotajoke.com**20110918003533
12287 Ignore-this: 6f135888d400a99a09b5f9a4be443b6e
12288]
12289[mutable/publish: add errback immediately after write, don't consume errors from other parts of the publisher
12290kevan@isnotajoke.com**20110917234708
12291 Ignore-this: 12bf6b0918a5dc5ffc30ece669fad51d
12292]
12293[.darcs-boringfile: minor cleanups.
12294david-sarah@jacaranda.org**20110920154918
12295 Ignore-this: cab78e30d293da7e2832207dbee2ffeb
12296]
12297[uri.py: fix two interface violations in verifier URI classes. refs #1474
12298david-sarah@jacaranda.org**20110920030156
12299 Ignore-this: 454ddd1419556cb1d7576d914cb19598
12300]
12301[Make platform-detection code tolerate linux-3.0, patch by zooko.
12302Brian Warner <warner@lothar.com>**20110915202620
12303 Ignore-this: af63cf9177ae531984dea7a1cad03762
12304 
12305 Otherwise address-autodetection can't find ifconfig. refs #1536
12306]
12307[test_web.py: fix a bug in _count_leases that was causing us to check only the lease count of one share file, not of all share files as intended.
12308david-sarah@jacaranda.org**20110915185126
12309 Ignore-this: d96632bc48d770b9b577cda1bbd8ff94
12310]
12311[docs: insert a newline at the beginning of known_issues.rst to see if this makes it render more nicely in trac
12312zooko@zooko.com**20110914064728
12313 Ignore-this: aca15190fa22083c5d4114d3965f5d65
12314]
12315[docs: remove the coding: utf-8 declaration at the to of known_issues.rst, since the trac rendering doesn't hide it
12316zooko@zooko.com**20110914055713
12317 Ignore-this: 941ed32f83ead377171aa7a6bd198fcf
12318]
12319[docs: more cleanup of known_issues.rst -- now it passes "rst2html --verbose" without comment
12320zooko@zooko.com**20110914055419
12321 Ignore-this: 5505b3d76934bd97d0312cc59ed53879
12322]
12323[docs: more formatting improvements to known_issues.rst
12324zooko@zooko.com**20110914051639
12325 Ignore-this: 9ae9230ec9a38a312cbacaf370826691
12326]
12327[docs: reformatting of known_issues.rst
12328zooko@zooko.com**20110914050240
12329 Ignore-this: b8be0375079fb478be9d07500f9aaa87
12330]
12331[docs: fix formatting error in docs/known_issues.rst
12332zooko@zooko.com**20110914045909
12333 Ignore-this: f73fe74ad2b9e655aa0c6075acced15a
12334]
12335[merge Tahoe-LAFS v1.8.3 release announcement with trunk
12336zooko@zooko.com**20110913210544
12337 Ignore-this: 163f2c3ddacca387d7308e4b9332516e
12338]
12339[docs: release notes for Tahoe-LAFS v1.8.3
12340zooko@zooko.com**20110913165826
12341 Ignore-this: 84223604985b14733a956d2fbaeb4e9f
12342]
12343[tests: bump up the timeout in this test that fails on FreeStorm's CentOS in order to see if it is just very slow
12344zooko@zooko.com**20110913024255
12345 Ignore-this: 6a86d691e878cec583722faad06fb8e4
12346]
12347[interfaces: document that the 'fills-holes-with-zero-bytes' key should be used to detect whether a storage server has that behavior. refs #1528
12348david-sarah@jacaranda.org**20110913002843
12349 Ignore-this: 1a00a6029d40f6792af48c5578c1fd69
12350]
12351[CREDITS: more CREDITS for Kevan and David-Sarah
12352zooko@zooko.com**20110912223357
12353 Ignore-this: 4ea8f0d6f2918171d2f5359c25ad1ada
12354]
12355[merge NEWS about the mutable file bounds fixes with NEWS about work-in-progress
12356zooko@zooko.com**20110913205521
12357 Ignore-this: 4289a4225f848d6ae6860dd39bc92fa8
12358]
12359[doc: add NEWS item about fixes to potential palimpsest issues in mutable files
12360zooko@zooko.com**20110912223329
12361 Ignore-this: 9d63c95ddf95c7d5453c94a1ba4d406a
12362 ref. #1528
12363]
12364[merge the NEWS about the security fix (#1528) with the work-in-progress NEWS
12365zooko@zooko.com**20110913205153
12366 Ignore-this: 88e88a2ad140238c62010cf7c66953fc
12367]
12368[doc: add NEWS entry about the issue which allows unauthorized deletion of shares
12369zooko@zooko.com**20110912223246
12370 Ignore-this: 77e06d09103d2ef6bb51ea3e5d6e80b0
12371 ref. #1528
12372]
12373[doc: add entry in known_issues.rst about the issue which allows unauthorized deletion of shares
12374zooko@zooko.com**20110912223135
12375 Ignore-this: b26c6ea96b6c8740b93da1f602b5a4cd
12376 ref. #1528
12377]
12378[storage: more paranoid handling of bounds and palimpsests in mutable share files
12379zooko@zooko.com**20110912222655
12380 Ignore-this: a20782fa423779ee851ea086901e1507
12381 * storage server ignores requests to extend shares by sending a new_length
12382 * storage server fills exposed holes (created by sending a write vector whose offset begins after the end of the current data) with 0 to avoid "palimpsest" exposure of previous contents
12383 * storage server zeroes out lease info at the old location when moving it to a new location
12384 ref. #1528
12385]
12386[storage: test that the storage server ignores requests to extend shares by sending a new_length, and that the storage server fills exposed holes with 0 to avoid "palimpsest" exposure of previous contents
12387zooko@zooko.com**20110912222554
12388 Ignore-this: 61ebd7b11250963efdf5b1734a35271
12389 ref. #1528
12390]
12391[immutable: prevent clients from reading past the end of share data, which would allow them to learn the cancellation secret
12392zooko@zooko.com**20110912222458
12393 Ignore-this: da1ebd31433ea052087b75b2e3480c25
12394 Declare explicitly that we prevent this problem in the server's version dict.
12395 fixes #1528 (there are two patches that are each a sufficient fix to #1528 and this is one of them)
12396]
12397[storage: remove the storage server's "remote_cancel_lease" function
12398zooko@zooko.com**20110912222331
12399 Ignore-this: 1c32dee50e0981408576daffad648c50
12400 We're removing this function because it is currently unused, because it is dangerous, and because the bug described in #1528 leaks the cancellation secret, which allows anyone who knows a file's storage index to abuse this function to delete shares of that file.
12401 fixes #1528 (there are two patches that are each a sufficient fix to #1528 and this is one of them)
12402]
12403[storage: test that the storage server does *not* have a "remote_cancel_lease" function
12404zooko@zooko.com**20110912222324
12405 Ignore-this: 21c652009704652d35f34651f98dd403
12406 We're removing this function because it is currently unused, because it is dangerous, and because the bug described in #1528 leaks the cancellation secret, which allows anyone who knows a file's storage index to abuse this function to delete shares of that file.
12407 ref. #1528
12408]
12409[immutable: test whether the server allows clients to read past the end of share data, which would allow them to learn the cancellation secret
12410zooko@zooko.com**20110912221201
12411 Ignore-this: 376e47b346c713d37096531491176349
12412 Also test whether the server explicitly declares that it prevents this problem.
12413 ref #1528
12414]
12415[Retrieve._activate_enough_peers: rewrite Verify logic
12416Brian Warner <warner@lothar.com>**20110909181150
12417 Ignore-this: 9367c11e1eacbf025f75ce034030d717
12418]
12419[Retrieve: implement/test stopProducing
12420Brian Warner <warner@lothar.com>**20110909181150
12421 Ignore-this: 47b2c3df7dc69835e0a066ca12e3c178
12422]
12423[move DownloadStopped from download.common to interfaces
12424Brian Warner <warner@lothar.com>**20110909181150
12425 Ignore-this: 8572acd3bb16e50341dbed8eb1d90a50
12426]
12427[retrieve.py: remove vestigal self._validated_readers
12428Brian Warner <warner@lothar.com>**20110909181150
12429 Ignore-this: faab2ec14e314a53a2ffb714de626e2d
12430]
12431[Retrieve: rewrite flow-control: use a top-level loop() to catch all errors
12432Brian Warner <warner@lothar.com>**20110909181150
12433 Ignore-this: e162d2cd53b3d3144fc6bc757e2c7714
12434 
12435 This ought to close the potential for dropped errors and hanging downloads.
12436 Verify needs to be examined, I may have broken it, although all tests pass.
12437]
12438[Retrieve: merge _validate_active_prefixes into _add_active_peers
12439Brian Warner <warner@lothar.com>**20110909181150
12440 Ignore-this: d3ead31e17e69394ae7058eeb5beaf4c
12441]
12442[Retrieve: remove the initial prefix-is-still-good check
12443Brian Warner <warner@lothar.com>**20110909181150
12444 Ignore-this: da66ee51c894eaa4e862e2dffb458acc
12445 
12446 This check needs to be done with each fetch from the storage server, to
12447 detect when someone has changed the share (i.e. our servermap goes stale).
12448 Doing it just once at the beginning of retrieve isn't enough: a write might
12449 occur after the first segment but before the second, etc.
12450 
12451 _try_to_validate_prefix() was not removed: it will be used by the future
12452 check-with-each-fetch code.
12453 
12454 test_mutable.Roundtrip.test_corrupt_all_seqnum_late was disabled, since it
12455 fails until this check is brought back. (the corruption it applies only
12456 touches the prefix, not the block data, so the check-less retrieve actually
12457 tolerates it). Don't forget to re-enable it once the check is brought back.
12458]
12459[MDMFSlotReadProxy: remove the queue
12460Brian Warner <warner@lothar.com>**20110909181150
12461 Ignore-this: 96673cb8dda7a87a423de2f4897d66d2
12462 
12463 This is a neat trick to reduce Foolscap overhead, but the need for an
12464 explicit flush() complicates the Retrieve path and makes it prone to
12465 lost-progress bugs.
12466 
12467 Also change test_mutable.FakeStorageServer to tolerate multiple reads of the
12468 same share in a row, a limitation exposed by turning off the queue.
12469]
12470[rearrange Retrieve: first step, shouldn't change order of execution
12471Brian Warner <warner@lothar.com>**20110909181149
12472 Ignore-this: e3006368bfd2802b82ea45c52409e8d6
12473]
12474[CLI: test_cli.py -- remove an unnecessary call in test_mkdir_mutable_type. refs #1527
12475david-sarah@jacaranda.org**20110906183730
12476 Ignore-this: 122e2ffbee84861c32eda766a57759cf
12477]
12478[CLI: improve test for 'tahoe mkdir --mutable-type='. refs #1527
12479david-sarah@jacaranda.org**20110906183020
12480 Ignore-this: f1d4598e6c536f0a2b15050b3bc0ef9d
12481]
12482[CLI: make the --mutable-type option value for 'tahoe put' and 'tahoe mkdir' case-insensitive, and change --help for these commands accordingly. fixes #1527
12483david-sarah@jacaranda.org**20110905020922
12484 Ignore-this: 75a6df0a2df9c467d8c010579e9a024e
12485]
12486[cli: make --mutable-type imply --mutable in 'tahoe put'
12487Kevan Carstensen <kevan@isnotajoke.com>**20110903190920
12488 Ignore-this: 23336d3c43b2a9554e40c2a11c675e93
12489]
12490[SFTP: add a comment about a subtle interaction between OverwriteableFileConsumer and GeneralSFTPFile, and test the case it is commenting on.
12491david-sarah@jacaranda.org**20110903222304
12492 Ignore-this: 980c61d4dd0119337f1463a69aeebaf0
12493]
12494[improve the storage/mutable.py asserts even more
12495warner@lothar.com**20110901160543
12496 Ignore-this: 5b2b13c49bc4034f96e6e3aaaa9a9946
12497]
12498[storage/mutable.py: special characters in struct.foo arguments indicate standard as opposed to native sizes, we should be using these characters in these asserts
12499wilcoxjg@gmail.com**20110901084144
12500 Ignore-this: 28ace2b2678642e4d7269ddab8c67f30
12501]
12502[docs/write_coordination.rst: fix formatting and add more specific warning about access via sshfs.
12503david-sarah@jacaranda.org**20110831232148
12504 Ignore-this: cd9c851d3eb4e0a1e088f337c291586c
12505]
12506[test_mutable.Version: consolidate some tests, reduce runtime from 19s to 15s
12507warner@lothar.com**20110831050451
12508 Ignore-this: 64815284d9e536f8f3798b5f44cf580c
12509]
12510[mutable/retrieve: handle the case where self._read_length is 0.
12511Kevan Carstensen <kevan@isnotajoke.com>**20110830210141
12512 Ignore-this: fceafbe485851ca53f2774e5a4fd8d30
12513 
12514 Note that the downloader will still fetch a segment for a zero-length
12515 read, which is wasteful. Fixing that isn't specifically required to fix
12516 #1512, but it should probably be fixed before 1.9.
12517]
12518[NEWS: added summary of all changes since 1.8.2. Needs editing.
12519Brian Warner <warner@lothar.com>**20110830163205
12520 Ignore-this: 273899b37a899fc6919b74572454b8b2
12521]
12522[test_mutable.Update: only upload the files needed for each test. refs #1500
12523Brian Warner <warner@lothar.com>**20110829072717
12524 Ignore-this: 4d2ab4c7523af9054af7ecca9c3d9dc7
12525 
12526 This first step shaves 15% off the runtime: from 139s to 119s on my laptop.
12527 It also fixes a couple of places where a Deferred was being dropped, which
12528 would cause two tests to run in parallel and also confuse error reporting.
12529]
12530[Let Uploader retain History instead of passing it into upload(). Fixes #1079.
12531Brian Warner <warner@lothar.com>**20110829063246
12532 Ignore-this: 3902c58ec12bd4b2d876806248e19f17
12533 
12534 This consistently records all immutable uploads in the Recent Uploads And
12535 Downloads page, regardless of code path. Previously, certain webapi upload
12536 operations (like PUT /uri/$DIRCAP/newchildname) failed to pass the History
12537 object and were left out.
12538]
12539[Fix mutable publish/retrieve timing status displays. Fixes #1505.
12540Brian Warner <warner@lothar.com>**20110828232221
12541 Ignore-this: 4080ce065cf481b2180fd711c9772dd6
12542 
12543 publish:
12544 * encrypt and encode times are cumulative, not just current-segment
12545 
12546 retrieve:
12547 * same for decrypt and decode times
12548 * update "current status" to include segment number
12549 * set status to Finished/Failed when download is complete
12550 * set progress to 1.0 when complete
12551 
12552 More improvements to consider:
12553 * progress is currently 0% or 100%: should calculate how many segments are
12554   involved (remembering retrieve can be less than the whole file) and set it
12555   to a fraction
12556 * "fetch" time is fuzzy: what we want is to know how much of the delay is not
12557   our own fault, but since we do decode/decrypt work while waiting for more
12558   shares, it's not straightforward
12559]
12560[Teach 'tahoe debug catalog-shares about MDMF. Closes #1507.
12561Brian Warner <warner@lothar.com>**20110828080931
12562 Ignore-this: 56ef2951db1a648353d7daac6a04c7d1
12563]
12564[debug.py: remove some dead comments
12565Brian Warner <warner@lothar.com>**20110828074556
12566 Ignore-this: 40e74040dd4d14fd2f4e4baaae506b31
12567]
12568[hush pyflakes
12569Brian Warner <warner@lothar.com>**20110828074254
12570 Ignore-this: bef9d537a969fa82fe4decc4ba2acb09
12571]
12572[MutableFileNode.set_downloader_hints: never depend upon order of dict.values()
12573Brian Warner <warner@lothar.com>**20110828074103
12574 Ignore-this: caaf1aa518dbdde4d797b7f335230faa
12575 
12576 The old code was calculating the "extension parameters" (a list) from the
12577 downloader hints (a dictionary) with hints.values(), which is not stable, and
12578 would result in corrupted filecaps (with the 'k' and 'segsize' hints
12579 occasionally swapped). The new code always uses [k,segsize].
12580]
12581[layout.py: fix MDMF share layout documentation
12582Brian Warner <warner@lothar.com>**20110828073921
12583 Ignore-this: 3f13366fed75b5e31b51ae895450a225
12584]
12585[teach 'tahoe debug dump-share' about MDMF and offsets. refs #1507
12586Brian Warner <warner@lothar.com>**20110828073834
12587 Ignore-this: 3a9d2ef9c47a72bf1506ba41199a1dea
12588]
12589[test_mutable.Version.test_debug: use splitlines() to fix buildslaves
12590Brian Warner <warner@lothar.com>**20110828064728
12591 Ignore-this: c7f6245426fc80b9d1ae901d5218246a
12592 
12593 Any slave running in a directory with spaces in the name was miscounting
12594 shares, causing the test to fail.
12595]
12596[test_mutable.Version: exercise 'tahoe debug find-shares' on MDMF. refs #1507
12597Brian Warner <warner@lothar.com>**20110828005542
12598 Ignore-this: cb20bea1c28bfa50a72317d70e109672
12599 
12600 Also changes NoNetworkGrid to put shares in storage/shares/ .
12601]
12602[test_mutable.py: oops, missed a .todo
12603Brian Warner <warner@lothar.com>**20110828002118
12604 Ignore-this: fda09ae86481352b7a627c278d2a3940
12605]
12606[test_mutable: merge davidsarah's patch with my Version refactorings
12607warner@lothar.com**20110827235707
12608 Ignore-this: b5aaf481c90d99e33827273b5d118fd0
12609]
12610[Make the immutable/read-only constraint checking for MDMF URIs identical to that for SSK URIs. refs #393
12611david-sarah@jacaranda.org**20110823012720
12612 Ignore-this: e1f59d7ff2007c81dbef2aeb14abd721
12613]
12614[Additional tests for MDMF URIs and for zero-length files. refs #393
12615david-sarah@jacaranda.org**20110823011532
12616 Ignore-this: a7cc0c09d1d2d72413f9cd227c47a9d5
12617]
12618[Additional tests for zero-length partial reads and updates to mutable versions. refs #393
12619david-sarah@jacaranda.org**20110822014111
12620 Ignore-this: 5fc6f4d06e11910124e4a277ec8a43ea
12621]
12622[test_mutable.Version: factor out some expensive uploads, save 25% runtime
12623Brian Warner <warner@lothar.com>**20110827232737
12624 Ignore-this: ea37383eb85ea0894b254fe4dfb45544
12625]
12626[SDMF: update filenode with correct k/N after Retrieve. Fixes #1510.
12627Brian Warner <warner@lothar.com>**20110827225031
12628 Ignore-this: b50ae6e1045818c400079f118b4ef48
12629 
12630 Without this, we get a regression when modifying a mutable file that was
12631 created with more shares (larger N) than our current tahoe.cfg . The
12632 modification attempt creates new versions of the (0,1,..,newN-1) shares, but
12633 leaves the old versions of the (newN,..,oldN-1) shares alone (and throws a
12634 assertion error in SDMFSlotWriteProxy.finish_publishing in the process).
12635 
12636 The mixed versions that result (some shares with e.g. N=10, some with N=20,
12637 such that both versions are recoverable) cause problems for the Publish code,
12638 even before MDMF landed. Might be related to refs #1390 and refs #1042.
12639]
12640[layout.py: annotate assertion to figure out 'tahoe backup' failure
12641Brian Warner <warner@lothar.com>**20110827195253
12642 Ignore-this: 9b92b954e3ed0d0f80154fff1ff674e5
12643]
12644[Add 'tahoe debug dump-cap' support for MDMF, DIR2-CHK, DIR2-MDMF. refs #1507.
12645Brian Warner <warner@lothar.com>**20110827195048
12646 Ignore-this: 61c6af5e33fc88e0251e697a50addb2c
12647 
12648 This also adds tests for all those cases, and fixes an omission in uri.py
12649 that broke parsing of DIR2-MDMF-Verifier and DIR2-CHK-Verifier.
12650]
12651[MDMF: more writable/writeable consistentifications
12652warner@lothar.com**20110827190602
12653 Ignore-this: 22492a9e20c1819ddb12091062888b55
12654]
12655[MDMF: s/Writable/Writeable/g, for consistency with existing SDMF code
12656warner@lothar.com**20110827183357
12657 Ignore-this: 9dd312acedbdb2fc2f7bef0d0fb17c0b
12658]
12659[setup.cfg: remove no-longer-supported test_mac_diskimage alias. refs #1479
12660david-sarah@jacaranda.org**20110826230345
12661 Ignore-this: 40e908b8937322a290fb8012bfcad02a
12662]
12663[test_mutable.Update: increase timeout from 120s to 400s, slaves are failing
12664Brian Warner <warner@lothar.com>**20110825230140
12665 Ignore-this: 101b1924a30cdbda9b2e419e95ca15ec
12666]
12667[tests: fix check_memory test
12668zooko@zooko.com**20110825201116
12669 Ignore-this: 4d66299fa8cb61d2ca04b3f45344d835
12670 fixes #1503
12671]
12672[TAG allmydata-tahoe-1.9.0a1
12673warner@lothar.com**20110825161122
12674 Ignore-this: 3cbf49f00dbda58189f893c427f65605
12675]
12676Patch bundle hash:
12677fdb52eef34481d24bba91152e012247d14a2b902