Ticket #1195: storage-refactor.dpatch

File storage-refactor.dpatch, 19.7 KB (added by zooko, at 2010-09-10T18:58:09Z)
Line 
1Fri Sep 10 11:35:20 MDT 2010  zooko@zooko.com
2  * fileutil: copy in the get_disk_stats() and get_available_space() functions from storage/server.py
3
4Fri Sep 10 11:36:29 MDT 2010  zooko@zooko.com
5  * storage: use fileutil's version of get_disk_stats() and get_available_space(), use mockery/fakery in tests, enable large share test on platforms with sparse files and if > 4 GiB of disk space is currently available
6
7New patches:
8
9[fileutil: copy in the get_disk_stats() and get_available_space() functions from storage/server.py
10zooko@zooko.com**20100910173520
11 Ignore-this: 8b15569715f710f4fc5092f7ca109253
12] {
13hunk ./src/allmydata/test/test_util.py 506
14             finally:
15                 os.chdir(saved_cwd)
16 
17+    def test_disk_stats(self):
18+        avail = fileutil.get_available_space('.', 2**14)
19+        if avail == 0:
20+            raise unittest.SkipTest("This test will spuriously fail there is no disk space left.")
21+
22+        disk = fileutil.get_disk_stats('.', 2**13)
23+        self.failUnless(disk['total'] > 0, disk['total'])
24+        self.failUnless(disk['used'] > 0, disk['used'])
25+        self.failUnless(disk['free_for_root'] > 0, disk['free_for_root'])
26+        self.failUnless(disk['free_for_nonroot'] > 0, disk['free_for_nonroot'])
27+        self.failUnless(disk['avail'] > 0, disk['avail'])
28+
29+    def test_disk_stats_avail_nonnegative(self):
30+        # This test will spuriously fail if you have more than 2^128
31+        # bytes of available space on your filesystem.
32+        disk = fileutil.get_disk_stats('.', 2**128)
33+        self.failUnlessEqual(disk['avail'], 0)
34+
35 class PollMixinTests(unittest.TestCase):
36     def setUp(self):
37         self.pm = pollmixin.PollMixin()
38hunk ./src/allmydata/util/fileutil.py 308
39     # there is always at least one Unicode path component.
40     return os.path.normpath(path)
41 
42+windows = False
43+try:
44+    import win32api, win32con
45+except ImportError:
46+    pass
47+else:
48+    windows = True
49+    # <http://msdn.microsoft.com/en-us/library/ms680621%28VS.85%29.aspx>
50+    win32api.SetErrorMode(win32con.SEM_FAILCRITICALERRORS |
51+                          win32con.SEM_NOOPENFILEERRORBOX)
52+
53+def get_disk_stats(whichdir, reserved_space=0):
54+    """Return disk statistics for the storage disk, in the form of a dict
55+    with the following fields.
56+      total:            total bytes on disk
57+      free_for_root:    bytes actually free on disk
58+      free_for_nonroot: bytes free for "a non-privileged user" [Unix] or
59+                          the current user [Windows]; might take into
60+                          account quotas depending on platform
61+      used:             bytes used on disk
62+      avail:            bytes available excluding reserved space
63+    An AttributeError can occur if the OS has no API to get disk information.
64+    An EnvironmentError can occur if the OS call fails.
65+
66+    whichdir is a directory on the filesystem in question -- the
67+    answer is about the filesystem, not about the directory, so the
68+    directory is used only to specify which filesystem.
69+
70+    reserved_space is how many bytes to subtract from the answer, so
71+    you can pass how many bytes you would like to leave unused on this
72+    filesystem as reserved_space.
73+    """
74+
75+    if windows:
76+        # For Windows systems, where os.statvfs is not available, use GetDiskFreeSpaceEx.
77+        # <http://docs.activestate.com/activepython/2.5/pywin32/win32api__GetDiskFreeSpaceEx_meth.html>
78+        #
79+        # Although the docs say that the argument should be the root directory
80+        # of a disk, GetDiskFreeSpaceEx actually accepts any path on that disk
81+        # (like its Win32 equivalent).
82+
83+        (free_for_nonroot, total, free_for_root) = win32api.GetDiskFreeSpaceEx(whichdir)
84+    else:
85+        # For Unix-like systems.
86+        # <http://docs.python.org/library/os.html#os.statvfs>
87+        # <http://opengroup.org/onlinepubs/7990989799/xsh/fstatvfs.html>
88+        # <http://opengroup.org/onlinepubs/7990989799/xsh/sysstatvfs.h.html>
89+        s = os.statvfs(whichdir)
90+
91+        # on my mac laptop:
92+        #  statvfs(2) is a wrapper around statfs(2).
93+        #    statvfs.f_frsize = statfs.f_bsize :
94+        #     "minimum unit of allocation" (statvfs)
95+        #     "fundamental file system block size" (statfs)
96+        #    statvfs.f_bsize = statfs.f_iosize = stat.st_blocks : preferred IO size
97+        # on an encrypted home directory ("FileVault"), it gets f_blocks
98+        # wrong, and s.f_blocks*s.f_frsize is twice the size of my disk,
99+        # but s.f_bavail*s.f_frsize is correct
100+
101+        total = s.f_frsize * s.f_blocks
102+        free_for_root = s.f_frsize * s.f_bfree
103+        free_for_nonroot = s.f_frsize * s.f_bavail
104+
105+    # valid for all platforms:
106+    used = total - free_for_root
107+    avail = max(free_for_nonroot - reserved_space, 0)
108+
109+    return { 'total': total, 'free_for_root': free_for_root,
110+             'free_for_nonroot': free_for_nonroot,
111+             'used': used, 'avail': avail, }
112+
113+def get_available_space(whichdir, reserved_space):
114+    """Returns available space for share storage in bytes, or None if no
115+    API to get this information is available.
116+
117+    whichdir is a directory on the filesystem in question -- the
118+    answer is about the filesystem, not about the directory, so the
119+    directory is used only to specify which filesystem.
120+
121+    reserved_space is how many bytes to subtract from the answer, so
122+    you can pass how many bytes you would like to leave unused on this
123+    filesystem as reserved_space.
124+    """
125+    try:
126+        return get_disk_stats(whichdir, reserved_space)['avail']
127+    except AttributeError:
128+        return None
129+    except EnvironmentError:
130+        log.msg("OS call to get disk statistics failed")
131+        return 0
132+
133}
134[storage: use fileutil's version of get_disk_stats() and get_available_space(), use mockery/fakery in tests, enable large share test on platforms with sparse files and if > 4 GiB of disk space is currently available
135zooko@zooko.com**20100910173629
136 Ignore-this: 1304f1164c661de6d5304f993eb9b27b
137] 
138<
139[fileutil: copy in the get_disk_stats() and get_available_space() functions from storage/server.py
140zooko@zooko.com**20100910173520
141 Ignore-this: 8b15569715f710f4fc5092f7ca109253
142] 
143> {
144hunk ./src/allmydata/storage/server.py 39
145     implements(RIStorageServer, IStatsProducer)
146     name = 'storage'
147     LeaseCheckerClass = LeaseCheckingCrawler
148-    windows = False
149-
150-    try:
151-        import win32api, win32con
152-        windows = True
153-        # <http://msdn.microsoft.com/en-us/library/ms680621%28VS.85%29.aspx>
154-        win32api.SetErrorMode(win32con.SEM_FAILCRITICALERRORS |
155-                              win32con.SEM_NOOPENFILEERRORBOX)
156-    except ImportError:
157-        pass
158 
159     def __init__(self, storedir, nodeid, reserved_space=0,
160                  discard_storage=False, readonly_storage=False,
161hunk ./src/allmydata/storage/server.py 153
162     def _clean_incomplete(self):
163         fileutil.rm_dir(self.incomingdir)
164 
165-    def get_disk_stats(self):
166-        """Return disk statistics for the storage disk, in the form of a dict
167-        with the following fields.
168-          total:            total bytes on disk
169-          free_for_root:    bytes actually free on disk
170-          free_for_nonroot: bytes free for "a non-privileged user" [Unix] or
171-                              the current user [Windows]; might take into
172-                              account quotas depending on platform
173-          used:             bytes used on disk
174-          avail:            bytes available excluding reserved space
175-        An AttributeError can occur if the OS has no API to get disk information.
176-        An EnvironmentError can occur if the OS call fails."""
177-
178-        if self.windows:
179-            # For Windows systems, where os.statvfs is not available, use GetDiskFreeSpaceEx.
180-            # <http://docs.activestate.com/activepython/2.5/pywin32/win32api__GetDiskFreeSpaceEx_meth.html>
181-            #
182-            # Although the docs say that the argument should be the root directory
183-            # of a disk, GetDiskFreeSpaceEx actually accepts any path on that disk
184-            # (like its Win32 equivalent).
185-
186-            (free_for_nonroot, total, free_for_root) = self.win32api.GetDiskFreeSpaceEx(self.storedir)
187-        else:
188-            # For Unix-like systems.
189-            # <http://docs.python.org/library/os.html#os.statvfs>
190-            # <http://opengroup.org/onlinepubs/7990989799/xsh/fstatvfs.html>
191-            # <http://opengroup.org/onlinepubs/7990989799/xsh/sysstatvfs.h.html>
192-            s = os.statvfs(self.storedir)
193-
194-            # on my mac laptop:
195-            #  statvfs(2) is a wrapper around statfs(2).
196-            #    statvfs.f_frsize = statfs.f_bsize :
197-            #     "minimum unit of allocation" (statvfs)
198-            #     "fundamental file system block size" (statfs)
199-            #    statvfs.f_bsize = statfs.f_iosize = stat.st_blocks : preferred IO size
200-            # on an encrypted home directory ("FileVault"), it gets f_blocks
201-            # wrong, and s.f_blocks*s.f_frsize is twice the size of my disk,
202-            # but s.f_bavail*s.f_frsize is correct
203-
204-            total = s.f_frsize * s.f_blocks
205-            free_for_root = s.f_frsize * s.f_bfree
206-            free_for_nonroot = s.f_frsize * s.f_bavail
207-
208-        # valid for all platforms:
209-        used = total - free_for_root
210-        avail = max(free_for_nonroot - self.reserved_space, 0)
211-
212-        return { 'total': total, 'free_for_root': free_for_root,
213-                 'free_for_nonroot': free_for_nonroot,
214-                 'used': used, 'avail': avail, }
215-
216     def get_stats(self):
217         # remember: RIStatsProvider requires that our return dict
218         # contains numeric values.
219hunk ./src/allmydata/storage/server.py 163
220                 stats['storage_server.latencies.%s.%s' % (category, name)] = v
221 
222         try:
223-            disk = self.get_disk_stats()
224+            disk = fileutil.get_disk_stats(self.storedir, self.reserved_space)
225             writeable = disk['avail'] > 0
226 
227             # spacetime predictors should use disk_avail / (d(disk_used)/dt)
228hunk ./src/allmydata/storage/server.py 195
229 
230         if self.readonly_storage:
231             return 0
232-        try:
233-            return self.get_disk_stats()['avail']
234-        except AttributeError:
235-            return None
236-        except EnvironmentError:
237-            log.msg("OS call to get disk statistics failed", level=log.UNUSUAL)
238-            return 0
239+        return fileutil.get_available_space(self.storedir, self.reserved_space)
240 
241     def allocated_size(self):
242         space = 0
243hunk ./src/allmydata/test/test_storage.py 1
244+import time, os.path, platform, stat, re, simplejson, struct
245 
246hunk ./src/allmydata/test/test_storage.py 3
247-import time, os.path, stat, re, simplejson, struct
248+from allmydata.util import log
249+
250+import mock
251 
252 from twisted.trial import unittest
253 
254hunk ./src/allmydata/test/test_storage.py 233
255         return self._do_test_readwrite("test_readwrite_v2",
256                                        0x44, WriteBucketProxy_v2, ReadBucketProxy)
257 
258-class FakeDiskStorageServer(StorageServer):
259-    DISKAVAIL = 0
260-    def get_disk_stats(self):
261-        return { 'free_for_nonroot': self.DISKAVAIL, 'avail': max(self.DISKAVAIL - self.reserved_space, 0), }
262-
263 class Server(unittest.TestCase):
264 
265     def setUp(self):
266hunk ./src/allmydata/test/test_storage.py 266
267                                           sharenums, size, canary)
268 
269     def test_large_share(self):
270+        syslow = platform.system().lower()
271+        if 'cygwin' in syslow or 'windows' in syslow or 'darwin' in syslow:
272+            raise unittest.SkipTest("If your filesystem doesn't support efficient sparse files then it is very expensive (Mac OS X and Windows don't support efficient sparse files).")
273+
274+        avail = fileutil.get_available_space('.', 2**14)
275+        if avail <= 4*2**30:
276+            raise unittest.SkipTest("This test will spuriously fail if you have less than 4 GiB free on your filesystem.")
277+
278         ss = self.create("test_large_share")
279 
280         already,writers = self.allocate(ss, "allocate", [0], 2**32+2)
281hunk ./src/allmydata/test/test_storage.py 288
282         readers = ss.remote_get_buckets("allocate")
283         reader = readers[shnum]
284         self.failUnlessEqual(reader.remote_read(2**32, 2), "ab")
285-    test_large_share.skip = "This test can spuriously fail if you have less than 4 GiB free on your filesystem, and if your filesystem doesn't support efficient sparse files then it is very expensive (Mac OS X and Windows don't support efficient sparse files)."
286 
287     def test_dont_overfill_dirs(self):
288         """
289hunk ./src/allmydata/test/test_storage.py 431
290         self.failUnlessEqual(already, set())
291         self.failUnlessEqual(set(writers.keys()), set([0,1,2]))
292 
293-    def test_reserved_space(self):
294-        ss = self.create("test_reserved_space", reserved_space=10000,
295-                         klass=FakeDiskStorageServer)
296-        # the FakeDiskStorageServer doesn't do real calls to get_disk_stats
297-        ss.DISKAVAIL = 15000
298+    @mock.patch('allmydata.util.fileutil.get_disk_stats')
299+    def test_reserved_space(self, mock_get_disk_stats):
300+        reserved_space=10000
301+        mock_get_disk_stats.return_value = {
302+            'free_for_nonroot': 15000,
303+            'avail': max(15000 - reserved_space, 0),
304+            }
305+
306+        ss = self.create("test_reserved_space", reserved_space=reserved_space)
307         # 15k available, 10k reserved, leaves 5k for shares
308 
309         # a newly created and filled share incurs this much overhead, beyond
310hunk ./src/allmydata/test/test_storage.py 478
311 
312         allocated = 1001 + OVERHEAD + LEASE_SIZE
313 
314-        # we have to manually increase DISKAVAIL, since we're not doing real
315+        # we have to manually increase available, since we're not doing real
316         # disk measurements
317hunk ./src/allmydata/test/test_storage.py 480
318-        ss.DISKAVAIL -= allocated
319+        mock_get_disk_stats.return_value = {
320+            'free_for_nonroot': 15000 - allocated,
321+            'avail': max(15000 - allocated - reserved_space, 0),
322+            }
323 
324         # now there should be ALLOCATED=1001+12+72=1085 bytes allocated, and
325         # 5000-1085=3915 free, therefore we can fit 39 100byte shares
326hunk ./src/allmydata/test/test_storage.py 497
327         ss.disownServiceParent()
328         del ss
329 
330-    def test_disk_stats(self):
331-        # This will spuriously fail if there is zero disk space left (but so will other tests).
332-        ss = self.create("test_disk_stats", reserved_space=0)
333-
334-        disk = ss.get_disk_stats()
335-        self.failUnless(disk['total'] > 0, disk['total'])
336-        self.failUnless(disk['used'] > 0, disk['used'])
337-        self.failUnless(disk['free_for_root'] > 0, disk['free_for_root'])
338-        self.failUnless(disk['free_for_nonroot'] > 0, disk['free_for_nonroot'])
339-        self.failUnless(disk['avail'] > 0, disk['avail'])
340-
341-    def test_disk_stats_avail_nonnegative(self):
342-        ss = self.create("test_disk_stats_avail_nonnegative", reserved_space=2**64)
343-
344-        disk = ss.get_disk_stats()
345-        self.failUnlessEqual(disk['avail'], 0)
346-
347     def test_seek(self):
348         basedir = self.workdir("test_seek_behavior")
349         fileutil.make_dirs(basedir)
350hunk ./src/allmydata/test/test_storage.py 2459
351         d = self.render1(page, args={"t": ["json"]})
352         return d
353 
354-class NoDiskStatsServer(StorageServer):
355-    def get_disk_stats(self):
356-        raise AttributeError
357-
358-class BadDiskStatsServer(StorageServer):
359-    def get_disk_stats(self):
360-        raise OSError
361-
362 class WebStatus(unittest.TestCase, pollmixin.PollMixin, WebRenderingMixin):
363 
364     def setUp(self):
365hunk ./src/allmydata/test/test_storage.py 2500
366         d = self.render1(page, args={"t": ["json"]})
367         return d
368 
369-    def test_status_no_disk_stats(self):
370+    @mock.patch('allmydata.util.fileutil.get_disk_stats')
371+    def test_status_no_disk_stats(self, mock_get_disk_stats):
372+        mock_get_disk_stats.side_effect = AttributeError()
373+
374         # Some platforms may have no disk stats API. Make sure the code can handle that
375         # (test runs on all platforms).
376         basedir = "storage/WebStatus/status_no_disk_stats"
377hunk ./src/allmydata/test/test_storage.py 2508
378         fileutil.make_dirs(basedir)
379-        ss = NoDiskStatsServer(basedir, "\x00" * 20)
380+        ss = StorageServer(basedir, "\x00" * 20)
381         ss.setServiceParent(self.s)
382         w = StorageStatus(ss)
383         html = w.renderSynchronously()
384hunk ./src/allmydata/test/test_storage.py 2519
385         self.failUnlessIn("Space Available to Tahoe: ?", s)
386         self.failUnless(ss.get_available_space() is None)
387 
388-    def test_status_bad_disk_stats(self):
389+    @mock.patch('allmydata.util.fileutil.get_disk_stats')
390+    def test_status_bad_disk_stats(self, mock_get_disk_stats):
391+        mock_get_disk_stats.side_effect = OSError()
392+
393         # If the API to get disk stats exists but a call to it fails, then the status should
394         # show that no shares will be accepted, and get_available_space() should be 0.
395         basedir = "storage/WebStatus/status_bad_disk_stats"
396hunk ./src/allmydata/test/test_storage.py 2527
397         fileutil.make_dirs(basedir)
398-        ss = BadDiskStatsServer(basedir, "\x00" * 20)
399+        ss = StorageServer(basedir, "\x00" * 20)
400         ss.setServiceParent(self.s)
401         w = StorageStatus(ss)
402         html = w.renderSynchronously()
403hunk ./src/allmydata/test/test_storage.py 2579
404         self.failUnlessEqual(w.render_abbrev_space(None, 10e6), "10.00 MB")
405         self.failUnlessEqual(remove_prefix("foo.bar", "foo."), "bar")
406         self.failUnlessEqual(remove_prefix("foo.bar", "baz."), None)
407-
408hunk ./src/allmydata/test/test_system.py 1784
409         d.addCallback(_got_lit_filenode)
410         return d
411 
412-
413}
414
415Context:
416
417[test: make tests stop relying on pyutil version class accepting the string 'unknown' for its version, and make them forward-compatible with the future Python Rational Version Numbering standard
418zooko@zooko.com**20100910154135
419 Ignore-this: d051b071f33595493be5df218f5015a6
420] 
421[setup: copy in this fix from zetuptoolz and the accompanying new version number of zetuptoolz: http://tahoe-lafs.org/trac/zetuptoolz/ticket/1
422zooko@zooko.com**20100910061411
423 Ignore-this: cb0ddce66b2a71666df3e22375fa581a
424] 
425[immutable download: have the finder inform its share consumer "no more shares" in a subsequent tick, thus avoiding accidentally telling it "no more shares" now and then telling it "here's another share" in a subsequent tick
426Brian Warner <warner@lothar.com>**20100910043038
427 Ignore-this: 47595fb2b87867d3d75695d51344c484
428 fixes #1191
429 Patch by Brian. This patch description was actually written by Zooko, but I forged Brian's name on the "author" field so that he would get credit for this patch in revision control history.
430] 
431[immutable downloader: add a test specifically of whether the finder sometimes announces "no more shares ever" and then announces a newly found share
432zooko@zooko.com**20100909041654
433 Ignore-this: ec0d5febc499f974b167465290770abd
434 (The current code fails this test, ref #1191.)
435] 
436[docs/frontends/FTP-and-SFTP.txt : ftpd and sftpd doesn't listen on loopback interface only
437marc.doudiet@nimag.net**20100813140853
438 Ignore-this: 5b5dfd0e5991a2669fe41ba13ea21bd4
439] 
440[tests: assign the storage servers to a fixed order which triggers a bug in new downloader every time this test is run (formerly this test would detect the bug in new-downloader only sporadically)
441zooko@zooko.com**20100904041515
442 Ignore-this: 33155dcc03e84217ec5541addd3a16fc
443 If you are investigating the bug in new-downloader, one way to investigate might be to change this ordering to a different fixed order (e.g. rotate by 4 instead of rotate by 5) and observe how the behavior of new-downloader differs in that case.
444] 
445[TAG allmydata-tahoe-1.8.0c3
446zooko@zooko.com**20100902212140
447 Ignore-this: e4550de37f57e5c1a591e549a104565d
448] 
449Patch bundle hash:
450c8607a41f8ddd2d945ae8ed6d309413d178c6c84