Ticket #999: sservertests.darcs.patch

Fri Mar 25 14:35:14 MDT 2011  wilcoxjg@gmail.com
  * storage: new mocking tests of storage server read and write
  There are already tests of read and functionality in test_storage.py, but those tests let the code under test use a real filesystem whereas these tests mock all file system calls.

New patches:

[storage: new mocking tests of storage server read and write
wilcoxjg@gmail.com**20110325203514
 Ignore-this: df65c3c4f061dd1516f88662023fdb41
 There are already tests of read and functionality in test_storage.py, but those tests let the code under test use a real filesystem whereas these tests mock all file system calls.
] {
addfile ./src/allmydata/test/test_server.py
hunk ./src/allmydata/test/test_server.py 1
+from twisted.trial import unittest
+
+from StringIO import StringIO
+
+from allmydata.test.common_util import ReallyEqualMixin
+
+import mock
+
+# This is the code that we're going to be testing.
+from allmydata.storage.server import StorageServer
+
+# The following share file contents was generated with
+# storage.immutable.ShareFile from Tahoe-LAFS v1.8.2
+# with share data == 'a'.
+share_data = 'a\x00\x00\x00\x00xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy\x00(\xde\x80'
+share_file_data = '\x00\x00\x00\x01\x00\x00\x00\x01\x00\x00\x00\x01' + share_data
+
+sharefname = 'testdir/shares/or/orsxg5dtorxxeylhmvpws3temv4a/0'
+
+class TestServerConstruction(unittest.TestCase, ReallyEqualMixin):
+    @mock.patch('__builtin__.open')
+    def test_create_server(self, mockopen):
+        """ This tests whether a server instance can be constructed. """
+
+        def call_open(fname, mode):
+            if fname == 'testdir/bucket_counter.state':
+                raise IOError(2, "No such file or directory: 'testdir/bucket_counter.state'")
+            elif fname == 'testdir/lease_checker.state':
+                raise IOError(2, "No such file or directory: 'testdir/lease_checker.state'")
+            elif fname == 'testdir/lease_checker.history':
+                return StringIO()
+        mockopen.side_effect = call_open
+
+        # Now begin the test.
+        s = StorageServer('testdir', 'testnodeidxxxxxxxxxx')
+
+        # You passed!
+
+class TestServer(unittest.TestCase, ReallyEqualMixin):
+    @mock.patch('__builtin__.open')
+    def setUp(self, mockopen):
+        def call_open(fname, mode):
+            if fname == 'testdir/bucket_counter.state':
+                raise IOError(2, "No such file or directory: 'testdir/bucket_counter.state'")
+            elif fname == 'testdir/lease_checker.state':
+                raise IOError(2, "No such file or directory: 'testdir/lease_checker.state'")
+            elif fname == 'testdir/lease_checker.history':
+                return StringIO()
+        mockopen.side_effect = call_open
+
+        self.s = StorageServer('testdir', 'testnodeidxxxxxxxxxx')
+
+
+    @mock.patch('time.time')
+    @mock.patch('os.mkdir')
+    @mock.patch('__builtin__.open')
+    @mock.patch('os.listdir')
+    @mock.patch('os.path.isdir')
+    def test_write_share(self, mockisdir, mocklistdir, mockopen, mockmkdir, mocktime):
+        """Handle a report of corruption."""
+
+        def call_listdir(dirname):
+            self.failUnlessReallyEqual(dirname, 'testdir/shares/or/orsxg5dtorxxeylhmvpws3temv4a')
+            raise OSError(2, "No such file or directory: 'testdir/shares/or/orsxg5dtorxxeylhmvpws3temv4a'")
+
+        mocklistdir.side_effect = call_listdir
+
+        class MockFile:
+            def __init__(self):
+                self.buffer = ''
+                self.pos = 0
+            def write(self, instring):
+                begin = self.pos
+                padlen = begin - len(self.buffer)
+                if padlen > 0:
+                    self.buffer += '\x00' * padlen
+                end = self.pos + len(instring)
+                self.buffer = self.buffer[:begin]+instring+self.buffer[end:]
+                self.pos = end
+            def close(self):
+                pass
+            def seek(self, pos):
+                self.pos = pos
+            def read(self, numberbytes):
+                return self.buffer[self.pos:self.pos+numberbytes]
+            def tell(self):
+                return self.pos
+
+        mocktime.return_value = 0
+
+        sharefile = MockFile()
+        def call_open(fname, mode):
+            self.failUnlessReallyEqual(fname, 'testdir/shares/incoming/or/orsxg5dtorxxeylhmvpws3temv4a/0' )
+            return sharefile
+
+        mockopen.side_effect = call_open
+        # Now begin the test.
+        alreadygot, bs = self.s.remote_allocate_buckets('teststorage_index', 'x'*32, 'y'*32, set((0,)), 1, mock.Mock())
+        print bs
+        bs[0].remote_write(0, 'a')
+        self.failUnlessReallyEqual(sharefile.buffer, share_file_data)
+
+
+    @mock.patch('os.path.exists')
+    @mock.patch('os.path.getsize')
+    @mock.patch('__builtin__.open')
+    @mock.patch('os.listdir')
+    def test_read_share(self, mocklistdir, mockopen, mockgetsize, mockexists):
+        """ This tests whether the code correctly finds and reads
+        shares written out by old (Tahoe-LAFS <= v1.8.2)
+        servers. There is a similar test in test_download, but that one
+        is from the perspective of the client and exercises a deeper
+        stack of code. This one is for exercising just the
+        StorageServer object. """
+
+        def call_listdir(dirname):
+            self.failUnlessReallyEqual(dirname,'testdir/shares/or/orsxg5dtorxxeylhmvpws3temv4a')
+            return ['0']
+
+        mocklistdir.side_effect = call_listdir
+
+        def call_open(fname, mode):
+            self.failUnlessReallyEqual(fname, sharefname)
+            self.failUnless('r' in mode, mode)
+            self.failUnless('b' in mode, mode)
+
+            return StringIO(share_file_data)
+        mockopen.side_effect = call_open
+
+        datalen = len(share_file_data)
+        def call_getsize(fname):
+            self.failUnlessReallyEqual(fname, sharefname)
+            return datalen
+        mockgetsize.side_effect = call_getsize
+
+        def call_exists(fname):
+            self.failUnlessReallyEqual(fname, sharefname)
+            return True
+        mockexists.side_effect = call_exists
+
+        # Now begin the test.
+        bs = self.s.remote_get_buckets('teststorage_index')
+
+        self.failUnlessEqual(len(bs), 1)
+        b = bs[0]
+        self.failUnlessReallyEqual(b.remote_read(0, datalen), share_data)
+        # If you try to read past the end you get the as much data as is there.
+        self.failUnlessReallyEqual(b.remote_read(0, datalen+20), share_data)
+        # If you start reading past the end of the file you get the empty string.
+        self.failUnlessReallyEqual(b.remote_read(datalen+1, 3), '')
}

Context:

[test: increase timeout on a network test because Francois's ARM machine hit that timeout
zooko@zooko.com**20110317165909
 Ignore-this: 380c345cdcbd196268ca5b65664ac85b
 I'm skeptical that the test was proceeding correctly but ran out of time. It seems more likely that it had gotten hung. But if we raise the timeout to an even more extravagant number then we can be even more certain that the test was never going to finish.
] 
[docs/configuration.rst: add a "Frontend Configuration" section
Brian Warner <warner@lothar.com>**20110222014323
 Ignore-this: 657018aa501fe4f0efef9851628444ca
 
 this points to docs/frontends/*.rst, which were previously underlinked
] 
[web/filenode.py: avoid calling req.finish() on closed HTTP connections. Closes #1366
"Brian Warner <warner@lothar.com>"**20110221061544
 Ignore-this: 799d4de19933f2309b3c0c19a63bb888
] 
[Add unit tests for cross_check_pkg_resources_versus_import, and a regression test for ref #1355. This requires a little refactoring to make it testable.
david-sarah@jacaranda.org**20110221015817
 Ignore-this: 51d181698f8c20d3aca58b057e9c475a
] 
[allmydata/__init__.py: .name was used in place of the correct .__name__ when printing an exception. Also, robustify string formatting by using %r instead of %s in some places. fixes #1355.
david-sarah@jacaranda.org**20110221020125
 Ignore-this: b0744ed58f161bf188e037bad077fc48
] 
[Refactor StorageFarmBroker handling of servers
Brian Warner <warner@lothar.com>**20110221015804
 Ignore-this: 842144ed92f5717699b8f580eab32a51
 
 Pass around IServer instance instead of (peerid, rref) tuple. Replace
 "descriptor" with "server". Other replacements:
 
  get_all_servers -> get_connected_servers/get_known_servers
  get_servers_for_index -> get_servers_for_psi (now returns IServers)
 
 This change still needs to be pushed further down: lots of code is now
 getting the IServer and then distributing (peerid, rref) internally.
 Instead, it ought to distribute the IServer internally and delay
 extracting a serverid or rref until the last moment.
 
 no_network.py was updated to retain parallelism.
] 
[TAG allmydata-tahoe-1.8.2
warner@lothar.com**20110131020101] 
Patch bundle hash:
7eccf95ba9716bac521b34cec4554836a7e5aa9f