[tahoe-lafs-trac-stream] [tahoe-lafs] #2015: leasedb: high file descriptor usage exposed by tests

[tahoe-lafs]

#2015: leasedb: high file descriptor usage exposed by tests
-------------------------+-------------------------------------------------
     Reporter:  daira    |      Owner:  markberger
         Type:  defect   |     Status:  new
     Priority:  major    |  Milestone:  1.11.0
    Component:  code-    |    Version:  1.10.0
  storage                |   Keywords:  leasedb tests reliability file-
   Resolution:           |  descriptor leak
Launchpad Bug:           |
-------------------------+-------------------------------------------------

Comment (by zooko):

 I learned that to recreate the big performance regression, you need to
 apply this patch to current 1819-cloud-merge:

 {{{
 diff --git a/src/allmydata/storage/leasedb.py
 b/src/allmydata/storage/leasedb.py
 index 02524e7..737d257 100644
 --- a/src/allmydata/storage/leasedb.py
 +++ b/src/allmydata/storage/leasedb.py
 @@ -128,7 +128,8 @@ class LeaseDB:
          (self._sqlite,
           self._db) = dbutil.get_db(dbfile,
 create_version=(LEASE_SCHEMA_V1, 1),
                                     # journal_mode="WAL",
 -                                   synchronous="OFF")
 +                                   # synchronous="NORMAL"
 +                                   )
          self._cursor = self._db.cursor()
          self.debug = False
          self.retained_history_entries = 10
 }}}

 I used my fdleakfinder (https://github.com/zooko/fdleakfinder) on the full
 test suite, with optimized sqlite settings (journal_mode="WAL",
 synchronous="NORMAL", see below), and with fd limit bumped up to the max
 that my linux allows -- 2000.

 fdleakfinder definitely reports that the sqlite files opened during test
 cases are left open instead of closed at the end of this test case. It is
 reporting the "high water mark", which is a list of all the fds that were
 open *at one particular* time. See attached output from fdleakfinder.


 {{{
 --- a/src/allmydata/storage/leasedb.py
 +++ b/src/allmydata/storage/leasedb.py
 @@ -127,8 +127,9 @@ class LeaseDB:

          (self._sqlite,
           self._db) = dbutil.get_db(dbfile,
 create_version=(LEASE_SCHEMA_V1, 1),
 -                                   # journal_mode="WAL",
 -                                   synchronous="OFF")
 +                                   journal_mode="WAL",

 +                                   synchronous="NORMAL"
 +                                   )
          self._cursor = self._db.cursor()
          self.debug = False
          self.retained_history_entries = 10
 }}}

-- 
Ticket URL: <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2015#comment:1>
tahoe-lafs <https://tahoe-lafs.org>
secure decentralized storage