multi backup error

[dawuud]

When performing two backups at once (tahoe backup) one of them crashed with a stack traceback ending in this error: OperationalError?: database is locked

I was running tahoe-lafs in Tails from wheezy-backports :

$ tahoe --version
allmydata-tahoe: 1.10.0
foolscap: 0.6.4
pycryptopp: 0.6.0.1206569328141510525648634803928199668821045408958
zfec: 1.4.5
Twisted: 12.0.0
Nevow: 0.10.0
zope.interface: unknown
python: 2.7.3
platform: Linux-debian_7.6-x86_64-32bit_ELF
pyOpenSSL: 0.13
simplejson: 2.5.2
pycrypto: 2.6
pyasn1: unknown
mock: 0.8.0
setuptools: 0.6 [distribute]

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/allmydata/scripts/runner.py", line 156, in run
    rc = runner(sys.argv[1:], install_node_control=install_node_control)
  File "/usr/lib/python2.7/dist-packages/allmydata/scripts/runner.py", line 141, in runner
    rc = cli.dispatch[command](so)
  File "/usr/lib/python2.7/dist-packages/allmydata/scripts/cli.py", line 574, in backup
    rc = tahoe_backup.backup(options)
  File "/usr/lib/python2.7/dist-packages/allmydata/scripts/tahoe_backup.py", line 325, in backup
    return bu.run()
  File "/usr/lib/python2.7/dist-packages/allmydata/scripts/tahoe_backup.py", line 118, in run
    new_backup_dircap = self.process(options.from_dir)
  File "/usr/lib/python2.7/dist-packages/allmydata/scripts/tahoe_backup.py", line 188, in process
    childcap = self.process(childpath)
  File "/usr/lib/python2.7/dist-packages/allmydata/scripts/tahoe_backup.py", line 188, in process
    childcap = self.process(childpath)
  File "/usr/lib/python2.7/dist-packages/allmydata/scripts/tahoe_backup.py", line 188, in process
    childcap = self.process(childpath)
  File "/usr/lib/python2.7/dist-packages/allmydata/scripts/tahoe_backup.py", line 194, in process
    childcap, metadata = self.upload(childpath)
  File "/usr/lib/python2.7/dist-packages/allmydata/scripts/tahoe_backup.py", line 313, in upload
    bdb_results.did_upload(filecap)
  File "/usr/lib/python2.7/dist-packages/allmydata/scripts/backupdb.py", line 122, in did_upload
    self.mtime, self.ctime, self.size)
  File "/usr/lib/python2.7/dist-packages/allmydata/scripts/backupdb.py", line 264, in did_upload_file
    fileid = self.get_or_allocate_fileid_for_cap(filecap)
  File "/usr/lib/python2.7/dist-packages/allmydata/scripts/backupdb.py", line 256, in get_or_allocate_fileid_for_cap
    c.execute("SELECT fileid FROM caps WHERE filecap=?", (filecap,))
OperationalError: database is locked

[daira]

See also #2062.

[zooko]

I'll bet we could fix this simply by setting ​PRAGMA busy_timeout to a large value (like 3000 milliseconds). This would, if I understand correctly, cause the second one of the tahoe backup processes that attempted to write to a table to block until the first one finished its write and committed the transaction. This would only take a tiny fraction of a second (especially if we turned on journal_mode="WAL" and synchronous="NORMAL", like we do ​in the leasedb branch).

The only question I have is: how would we test a patch which made this change!

[daira]

Replying to zooko:

I'll bet we could fix this simply by setting ​PRAGMA busy_timeout to a large value (like 3000 milliseconds). This would, if I understand correctly, cause the second one of the tahoe backup processes that attempted to write to a table to block until the first one finished its write and committed the transaction.

I think that's correct.

The only question I have is: how would we test a patch which made this change!

I wouldn't mind having the test just check that the intended pragmas are set (it should also check the existing pragmas, which I don't think are currently tested).