attempting more than four simultaneous put uploads seems to break client

[mirimir]

Our test client behaves oddly when doing more than four simultaneous uploads.

Our test grid (all 1.9.1) is on a private VPN. A remote dedicated server with one 10Mbps port hosts our OpenVPN server and a storage node, with helper enabled. The introducer, test client and nine other storage nodes are VMs (some local and some remote VPS). Although bandwidths in both directions are typically 1.5-2.5Mbps, latencies are large, with RTT averaging 800ms.

The client VM has one CPU, 512MB memory and 1.3GB swap. Versions are:

allmydata-tahoe: 1.9.1, foolscap: 0.6.3, pycryptopp: 0.5.25, zfec: 1.4.10, Twisted: 10.1.0, Nevow: 0.10.0, zope.interface: unknown, python: 2.6.6, platform: Linux-Ubuntu_10.10-i686-32bit_ELF, pyOpenSSL: 0.10, simplejson: 2.1.1, pycrypto: 2.0.1, pyasn1: unknown, mock: 0.8.0rc2, sqlite3: 2.4.1 [sqlite 3.7.2], setuptools: 0.6c16dev3

For single puts, the client uploads to the helper at ~70KBps, and the helper pushes to storage servers at ~200KBps nominal (660KBps total). For four simultaneous (forked) puts, uploading plateaus at ~200KBps, and pushing plateaus at ~250KBps nominal. However, for five or more simultaneous puts, something breaks. Although all ten shares of all files do get uploaded, some files don't get linked to the grid directory, and their upload operations are missing from "Recent Uploads and Downloads".

To create ten 1MB test files, we paste this into terminal:

dd if=/dev/urandom of=1MB1 bs=1K count=1K &
dd if=/dev/urandom of=1MB2 bs=1K count=1K &
dd if=/dev/urandom of=1MB3 bs=1K count=1K &
dd if=/dev/urandom of=1MB4 bs=1K count=1K &
dd if=/dev/urandom of=1MB5 bs=1K count=1K &
dd if=/dev/urandom of=1MB6 bs=1K count=1K &
dd if=/dev/urandom of=1MB7 bs=1K count=1K &
dd if=/dev/urandom of=1MB8 bs=1K count=1K &
dd if=/dev/urandom of=1MB9 bs=1K count=1K

To simultaneously put four files, we paste this into terminal:

~/allmydata-tahoe-1.9.1/bin/tahoe put --dir-cap=[Directory writecap] ~/1MB1 tahoe:1MB/1MB1 &
~/allmydata-tahoe-1.9.1/bin/tahoe put --dir-cap=[Directory writecap] ~/1MB2 tahoe:1MB/1MB2 &
~/allmydata-tahoe-1.9.1/bin/tahoe put --dir-cap=[Directory writecap] ~/1MB3 tahoe:1MB/1MB3 &
~/allmydata-tahoe-1.9.1/bin/tahoe put --dir-cap=[Directory writecap] ~/1MB4 tahoe:1MB/1MB4

We get this output (excerpted):

201 Created
URI:CHK:...:3:10:1048576
201 Created
URI:CHK:...:3:10:1048576
201 Created
URI:CHK:...:3:10:1048576
201 Created
URI:CHK:...:3:10:1048576

Browsing to tahoe:1MB, we see all four files:

Tahoe-LAFS Directory SI=foobar

Type     Filename    Size      Times
FILE     1MB1        1048576   lcr: 2012-03-27_HH:MM:SS.MMMMMM
                               lmo: 2012-03-27_HH:MM:SS.MMMMMM
FILE     1MB2        1048576   lcr: 2012-03-27_HH:MM:SS.MMMMMM
                               lmo: 2012-03-27_HH:MM:SS.MMMMMM
FILE     1MB3        1048576   lcr: 2012-03-27_HH:MM:SS.MMMMMM
                               lmo: 2012-03-27_HH:MM:SS.MMMMMM
FILE     1MB4        1048576   lcr: 2012-03-27_HH:MM:SS.MMMMMM
                               lmo: 2012-03-27_HH:MM:SS.MMMMMM

To simultaneously put five files, we paste this into terminal:

~/allmydata-tahoe-1.9.1/bin/tahoe put --dir-cap=[Directory writecap] ~/1MB5 tahoe:1MB/1MB5 &
~/allmydata-tahoe-1.9.1/bin/tahoe put --dir-cap=[Directory writecap] ~/1MB6 tahoe:1MB/1MB6 &
~/allmydata-tahoe-1.9.1/bin/tahoe put --dir-cap=[Directory writecap] ~/1MB7 tahoe:1MB/1MB7 &
~/allmydata-tahoe-1.9.1/bin/tahoe put --dir-cap=[Directory writecap] ~/1MB8 tahoe:1MB/1MB8 &
~/allmydata-tahoe-1.9.1/bin/tahoe put --dir-cap=[Directory writecap] ~/1MB9 tahoe:1MB/1MB9

We get this output (excerpted):

201 Created
URI:CHK:...:3:10:1048576
201 Created
URI:CHK:...:3:10:1048576
201 Created
URI:CHK:...:3:10:1048576
201 Created
URI:CHK:...:3:10:1048576
201 Created
URI:CHK:...:3:10:1048576

Browsing to tahoe:1MB, we only see one new file (not five):

Tahoe-LAFS Directory SI=foobar

Type     Filename    Size      Times
FILE     1MB1        1048576   lcr: 2012-03-27_HH:MM:SS.MMMMMM
                               lmo: 2012-03-27_HH:MM:SS.MMMMMM
FILE     1MB2        1048576   lcr: 2012-03-27_HH:MM:SS.MMMMMM
                               lmo: 2012-03-27_HH:MM:SS.MMMMMM
FILE     1MB3        1048576   lcr: 2012-03-27_HH:MM:SS.MMMMMM
                               lmo: 2012-03-27_HH:MM:SS.MMMMMM
FILE     1MB4        1048576   lcr: 2012-03-27_HH:MM:SS.MMMMMM
                               lmo: 2012-03-27_HH:MM:SS.MMMMMM
FILE     1MB8        1048576   lcr: 2012-03-27_HH:MM:SS.MMMMMM
                               lmo: 2012-03-27_HH:MM:SS.MMMMMM