[tahoe-lafs-trac-stream] [tahoe-lafs] #1696: attempting more than four simultaneous put uploads seems to break client

[tahoe-lafs]

#1696: attempting more than four simultaneous put uploads seems to break client
-------------------------------------------------+-------------------------
 Reporter:  mirimir                              |          Owner:  nobody
     Type:  defect                               |         Status:  new
 Priority:  minor                                |      Milestone:
Component:  unknown                              |  undecided
 Keywords:  integrity preservation reliability   |        Version:  1.9.1
  scalability error                              |  Launchpad Bug:
-------------------------------------------------+-------------------------
 Our test client behaves oddly when doing more than four simultaneous
 uploads.

 Our test grid (all 1.9.1) is on a private VPN. A remote dedicated server
 with one 10Mbps port hosts our OpenVPN server and a storage node, with
 helper enabled. The introducer, test client and nine other storage nodes
 are VMs (some local and some remote VPS). Although bandwidths in both
 directions are typically 1.5-2.5Mbps, latencies are large, with RTT
 averaging 800ms.

 The client VM has one CPU, 512MB memory and 1.3GB swap. Versions are:

 allmydata-tahoe: 1.9.1, foolscap: 0.6.3, pycryptopp: 0.5.25, zfec: 1.4.10,
 Twisted: 10.1.0, Nevow: 0.10.0, zope.interface: unknown, python: 2.6.6,
 platform: Linux-Ubuntu_10.10-i686-32bit_ELF, pyOpenSSL: 0.10, simplejson:
 2.1.1, pycrypto: 2.0.1, pyasn1: unknown, mock: 0.8.0rc2, sqlite3: 2.4.1
 [sqlite 3.7.2], setuptools: 0.6c16dev3

 For single puts, the client uploads to the helper at ~70KBps, and the
 helper pushes to storage servers at ~200KBps nominal (660KBps total). For
 four simultaneous (forked) puts, uploading plateaus at ~200KBps, and
 pushing plateaus at ~250KBps nominal. However, for five or more
 simultaneous puts, something breaks. Although all ten shares of all files
 do get uploaded, some files don't get linked to the grid directory, and
 their upload operations are missing from "Recent Uploads and Downloads".

 To create ten 1MB test files, we paste this into terminal:

 {{{
 dd if=/dev/urandom of=1MB1 bs=1K count=1K &
 dd if=/dev/urandom of=1MB2 bs=1K count=1K &
 dd if=/dev/urandom of=1MB3 bs=1K count=1K &
 dd if=/dev/urandom of=1MB4 bs=1K count=1K &
 dd if=/dev/urandom of=1MB5 bs=1K count=1K &
 dd if=/dev/urandom of=1MB6 bs=1K count=1K &
 dd if=/dev/urandom of=1MB7 bs=1K count=1K &
 dd if=/dev/urandom of=1MB8 bs=1K count=1K &
 dd if=/dev/urandom of=1MB9 bs=1K count=1K
 }}}

 To simultaneously put four files, we paste this into terminal:

 {{{
 ~/allmydata-tahoe-1.9.1/bin/tahoe put --dir-cap=[Directory writecap]
 ~/1MB1 tahoe:1MB/1MB1 &
 ~/allmydata-tahoe-1.9.1/bin/tahoe put --dir-cap=[Directory writecap]
 ~/1MB2 tahoe:1MB/1MB2 &
 ~/allmydata-tahoe-1.9.1/bin/tahoe put --dir-cap=[Directory writecap]
 ~/1MB3 tahoe:1MB/1MB3 &
 ~/allmydata-tahoe-1.9.1/bin/tahoe put --dir-cap=[Directory writecap]
 ~/1MB4 tahoe:1MB/1MB4
 }}}

 We get this output (excerpted):

 {{{
 201 Created
 URI:CHK:...:3:10:1048576
 201 Created
 URI:CHK:...:3:10:1048576
 201 Created
 URI:CHK:...:3:10:1048576
 201 Created
 URI:CHK:...:3:10:1048576
 }}}

 Browsing to tahoe:1MB, we see all four files:

 Tahoe-LAFS Directory SI=foobar

 {{{
 Type     Filename    Size      Times
 FILE     1MB1        1048576   lcr: 2012-03-27_HH:MM:SS.MMMMMM
                                lmo: 2012-03-27_HH:MM:SS.MMMMMM
 FILE     1MB2        1048576   lcr: 2012-03-27_HH:MM:SS.MMMMMM
                                lmo: 2012-03-27_HH:MM:SS.MMMMMM
 FILE     1MB3        1048576   lcr: 2012-03-27_HH:MM:SS.MMMMMM
                                lmo: 2012-03-27_HH:MM:SS.MMMMMM
 FILE     1MB4        1048576   lcr: 2012-03-27_HH:MM:SS.MMMMMM
                                lmo: 2012-03-27_HH:MM:SS.MMMMMM
 }}}

 To simultaneously put five files, we paste this into terminal:

 {{{
 ~/allmydata-tahoe-1.9.1/bin/tahoe put --dir-cap=[Directory writecap]
 ~/1MB5 tahoe:1MB/1MB5 &
 ~/allmydata-tahoe-1.9.1/bin/tahoe put --dir-cap=[Directory writecap]
 ~/1MB6 tahoe:1MB/1MB6 &
 ~/allmydata-tahoe-1.9.1/bin/tahoe put --dir-cap=[Directory writecap]
 ~/1MB7 tahoe:1MB/1MB7 &
 ~/allmydata-tahoe-1.9.1/bin/tahoe put --dir-cap=[Directory writecap]
 ~/1MB8 tahoe:1MB/1MB8 &
 ~/allmydata-tahoe-1.9.1/bin/tahoe put --dir-cap=[Directory writecap]
 ~/1MB9 tahoe:1MB/1MB9
 }}}

 We get this output (excerpted):

 {{{
 201 Created
 URI:CHK:...:3:10:1048576
 201 Created
 URI:CHK:...:3:10:1048576
 201 Created
 URI:CHK:...:3:10:1048576
 201 Created
 URI:CHK:...:3:10:1048576
 201 Created
 URI:CHK:...:3:10:1048576
 }}}

 Browsing to tahoe:1MB, we only see one new file (not five):

 Tahoe-LAFS Directory SI=foobar

 {{{
 Type     Filename    Size      Times
 FILE     1MB1        1048576   lcr: 2012-03-27_HH:MM:SS.MMMMMM
                                lmo: 2012-03-27_HH:MM:SS.MMMMMM
 FILE     1MB2        1048576   lcr: 2012-03-27_HH:MM:SS.MMMMMM
                                lmo: 2012-03-27_HH:MM:SS.MMMMMM
 FILE     1MB3        1048576   lcr: 2012-03-27_HH:MM:SS.MMMMMM
                                lmo: 2012-03-27_HH:MM:SS.MMMMMM
 FILE     1MB4        1048576   lcr: 2012-03-27_HH:MM:SS.MMMMMM
                                lmo: 2012-03-27_HH:MM:SS.MMMMMM
 FILE     1MB8        1048576   lcr: 2012-03-27_HH:MM:SS.MMMMMM
                                lmo: 2012-03-27_HH:MM:SS.MMMMMM
 }}}

-- 
Ticket URL: <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1696>
tahoe-lafs <https://tahoe-lafs.org>
secure decentralized storage