[tahoe-lafs-trac-stream] [tahoe-lafs] #1456: High latency for 'tahoe get' if 'tahoe put' in parallel

[tahoe-lafs]

#1456: High latency for 'tahoe get' if 'tahoe put' in parallel
-------------------------------------------------+-------------------------
 Reporter:  T_X                                  |          Owner:
     Type:  defect                               |  somebody
 Priority:  major                                |         Status:  new
Component:  code                                 |      Milestone:
 Keywords:  latency performance gateway vm kvm   |  undecided
  vpn trickle                                    |        Version:  1.8.2
                                                 |  Launchpad Bug:
-------------------------------------------------+-------------------------
 The two node setup, connected via a VPN and details of the setup can be
 found here: http://tahoe-lafs.org/pipermail/tahoe-
 dev/2011-July/006563.html.

 Notable was, that if the bandwidth is saturated with another program
 ('netcat6' in this case), then there was no latency issue for 'tahoe get'.
 However, with running 'tahoe put' over the same gateway, then the latency
 went up to about 20 seconds.

 I further tried to reproduce the issue within a single host in a KVM
 instance. One node was set up as a dedicated storage node, the other as a
 dedicated gateway with no storage. See tahoe.cfg attachments for the
 detailed node configurations.

 The scripted test procedure can be found in the attached test-run.sh.

 The results were the following:

 Test 0: Download with no parallel upload or limitations: 1.6s average

 Test 1: Parallel 'tahoe put', limited to 800kbit/s: 1.8s average

 Test 2: Parallel 'tahoe put', tahoe-2 (gateway) limited to 800kbit/s: 87s
 average

 Test 3: Parallel 'tahoe put', tahoe-1 (storage) limited to 800kbit/s: 12s
 average

 For latency of each individual download, see log attachements.

 Test 2's individual download times are also very noteworthy in particular.
 I'm not quite sure whether the 200s and 500s delays are another issue or
 whether it is an extreme manifestation of the same thing. When I'm
 repeating the test script, at least one of these highly delayed transfers
 occures in the beginning of test 2.

 For Tests 0 and 1, the 1.6s seem to be a little high for a local transfer
 of this tiny file, but ok, that latency is just about acceptable.

 My actual goal was to be able to run a common httpd and ftpd on top of
 sshfs using tahoe-lafs. However the results of most tests are too high and
 influence the usability badly. As there is no feedback for the final
 user's ftp and http client during the delays, Users might for one thing
 claim that the service is slow, might abort downloads or applications
 would abort on their own anyway (with the default settings wget times out
 after 30s for instance).

 During the tests, especially during the downloads that took more than
 200s, I didn't notice any cpu usage spikes. I hope that with trickle the
 limited bandwidth capabilities of the internet uplink as it was the case
 in the first tests over the VPN could somehow be simulated by that,
 hopefully with the similar issues in both test setups having the same
 cause(s).

-- 
Ticket URL: <http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1456>
tahoe-lafs <http://tahoe-lafs.org>
secure decentralized storage