[tahoe-dev] [tahoe-lafs] #962: Performance problems testing on a 45 disk Dual core 3.3Ghz 4G memory Box
tahoe-lafs
trac at allmydata.org
Mon Feb 22 17:08:26 PST 2010
#962: Performance problems testing on a 45 disk Dual core 3.3Ghz 4G memory Box
-------------------------+--------------------------------------------------
Reporter: stott | Owner: somebody
Type: enhancement | Status: new
Priority: major | Milestone: undecided
Component: website | Version: 1.5.0
Keywords: Performance | Launchpad_bug:
-------------------------+--------------------------------------------------
Comment(by stott):
System Configuration (http://blog.backblaze.com/2009/09/01/petabytes-on-a
-budget-how-to-build-cheap-cloud-storage)
1 Intel(R) Core(TM)2 Duo CPU E8600 @ 3.33GHz
1 80G boot drive mounted as /
45 Seagate 1.5 T drives mounted as /tahoe/node1-45
4G memory
eth0 is at 100MB/FULL
Default tahoe configuration.
SMEM :
root at firetruck1:~/app/smem-0.9# ./smem |egrep "PID|python"
PID User Command Swap USS PSS
RSS
13025 root egrep --color=auto PID|pyth 0 260 276
860
13024 root /usr/bin/python ./smem 0 4560 4609
6224
12402 www-data /usr/bin/python /usr/bin/tw 23028 33356 33391
35504
12546 www-data /usr/bin/python /usr/bin/tw 14796 47700 47754
50764
12537 www-data /usr/bin/python /usr/bin/tw 9412 53300 53351
56228
12590 www-data /usr/bin/python /usr/bin/tw 12 60732 60786
63796
12634 www-data /usr/bin/python /usr/bin/tw 0 60792 60846
63856
12645 www-data /usr/bin/python /usr/bin/tw 0 60796 60854
63868
12612 www-data /usr/bin/python /usr/bin/tw 8 60904 60958
63968
12601 www-data /usr/bin/python /usr/bin/tw 4 61192 61246
64256
12425 www-data /usr/bin/python /usr/bin/tw 668 61860 61911
64788
12447 www-data /usr/bin/python /usr/bin/tw 728 62240 62291
65168
12557 www-data /usr/bin/python /usr/bin/tw 12 62368 62422
65432
12480 www-data /usr/bin/python /usr/bin/tw 0 62384 62438
65448
12524 www-data /usr/bin/python /usr/bin/tw 156 62500 62552
65492
12623 www-data /usr/bin/python /usr/bin/tw 4 62500 62554
65564
12579 www-data /usr/bin/python /usr/bin/tw 8 62564 62616
65556
12513 www-data /usr/bin/python /usr/bin/tw 52 62632 62686
65696
12700 www-data /usr/bin/python /usr/bin/tw 0 62644 62698
65708
12491 www-data /usr/bin/python /usr/bin/tw 40 62772 62824
65764
12469 www-data /usr/bin/python /usr/bin/tw 264 62796 62850
65860
12656 www-data /usr/bin/python /usr/bin/tw 0 63004 63057
66060
12502 www-data /usr/bin/python /usr/bin/tw 0 63184 63238
66248
12667 www-data /usr/bin/python /usr/bin/tw 0 63264 63318
66328
12436 www-data /usr/bin/python /usr/bin/tw 92 63392 63446
66456
12458 www-data /usr/bin/python /usr/bin/tw 164 63780 63834
66844
12744 www-data /usr/bin/python /usr/bin/tw 0 64016 64070
67080
12788 www-data /usr/bin/python /usr/bin/tw 0 65400 65451
68328
12722 www-data /usr/bin/python /usr/bin/tw 0 66472 66526
69536
12678 www-data /usr/bin/python /usr/bin/tw 0 67036 67090
70100
12689 www-data /usr/bin/python /usr/bin/tw 0 67112 67166
70176
12568 www-data /usr/bin/python /usr/bin/tw 0 70172 70224
73164
12832 www-data /usr/bin/python /usr/bin/tw 0 71800 71854
74864
12898 www-data /usr/bin/python /usr/bin/tw 0 71760 71949
75168
12777 www-data /usr/bin/python /usr/bin/tw 0 72432 72486
75496
12854 www-data /usr/bin/python /usr/bin/tw 0 73000 73050
75920
12735 www-data /usr/bin/python /usr/bin/tw 0 73096 73150
76160
12711 www-data /usr/bin/python /usr/bin/tw 0 73308 73362
76372
12414 www-data /usr/bin/python /usr/bin/tw 908 73256 73449
76680
12799 www-data /usr/bin/python /usr/bin/tw 0 73652 73706
76716
12843 www-data /usr/bin/python /usr/bin/tw 0 74660 74714
77724
12865 www-data /usr/bin/python /usr/bin/tw 0 74888 74946
77960
12821 www-data /usr/bin/python /usr/bin/tw 0 76804 76858
79868
12887 www-data /usr/bin/python /usr/bin/tw 0 79180 79233
82236
12755 www-data /usr/bin/python /usr/bin/tw 0 79892 79946
82956
12766 www-data /usr/bin/python /usr/bin/tw 0 80548 80599
83532
12810 www-data /usr/bin/python /usr/bin/tw 0 86512 86563
89440
12876 www-data /usr/bin/python /usr/bin/tw 0 87592 87645
90648
================
Note,
After switching to
shares.needed = 32
shares.happy = 35
shares.total = 40
Performance has greatly increased. The 3.3 Blowup may have been the
largest contributing factor.
I'm still concerned about memory utilization per node as the nodes
increase per grid, but that may be a separate ticket.
Performace is still much less than I would like to see.
bash-3.2$ time curl -T 524stottlemyer.zip
http://10.20.0.151:3470/uri/URI:DIR2:whzhc6kxh5mfem4basdoioewji:4qpwpea2s46tg3lnd5znwsyqqenwjoddjxpsvr733sk7n4on6uua/
URI:CHK:fsmf5rn6me7mrueq2ri4zvboay:sd5lnugn2gnisrvodxtzp7zvkx4f2skyk6ri5ncs65rmq6qkdoqa:32:40:106554122
real 1m17.962s
bash-3.2$ ls -larth 524stottlemyer.zip
-rw-r--r-- 1 justin staff 102M Nov 20 15:44 524stottlemyer.zip
I'll have to collect more stats during an upload to present what is going
on. A snapshot indicated heavy CPU, nearly no swap in use at this time,
and disks were much better able to handle the load w/ great distribution.
Here is a brief network snapshot though time spent on internal operations
was much greater.
eth0: 0.003 In, 0.000 Out Mbps | 4 In, 0 Out pps
eth0: 40.860 In, 0.963 Out Mbps | 3709 In, 1912 Out pps
eth0: 92.249 In, 2.226 Out Mbps | 8409 In, 4354 Out pps
eth0: 37.314 In, 0.893 Out Mbps | 3430 In, 1761 Out pps
eth0: 0.006 In, 0.015 Out Mbps | 8 In, 3 Out pps
--
Ticket URL: <http://allmydata.org/trac/tahoe/ticket/962#comment:3>
tahoe-lafs <http://allmydata.org>
secure decentralized file storage grid
More information about the tahoe-dev
mailing list