[tahoe-dev] 1.6.0 When storing shares (from another machine), Python consumes --> 100% of local (storage node) CPU
Zooko O'Whielacronx
zookog at gmail.com
Thu Mar 18 08:10:52 PDT 2010
On Thursday, 2010-03-18, at 7:37 , Jody Harris wrote:
> This raised it's ugly head in last night's backup. Fortunately, the symptom showed up on the very last upload, so I found the timing data on the first file in the list.
> I will attempt to attach a PDF of the timings page to this message. If there is a ticket for this, or a ticket this should be attached to, let me know.
Thanks for the bug report! This is #983. I guess I should have Cc:'ed
you on that ticket since your earlier report was part of the process
of finding the bug.
This is actually my highest Tahoe-LAFS-hacking priority right now, and
the way I've been going about it is to write up this "StringChain"
package which I hope could be used to fix this problem. That's pretty
much done (footnote 1), and the next step is for me to write a patch
and off it to Brian for foolscap #149. (I'm not sure that StringChain
is the right way to fix foolscap #149 and I suspect Brian is even less
sure, but it is worth a try.)
I'm hoping to get a fixed foolscap into Ubuntu Lucid so that Lucid
users don't have this problem.
In the meantime, the work-around is not to use mutable files and to
keep your mutable directories nice and small. Immutable files perform
much better than mutable files do, for this reason (#983) as well as a
few other reasons. They also have the handy property that they are
immutable, so you can be sure that the file contents have not changed.
Another problem with mutable files is that they break the FTP and SFTP
frontends (#680, #941).
Could you remind me why you use mutable files in your backup scripts?
I remember the first time you posted to this list about why you are
using mutable files I thought "Huh, I hadn't thought of that.". But
now I don't remember your reason.
Regards,
Zooko
http://tahoe-lafs.org/trac/tahoe-lafs/ticket/680# Fix for mutable files with FTP
http://tahoe-lafs.org/trac/tahoe-lafs/ticket/941# SFTP frontend fails
when listing a directory containing a mutable file, because it relies
on node.get_size() to be an integer
http://tahoe-lafs.org/trac/tahoe-lafs/ticket/983# high CPU load on
storage servers when uploading large mutable file
http://foolscap.lothar.com/trac/ticket/149# O(n**2) CPU/malloc during
receive of large strings
footnote 1: benchmarks of StringChain vs. a few other data structures
N is bytes processed; results are in nanoseconds per byte; on my
MacBook Pro with 2.33 GHz Intel Core 2 Duo. To run the benchmarks
yourself just get the StringChain package
(http://tahoe-lafs.org/trac/stringchain ) and run the same command.
The important thing to note is that StringChain is scalable (speed
doesn't drop off as N increases) as well as fast.
The task named "_alternate_str" is the one that most closely
approximates what foolscap needs to do. The implementation named
"Stringy" is the on e that most closely approximates what foolscap
currently does.
$ python -OOu -c 'from stringchain.bench import bench; bench.slow_bench()'
impl: StringChain
task: _accumulate_then_one_gulp
N: 10000, ns/op: best: 2.79, 2th-best: 3.39, ave: 3.90,
2th-worst: 4.70, worst: 4.82 (of 5)
N: 50000, ns/op: best: 2.76, 2th-best: 2.78, ave: 2.95,
2th-worst: 2.94, worst: 3.36 (of 5)
N: 100000, ns/op: best: 2.33, 2th-best: 2.34, ave: 2.40,
2th-worst: 2.45, worst: 2.54 (of 5)
N: 500000, ns/op: best: 2.03, 2th-best: 2.10, ave: 2.26,
2th-worst: 2.28, worst: 2.70 (of 5)
N: 1000000, ns/op: best: 1.98, 2th-best: 2.01, ave: 2.30,
2th-worst: 2.45, worst: 2.68 (of 5)
N: 5000000, ns/op: best: 2.11, 2th-best: 2.14, ave: 2.18,
2th-worst: 2.20, worst: 2.25 (of 5)
task: _many_gulps_str
N: 10000, ns/op: best: 5.20, 2th-best: 5.51, ave: 6.60,
2th-worst: 6.10, worst: 10.61 (of 5)
N: 50000, ns/op: best: 3.24, 2th-best: 3.24, ave: 3.42,
2th-worst: 3.42, worst: 3.88 (of 5)
N: 100000, ns/op: best: 2.87, 2th-best: 2.88, ave: 2.96,
2th-worst: 2.93, worst: 3.22 (of 5)
N: 500000, ns/op: best: 2.80, 2th-best: 2.84, ave: 2.90,
2th-worst: 2.95, worst: 3.00 (of 5)
N: 1000000, ns/op: best: 2.70, 2th-best: 2.70, ave: 2.77,
2th-worst: 2.84, worst: 2.88 (of 5)
N: 5000000, ns/op: best: 2.70, 2th-best: 2.71, ave: 2.72,
2th-worst: 2.74, worst: 2.76 (of 5)
task: _alternate_str
N: 10000, ns/op: best: 6.79, 2th-best: 7.10, ave: 7.84,
2th-worst: 8.61, worst: 8.80 (of 5)
N: 50000, ns/op: best: 4.74, 2th-best: 4.78, ave: 4.83,
2th-worst: 4.88, worst: 4.90 (of 5)
N: 100000, ns/op: best: 4.34, 2th-best: 4.35, ave: 4.50,
2th-worst: 4.53, worst: 4.78 (of 5)
N: 500000, ns/op: best: 3.98, 2th-best: 4.12, ave: 4.25,
2th-worst: 4.36, worst: 4.50 (of 5)
N: 1000000, ns/op: best: 3.96, 2th-best: 4.00, ave: 4.07,
2th-worst: 4.02, worst: 4.36 (of 5)
N: 5000000, ns/op: best: 3.96, 2th-best: 3.97, ave: 4.07,
2th-worst: 4.18, worst: 4.19 (of 5)
impl: StringIOy
task: _accumulate_then_one_gulp
N: 10000, ns/op: best: 4.10, 2th-best: 4.29, ave: 4.84,
2th-worst: 5.29, worst: 5.51 (of 5)
N: 50000, ns/op: best: 5.06, 2th-best: 5.60, ave: 5.75,
2th-worst: 6.14, worst: 6.32 (of 5)
N: 100000, ns/op: best: 4.51, 2th-best: 4.52, ave: 4.60,
2th-worst: 4.67, worst: 4.69 (of 5)
N: 500000, ns/op: best: 3.80, 2th-best: 3.90, ave: 4.70,
2th-worst: 5.10, worst: 6.71 (of 5)
N: 1000000, ns/op: best: 4.19, 2th-best: 4.29, ave: 4.69,
2th-worst: 5.08, worst: 5.49 (of 5)
N: 5000000, ns/op: best: 5.27, 2th-best: 5.30, ave: 5.57,
2th-worst: 5.89, worst: 6.00 (of 5)
task: _many_gulps_str
N: 10000, ns/op: best: 4.70, 2th-best: 5.51, ave: 5.70,
2th-worst: 5.70, worst: 6.91 (of 5)
N: 50000, ns/op: best: 24.28, 2th-best: 24.34, ave: 27.87,
2th-worst: 25.36, worst: 40.90 (of 5)
N: 100000, ns/op: best: 38.64, 2th-best: 38.84, ave: 39.03,
2th-worst: 39.22, worst: 39.38 (of 5)
N: 500000, ns/op: best: 158.61, 2th-best: 159.18, ave: 162.47,
2th-worst: 165.73, worst: 168.40 (of 5)
N: 1000000, ns/op: best: 455.90, 2th-best: 459.57, ave: 471.79,
2th-worst: 488.72, worst: 490.74 (of 5)
N: 5000000, ns/op: best: 2730.39, 2th-best: 2734.04, ave: 2771.03,
2th-worst: 2734.04, worst: 2848.67 (of 3)
task: _alternate_str
N: 10000, ns/op: best: 6.99, 2th-best: 7.08, ave: 8.10,
2th-worst: 7.70, worst: 11.11 (of 5)
N: 50000, ns/op: best: 8.34, 2th-best: 8.34, ave: 8.52,
2th-worst: 8.38, worst: 9.18 (of 5)
N: 100000, ns/op: best: 22.37, 2th-best: 22.68, ave: 22.82,
2th-worst: 22.90, worst: 23.44 (of 5)
N: 500000, ns/op: best: 74.92, 2th-best: 75.96, ave: 76.61,
2th-worst: 76.87, worst: 78.45 (of 5)
N: 1000000, ns/op: best: 135.24, 2th-best: 136.14, ave: 139.87,
2th-worst: 137.46, worst: 153.85 (of 5)
N: 5000000, ns/op: best: 1000.30, 2th-best: 1003.59, ave: 1076.12,
2th-worst: 1003.59, worst: 1224.47 (of 3)
impl: Stringy
task: _accumulate_then_one_gulp
N: 10000, ns/op: best: 2.38, 2th-best: 2.69, ave: 3.01,
2th-worst: 3.00, worst: 4.20 (of 5)
N: 50000, ns/op: best: 15.86, 2th-best: 16.16, ave: 18.65,
2th-worst: 16.34, worst: 28.66 (of 5)
N: 100000, ns/op: best: 26.60, 2th-best: 26.80, ave: 30.51,
2th-worst: 34.00, worst: 35.07 (of 5)
N: 500000, ns/op: best: 102.15, 2th-best: 102.16, ave: 103.20,
2th-worst: 103.29, worst: 105.36 (of 5)
N: 1000000, ns/op: best: 266.58, 2th-best: 266.73, ave: 280.54,
2th-worst: 286.67, worst: 311.59 (of 5)
N: 5000000, ns/op: best: 1543.46, 2th-best: 1554.51, ave: 1553.37,
2th-worst: 1554.51, worst: 1562.16 (of 3)
task: _many_gulps_str
N: 10000, ns/op: best: 2.60, 2th-best: 2.91, ave: 3.15,
2th-worst: 3.50, worst: 3.72 (of 5)
N: 50000, ns/op: best: 15.06, 2th-best: 15.08, ave: 15.34,
2th-worst: 15.60, worst: 15.86 (of 5)
N: 100000, ns/op: best: 24.20, 2th-best: 24.21, ave: 25.17,
2th-worst: 24.88, worst: 27.80 (of 5)
N: 500000, ns/op: best: 98.93, 2th-best: 101.46, ave: 101.85,
2th-worst: 102.31, worst: 105.05 (of 5)
N: 1000000, ns/op: best: 264.17, 2th-best: 266.87, ave: 268.33,
2th-worst: 270.90, worst: 272.28 (of 5)
N: 5000000, ns/op: best: 1556.56, 1th-best: 1556.56, ave: 1586.65,
1th-worst: 1616.73, worst: 1616.73 (of 2)
task: _alternate_str
N: 10000, ns/op: best: 4.29, 2th-best: 4.70, ave: 5.48,
2th-worst: 6.01, worst: 6.39 (of 5)
N: 50000, ns/op: best: 5.16, 2th-best: 5.20, ave: 5.38,
2th-worst: 5.44, worst: 5.68 (of 5)
N: 100000, ns/op: best: 20.53, 2th-best: 20.65, ave: 20.77,
2th-worst: 20.71, worst: 21.30 (of 5)
N: 500000, ns/op: best: 81.43, 2th-best: 81.60, ave: 82.35,
2th-worst: 82.54, worst: 84.52 (of 5)
N: 1000000, ns/op: best: 143.76, 2th-best: 146.12, ave: 146.30,
2th-worst: 146.19, worst: 149.30 (of 5)
N: 5000000, ns/op: best: 917.51, 2th-best: 925.97, ave: 923.90,
2th-worst: 925.97, worst: 928.21 (of 3)
More information about the tahoe-dev
mailing list