[tahoe-lafs-trac-stream] [Tahoe-LAFS] #4079: Performance hit by a million cuts

[Tahoe-LAFS]

#4079: Performance hit by a million cuts
----------------------+----------------------------------------------
 Reporter:  itamarst  |          Owner:
     Type:  defect    |         Status:  new
 Priority:  normal    |      Milestone:  Performance and Benchmarking
Component:  unknown   |        Version:  n/a
 Keywords:            |  Launchpad Bug:
----------------------+----------------------------------------------
 Profiling shows just lots and lots and lots of tiny little actions.
 Question is, is there any obvious culprint beyond "it's Python", and if
 so, can we do anything about it.

 Things already investigated:

 * I measured overhead of using plain Deferreds. I'd estimate it's maybe 1%
 of CPU usage? So probably not that. In future Python versions this might
 become more significant depending how good the theoretical proposed JIT
 ends up being, as callbacks may prevent optimizations at that level. But
 that's not quite the same thing as overhead.

 Things to investigate:

 * inlineCallbacks probably has overhead too. I didn't measure it yet.
 * HTTP overhead. Twisted's HTTP parser probably isn't the fastest.
 * Chunk size. If chunking is done in too small of a unit, this increases
 all the overhead. Larger chunk sizes will help.

--
Ticket URL: <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/4079>
Tahoe-LAFS <https://Tahoe-LAFS.org>
secure decentralized storage