[tahoe-lafs-trac-stream] [Tahoe-LAFS] #4080: Increase chunk sizes to improve performance and parallelism

[Tahoe-LAFS]

#4080: Increase chunk sizes to improve performance and parallelism
-------------------------+----------------------------------------------
 Reporter:  itamarst     |          Owner:
     Type:  enhancement  |         Status:  new
 Priority:  normal       |      Milestone:  Performance and Benchmarking
Component:  unknown      |        Version:  n/a
 Keywords:               |  Launchpad Bug:
-------------------------+----------------------------------------------
 1. Processing data in larger chunk sizes can reduce overhead of all the
 little Python function calls for each chunk.
 2. Larger chunk sizes for functions that release GIL add additional
 opportunities for utilizing parallelism by shifting to a thread pool.
 Larger chunks of work are more efficient for this scheme.

--
Ticket URL: <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/4080>
Tahoe-LAFS <https://Tahoe-LAFS.org>
secure decentralized storage