[tahoe-dev] 1.6.0 When storing shares (from another machine), Python consumes --> 100% of local (storage node) CPU

[Brian Warner]

I've identified a O(n**2) CPU/string-allocation misbehavior in Foolscap
when receiving large strings. This could explain the memory and CPU
problems in Tahoe storage servers when you're uploading large mutable
files to them (where "large" means segments that are more than about
10MB, which for the default 3-of-10 means filesizes above 30MB). In
particular, uploading a 100MB mutable file at 3-of-10, leading to 33MB
blocks, appears to take about three minutes to unpack, using 100MB in
the process (on each server).

I still don't have an explanation for reports of slowdowns and large
memory consumption for large *immutable* files. Those use small blocks
(typically about 40KB) and shouldn't be hurt by this quadratic problem.
There are still a lot of string copies going on, and perhaps the way
that Python allocates memory is interacting badly with that behavior.

http://foolscap.lothar.com/trac/ticket/149 has some more details. The
fix for this will be inside Foolscap's receive-side token parser, and
Tahoe won't even notice.

When I've got a patch to try out, I'll follow up here, so that the folks
who have experienced this problem can tell me what improves and what
remains troublesome.

cheers,
 -Brian