Opened at 2007-12-18T22:36:57Z
Last modified at 2010-01-15T20:32:55Z
#236 new enhancement
measure the effect of a more memory-local zfec variant on a webapi server
| Reported by: | zooko | Owned by: | zooko |
|---|---|---|---|
| Priority: | minor | Milestone: | undecided |
| Component: | code-encoding | Version: | 0.7.0 |
| Keywords: | performance memory zfec large | Cc: | |
| Launchpad Bug: |
Description
The preliminary results of the load tests ("the webapi servers are the busy points. Each webapi server is on its own machine. Each one is consuming anywhere from 30% to 70% CPU.") suggest that zfec might be a bottleneck in a complete system (grid plus webapi servers). I have some patches to zfec which constrain its use of memory to be more localized in order to fit into smaller caches.
I will measure whethermany zfec processes running in parallel on one server-sized machine (an Athlon64 2.4 GHz, 512 KiB L2 cache, 4 GiB DDR2 RAM) benefit or suffer from this locality optimization.
Change History (6)
comment:1 Changed at 2008-03-08T00:53:51Z by warner
comment:2 Changed at 2008-06-01T21:05:08Z by warner
- Milestone changed from eventually to undecided
comment:3 Changed at 2009-12-13T03:06:09Z by davidsarah
- Keywords performance memory zfec added; scaling removed
comment:4 Changed at 2009-12-13T03:08:01Z by davidsarah
- Keywords scalability removed
comment:5 Changed at 2009-12-13T05:04:01Z by davidsarah
- Priority changed from major to minor
comment:6 Changed at 2010-01-15T20:32:55Z by davidsarah
- Keywords large added
Note: See
TracTickets for help on using
tickets.
FYI, the new upload/download status timing pages tell me that for a 5MB 3-of-10 download, zfec is running at 89MBps, whereas AES is running at 5.6MBps. So at least for one-at-a-time downloads, AES is far more of a bottleneck than zfec.
For a machine that is in colo, a 2MB download runs zfec at 587MBps and AES at 12MBps. A 16MB download runs zfec at 320MBps and AES at 12MBps.