[tahoe-lafs-trac-stream] [Tahoe-LAFS] #1638: S3 backend: Upload of large files consumes memory > twice the size of the file

[Tahoe-LAFS]

#1638: S3 backend: Upload of large files consumes memory > twice the size of the
file
-------------------------+-------------------------------------------------
     Reporter:  zancas   |      Owner:
         Type:  defect   |     Status:  closed
     Priority:  major    |  Milestone:  eventually
    Component:  code-    |    Version:  cloud-branch
  storage                |   Keywords:  reliability availability
   Resolution:  fixed    |  scalability usability performance memory large
Launchpad Bug:           |  s3-backend
-------------------------+-------------------------------------------------
Changes (by daira):

 * version:  1.9.0 => cloud-branch
 * milestone:  1.10.1 => eventually


Old description:

> During upload greater than 2 (~2.5) times the size of shares (both
> mutable and immutable) are consumed on the storage server.  For
> storageservers running with tight memory constraints and/or large shares
> this is problematic.
>   Attached is a 1-second-resolution time-series of memory usage generated
> by ps for a 200 MB immutable file upload to an amazon EC2 t1.micro.
>
>   The exact command:  ps -Ovsize,rss -C tahoe

New description:

 During upload greater than 2 (~2.5) times the size of shares (both mutable
 and immutable) are consumed on the storage server.  For storageservers
 running with tight memory constraints and/or large shares this is
 problematic.
   Attached is a 1-second-resolution time-series of memory usage generated
 by ps for a 200 MB immutable file upload to an amazon EC2 t1.micro.

   The exact command:  ps -Ovsize,rss -C tahoe

--

--
Ticket URL: <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1638#comment:5>
Tahoe-LAFS <https://Tahoe-LAFS.org>
secure decentralized storage