[tahoe-dev] Recommendations for minimal RAM usage ?

[Johannes Nix]

Hello Zooko,

> It is cool that the Tahoe-LAFS storage server works at all on that
> tiny NAS device! Does it work okay? Is your desire to reduce the
> memory footprint to eliminate some failures, or to improve
> performance, or to allow concurrent use of other processes on the same
> device, or to move it to an even smaller device, or something else?

I think it's cool as well. What I like
especially is the idea that people can put such
an inexpensive  box as personal server to share information
and make themselves more independent from
data-gathering companies. It would also be
quite easy to provide a ready-to use image or
software package.

For me, the server worked very well in the last
eight days.  Today however, we've seen on VG2 huge latencies of 
about 6 seconds, which occured when I did a backup upload at
the same time. I ran also an NFS daemon to
upload data and maybe that was sometimes becoming
too much. Neither NFS nor tahoe-backup uses
much RAM but I suspect that this leads occasionally 
to a kind of memory gridlock in kernel space; NFS
can sometimes stall for many seconds.

So, I think the reliability and speed could be increased
when reducing RAM further. 

> > The storage-server side of Tahoe doesn't really do that much, so
> > I'd be pleased as punch if it could fit into 10 or 20MB. To do
> > that, I think we'd need to avoid loading all of the code when we're
> > not running a client 

> 
> Wouldn't the code (.py's, .pyc's, or .so's) tend to be swapped out if
> it unused? Or maybe unused code gets interspersed with code that is
> needed, so it ends up bloating the memory usage?

Memory is managed by the OS in 4k size pages. Entirely unused
pages are simply swapped out. A reduction in the resident set size,
in the amount of file buffering required or a grouping of often used
code / data could probably achieve some speed-up.
The simplest measure could be to import modules
in a grouped order, with server and client
functions in different groups.

How difficult would it be to rip out a server-only
program? This could run with increased
priority. Is there a fundamental reason that
both processes run in the same interpreter instance? I've
experimented some time with real-time audio processing 
in Python, and a low-hanging fruit for multiprocessor
systems is to separate different processes which
are not serialized by Python's GIL (Global Interpreter Lock).

Of course, my NAS has only one core but most future
desktop and server systems will have much more than one.

Regards,

Johannes



Am Sun, 4 Mar 2012 21:33:08 -0700
schrieb "Zooko Wilcox-O'Hearn" <zooko at zooko.com>:

> Dear Johannes Nix:
> 
> It is cool that the Tahoe-LAFS storage server works at all on that
> tiny NAS device! Does it work okay? Is your desire to reduce the
> memory footprint to eliminate some failures, or to improve
> performance, or to allow concurrent use of other processes on the same
> device, or to move it to an even smaller device, or something else?
> 
> The numbers you give sound about right. I measured resident RAM usage
> on 32-bit x86 servers a couple of times, and found that it started at
> around 32 MB and the high water mark of resident RAM usage was about
> 56 MB.
> 
> I don't know of any technique that could further reduce it, aside from
> profiling the memory usage and reporting back to this list. If your
> profiling turned up some "low hanging fruit" of wasted memory, then
> perhaps some Tahoe-LAFS developer would be motivated to try reducing
> that.
> 
> Regards,
> 
> Zooko
> _______________________________________________
> tahoe-dev mailing list
> tahoe-dev at tahoe-lafs.org
> http://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev