[tahoe-dev] 1,000,000+ files?

[Brian Warner]

On Sun, 4 May 2008 13:31:01 -0700
Chris Goffinet <cg at chrisgoffinet.com> wrote:

> I remember reading in a ticket that if you had over 9600 files in a  
> folder, you started to notice high cpu load. Has this issue been  
> addressed? If I were to start storing over 1,000,000 files any  
> techniques you recommend? I originally was hoping to just store them  
> in 1 directory as it was the easiest (i wouldn't be doing lookups  
> asking for a list of all files).

Yeah, tahoe directories can't handle massive numbers of children.

The specific memory-consumption problem we observed (#379) was an
edge-effect: two different size limits, such that a directory got past one
size check but got stopped on the other one. (also, it interacted with a
couple of other problems, the most fundamental of which is inside Twisted,
ticket Twisted#2466).

But the limit remains. Our mutable file implementation is still in its
infancy: we haven't built everything that we've designed, and we haven't
designed everything that we plan to build. Our mutable-file roadmap specifies
three levels:

 SDMF : small distributed mutable files : read/write all-at-once
 MDMF : medium : read/write individual segments
 LDMF : large : versioning, insert/delete span

We've only implement SDMF so far, and we've only implemented a subset of it
(limited to a single segment), which limits the size of a mutable file to a
few megabytes. This is enough for our target requirement of about 10000
children in a directory, but when you try to add children beyond that, you'll
get size-exceeded errors.

The next step for mutable files (now that the ServerMap refactoring is in
place) is to finish multi-segment SDMF, which will remove that size limit,
but which will still obligate users to read and write the whole file at once,
which could be a drag if you have a million entries in a 300MB dirnode and
you only care about reading the first one.

MDMF will let you read just a small portion of it instead of the whole thing.
But dirnodes will need to be revamped to make this useful, by adding some
sort of index at the beginning of the dirnode, so you can figure out *which*
portion to retrieve.


So, if you have a million files that you need to put somewhere in Tahoe 1.0,
I can suggest two things:

 * use a multiple-level directory hierarchy, to reduce the size of any
   individual directory below the size limits. A two-level structure would be
   plenty (using 1001 directories total: a single root dir that contains 1000
   subdirectories, then 1000 files in each subdir).

 * serialize the mapping from child name to child URI into a single file
   yourself, then upload this file. This would be immutable, and you'd have
   to handle the serialization/deserialization and interpretation of this
   file by yourself. But we can handle foolishly large immutable files, so
   ought to at least work.

Remember, a directory is really just a convenient tool that maps names to
URIs. Any tool that helps you organize your URIs will work, and if your
tool's state can be serialized into a sequence of bytes, then it can be
serialized and stored in tahoe somewhere.

hope that helps,
 -Brian