[tahoe-dev] File size problem

[Brian Warner]

> How do mutable and immutable files (for /regular/ files) actually
> differ? When I tested with an immutable file, I'd create one, delete
> it and create a new file with different content (but the same name).

Mutable files retain their identity (their URI) even when their contents
change. Immutable files do not. These URI identifiers are unrelated to the
directory pathnames which we typically use to handle a collection of files.

One way to think about this is to simplify tahoe into the following
functions:

 immutable_uri = create_immutable_file(contents)

 mutable_uri = create_mutable_file(initial_contents="")
 modify_mutable_file(mutable_uri, new_contents)

 contents = download_file(uri)

 directory_uri = create_empty_directory()
 add_to_directory(directory_uri, childname, childuri)
 remove_from_directory(directory_uri, childname)
 children = list_directory(directory_uri)

These, plus some mechanism to remember a "starting point" directory_uri
(sometimes referred to as a "rootcap": a root directory capability string),
encompasses pretty much the entire Tahoe CLI.

Each directory is really just a serialized mapping (from child name to child
URI), stored in a mutable file. By keeping it in a mutable file, changing a
child directory doesn't require us to go and modify the parent. If we had
immutable directories instead, then each modification would require the
parent to be replaced, followed by the grandparent, etc, up to and including
the rootcap. Many local filesystems use this approach to get strong
consistency properties.. I think "WAFL" is the right term. Tahoe could do
WAFL if we had immutable directories, but it seemed more important to have
mutable directories first.

Using directories and pathnames tends to hide the difference between mutable
and immutable files. If you have a directory (which let's say has a URI of
URI:DIR2:1234), and you use 'tahoe add-alias' to remember it as "foo", then
you might list its (empty) contents with:

 tahoe ls foo:

Now if you upload "Hello World" into a new immutable file, and attach that to
our directory as a child named "bar", you might use:

 echo "Hello World" | tahoe put - foo:bar

What that really does is:

 # baruri = create_immutable_file("Hello World")
 # add_to_directory(foouri, "bar", baruri)

If you then replace that file with a different one, containing "Goodbye
World", you might use:

 echo "Goodbye World" | tahoe put - foo:bar

Which really does:

 # byeuri = create_immutable_file("Goodbye World")
 # remove_from_directory(foouri, "bar")
 # add_to_directory(foouri, "bar", byeuri)


On the other hand, if you did all of this with mutable files, you'd have:

 echo "Hello World" |tahoe put --mutable - foo:bar
 # baruri = create_mutable_file("Hello World")
 # add_to_directory(foouri, "bar", baruri)

 echo "Goodbye World" | tahoe put - foo:bar
    (note: since foo:bar is a mutable file, the --mutable is unnecessary)
 # baruri = list_directory(foouri)["bar"]
 # modify_mutable_file(baruri, "Goodbye World")


If you only ever use directories and child names, then you probably don't
need to pay attention to the distinction between mutable and immutable files.
Unless, of course, you want to use files larger than 3.5MB :-).
 
> (I noticed that `du -ms' on the node's directories didn't get smaller.
> When deleting files, are their fragments actually deleted, or marked
> to be deletable in some way?)

Nope, unfortunately. We do not yet have automatic garbage collection. Without
a confident way to know that nobody remembers a reference to e.g. the old
'baruri' file, we can't safely delete it.

We have some plans to address this: check the mailing list for posts about
"GC" or "accounting" for details.

> > >   * When is `ps axflwww'ed the process' memory usage, I saw that
> > > the python instance that ran the connection where I uploaded the
> > > 31MB file grew beyond 300MB of VSZ.
> > 
> > That sounds like a bug in the code that's rejecting the too-large
> > mutable file. Which version were you running? (1.2.0 or current
> > trunk?). If it was current trunk, I'll look more closely at the
> 
> I `apt-get install'ed the "allmydata-tahoe" package; right now, that's
> version  1.2.0-r3241. Since I tested some three days ago, it might
> have been a few revisions earlier than 3241.

Hrm, that's effectively trunk, so it sounds like a modern bug that needs to
be addressed. I'll try to look into it.

> There's also a "tahoe-prod" package at version 3.0.0-r2758. Should I
> re-do the testing with that one?

Nope. We have three branches of tahoe:

 trunk (package name 'allmydata-tahoe'): general hacking, main development

 server (package name 'tahoe-server'): more stable, runs allmydata.com servers

 prod (package name 'tahoe-prod'): more stable, used for allmydata win32 client

The non-trunk packages are really only there for the convenience of
allmydata.com deployment. Most allmydata.org users should stick to trunk.

> For what is the key-generator actually used?

The key-generator is a small process that offloads the CPU-intensive work of
generating RSA public/private keypairs from a webapi node. Each mutable file
requires a new keypair (a mutable-file write-URI is effectively an RSA
private key), so every time you create a new directory, you must generate a
new keypair. On many systems (depending on the CPU) it can take two or three
seconds to create a 2048-bit RSA keypair, and since the Tahoe process is
single-threaded, it can't do anything else during that time.

So the key-generator just sits off to the side and maintains a pool of
keypairs. A Tahoe client node (which is probably running a webapi service)
can be configured to ask the key-generator for a key instead of making one by
itself. The key-generator tries put off refilling its pool until nobody is
using it. If it's lucky, no request will ever be stalled by the RSA
key-generation time. If it isn't, then at least only the "mkdir" request will
be stalled: other requests can still be serviced without delay.

We built the key-generator to improve service for allmydata.com users (most
of whom are sharing a small number of webapi nodes). Unless you're creating a
lot of directories at once, it probably won't help you very much. On the
other hand, it can't hurt.

> Doesn't come up right after creating one:

> spinne:/mnt/tahoe# tahoe start -C foo
> key_generator.KeyGeneratorService(2048) File
> "/usr/lib/python2.5/site-packages/allmydata/key_generator.py", line
> 82, in __init__ self.tub =
> foolscap.Tub(certFile=os.path.join(self.basedir,
> 'key_generator.pem')) File "/usr/lib/python2.5/posixpath.py", line
> 62, in join elif path == '' or path.endswith('/'):
> exceptions.AttributeError: 'int' object has no attribute 'endswith'

Sigh. We've been using the same key-generator for seven months, and I think
the 'tahoe create-key-generator' code has bitrotted. I've created ticket #542
to track this: http://allmydata.org/trac/tahoe/ticket/542 .

To actually use the key-generator (if it worked), you'd need to copy the
key-generator's "key_generator.furl" into the client node directory, and
restart the node.

> The create-stats-gatherer command works (though I have to find out
> what is't actually used for, as well as the create-key-generator.

The stats-gatherer polls a bunch of storage nodes every 15 minutes for their
statistics (how many files have been uploaded/downloaded, how much CPU
they're using, etc). It then provides data to a set of munin plugins to
create graphs of these values.

> Maybe I'll draw a nice little sheet with the basics about upload
> helper, access/storage nodes, introducer, etc. one of these days.)

That'd be wonderful!

cheers,
 -Brian