[tahoe-dev] maximum file sizes: 2GiB with helper, 12GiB without

[Brian Warner]

FYI, I fixed the RemoteInterface-side limitations. Both client and helper
must be updated to gain the benefits, but there are no compability issues
introduced by this change.

So the current filesize restriction is 12GiB, caused by the 4-byte share-size
field in the on-disk immutable share storage format, and the default 3-of-10
encoding: the actual limit is k*sharesize. This size is achieveable by both
Helper and non-Helper uploads.

Fixing that will require a couple days of work, to rev the on-disk layout and
provide backwards-compatibility code for existing shares, and we'll probably
schedule this work for a month or so from now. That would leave us with the
next restriction, the hash tree offsets in the immutable share format. For
128KiB segments (the current default), this limits us to 8.8TB per file.

To fix that one, we'll need to rev the share format, which would probably
take about two weeks (this is the format used by clients, rather than storage
servers, so there is more compatibility code to write). That would raise our
file size limit to a number that is too long to fit in the margins of this
email :-).

 -Brian