[tahoe-dev] How large can a Small Distributed Mutable File be?

[zooko]

Folks:

Brian's excellent design document mutable.txt [1] describes the  
design of Small Distributed Mutable Files, which are the next big  
feature in Tahoe (as per the Roadmap [2]).

I'm reading that document now, and it suggests that you should use  
the SDMF feature only for files < 1 MB in size.  This raises the  
question in my mind -- what's the practical limit?

There is no hard-coded limit in the SDMF format, but changing  
anything -- even changing one byte or appending one byte -- in an  
SDMF is implemented as re-encrypting, erasure coding, Merkle-tree- 
building, and uploading the entire file.  That's why we put "Small"  
in the name.

So it is a matter of users -- is there somebody out there who will be  
happy to do all that every time they change their file, even though  
their file is 1 GB?  Are there people who will be unsatisified with  
the total performance of SDMFs even if their files are only 100 KB?

I guess a big part of it is how often you change the file, and  
whether you need to wait for the new change to be written before you  
can go on and do something else.

I guess we will learn more about this question as soon as SDMFs are  
complete enough to benchmark, which I think will be today.

Regards,

Zooko

[1] http://allmydata.org/trac/tahoe/browser/docs/mutable.txt
[2] http://allmydata.org/trac/tahoe/roadmap