id	summary	reporter	owner	description	type	status	priority	milestone	component	version	resolution	keywords	cc	launchpad_bug
85	store shares in single files, instead of 7 files and a directory	warner	warner	"At the moment we have significant filesystem overhead in our share storage, mostly because common filesystems like ext3 consume a whole disk block (4096 bytes) even for 1-byte files. Since we use 7 files in a separate directory for each share, that means 8*4096=32768 bytes consumed per share, even for 1-byte shares. As a result, the lower bound on storage consumed occurs at 102400 bytes (i.e. all files 102400 bytes or smaller consume the same amount of storage), at which point the storage consumed is 3276800 (3.3MB).

Our storage format was chosen for simplicity and ease of implementation, but this represents a huge overhead. So the plan is to combine all 7 files into a single one, and to not put it in its own directory. That will reduce the minimum share size to one disk block (4096) instead of 8 (32768), and will bring the lower bound on storage to a filesize of 81250, at which point the storage consumed will be 409600 (410kB), an 8x improvement.

Reducing this filesystem-blocksize overhead below that would involve packing multiple shares (for different URIs) into a single file, which complicates the deletion and indexing of them. It might be useful, but hopefully we can avoid this step.

Also, we need to figure out a good place to put leases, once we implement them, but they can probably live in a separate database with different packing and access requirements.
"	defect	closed	major	0.5.0	code	0.4.0	fixed	storage