[tahoe-dev] Convergence statistics

rob kinninmont robk-tahoe at allmydata.com
Mon Mar 31 13:01:14 PDT 2008 


Below and attached are some statistics on the extent of convergence
in the Allmydata 2.0 network ('mountain view', the predecessor to
tahoe)


the results are relatively roughly formatted, but I'm sharing the
data because I imagine it's useful as it.

this is a histogram of the savings convergence provides broken
down by file size.

in summary convergence saves 0.91% (20Gb) of the file data

this is measuring file data itself, and not accounting for the
overhead of expansion storage costs, disk block consumption etc.

this analysis is assuming convergence within the domain of each
account, i.e. multiple instances of a file stored within an
account have no effect, this is measuring convergence only of
files across account boundaries.


briefly the stats in this table are, for each filesize range:

totu: total data stored by users (non convergent case)
conv: total data actually stored in mesh (convergent case)
saved: the net win convergence provides

% of space saved (within file size range) by convergence
% that saving represents of saving totalled over all filesizes

number of user account references to files of that size
(i.e. total files in non-convergent case)

number of instances of files actually stored
(i.e. total files in convergent case)

number of 'shared references'
(i.e. count of references in accounts 'saved' by convergence)

n_converged: number of distinct files actually converged
between multiple accounts


the graphs present the space usage ('conv'/'saved') and the
counts of files, illustrating the extent of convergence's
impact on the data stored in the mesh


hope this helps :-)
cheers,
rob



non-conv usage: 2221.86 GB (2221858411002) in 1219287 files
convergent usage: 2201.56 GB (2201564894911) in 1183035 files
space saved: 20.29 GB (20293516091) (0.91% of non-convergent total)
files from      1 B to      1 B:    totu:     57 B conv:     17 B  
saved:     40 B (70.18% , 0.00% of total) (57, 17, 40, n_converged: 10)
files from      1 B to      3 B:    totu:    371 B conv:    164 B  
saved:    207 B (55.80% , 0.00% of total) (154, 67, 87, n_converged: 25)
files from      3 B to      5 B:    totu:    457 B conv:    337 B  
saved:    120 B (26.26% , 0.00% of total) (104, 75, 29, n_converged: 11)
files from      5 B to     10 B:    totu:   4871 B conv:   4711 B  
saved:    160 B (3.28% , 0.00% of total) (579, 557, 22, n_converged: 15)
files from     10 B to     17 B:    totu:    11 KB conv:    11 KB  
saved:    673 B (5.61% , 0.00% of total) (1015, 964, 51, n_converged:  
38)
files from     17 B to     31 B:    totu:   197 KB conv:   193 KB  
saved:   3302 B (1.68% , 0.00% of total) (7404, 7270, 134,  
n_converged: 78)
files from     31 B to     56 B:    totu:   209 KB conv:   170 KB  
saved:    38 KB (18.61% , 0.00% of total) (4599, 3730, 869,  
n_converged: 315)
files from     56 B to    100 B:    totu:   740 KB conv:   661 KB  
saved:    78 KB (10.66% , 0.00% of total) (9447, 8417, 1030,  
n_converged: 506)
files from    100 B to    177 B:    totu:  2238 KB conv:  2034 KB  
saved:   204 KB (9.14% , 0.00% of total) (15924, 14451, 1473,  
n_converged: 797)
files from    177 B to    316 B:    totu:     6 MB conv:  5816 KB  
saved:   386 KB (6.23% , 0.00% of total) (25345, 23714, 1631,  
n_converged: 990)
files from    316 B to    562 B:    totu:    15 MB conv:    14 MB  
saved:   711 KB (4.60% , 0.00% of total) (35391, 33720, 1671,  
n_converged: 1111)
files from    562 B to   1000 B:    totu:    28 MB conv:    27 MB  
saved:  1855 KB (6.42% , 0.00% of total) (37722, 35318, 2404,  
n_converged: 1614)
files from   1000 B to   1778 B:    totu:    77 MB conv:    70 MB  
saved:     6 MB (8.89% , 0.00% of total) (60574, 54615, 5959,  
n_converged: 4898)
files from   1778 B to   3162 B:    totu:   126 MB conv:   120 MB  
saved:     6 MB (5.17% , 0.00% of total) (52629, 49907, 2722,  
n_converged: 2099)
files from   3162 B to   5623 B:    totu:   218 MB conv:   207 MB  
saved:    11 MB (5.28% , 0.00% of total) (51433, 48714, 2719,  
n_converged: 2177)
files from   5623 B to    10 KB:    totu:   448 MB conv:   424 MB  
saved:    24 MB (5.52% , 0.00% of total) (58370, 55133, 3237,  
n_converged: 2225)
files from    10 KB to    17 KB:    totu:   938 MB conv:   898 MB  
saved:    40 MB (4.28% , 0.00% of total) (69688, 66652, 3036,  
n_converged: 1984)
files from    17 KB to    31 KB:    totu:  2273 MB conv:  2223 MB  
saved:    49 MB (2.20% , 0.00% of total) (94429, 92319, 2110,  
n_converged: 1571)
files from    31 KB to    56 KB:    totu:  3812 MB conv:  3745 MB  
saved:    67 MB (1.76% , 0.00% of total) (89307, 87707, 1600,  
n_converged: 1153)
files from    56 KB to   100 KB:    totu:  5748 MB conv:  5656 MB  
saved:    92 MB (1.61% , 0.00% of total) (76795, 75582, 1213,  
n_converged: 898)
files from   100 KB to   177 KB:    totu:  8.50 GB conv:  8.37 GB  
saved:   126 MB (1.49% , 0.01% of total) (63136, 62186, 950,  
n_converged: 698)
files from   177 KB to   316 KB:    totu: 12.79 GB conv: 12.61 GB  
saved:   183 MB (1.43% , 0.01% of total) (53630, 52889, 741,  
n_converged: 557)
files from   316 KB to   562 KB:    totu: 27.92 GB conv: 27.51 GB  
saved:   401 MB (1.44% , 0.02% of total) (63890, 62954, 936,  
n_converged: 662)
files from   562 KB to  1000 KB:    totu: 69.92 GB conv: 69.58 GB  
saved:   339 MB (0.49% , 0.02% of total) (93194, 92729, 465,  
n_converged: 306)
files from  1000 KB to  1778 KB:    totu: 95.45 GB conv: 95.07 GB  
saved:   379 MB (0.40% , 0.02% of total) (72178, 71896, 282,  
n_converged: 184)
files from  1778 KB to  3162 KB:    totu:143.46 GB conv:143.09 GB  
saved:   369 MB (0.26% , 0.02% of total) (60224, 60068, 156,  
n_converged: 113)
files from  3162 KB to  5623 KB:    totu:277.76 GB conv:276.99 GB  
saved:   767 MB (0.28% , 0.03% of total) (65528, 65351, 177,  
n_converged: 134)
files from  5623 KB to    10 MB:    totu:235.74 GB conv:233.34 GB  
saved:  2402 MB (1.02% , 0.11% of total) (33821, 33529, 292,  
n_converged: 170)
files from    10 MB to    17 MB:    totu:107.75 GB conv:106.56 GB  
saved:  1188 MB (1.10% , 0.05% of total) (8469, 8381, 88,  
n_converged: 65)
files from    17 MB to    31 MB:    totu:165.88 GB conv:164.15 GB  
saved:  1725 MB (1.04% , 0.08% of total) (6516, 6443, 73,  
n_converged: 49)
files from    31 MB to    56 MB:    totu:133.95 GB conv:132.55 GB  
saved:  1400 MB (1.05% , 0.06% of total) (3159, 3130, 29,  
n_converged: 27)
files from    56 MB to   100 MB:    totu:116.27 GB conv:115.50 GB  
saved:   766 MB (0.66% , 0.03% of total) (1589, 1578, 11,  
n_converged: 10)
files from   100 MB to   177 MB:    totu:167.52 GB conv:166.10 GB  
saved:  1415 MB (0.85% , 0.06% of total) (1371, 1360, 11,  
n_converged: 9)
files from   177 MB to   316 MB:    totu:202.68 GB conv:202.47 GB  
saved:   209 MB (0.10% , 0.01% of total) (861, 860, 1, n_converged: 1)
files from   316 MB to   562 MB:    totu:201.39 GB conv:201.39 GB  
saved:      0 B (0.00% , 0.00% of total) (477, 477, 0, n_converged: 0)
files from   562 MB to  1000 MB:    totu:152.27 GB conv:151.53 GB  
saved:   733 MB (0.48% , 0.03% of total) (213, 212, 1, n_converged: 1)
files from  1000 MB to  1778 MB:    totu: 71.27 GB conv: 69.13 GB  
saved:  2147 MB (3.01% , 0.10% of total) (57, 55, 2, n_converged: 2)
files from  1778 MB to  3162 MB:    totu: 17.65 GB conv: 12.22 GB  
saved:  5434 MB (30.78% , 0.24% of total) (8, 6, 2, n_converged: 1)
files from  3162 MB to  5623 MB:    totu:      0 B conv:      0 B  
saved:      0 B (0.00% , 0.00% of total) (0, 0, 0, n_converged: 0)

-------------- next part --------------
A non-text attachment was scrubbed...
Name: convergence.tgz
Type: application/octet-stream
Size: 14590 bytes
Desc: not available
Url : http://allmydata.org/pipermail/tahoe-dev/attachments/20080331/cabde190/attachment-0001.obj

More information about the tahoe-dev mailing list