[tahoe-dev] testnet anomaly report

[Brian Warner]

I've been working on some tools to look at a set of storage servers and
identify anomalous shares:

 * CHK shares for which more than one set of k-of-N encoding parameters
   were used
 * SDMF files for which more than one set of k-of-N parameters were used
 * SDMF files for which we see multple versions present in the grid

The first and second could occur if the same file were uploaded with
different parameters: for CHK files, this should happen a lot less now
(hopefully never) that our Content Hash Key value includes the encoding
parameters in the hash.

The third happens when some storage servers are offline when a new version of
a directory is uploaded. Our SDMF mutable-file upload algorithm is designed
to handle this case, but ticket #272 points out that we haven't implemented
the necessary recovery code yet. So these files are a problem (at least when
there are fewer than 'k' shares of the newer version available).

The new 'tahoe catalog-shares NODEDIRS..' command walks the given node
directories and emits one line for each share that it discovers. This line
summarizes the file: what storage index is it for, what encoding parameters
were used, etc. The 'misc/find-share-anomalies.py' tool then analyzes a
collection of these "share catalogs" and produces a report with any problems
that it sees.

I built a share catalog about an hour ago and ran the anomaly tool over the
results. The resulting report is attached to this message, with a summary
(produced with 'grep -v " "') of the troublesome storage index values inlined
here:


CHK multiple encodings:
 16a9zdg5ccjdawwd81cepti3rw
 3a5rjfz15horsuxbynyepowyxa
 3b457f9rtx9bbi4cfbaty81xsa
 44kai1tui348689nrw8fjegc8c
 5u7byy7awoe3jxf4zwshbehedc
 68g6c6umnemx67bsjmmt9auf9w
 998i58ckhnaxtc8z158j8u84ur
 detktbkm8qwzs585j6txaf5ybc
 gh3h1p9q9nar7zqkeqiokq4xto
 inj1mu6wgfnrrmfnppgxmzwhec
 jc96f1kb5wfsjmfsfdeurq7ohy
 nobodcjahtjbsfp9hn8w3h1pqw
 x4kemmi948sai6nbjsf1kxdzqe
 yxusm7susgmwtaok9459dz314e

SDMF multiple versions:
 1hze8idrkgsix455werhonqw8r
 4mbw74wnrzg9971p7yqezamdxy
 6653wrq68yyn1gsg79rntxpx6w
 as3ojdp1uo9ue8m16at94pubsh
 b4egmzyd59zumayjj7wdnbyuwr
 b51ni3x4f54nzor1zuy6zj85dr
 c7itaz3h8orc4ueq1zey6wr5cr
 cgjcez4bjdm5ayk8onodgpgejc
 czejfukk9mmy7fe7mm5fbmkuah
 dk5seyn4rueaeq4owiihxnb6jr
 fw677bowff36rwmkpnffpf5ewa
 g9idg8u9r7frxyuq8tz567hana
 heqeiqidwpuemhdnrgkdjqkbzr
 odhn3paaycdidpatfmrowjbhhr
 oo9npgn5xwfwp5uozwhqh3pzjh
 smuqyechtpqe1bmrqi4xp7p18e
 t1aj6bghrta157prg5pkrz9kpr
 tmoi9qjq5oudpj517ktyizxfpr
 uq3r7febbao7k8z1htghr373io
 we59fyq31434zumrczccgzabne
 xug7zwkw4cdy9q6rko4jaodrjr
 y4sa84ux1n69uq6j8ef9b1wimh


If you're experiencing problems modifying a directory, you might want to see
if it's in this list (pass the URI:DIR2: cap to 'tahoe dump-cap' to extract
the right storage index).

cheers,
 -Brian
-------------- next part --------------
A non-text attachment was scrubbed...
Name: testnet-anomalies.out
Type: application/octet-stream
Size: 76140 bytes
Desc: not available
Url : http://allmydata.org/pipermail/tahoe-dev/attachments/20080211/9e17d1aa/attachment-0001.obj