[tahoe-dev] [mm] unparseable directory

[Ben Laurie]

zooko wrote:
> On May 5, 2008, at 11:00 AM, Ben Laurie wrote:
> 
>> OK, so I run:
>>
>>  curl http://127.0.0.1:8124/uri/URI%3ASSK-RO% 
>> 3Apgj2oefhipe2uzulz7kyvyfcbe% 
>> 3A7xoalidet7oignxsnhra77ez7qxh6ro5humhumncabv4cghdkc2q > ssk-ro
> ...
>> I get 332 bytes, the sha1sum is
>>
>> $ sha1 ssk-ro
>> SHA1 (ssk-ro) = c7ccfeb56c6a40a090709f915307fba80fd909c8
>>
>> and I attach the contents of the file, which is just garbage.
> 
> Yep, so we have a bonafide data integrity failure, where the  
> capability that you used, above, should have yielded a valid version  
> of the mutable file contents or else failed clearly, but instead it  
> returned garbage.  "Valid" in that sentence means that the file  
> contents  (or more precisely a deterministic function of the file  
> contents) were digitally signed with an RSA key, the secure hash of  
> the public component of which is embedded into the capability that  
> you used.
> 
> Since we haven't seen anything like this in our automated tests or  
> our other usage, I'm inclined to suspect a build failure (especially  
> in the C/C++ code that does erasure decoding, decryption, and digital  
> signature verification) on your system.
> 
> The code path for integrity verification of SSK ("Sub-Space Key", as  
> called by Freenet, which in our docs are called "Small Distributed  
> Mutable Files") is like this:
> 
> http://allmydata.org/trac/tahoe/browser/docs/mutable.txt?rev=2514#L163
> 
> The access pattern for read is:
>   * hash read-key to get storage index
>   * use storage index to locate 'k' shares with identical 'R' values
>     * either get one share, read 'k' from it, then read k-1 shares
>     * or read, say, 5 shares, discover k, either get more or be finished
>     * or copy k into the URIs
>   * read verification key
>   * hash verification key, compare against verification key hash
>   * read seqnum, R, encoding parameters, signature
>   * verify signature against verification key
>   * read share data, compute block-hash Merkle tree and root "r"
>   * read share hash chain (leading from "r" to "R")
>   * validate share hash chain up to the root "R"
>   * submit share data to erasure decoding
>   * decrypt decoded data with read-key
>   * submit plaintext to application
> 
> Hm...  Thinking over this it seems like the only way that all these  
> steps could complete with apparent success and then give you garbage  
> would be if there is a bug in the digital signature checking, the  
> erasure decoding, the decryption, or if you downloaded shares which  
> were actually from different encodings of the file or different  
> versions of the file but which incorrectly had identical version  
> numbers and root hashes.

Since you don't do an integrity check after decryption (which seems 
unwise), all that needs to fail is the decryption.

> Could you use the wui (point your web browser to http:// 
> 127.0.0.1:8124 ) and look at the "recent uploads and downloads" page  
> and see which shares from which servers were used for fetching that  
> mutable file?

I don't see how to do that? I see a "retrieve" operation, but clicking 
on "Done" doesn't show me any servers...

> Also, could you paste the output of "bin/tahoe --version"?

$ bin/tahoe --version
allmydata: 1.0.0-r2514, foolscap: 0.2.5, pycryptopp: 0.3.0, zfec: 1.3.4, 
twisted: 2.5.0, nevow: 0.9.18, simplejson: 1.7.3, pyopenssl: 0.6, 
setuptools: 0.6c8

-- 
http://www.apache-ssl.org/ben.html           http://www.links.org/

"There is no limit to what a man can do or how far he can go if he
doesn't mind who gets the credit." - Robert Woodruff