[tahoe-dev] [mm] unparseable directory

[zooko]

On May 5, 2008, at 11:00 AM, Ben Laurie wrote:

> OK, so I run:
>
>  curl http://127.0.0.1:8124/uri/URI%3ASSK-RO% 
> 3Apgj2oefhipe2uzulz7kyvyfcbe% 
> 3A7xoalidet7oignxsnhra77ez7qxh6ro5humhumncabv4cghdkc2q > ssk-ro
...
> I get 332 bytes, the sha1sum is
>
> $ sha1 ssk-ro
> SHA1 (ssk-ro) = c7ccfeb56c6a40a090709f915307fba80fd909c8
>
> and I attach the contents of the file, which is just garbage.

Yep, so we have a bonafide data integrity failure, where the  
capability that you used, above, should have yielded a valid version  
of the mutable file contents or else failed clearly, but instead it  
returned garbage.  "Valid" in that sentence means that the file  
contents  (or more precisely a deterministic function of the file  
contents) were digitally signed with an RSA key, the secure hash of  
the public component of which is embedded into the capability that  
you used.

Since we haven't seen anything like this in our automated tests or  
our other usage, I'm inclined to suspect a build failure (especially  
in the C/C++ code that does erasure decoding, decryption, and digital  
signature verification) on your system.

The code path for integrity verification of SSK ("Sub-Space Key", as  
called by Freenet, which in our docs are called "Small Distributed  
Mutable Files") is like this:

http://allmydata.org/trac/tahoe/browser/docs/mutable.txt?rev=2514#L163

The access pattern for read is:
  * hash read-key to get storage index
  * use storage index to locate 'k' shares with identical 'R' values
    * either get one share, read 'k' from it, then read k-1 shares
    * or read, say, 5 shares, discover k, either get more or be finished
    * or copy k into the URIs
  * read verification key
  * hash verification key, compare against verification key hash
  * read seqnum, R, encoding parameters, signature
  * verify signature against verification key
  * read share data, compute block-hash Merkle tree and root "r"
  * read share hash chain (leading from "r" to "R")
  * validate share hash chain up to the root "R"
  * submit share data to erasure decoding
  * decrypt decoded data with read-key
  * submit plaintext to application

Hm...  Thinking over this it seems like the only way that all these  
steps could complete with apparent success and then give you garbage  
would be if there is a bug in the digital signature checking, the  
erasure decoding, the decryption, or if you downloaded shares which  
were actually from different encodings of the file or different  
versions of the file but which incorrectly had identical version  
numbers and root hashes.

Could you use the wui (point your web browser to http:// 
127.0.0.1:8124 ) and look at the "recent uploads and downloads" page  
and see which shares from which servers were used for fetching that  
mutable file?

Also, could you paste the output of "bin/tahoe --version"?

Thanks!

Regards,

Zooko