[tahoe-dev] unparseable directory

[zooko]

Ben:

Thank you very much for your persistence in trying to make Tahoe work  
on your system.


On May 3, 2008, at 10:44 AM, Ben Laurie wrote:

> /disk1.1/usr/home/ben/darcs-work/tahoe/src/allmydata/dirnode.py,  
> line 181 in _unpack_contents

Yikes, this is getting weird.  Here is the line 181 in question:

http://allmydata.org/trac/tahoe/browser/src/allmydata/dirnode.py? 
rev=2514#L181

(Note the rev=2514, which I inserted because an earlier stack trace  
that you posted showed "load_entry_point('allmydata-tahoe==1.0.0- 
r2514', ".)

Which is calling split_netstring() on one of the entries in the  
directory.  split_netstring() is this simple function:

http://allmydata.org/trac/tahoe/browser/src/allmydata/dirnode.py? 
rev=2514#L16

and according to your recent stack trace, there is no ":" character  
in the entry:

> Function split_netstring in file /disk1.1/usr/home/ben/darcs-work/ 
> tahoe/src/allmydata/dirnode.py at line 16
> /disk1.1/usr/home/ben/darcs-work/tahoe/src/allmydata/dirnode.py,  
> line 24 in split_netstring
> 22 assert numstrings >= 0
> 23 while data:
> 24 colon = data.index(":")
> 25 length = int(data[:colon])
> Locals
> numstrings1
> data��"�>/�w��T�GsB���f�����n\A ZӂƟ| 
> t[Yh��
Yk�/T�)�!���Xc�#�k��}��˨^  
> XZ���㧿)����>w� ������o��E<7Σ 
> \"JI�s��,
�KXy�����v 
> +�D5k��L�W�����G�V��@wF�-?n�D�<} 
> ����z���yV���U�e�ׂ� 
> +���qΖ_ľgj�瀆�ՋÄY��My0�
�%4��/ 
> A�X���h�q`j�E���v���gw��I�ņJy�KE 
> �[��Gʓ���] 
> *��A�x���&�_G�A&���֒�������7/ 
> ��#_*R�W
> <type 'exceptions.ValueError'>: substring not found
>

This is quite disturbing, as it suggests that your Tahoe node is  
trying to parse corrupted or invalid data, but any such data should  
have been detected by integrity checks before parsing.

Now, the capability that we're using is this:

"URI:DIR2-RO:pgj2oefhipe2uzulz7kyvyfcbe: 
7xoalidet7oignxsnhra77ez7qxh6ro5humhumncabv4cghdkc2q"

(Note: no spaces or linebreaks between those quotes!)

When I do:

-------
curl http://127.0.0.1:8123/uri/URI%3ADIR2-RO% 
3Apgj2oefhipe2uzulz7kyvyfcbe% 
3A7xoalidet7oignxsnhra77ez7qxh6ro5humhumncabv4cghdkc2q/?t=json
-------

(Note: no line breaks between those "-------", and only one space:  
between "curl" and the URL.)

I get the directory contents in JSON form which (with its line-breaks  
removed by hitting "J" in vi) looks like this:

[ "dirnode", { "ro_uri": "URI:DIR2-RO:pgj2oefhipe2uzulz7kyvyfcbe: 
7xoalidet7oignxsnhra77ez7qxh6ro5humhumncabv4cghdkc2q", "children":  
{ "wiki.html": [ "filenode", { "metadata": { "ctime":  
1209507649.88223, "mtime": 1209507973.6173389 }, "ro_uri":  
"URI:CHK:rcfs25cqwnknhhvfv6lvtjujfy:tugmdpcsxjaglvz74baq6ehl7su6sbwxzjsd 
sf23ahihth6gko7a:3:10:313468", "size": 313468 } ] } } ]

When you do the equivalent on your machine, you get the above parse  
error.  Scary!

Now we can change the capability by replacing "DIR2-RO to "SSK-SO",  
and then leave off the '?t=json' part of the URL, and this gets the  
unparsed contents of the directory:

-------
curl http://127.0.0.1:8123/uri/URI%3ASSK-RO% 
3Apgj2oefhipe2uzulz7kyvyfcbe% 
3A7xoalidet7oignxsnhra77ez7qxh6ro5humhumncabv4cghdkc2q
-------

The length of that file is 332 bytes, the sha1sum is  
e26203d9711a8889d2ab02405fd0bf9663790a16, and the contents are  
attached and appended after "-------".

Could you please run the equivalent curl command to fetch the  
directory data and tell me its length and sha1sum?  Note that the  
stack trace that you already posted shows a value for the data field  
which does not look like the data that I get from fetching that URL:

> data��"�>/�w��T�GsB���f�����n\A ZӂƟ| 
> t[Yh��
Yk�/T�)�!���Xc�#�k��}��˨^  
> XZ���㧿)����>w� ������o��E<7Σ 
> \"JI�s��,
�KXy�����v 
> +�D5k��L�W�����G�V��@wF�-?n�D�<} 
> ����z���yV���U�e�ׂ� 
> +���qΖ_ľgj�瀆�ՋÄY��My0�
�%4��/ 
> A�X���h�q`j�E���v���gw��I�ņJy�KE 
> �[��Gʓ���] 
> *��A�x���&�_G�A&���֒�������7/ 
> ��#_*R�W

Thanks!

Regards,

Zooko

-------------- next part --------------
A non-text attachment was scrubbed...
Name: directory.data
Type: application/octet-stream
Size: 332 bytes
Desc: not available
Url : http://allmydata.org/pipermail/tahoe-dev/attachments/20080505/407c6790/attachment.obj 
-------------- next part --------------



-------
327:9:wiki.html, 
99:URI:CHK:rcfs25cqwnknhhvfv6lvtjujfy:tugmdpcsxjaglvz74baq6ehl7su6sbwxzj 
sdsf23ahihth6gko7a:3:10:313468,147:M??$u<q??8????? 
???V=*?^8b>?S99mI??????_?xw?eng?[???ZkR???w
????*H?????C?]@????'??;1??L????v#9A????K?}?;@?=???? H???? 
?3??c????0??Q4?????,56:{"ctime": 1209507649.88223, "mtime":  
1209507973.6173389},,