TypeError due to "size" being None in pyfilesystem+dokan

[zooko]

I saw this bug report to pyfilesystem:

​http://code.google.com/p/pyfilesystem/issues/detail?id=96

tahoe-lafs 1.9.0 + pyfilesystem 0.4 (and svn r723) + dokan 0.6.0
OS Windows 7 x64 Russian + Python 2.7.2 x86

Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 313, in 'calling callback function'
  File "C:\Program Files (x86)\Python27\lib\site-packages\fs\expose\dokan\__init__.py", line 256, in wrapper
    return func(self,*args)
  File "C:\Program Files (x86)\Python27\lib\site-packages\fs\expose\dokan\__init__.py", line 173, in wrapper
    res = func(*args,**kwds)
  File "C:\Program Files (x86)\Python27\lib\site-packages\fs\errors.py", line 191, in wrapper
    return func(*args,**kwds)
  File "C:\Program Files (x86)\Python27\lib\site-packages\fs\expose\dokan\__init__.py", line 604, in FindFilesWithPattern
    data = self._info2finddataw(fpath,finfo,None)
  File "C:\Program Files (x86)\Python27\lib\site-packages\fs\expose\dokan\__init__.py", line 794, in _info2finddataw
    data.nFileSizeHigh = info.get("size",0) >> 32
TypeError: unsupported operand type(s) for >>: 'NoneType' and 'int'

As far as I know, pyfilesystem doesn't have a strong test suite, so it will require more effort from someone to figure out where exactly the problem lies in pyfilesystem, Tahoe-LAFS, dokan, or the interfaces between them...

[ClashTheBunny]

I'm thinking this is related to my same issue. I have a Tahoe-LAFS mounted through PyFS on Linux and I was using the same one earlier on Windows with docan. I saw similar things on docan, but didn't write them down. I mount the directory and do an 'ls'. This is the error I'm seeing on Linux:

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/fs/expose/fuse/fuse_ctypes.py", line 294, in _wrapper_
    return func(*args, **kwargs) or 0
  File "/usr/local/lib/python2.7/dist-packages/fs/expose/fuse/fuse_ctypes.py", line 425, in readdir
    set_st_attrs(st, attrs)
  File "/usr/local/lib/python2.7/dist-packages/fs/expose/fuse/fuse_ctypes.py", line 239, in set_st_attrs
    setattr(st, key, val)
TypeError: an integer is required

This is due to me having some MDMF files in the directory. When I unlink them from the directory, I get immediate results from my 'ls'. I think the problem is that MDMF files don't have a "size" always. In the JSON for the DIR:CAP, I have the file size listed as:

"size": null

When I run a "check/renew/verify every bit" on the MDMF, it spits out the size for a while and then the 'ls' works for a time.

The change to fix this should either be in TahoeLAFS to output something like 0 for a null sized file, or at least an integer, like "-1", or PyFS should be updated to check "size" for null and replace it with 0 or a "-1".

UPDATE:

I did get it to do it again on Windows, and it seems to be the same error:

Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 314, in 'calling callback function'
  File "C:\Python27\lib\site-packages\fs\expose\dokan\__init__.py", line 256, in
 wrapper
    return func(self,*args)
  File "C:\Python27\lib\site-packages\fs\expose\dokan\__init__.py", line 173, in
 wrapper
    res = func(*args,**kwds)
  File "C:\Python27\lib\site-packages\fs\errors.py", line 191, in wrapper
    return func(*args,**kwds)
  File "C:\Python27\lib\site-packages\fs\expose\dokan\__init__.py", line 604, in
 FindFilesWithPattern
    data = self._info2finddataw(fpath,finfo,None)
  File "C:\Python27\lib\site-packages\fs\expose\dokan\__init__.py", line 794, in
 _info2finddataw
    data.nFileSizeHigh = info.get("size",0) >> 32
TypeError: unsupported operand type(s) for >>: 'NoneType' and 'int'

Also, this may be a good idea to fix in the web interface. I don't know why size is ? for MDMF files when I look at a DIRCAP through the web interface. Is my file there? Is it corrupted? Is it just hard to calculate the size? Maybe a message other than "?" would be better

[davidsarah]

The '?' in the WUI is intended behaviour. It is too inefficient to calculate the size for every mutable file in a directory listing; it would require querying each file individually. Similarly, it's intended that the JSON output not include the actual file size. Representing the lack of a known file size by "size": null is dubious, though (even though it is valid JSON). We normally represent unknown information by omitting a field (which the pyFilesystem Tahoe-LAFS backend would be able to handle, using info.get("size",0)). For example, that's what we do for the "mutable" field on unknown nodes.

webapi.rst​ fails to document how an unknown file size is represented. tahoe_ls.py appears to assume that an unknown size is represented by omitting the "size" field. (So, I'm not sure why tahoe ls correctly prints "?" rather than "null" for mutable files. Perhaps "size": null is only included sometimes?)

In any case, the bug in the description doesn't seem to be related to the WUI. If we decide that omitting the "size" field is correct, then it's a bug in the web-API with JSON output; if we decide that "size": null is correct, it's a bug in the pyFilesystem Tahoe-LAFS backend. Either way, webapi.rst also needs to be clarified.

[davidsarah]

Replying to davidsarah:

We normally represent unknown information by omitting a field (which the pyFilesystem Tahoe-LAFS backend would be able to handle, using info.get("size",0)).

Correction: a missing "size" field will actually be coerced to 0 earlier, in this code: ​http://code.google.com/p/pyfilesystem/source/browse/trunk/fs/contrib/tahoelafs/util.py#79

[daira]

I'll accept this for 1.11. I intend to make it omit the "size" field.