#1111 closed enhancement (invalid)

PyFilesystem backend

Reported by: zooko Owned by: zooko
Priority: major Milestone: soon
Component: contrib Version: 1.7.0
Keywords: pyfilesystem fuse test-needed docs dokan Cc: slush
Launchpad Bug:

Description

PyFilesystem is a python library to abstract various sorts of filesystemish things:

http://code.google.com/p/pyfilesystem/

It just got included in Debian:

http://packages.debian.org/sid/python-fs

This ticket is a wish that someone would make a Tahoe-LAFS backend for PyFilesystem and contribute it to PyFilesystem.

Change History (14)

comment:1 Changed at 2010-07-18T01:47:23Z by warner

Looks pretty cool. From what I can tell, PyFilesystem is entirely synchronous, so the tahoe backend would really be a wrapper around the httplib-based webapi interfaces like those used in tahoe ls and tahoe get (as opposed to something you might use inside the Tahoe client node process). When you instantiate a tahoe-backed FS object, the arguments would probably be a nodeurl and a rootcap.

Neat: PyFilesystem has code to expose the FS you create to FUSE and SFTP, among other frontends.

comment:2 Changed at 2011-01-27T08:12:33Z by zooko

  • Cc slush added
  • Owner set to slush

Hey look! slush (marek@…) did this!

http://code.google.com/p/pyfilesystem/issues/detail?id=28

I haven't had time to read through the whole ticket or the code yet. This is cool! Has anyone used it for anything?

comment:3 follow-up: Changed at 2011-01-27T08:12:41Z by zooko

  • Resolution set to fixed
  • Status changed from new to closed

comment:4 in reply to: ↑ 3 Changed at 2011-01-27T21:43:02Z by slush

Has anyone used it for anything?

Ehm, me :-). I'm using it with Dokan (virtual drive for Windows) as my main interface for TahoeLAFS.

Those four lines does the trick:

from fs.contrib.tahoefs import TahoeFS
from fs.expose import dokan
fs = TahoeFS('URI:DIR2:<your dircap>')
mp = dokan.mount(fs, "f", foreground=True)
Last edited at 2011-01-27T21:43:39Z by slush (previous) (diff)

comment:5 Changed at 2011-01-27T22:47:55Z by zooko

Awesome.

comment:6 Changed at 2011-01-28T00:28:59Z by davidsarah

  • Keywords pyfilesystem fuse test-needed docs added
  • Milestone changed from undecided to soon
  • Resolution fixed deleted
  • Status changed from closed to reopened

This is way cool! But it needs to be reviewed for security, correctness, test coverage, etc., so I'm reopening the ticket for that.

comment:7 Changed at 2011-01-28T05:08:24Z by zooko

I imagine that it is the responsibility of the maintainers of pyfilesystem to ensure that. However, I would be happy to contribute some help. Maybe we should ask them if they want such help, and if so open a ticket on the pyfilesystem issue tracker to track whether the patch has tests and review?

comment:8 Changed at 2011-01-28T21:26:21Z by slush

TahoeFS is using standard PyFS test coverage (http://code.google.com/p/pyfilesystem/source/browse/trunk/fs/contrib/tahoefs/test_tahoefs.py) and pass it without any troubles on VolunteerGrid. I switched off only those tests which require atomic file operations, which is not available on TahoeLAFS. I'm open to any code review and improvements. I enjoyed hacking TahoeLAFS and PyFS together during my summer holiday in Italy, so maybe good wine and beautiful Italian women affected quality of final code ;).

comment:9 follow-up: Changed at 2011-01-28T21:30:38Z by slush

If PyFS need some help, it is better RemoteFileBuffer?, which manage loading and storing files to remote filesystems (in general, not only TahoeLAFS). Opening large remotte files directly from applications (movie) is still with troubles, because seeking in remote file needs loading full content before. I have improvements of this in my head, but time is my problem now.

comment:10 in reply to: ↑ 9 Changed at 2011-01-29T00:38:58Z by davidsarah

Replying to slush:

If PyFS need some help, it is better RemoteFileBuffer?, which manage loading and storing files to remote filesystems (in general, not only TahoeLAFS). Opening large remotte files directly from applications (movie) is still with troubles, because seeking in remote file needs loading full content before. I have improvements of this in my head, but time is my problem now.

(RemoteFileBuffer source is here.)

The SFTP frontend has OverwriteableFileConsumer which is doing essentially the same thing. Like RemoteFileBuffer, reading a chunk waits until all of the data up to the end of that chunk has been downloaded; segments are never downloaded out-of-order. Unlike RemoteFileBuffer, it supports writing chunks of the file that haven't been read yet.

Out-of-order downloads would require use of HTTP range requests, if the connection to the web-API gateway is over HTTP.

Version 0, edited at 2011-01-29T00:38:58Z by davidsarah (next)

comment:11 Changed at 2011-04-19T20:21:33Z by zooko

  • Owner changed from slush to davidsarah
  • Status changed from reopened to new

What's the status of this ticket? I say we close this as "fixed", or perhaps as "this was never Tahoe-LAFS's job in the first place". :-)

If there are other changes that we could make in Tahoe-LAFS to improve the behavior or performance or testability of the PyFilesystem backend, we can open a new ticket for that. If there are changes that we could make in PyFilesystem to improve things then we could open a ticket in PyFilesystem's issue tracker.

comment:12 Changed at 2011-06-01T11:50:47Z by zooko

  • Owner changed from davidsarah to zooko
  • Status changed from new to assigned

comment:13 Changed at 2011-09-02T20:07:02Z by zooko

  • Keywords dokan added

comment:14 Changed at 2011-09-28T20:14:12Z by zooko

  • Resolution set to invalid
  • Status changed from assigned to closed

Well, Tahoe-LAFS appears to be compatible with pyfilesystem. Closing this ticket as invalid because it didn't really require any changes to Tahoe-LAFS as such. If anybody wants to cooperate with us on improving Tahoe-LAFS+pyfilesystem integration (for example, setting up a buildbot to run automated tests!), just get in touch!

Note: See TracTickets for help on using tickets.