[tahoe-dev] tahoe now has an FTP server!

[Brian Warner]

Just an update: I was able to get the FTP server code working and
pushed into Tahoe trunk (ticket #512), and it seems to work pretty well.

The limitation is that it needs a patch to Twisted, otherwise the uploading
'PUT' command will finish before the file has actually been uploaded and
linked. Without such a patch, most advanced FTP clients will probably be
unhappy, since they tend to list the directory as soon as the PUT finishes,
and if they don't see the file that they think they just uploaded, they'll be
confused.

On the other hand, it doesn't need a special version of Twisted.. the FTP
server code has been around since Twisted-2.4.0 .

There are two user-handling mechanisms available. The first is a simple
flat-file, with lines of username/password/rootcap, which is quite suitable
for local deployment. The second mechanism uses a web-service that is
unsurprisingly similar to the allmydata.com authentication scheme, which
submits a username/password pair to a database-checking script named
"native_client.php", and gets back a rootcap.

We're currently testing this against a number of FTP clients. Typical BSD
ftp, ncftp, and a couple of Mac ftp clients seem ok so far. Windows will
mount an FTP server as a regular virtual drive, and that appears to work
pretty well. The Mac OS-X Finder will mount an FTP server as a virtual drive,
but read-only, and only if there is no "@" in the username. Since
allmydata.com sues email addresses as usernames, this is a bit of a drag.
We're still investigating.

Rob Kinninmont asked:

> what does it take to get from there to sftp?

Not as much work as one might think, and too much work to get done this week.

Twisted's FTP code is split into two pieces. The lower-layer
protocol-handling stuff is in twisted/protocols/ftp.py, and requires that you
give it an object which implements IFTPShell (with methods like list(),
makeDirectory(), and openForReading() ). I wrote an implementation of
IFTPShell that uses Tahoe client calls on the backend.. that's the one in
Tahoe trunk now.

As Rob pointed out, there is a higher-layer handler in Twisted, in
twisted/vfs/adapters/ftp.py, which wants an object that implements
IFileSystem. There are adapters here for both FTP and SFTP, so if we had a
class that provided IFileSystem and used Tahoe calls on the backend, then we
could support both protocols.

If I'd looked more closely at the split, I might have written an IFileSystem
class instead of an IFTPShell class. On the other hand, it turns out I was
wrong about the FTP server support not being in the latest Twisted release..
the FTP server in twisted has been around forever, at least as far back as
2.4.0. It's the VFS component that is not yet in a release, but we're not
using that.

On the other other hand, to get FTP uploads to wait until the backend upload
has actually finished, we need a patch to the FTP code. So, while our
limitation to FTP (and not SFTP) means we don't need Twisted SVN-HEAD, our
desire to have 'PUT' wait until the file is actually uploaded and linked
means that we need to run this code against a patched Twisted-8.1.0 (or any
other suitable release).

I plan to look into the SFTP/vfs/IFileSystem support soon, maybe next week,
since I would like us to have SFTP support. I'd like to avoid duplicating
code with the FTP layer, but on the other hand I don't want to require an
unreleased version of Twisted to use just the FTP service. Once the next
Twisted release happens (particularly if I can convince someone to review my
wait-for-upload patch), things will be a lot better.


cheers,
 -Brian