investigate best FUSE+sshfs options to use for performance and correctness of SFTP via sshfs

[davidsarah]

It looks as though at least direct_io and big_writes may be beneficial, so that writes are not limited to 4 KiB blocks.

[bj0]

I started doing a couple random tests I could think of. I didn't repeat many of the tests, since they took a bit of time and it's all manual, but it sort of gives an idea:

Most of these tests were copying a large file (728MB .iso file) to and from a tahoe introducer/storage client running inside a VirtualBox? VM on the same host (both guest and host running ubuntu). Most of the copying was done with "time rsync -rhPa ", and when copying a large file to tahoe, after the transfer the command hangs for another minute or so. I checked the flog during this time and there was activity (read i think) so it may be that rsync tries to checksum the file after transfer, i'm not sure.

To verify the file was being transfered correctly, I also did "time md5sum mnt/iso". I would expect this to be similar to simply reading the file, but for some reason it performed differently...

with it mounted as: "sshfs -p PORT server:/ mnt/"

• rsync .iso TO tahoe: 4m51s to transfer ( 2.39 MB/s ), 6m14s total
• rsync .iso FROM tahoe: 5m42s ( 2.13 MB/s )
• cp .iso FROM tahoe: 5m31s
• time md5sum iso:
  • first try: 7m46s
  • second try: 5m13s

with it mounted as: "sshfs -p PORT -o direct_io,big_writes server:/ mnt/"

• rsync .iso TO tahoe: 3m37s to transfer ( 3.19 MB/s ), 5m1s total
• rsync .iso FROM tahoe: 5m15s ( 2.31 MB/s )
• cp .iso FROM tahoe: 5m31s
• time md5sum iso:
  • first try: 10m40s
  • second try: 9m25s

obviously I expect the values to fluctuate a bit, but it seems like direct_io,big_writes bumps up the write speed a bit, without really affecting the read speed. I'm not really sure why it hit md5sum time so bad...

I also tried to rsync a large directory of source files (4mb, 981 files) to tahoe, but it seems to be acting odd, and stalls a lot, resulting in a very long transfer time (120 - 140 minutes). This happened with and without the options.

[zooko]

Dear bj0: thanks for the report!

What version(s) of Tahoe-LAFS were you using? If you have just been tracking the official trunk repo at ​http://tahoe-lafs.org/source/tahoe-lafs/trunk and haven't applied any other patches, then you can find out by running make make-version.

[davidsarah]

Thanks bj0.

big_writes should only affect writes, and I can't immediately see why it would have anything but a beneficial effect. direct_io might affect both reads and writes, and could cause some loss of performance for applications whose performance depends on kernel caching. Can you try the same tests with -o big_writes only?

[I initially suggested both because ​http://xtreemfs.blogspot.com/2008/08/fuse-performance.html said that direct_io was needed (at least for some version of sshfs FUSE and Linux tested in 2008) to support writing in blocks greater than 4 KiB. However, point 2 in ​http://article.gmane.org/gmane.comp.file-systems.fuse.devel/5292 suggests that this restriction might have been lifted.]

What Linux kernel version and sshfs version did you use?

[bj0]

make make-version returned: setup.py darcsver: wrote '1.8.0c2-r4702' into src/allmydata/_version.py

sshfs Version: 2.2-1build1 (from ubuntu repo)

client uname -a: Linux nazgul 2.6.32-020632-generic #020632 SMP Thu Dec 3 10:09:58 UTC 2009 x86_64 GNU/Linux server uname -a: Linux testbuntu 2.6.32-24-generic #41-Ubuntu SMP Thu Aug 19 01:12:52 UTC 2010 i686 GNU/Linux

I was going to try the tests without direct_io, but I seem to be having trouble with my vm...