Changes between Initial Version and Version 1 of Tutorial

Timestamp:

2010-08-04T18:29:24Z (15 years ago)

Author:

guyzmo

Comment:

--

Tutorial

@@ +1 @@
+{{{
+                ===============================
+                Tutorial for Tahoe-LAFS (Draft) 
+                ===============================
+}}}
+= Hosts setup =
+
+In this tutorial we're gonna setup 3 nodes, whatever they are (Host+VM1+VM2)
+
+IPs and hosts are:
+ * 172.16.23.1 (host.local : Mac OS X 10.6.4)
+ * 172.16.23.128 (vm1.local : Debian GNU/Linux 5.0)
+ * 172.16.23.130 (vm2.local : Debian GNU/Linux 5.0)
+
+I'll use a self-explanatory prompt for each host.
+
+= Installation on each node =
+
+I like pypi so much I find it easier to do the download, extraction, 
+get dependencies and build using it.
+{{{
+ # easy_install -U allmydata-tahoe
+}}}
+it failed on the freshly installed debian, with that error :
+{{{
+ ImportError: No module named darcsver.setuptools_command
+}}}
+A fast look on google, showed that installing setuptools_darcs and 
+darcsver don't build as expected and may be missing at install time. 
+The fix is to install them first, and only then install tahoe.
+{{{
+ # easy_install -U setuptools_darcs darcsver
+ # easy_install -U allmydata-tahoe
+}}}
+then execute tahoe --vesion to check everything is installed as
+expected :
+
+on debian :
+{{{
+ % tahoe --version
+allmydata-tahoe: 1.7.1, foolscap: 0.5.1, pycryptopp: 0.5.17-r683, zfec: 1.4.7, Twisted: 10.0.0, Nevow: 0.10.0, zope.interface: 3.5.1, python: 2.6.1, platform: Darwin-10.4.0-i386-64bit, sqlite: 3.6.12, simplejson: 2.1.1, argparse: 1.1, pycrypto: 2.1.0, pyOpenSSL: 0.7, pyutil: 1.7.7, zbase32: 1.1.1, setuptools: 0.6c15dev, pyasn1: 0.0.11a, pysqlite: 2.4.1
+}}}
+on MacOSX :
+{{{
+ % tahoe --version
+allmydata-tahoe: 1.7.1, foolscap: 0.5.1, pycryptopp: 0.5.19, zfec: 1.4.7, Twisted: 10.0.0, Nevow: 0.10.0, zope.interface: 3.6.1, python: 2.5.2, platform: Linux-debian_5.0-x86_64-64bit, sqlite: 3.5.9, simplejson: 2.1.1, argparse: 1.1, pycrypto: 2.1.0, pyOpenSSL: 0.7, pyutil: 1.7.7, zbase32: 1.1.1, setuptools: 0.6c12dev, pyasn1: 0.0.11a, pysqlite: 2.3.2
+}}}
+now Tahoe is installed \o/
+
+= Instanciate the nodes =
+
+So, after reading running.html, configuration.txt and architecture.txt, we'll first 
+create an introducer node, that will get every other nodes to know each other. The
+documentation says that we have to create and launch the introducer as follows (that
+we do on vm1.local)
+{{{
+ vm1% mkdir tahoe-introducer
+ vm1% tahoe create-introducer tahoe-introducer
+ Introducer created in tahoe-introducer
+ vm1% tahoe start tahoe-introducer
+ STARTING /home/guyzmo/tahoe-introducer
+ introducer node probably started
+ vm1% cat tahoe-introducer/introducer.furl 
+ pb://6oibvp5whrb3v3ge7ajuyapooswsda2e@172.16.23.128:33272,127.0.0.1:33272/introducer
+}}}
+Then we copy that last string we got from 'cat', and paste it in tahoe.cfg,
+as follows (edit it using your favorite editor, mine is vim) :
+{{{
+ vm2% vim ~/.tahoe/tahoe.cfg
+-------------8<----------------------8<----------------
+ 22 [client]
+ 23 introducer.furl = pb://6oibvp5whrb3v3ge7ajuyapooswsda2e@172.16.23.128:33272,127.0.0.1:33272/introducer
+ 24 helper.furl =
+ 25 #key_generator.furl =
+ 26 #stats_gatherer.furl =
+ 27 #shares.needed = 3
+ 28 #shares.happy = 7
+ 29 #shares.total = 10
+------------->8---------------------->8----------------
+}}}
+exit, and then do the same on 'host'.
+
+finally, execute on 'vm2' and 'host' the following :
+{{{
+ vm2% tahoe start
+STARTING /home/guyzmo/.tahoe
+client node probably started
+
+ host% tahoe start
+STARTING /home/guyzmo/.tahoe
+client node probably started
+}}}
+wait a bit and do :
+{{{
+ % ps aux | grep tahoe
+myuser    5360  1.2  7.4 158820 28348 ?        S<   02:13   0:00 /usr/bin/python /usr/bin/twistd -y tahoe-client.tac --logfile logs/twistd.log
+}}}
+and you shall see an output that looks alike. If not, tahoe failed to start.
+You've got to check ~/.tahoe/logs/ what got wrong.
+
+= Configure the nodes =
+
+Now, let's do some configuration. On each node edit tahoe.cfg :
+{{{
+ host% vim ~/.tahoe/tahoe.cfg
+-------------8<----------------------8<----------------
+ 10 [node]
+ 11 nickname = client-host
+ 12 web.port = tcp:3456:interface=172.16.23.1
+------------->8---------------------->8----------------
+ host% tahoe restart
+}}}
+
+{{{
+ vm1% vim ~/tahoe-introducer/tahoe.cfg
+-------------8<----------------------8<----------------
+ 10 [node]
+ 11 nickname = introducer-vm1
+ 12 web.port = tcp:3456:interface=172.16.23.128
+------------->8---------------------->8----------------
+ vm1% tahoe restart ~/tahoe-introducer
+
+ vm2% vim ~/.tahoe/tahoe.cfg
+-------------8<----------------------8<----------------
+ 10 [node]
+ 11 nickname = client-vm2
+ 12 web.port = tcp:3456:interface=172.16.23.130
+------------->8---------------------->8----------------
+ vm2% tahoe restart
+}}}
+
+Then get your favorite browser and browse to the following URLs, you
+should get pages similar to the ones below
+
+  * http://172.16.23.1:3456/      -> http://m0g.net/~guyzmo/172.16.23.1_3456_1.png
+  * http://172.16.23.128:3456/    -> http://m0g.net/~guyzmo/172.16.23.128_3456_1.png
+  * http://172.16.23.130:3456/    -> http://m0g.net/~guyzmo/172.16.23.130_3456_1.png
+
+Now it's time to be able to store something on the grid, as
+the whole purpose of Tahoe-LAFS, and thus of this tutorial is
+to get a grid to store files.
+
+On host and vm2, edit again ~/.tahoe/tahoe.cfg and modify it
+as follows :
+
+{{{
+ vm2% vim ~/.tahoe/tahoe.cfg
+-------------8<----------------------8<----------------
+ 31 [storage]
+ 32 enabled = true
+ 33 readonly = false
+ 34 reserved_space = 1GB # here you tell the storage server how much space you would like to use
+------------->8---------------------->8----------------
+ vm2% tahoe restart
+}}}
+
+and the same on host.
+
+The result would be as follows :
+
+   * http://172.16.23.1:3456/      -> http://m0g.net/~guyzmo/172.16.23.1_3456_2.png
+   * http://172.16.23.128:3456/    -> http://m0g.net/~guyzmo/172.16.23.128_3456_2.png
+   * http://172.16.23.130:3456/    -> http://m0g.net/~guyzmo/172.16.23.130_3456_2.png
+
+Then, you've got a working Tahoe-LAFS grid working !
+Is that all ? Well, no, it's time to learn how to use it.
+
+= File capabilities ? =
+
+Now it's time for a bit of litterature.
+
+TODO
+
+= Access your shares =
+
+You can store and access files on Tahoe-LAFS using a lot of 
+different ways. The fatest is to either use the Web UI, or
+the CLI though you need to remember the caps of the files.
+
+Using the WUI is straight forward, connect to a storage client,
+use the forms to add/get/remove files and directories. Just
+always remember that once you created a directory, you need to
+save the file's URI, or you won't be able to find it again later
+on.
+
+== Using the CLI ==
+
+The CLI behaves mostly like a standard shell, with all
+commands prefixed by 'tahoe', the same way version control
+system work.
+
+If you execute for the first time :
+
+{{{
+ % tahoe ls
+error: No alias specified, and the default 'tahoe' alias doesn't exist. To create it, use 'tahoe create-alias tahoe'.
+}}}
+
+you'll get that error. That's why you need to create a 
+'root' directory for CLI commands that will be aliased
+to the 'tahoe:' prefix.
+
+{{{
+ % tahoe create-alias tahoe
+[...]
+socket.error: [Errno 61] Connection refused
+}}}
+
+What happens here is that the Tahoe client tries to connect to the URL 
+given in the ~/.tahoe/node.url file. So, you need to edit ~/.tahoe/node.url
+{{{
+ host% vim ~/.tahoe/node.url
+-------------8<----------------------8<----------------
+ 1 http://172.16.23.1:3456/
+------------->8---------------------->8----------------
+}}}
+{{{
+ vm2% vim ~/.tahoe/node.url
+-------------8<----------------------8<----------------
+ 1 http://172.16.23.1:3456/
+------------->8---------------------->8----------------
+}}}
+
+HINT: each time you do 'tahoe start' node.url gets reset to 127.0.0.1. If you 
+      want to intensively use the CLI, change 
+          'web.port = tcp:3456:interface=172.16.23.1'
+      to 
+          'web.port = tcp:3456'
+      in tahoe.cfg. Specifying the interface to listen on is usually a good security
+      policy, though twisted supports only one interface declaration, or all interfaces
+      if you don't declare it.
+
+Now, you can run the previous command :
+
+{{{
+ % tahoe create-alias tahoe
+Alias 'tahoe' created
+}}}
+
+What it has actually done, is that it edited the ~/.tahoe/private/aliases file
+linking the keyword you gave with a file cap :
+
+{{{
+ vm2% cat ~/.tahoe/private/aliases
+tahoe: URI:DIR2:bfe7inwcduobfnnrxocqechr4q:huurbpkrtajzdrq6okvxxppj5negccz4payh6gxes5nphe6djbeq
+}}}
+and
+{{{
+ host% cat ~/.tahoe/private/aliases
+tahoe: URI:DIR2:ozgebrm5oe52hhcnxhbc6oy6xu:7cyu73izd6q6w4kwngyxxdu7ttuzwy5kl2uybcrqohhen362crla
+}}}
+so, as you can see, different tahoe CLI won't work in the same base directories.
+If you really want to enable this, copy the same aliases file on every account
+you want to use the CLI on.
+
+Finally, to use the CLI, have a look at 'tahoe help' for a full description
+of the commands.
+
+For example, here are some basic commands:
+
+{{{
+ % tahoe cp README.txt tahoe:
+ Success: file copied
+ % tahoe ls -l tahoe:
+ -r-- 1348 Aug 04 12:52 README.txt
+ % tahoe get tahoe:README.txt | head
+ ===============================
+ Tahoe-LAFS
+ ===============================
+
+ Tahoe-LAFS is a Free Software/Open Source decentralized data store. It
+ distributes your filesystem across multiple servers, and even if some of the
+ servers fail or are taken over by an attacker, the entire filesystem continues
+ to work correctly and to preserve your privacy and security.
+ 
+ To get started please see `quickstart.html`_.
+}}}
+
+== Using the FTP interface ==
+
+As Tahoe does not have user accounts management, to enable FTP access,
+we need first to enable authentication and associate root caps to each
+user allowed to access the system.
+
+/!\ Be sure to have installed twisted version >=10.1 before trying FTP/sFTP 
+interface, tahoe's ftp feature is not compatible with prior versions.
+
+For more informations about FTP/sFTP configuration refer to the documentation:
+ * http://tahoe-lafs.org/source/tahoe-lafs/trunk/docs/frontends/FTP-and-SFTP.txt
+ * http://tahoe-lafs.org/trac/tahoe-lafs/wiki/SftpFrontend
+
+=== Account file creation ===
+
+The password file is using a very simple format, each non blank/commented
+line is following this template :
+{{{
+ USER PASSWORD ROOTCAP
+}}}
+so we're gonna setup two users the following way on host:
+{{{
+ host% tahoe create-alias alice
+ Alias 'alice' created
+ host% tahoe create-alias bob
+ Alias 'bob' created
+ host% cat ~/.tahoe/private/aliases
+ tahoe: URI:DIR2:ozgebrm5oe52hhcnxhbc6oy6xu:7cyu73izd6q6w4kwngyxxdu7ttuzwy5kl2uybcrqohhen362crla
+ alice: URI:DIR2:dhl2wcuoppuirtv3gk2lslhx6i:nur5sar5lwfxal63gcjaysu4mxt33l35uw4bqtokby3airqhwehq
+ bob: URI:DIR2:r2dwmysrl4wigiov52fmifs2mi:2fdu2l3jxuqupepefmkd3y5vftqatc62vl6elmb5qlgi3gmvpx6a
+ host% cat > ~/.tahoe/private/ftp.accounts
+ alice h3rp4ssw0rd URI:DIR2:dhl2wcuoppuirtv3gk2lslhx6i:nur5sar5lwfxal63gcjaysu4mxt33l35uw4bqtokby3airqhwehq
+ bob h1sp4ssw0rd URI:DIR2:r2dwmysrl4wigiov52fmifs2mi:2fdu2l3jxuqupepefmkd3y5vftqatc62vl6elmb5qlgi3gmvpx6a
+ host% 
+}}}
+and two users on vm2:
+{{{
+ vm2% tahoe create-alias alice
+ Alias 'alice' created
+ vm2% tahoe add-alias bob URI:DIR2:r2dwmysrl4wigiov52fmifs2mi:2fdu2l3jxuqupepefmkd3y5vftqatc62vl6elmb5qlgi3gmvpx6a
+ vm2% cat ~/.tahoe/private/aliases
+ tahoe: URI:DIR2:ozgebrm5oe52hhcnxhbc6oy6xu:7cyu73izd6q6w4kwngyxxdu7ttuzwy5kl2uybcrqohhen362crla
+ alice: URI:DIR2:x3y3i3s6h5l7gspse6ol4b7goy:zinwdjlffv23gbv5umyjqcdg3trfmvxdwc77cwae3cpxw6mhtq5a
+ bob: URI:DIR2:r2dwmysrl4wigiov52fmifs2mi:2fdu2l3jxuqupepefmkd3y5vftqatc62vl6elmb5qlgi3gmvpx6a
+ vm2% cat > ~/.tahoe/private/ftp.accounts
+ alice h3rp4ssw0rd URI:DIR2:x3y3i3s6h5l7gspse6ol4b7goy:zinwdjlffv23gbv5umyjqcdg3trfmvxdwc77cwae3cpxw6mhtq5a
+ bob h1sp4ssw0rd URI:DIR2:r2dwmysrl4wigiov52fmifs2mi:2fdu2l3jxuqupepefmkd3y5vftqatc62vl6elmb5qlgi3gmvpx6a
+ vm2% 
+}}}
+
+If you look carefully to both logs, alice gets two root caps on each target host,
+while bob is getting the same root cap on both. Of course, the alias creation is 
+optional, but handy for the grid manager to control what the users have.
+
+now, let's update the configuration :
+{{{
+ % vim ~/.tahoe/tahoe.cfg
+-------------8<----------------------8<----------------
+  22 [ftpd]
+  23 enabled = true
+  24 port = 8021
+  25 accounts.file = private/ftp.accounts
+------------->8---------------------->8----------------
+ % tahoe restart
+ STOPPING /home/guyzmo/.tahoe
+ process 3063 is dead
+ STARTING /home/guyzmo/.tahoe
+ client node probably started
+}}}
+
+and then, connect your FTP client to tahoe's FTP server:
+
+{{{
+ % tahoe cp foobar.jpg bob:
+Success: files copied 
+ % lftp bob:h1sp4ssw0rd@172.16.23.1:8021
+ lftp bob@172.16.23.1:/> ls
+ -rw-------   1 bob       bob                857648 Jan 01  1970 foobar.jpg
+ lftp bob@172.16.23.1:/> exit
+ % lftp alice:h3rp4ssw0rd@172.16.23.1:8021
+ lftp alice@172.16.23.1:~> put cliclac.jpg 
+ 857648 bytes transferred                           
+ lftp alice@172.16.23.1:~> ls
+ -rw-------   1 alice     alice              857648 Jan 01  1970 foobar.jpg
+ lftp alice@172.16.23.1:/> exit
+ % lftp bob:h1sp4ssw0rd@172.16.23.130:8021
+ lftp bob@172.16.23.130:~> ls
+ -rw-------   1 bob       bob                857648 Jan 01  1970 foobar.jpg
+ lftp bob@172.16.23.130:/> exit
+ % lftp alice:h3rp4ssw0rd@172.16.23.130:8021
+ lftp alice@172.16.23.130:~> ls
+ lftp alice@172.16.23.130:/> exit
+}}}
+
+et voilà !
+
+== Configure sFTP access ==
+
+The procedure is slightly the same as for ftp access. You need to have the
+ftp.accounts files placed in ~/.tahoe/private/, and we'll consider you already
+have the one written in the previous section.
+
+/!\ As for FTP access, you need to have twisted version >=10.1.0 installed
+before trying to use tahoe's sFTP feature.
+
+{{{
+ % ssh-keygen -f ~/.tahoe/private/ssh_host_rsa_key
+ Generating public/private rsa key pair.
+ Enter passphrase (empty for no passphrase): 
+ Enter same passphrase again: 
+ Your identification has been saved in /home/guyzmo/.tahoe/private/ssh_host_rsa_key.
+ Your public key has been saved in /home/guyzmo/.tahoe/private/ssh_host_rsa_key.pub.
+ The key fingerprint is:
+ 3f:a6:c7:38:19:3a:19:3c:ee:77:d5:5a:3d:2f:86:6c guyzmo@vm2
+ The key's randomart image is:
+ +--[ RSA 2048]----+
+ |                 |
+ |                 |
+ |                 |
+ |                 |
+ |     .  S    . . |
+ |      + ..  . o..|
+ |     . = =+o +  o|
+ |      = =o+.E o .|
+ |     ..o.+ . . . |
+ +-----------------+
+ % vim ~/.tahoe/tahoe.cfg
+-------------8<----------------------8<----------------
+  27 [sftpd]
+  28 enabled = true
+  29 port = 8022
+  30 host_pubkey_file = private/ssh_host_rsa_key.pub
+  31 host_privkey_file = private/ssh_host_rsa_key
+  32 accounts.file = private/ftp.accounts
+------------->8---------------------->8----------------
+ % tahoe restart
+ STOPPING /home/guyzmo/.tahoe
+ process 3101 is dead
+ STARTING /home/guyzmo/.tahoe
+ client node probably started
+ % mkdir ./mnt
+ % mount_sshfs -p 8022 bob@172.16.23.1:/ ./mnt/
+ % ls ./mnt/
+ total 1676
+ -rw-rw-rw-  luser  staff  857648 Aug  4 17:37 foobar.jpg
+}}}
+
+Now, you have a 3 nodes grid running with 1 introducer, 2 storage servers
+and 4 ways to access your files : a WUI, a CLI and accounts for FTP/sFTP 
+access.