Ticket #778: docs3.txt

Fri Dec  4 19:38:13 PST 2009  Kevan Carstensen <kevan@isnotajoke.com>
  * Update 'docs/configuration.txt' to reflect the servers_of_happiness behavior.

Fri Dec  4 19:40:05 PST 2009  "Kevan Carstensen" <kevan@isnotajoke.com>
  * Alter wording in 'interfaces.py' to be correct wrt #778

Tue Apr 27 17:24:55 PDT 2010  Kevan Carstensen <kevan@isnotajoke.com>
  * Alter the wording in docs/architecture.txt to more accurately describe the servers_of_happiness behavior.

Thu May 13 17:38:52 PDT 2010  kevan@isnotajoke.com
  * Update 'docs/architecture.txt' to reflect readonly share discovery

New patches:

[Update 'docs/configuration.txt' to reflect the servers_of_happiness behavior.
Kevan Carstensen <kevan@isnotajoke.com>**20091205033813
 Ignore-this: 5e1cb171f8239bfb5b565d73c75ac2b8
] hunk ./docs/configuration.txt 275
  larger than 256, because of the 8-bit erasure-coding algorithm that Tahoe
  uses.
 
- If servers are lost during an upload, shares.happy determines whether the
- upload is considered successful or not. If at least "shares.happy" shares
- were placed, the upload is declared a success, otherwise it is declared a
- failure. The default value is 7. This value must not be smaller than k nor
- larger than N.
+ shares.happy allows you control over the distribution of your file. An upload
+ is only considered successful if shares are placed on at least 'shares.happy'
+ distinct servers, the correct functioning of at least k of which is sufficient
+ to guarantee the availability of the uploaded file. This value should not be
+ larger than the number of servers on your grid.
 
 
 == Storage Server Configuration ==
[Alter wording in 'interfaces.py' to be correct wrt #778
"Kevan Carstensen" <kevan@isnotajoke.com>**20091205034005
 Ignore-this: c9913c700ac14e7a63569458b06980e0
] hunk ./src/allmydata/interfaces.py 1275
     def set_params(params):
         """Override the default encoding parameters. 'params' is a tuple of
         (k,d,n), where 'k' is the number of required shares, 'd' is the
-        shares_of_happiness, and 'n' is the total number of shares that will
+        servers_of_happiness, and 'n' is the total number of shares that will
         be created.
 
         Encoding parameters can be set in three ways. 1: The Encoder class
[Alter the wording in docs/architecture.txt to more accurately describe the servers_of_happiness behavior.
Kevan Carstensen <kevan@isnotajoke.com>**20100428002455
 Ignore-this: 6eff7fa756858a1c6f73728d989544cc
] {
hunk ./docs/architecture.txt 180
 up with as many shares as we desire.
 
 If we are unable to place every share that we want, but we still managed to
-place a quantity known as "shares of happiness", we'll do the upload anyways.
-If we cannot place at least this many, the upload is declared a failure.
+place a quantity known as "servers of happiness" that each map to a unique
+server, we'll do the upload anyways. If we cannot place at least this many
+in this way, the upload is declared a failure.
 
 The current defaults use k=3, shares_of_happiness=7, and N=10, meaning that
hunk ./docs/architecture.txt 185
-we'll try to place 10 shares, we'll be happy if we can place 7, and we need
-to get back any 3 to recover the file. This results in a 3.3x expansion
-factor. In general, you should set N about equal to the number of nodes in
-your grid, then set N/k to achieve your desired availability goals.
+we'll try to place 10 shares, we'll be happy if we can place shares on enough
+servers that there are 7 different servers, the correct functioning of any 3 of
+which guarantee the availability of the file, and we need to get back any 3 to
+recover the file. This results in a 3.3x expansion factor. On a small grid, you
+should set N about equal to the number of storage servers in your grid; on a
+large grid, you might set it to something smaller to avoid the overhead of
+contacting every server to place a file. In either case, you should then set k
+such that N/k reflects your desired availability goals. The correct value for
+servers_of_happiness will depend on how you use Tahoe-LAFS. In a friendnet with
+a variable number of servers, it might make sense to set it to the smallest
+number of servers that you expect to have online and accepting shares at any
+given time. In a stable environment without much server churn, it may make
+sense to set servers_of_happiness = N.
 
 When downloading a file, the current version just asks all known servers for
 any shares they might have. Once it has received enough responses that it
}
[Update 'docs/architecture.txt' to reflect readonly share discovery
kevan@isnotajoke.com**20100514003852
 Ignore-this: 7ead71b34df3b1ecfdcfd3cb2882e4f9
] {
hunk ./docs/architecture.txt 149
 set of all peer nodes (by sorting the peer nodes by
 HASH(storage_index+peerid)). Each file gets a different permutation, which
 (on average) will evenly distribute shares among the grid and avoid hotspots.
+We first remove any peer nodes that cannot hold an encoded share for our file,
+and then ask some of the peers that we have removed if they are already
+holding encoded shares for our file; we use this information later. This step
+helps conserve space, time, and bandwidth by making the upload process less
+likely to upload encoded shares that already exist.
 
hunk ./docs/architecture.txt 155
-We use this permuted list of nodes to ask each node, in turn, if it will hold
-a share for us, by sending an 'allocate_buckets() query' to each one. Some
-will say yes, others (those who are full) will say no: when a node refuses
-our request, we just take that share to the next node on the list. We keep
-going until we run out of shares to place. At the end of the process, we'll
-have a table that maps each share number to a node, and then we can begin the
-encode+push phase, using the table to decide where each share should be sent.
+We then use this permuted list of nodes to ask each node, in turn, if it will
+hold a share for us, by sending an 'allocate_buckets() query' to each one.
+Some will say yes, others (those who have become full since the start of peer
+selection) will say no: when a node refuses our request, we just take that
+share to the next node on the list. We keep going until we run out of shares
+to place. At the end of the process, we'll have a table that maps each share
+number to a node, and then we can begin the encode+push phase, using the table
+to decide where each share should be sent.
 
 Most of the time, this will result in one share per node, which gives us
 maximum reliability (since it disperses the failures as widely as possible).
}

Context:

[Dependency on Windmill test framework is not needed yet.
david-sarah@jacaranda.org**20100504161043
 Ignore-this: be088712bec650d4ef24766c0026ebc8
] 
[tests: pass z to tar so that BSD tar will know to ungzip
zooko@zooko.com**20100504090628
 Ignore-this: 1339e493f255e8fc0b01b70478f23a09
] 
[setup: update comments and URLs in setup.cfg
zooko@zooko.com**20100504061653
 Ignore-this: f97692807c74bcab56d33100c899f829
] 
[setup: reorder and extend the show-tool-versions script, the better to glean information about our new buildslaves
zooko@zooko.com**20100504045643
 Ignore-this: 836084b56b8d4ee8f1de1f4efb706d36
] 
[CLI: Support for https url in option --node-url
Francois Deppierraz <francois@ctrlaltdel.ch>**20100430185609
 Ignore-this: 1717176b4d27c877e6bc67a944d9bf34
 
 This patch modifies the regular expression used for verifying of '--node-url'
 parameter.  Support for accessing a Tahoe gateway over HTTPS was already
 present, thanks to Python's urllib.
 
] 
[backupdb.did_create_directory: use REPLACE INTO, not INSERT INTO + ignore error
Brian Warner <warner@lothar.com>**20100428050803
 Ignore-this: 1fca7b8f364a21ae413be8767161e32f
 
 This handles the case where we upload a new tahoe directory for a
 previously-processed local directory, possibly creating a new dircap (if the
 metadata had changed). Now we replace the old dirhash->dircap record. The
 previous behavior left the old record in place (with the old dircap and
 timestamps), so we'd never stop creating new directories and never converge
 on a null backup.
] 
["tahoe webopen": add --info flag, to get ?t=info
Brian Warner <warner@lothar.com>**20100424233003
 Ignore-this: 126b0bb6db340fabacb623d295eb45fa
 
 Also fix some trailing whitespace.
] 
[docs: install.html http-equiv refresh to quickstart.html
zooko@zooko.com**20100421165708
 Ignore-this: 52b4b619f9dde5886ae2cd7f1f3b734b
] 
[docs: install.html -> quickstart.html
zooko@zooko.com**20100421155757
 Ignore-this: 6084e203909306bed93efb09d0e6181d
 It is not called "installing" because that implies that it is going to change the configuration of your operating system. It is not called "building" because that implies that you need developer tools like a compiler. Also I added a stern warning against looking at the "InstallDetails" wiki page, which I have renamed to "AdvancedInstall".
] 
[Fix another typo in tahoe_storagespace munin plugin
david-sarah@jacaranda.org**20100416220935
 Ignore-this: ad1f7aa66b554174f91dfb2b7a3ea5f3
] 
[Add dependency on windmill >= 1.3
david-sarah@jacaranda.org**20100416190404
 Ignore-this: 4437a7a464e92d6c9012926b18676211
] 
[licensing: phrase the OpenSSL-exemption in the vocabulary of copyright instead of computer technology, and replicate the exemption from the GPL to the TGPPL
zooko@zooko.com**20100414232521
 Ignore-this: a5494b2f582a295544c6cad3f245e91
] 
[munin-tahoe_storagespace
freestorm77@gmail.com**20100221203626
 Ignore-this: 14d6d6a587afe1f8883152bf2e46b4aa
 
 Plugin configuration rename
 
] 
[setup: add licensing declaration for setuptools (noticed by the FSF compliance folks)
zooko@zooko.com**20100309184415
 Ignore-this: 2dfa7d812d65fec7c72ddbf0de609ccb
] 
[setup: fix error in licensing declaration from Shawn Willden, as noted by the FSF compliance division
zooko@zooko.com**20100309163736
 Ignore-this: c0623d27e469799d86cabf67921a13f8
] 
[CREDITS to Jacob Appelbaum
zooko@zooko.com**20100304015616
 Ignore-this: 70db493abbc23968fcc8db93f386ea54
] 
[desert-island-build-with-proper-versions
jacob@appelbaum.net**20100304013858] 
[docs: a few small edits to try to guide newcomers through the docs
zooko@zooko.com**20100303231902
 Ignore-this: a6aab44f5bf5ad97ea73e6976bc4042d
 These edits were suggested by my watching over Jake Appelbaum's shoulder as he completely ignored/skipped/missed install.html and also as he decided that debian.txt wouldn't help him with basic installation. Then I threw in a few docs edits that have been sitting around in my sandbox asking to be committed for months.
] 
[TAG allmydata-tahoe-1.6.1
david-sarah@jacaranda.org**20100228062314
 Ignore-this: eb5f03ada8ea953ee7780e7fe068539
] 
Patch bundle hash:
1e8e2ad6a8b4ed848f8212b7f516a6e8801d7aee