[tahoe-dev] First pass at setting up Tahoe on Amazon Web Services (AWS) (was: Re: Changing n-of-k - where is that done?)

[Peter Secor]

No documentation yet, I'm just playing with it now and will send out 
updates to this list and/or on IRC. Once I get a nice working solution, 
I'll write up a HOWTO on it.

So far, the steps have been:
  * sign up for Amazon Web Services
  * find an appropriate image to run on the virtual machines (Ubuntu 
Karmic Koala)
  * learn about Elastic Block Store (persistent storage over reboots), 
need to learn more about snapshots to S3
  * learn about Elastic IP Addresses (allows mapping between public IP's 
to internal virtual machine IP's, note that data transfer between 
internal IP's is cheaper)
  * start up virtual server
  * install tahoe-lafs (sudo apt-get install tahoe-lafs)
  * provision introducer, helper, nodes, note using 10-of-12 encoding
  * start up the grid and test out uploads and downloads
  * set up a web client and accounting db (not complete yet)

Next up is determining what the best setup is for CPU's and disk space, 
maybe using a load balancer, and making a Windows build that will 
default to this grid. After that are the statistics and monitoring tools 
followed by performance and load tests.

Parallel to this is an economic analysis of what the costs are. I'll 
post this as a spreadsheet when it's actually legible, so far it's 
mostly scribbles on scratch paper and some rough calculations. For the 
particular application I'm working towards, the margins are high enough 
to make the rough analysis clearly indicate that this is a direction 
worth pursing.

Ps

On 12/24/09 9:12 PM, Jody Harris wrote:
> Peter,
>
> Is there documentation somewhere for duplicating what you have done?
>
> j
> ----
> - Think carefully.
> - Contra mundum - "Against the world" (St. Athanasius)
> - Credo ut intelliga - "I believe that I may know" (St. Augustin of Hippo)
>
>
> On Thu, Dec 24, 2009 at 9:59 PM, Peter Secor <secorp at allmydata.com
> <mailto:secorp at allmydata.com>> wrote:
>
>     Thanks again, I now have a grid running fully under Amazon Web Services
>     with 1.2x expansion (10-of-12) that persists over reboots of the virtual
>     boxes. Next up is to get a web client provisioned, then the system tools
>     for statistics and monitoring.
>
>     Peter
>
>     On 12/24/09 8:47 PM, Jody Harris wrote:
>      > I had a copy of the configuration.txt setting right next to me.... I
>      > cheated!
>      >
>      > j
>      > ----
>      > - Think carefully.
>      > - Contra mundum - "Against the world" (St. Athanasius)
>      > - Credo ut intelliga - "I believe that I may know" (St. Augustin
>     of Hippo)
>      >
>      >
>      > On Thu, Dec 24, 2009 at 9:41 PM, Peter Secor
>     <secorp at allmydata.com <mailto:secorp at allmydata.com>
>      > <mailto:secorp at allmydata.com <mailto:secorp at allmydata.com>>> wrote:
>      >
>      >     Thanks, the last time I was digging around it wasn't in the
>     config
>      >     file :)
>      >
>      >     Ps
>      >
>      >     On 12/24/09 8:23 PM, Jody Harris wrote:
>      > > It's in the tahoe.cfg
>      > >
>      > > under [client]:
>      > >
>      > > shares.needed
>      > > shares.total
>      > > shares.happy
>      > > ----
>      > > - Think carefully.
>      > > - Contra mundum - "Against the world" (St. Athanasius)
>      > > - Credo ut intelliga - "I believe that I may know" (St. Augustin
>      >     of Hippo)
>      > >
>      > >
>      > > On Thu, Dec 24, 2009 at 9:01 PM, Peter Secor
>      > <secorp at allmydata.com <mailto:secorp at allmydata.com>
>     <mailto:secorp at allmydata.com <mailto:secorp at allmydata.com>>
>      > > <mailto:secorp at allmydata.com <mailto:secorp at allmydata.com>
>     <mailto:secorp at allmydata.com <mailto:secorp at allmydata.com>>>> wrote:
>      > >
>      > >     I'm just about to start digging through the code to find it,
>      >     but as I'm
>      > >     experimenting with AWS, one thing I'm going to do is try to
>      >     set the
>      > >     expansion factor to 1.1 as the underlying system already has
>      >     redundancy
>      > >     built into it.
>      > >
>      > >     Does anybody know off-hand where it is?
>      > >
>      > >     Thanks,
>      > >     Peter
>      > >     _______________________________________________
>      > >     tahoe-dev mailing list
>      > > tahoe-dev at allmydata.org <mailto:tahoe-dev at allmydata.org>
>     <mailto:tahoe-dev at allmydata.org <mailto:tahoe-dev at allmydata.org>>
>      > <mailto:tahoe-dev at allmydata.org <mailto:tahoe-dev at allmydata.org>
>     <mailto:tahoe-dev at allmydata.org <mailto:tahoe-dev at allmydata.org>>>
>      > > http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev
>      > >
>      > >
>      > >
>      > >
>      > > _______________________________________________
>      > > tahoe-dev mailing list
>      > > tahoe-dev at allmydata.org <mailto:tahoe-dev at allmydata.org>
>     <mailto:tahoe-dev at allmydata.org <mailto:tahoe-dev at allmydata.org>>
>      > > http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev
>      >     _______________________________________________
>      >     tahoe-dev mailing list
>      > tahoe-dev at allmydata.org <mailto:tahoe-dev at allmydata.org>
>     <mailto:tahoe-dev at allmydata.org <mailto:tahoe-dev at allmydata.org>>
>      > http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev
>      >
>      >
>      >
>      >
>      > _______________________________________________
>      > tahoe-dev mailing list
>      > tahoe-dev at allmydata.org <mailto:tahoe-dev at allmydata.org>
>      > http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev
>     _______________________________________________
>     tahoe-dev mailing list
>     tahoe-dev at allmydata.org <mailto:tahoe-dev at allmydata.org>
>     http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev
>
>
>
>
> _______________________________________________
> tahoe-dev mailing list
> tahoe-dev at allmydata.org
> http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev