[tahoe-dev] Tahoe as glue filesystem

[Valentino Volonghi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Jul 7, 2008, at 4:33 PM, Duncan McGreggor wrote:

> You wounldn't, by any chance, be working with Tommi on this, would  
> you?
>
> In particular, I'm referring to his article on an adjusted mapreduce:
>  http://eagain.net/articles/incremental-mapreduce/


We actually talked very very very briefly about this via email but we  
are
using 2 completely different approaches. He plans to use web servers
(pylons I think) as worker nodes and they talk via REST+JSON with the
thing submitting jobs and I suppose also between each other.

In my view I'd have a process pool in every node and every node  
understands
AMP (but being twisted I might decide that it understands REST+JSON too)
and I'd have Map, Reduce, Blabla commands that start this process,  
something
like:

l = []
destination = "pburl://destination"
destination2 = "pburl://final_storage"
for source in sources:
     d = cluster.callRemote(Map, source=source, dest=destination
     l.append(d)

defer.DeferredList(l
    ).addCallback(lambda _:
           cluster.callRemote(Reduce, source=destination,  
dest=destination2))


If one day I manage to have this in my code I'd be really really  
happy :).

- --
Valentino Volonghi aka Dialtone
Now running MacOS X 10.5
Home Page: http://www.twisted.it
http://www.adroll.com

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)

iEYEARECAAYFAkhyrpAACgkQ9Llz28widGUk8wCfV3QXaLhKE3AyDVe1iLeUO6gx
LbAAn3jqrFDcr/bGo/AVJLSg+lIq916q
=gptl
-----END PGP SIGNATURE-----