[tahoe-dev] Tahoe Performance

[Wassim Drira]

Hi,

Brian, thank you for your detailed reply.

I'd suggest a test framework which uses multiple simultaneous "tahoe
> put" commands, each in a separate process (ideally on a different
> machine than the Tahoe node, to factor out contention for CPU and local
> disk between your test harness and the Tahoe node).
>

You found in attachement a script that sends multiple files simultaneously.
I done those tests on my laptop (test1 -> test3) and on the tahoe test
platform (test4). The results are better than uploading sequentially small
files but when files are larger this adds an overhead to each upload at
least ~2s (1MB files).

Parallelism is easy to add in my application :) but adding my code into the
tahoe node is not obvious (I need to convert my code java -> python)



> The quickest way to measure the HTTP roundtrip latency is to do "time
> echo -n '' |tahoe put". That uploads a zero-byte file, which Tahoe does
> by creating a "URI:LIT" filecap, which is not distributed and doesn't
> even talk to the storage servers. Then compare the time it takes to do
> that against uploading a 100-byte file, which *is* distributed and does
> the usual tahoe upload work. On my laptop, the zero-byte file takes
> 971ms to upload,
>

This is strange!

Regards,
Wassim
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://allmydata.org/pipermail/tahoe-dev/attachments/20100412/911284c2/attachment.htm 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: _testTahoeThreads.sh
Type: application/x-sh
Size: 1561 bytes
Desc: not available
Url : http://allmydata.org/pipermail/tahoe-dev/attachments/20100412/911284c2/attachment.sh 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: put_tahoe.sh
Type: application/x-sh
Size: 174 bytes
Desc: not available
Url : http://allmydata.org/pipermail/tahoe-dev/attachments/20100412/911284c2/attachment-0001.sh 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: log2.csv
Type: text/csv
Size: 12403 bytes
Desc: not available
Url : http://allmydata.org/pipermail/tahoe-dev/attachments/20100412/911284c2/attachment.csv