[volunteergrid2-l] Fw: apparent correlation of server node response times

[Johannes Nix]

Hello,

I'd like to re-send an earlier post for Brian.

What had called my attention was that not only some nodes
take occasionally a long time for map-update, but that also
if one node takes a time of several seconds,
it seems way more frequent that other server nodes
do too. Response times between server nodes
are correlated.

This is surprising for me, because my mental concept of
the network is that server nodes are mutually
independent; Even if one node behaves really bad, others should
behave as always.

Of course, this can be rooted also in slow connection
speed at the client's end. This would not be
discernible from one client's side.

Do you others see the same correlation?

Cheers,

Johannes

....................................

Begin forwarded message:

To: Steve Dodson <steve.dodson at gmail.com>, a small group of people
sharing storage resources with Tahoe-LAFS
<volunteergrid2-l at tahoe-lafs.org> Subject: Re: [volunteergrid2-l] How's
the grid


Hello,

> I am also experiencing poor grid performance.  I uploaded my first
> files today (hooray!) which went about as I expected. Following the
> upload, however, when I try to list the Archives directory, the
> operation takes just over 17 minutes to complete:


I have been looking into this a while.

What seems to happen is that a few mapupdate MODE_READ executions
occasionally take a very long time. It's of course plausible that a
system can't response fast if it is trashing. I have not found,
however any further correlation with load on cerezal (inxoy6ui),
which is currently doing nothing else.

Another (weak) observation is that such long response times
seem to cluster in some way. Is is possible that there
is some non-obvious interdependency on the network?

Johannes


.................................................................
Mutable File Servermap Update Status

    Started: 07:47:29 06-Mar-2012
    Finished: 07:48:11 06-Mar-2012
    Storage Index: jlsl7mexe6eytco6lv7mz2rzda
    Helper?: No
    Progress: 100.0%
    Status: Finished

Update Results

    Timings:
        Total: 42 seconds
            Initial Queries: 16ms
            Cumulative Verify: 0us
        Per-Server Response Times:
            [baeq4skx]: 496ms
            [b55ww7wa]: 190ms
            [gbytbnxw]: 516ms
            [g4xvpwqa]: 368ms
            [hpd3hn75]: 269ms
            [huadamis]: 355ms
            [inxoy6ui]: 367ms
            [jbrse33y]: 494ms
            [kvj2xrmm]: 42 seconds
            [otedzi6b]: 376ms
            [pmitrhwg]: 1.08s
            [qivwuhf6]: late(48 seconds)
            [vifimgiw]: 435ms
.................................................................

Mutable File Servermap Update Status

    Started: 00:55:51 06-Mar-2012
    Finished: 00:55:52 06-Mar-2012
    Storage Index: pu2rajxiq3v4jlzmuyzbxyhisa
    Helper?: No
    Progress: 100.0%
    Status: Finished

Update Results

    Timings:
        Total: 388ms
            Initial Queries: 15ms
            Cumulative Verify: 0us
        Per-Server Response Times:
            [baeq4skx]: 304ms
            [b55ww7wa]: 149ms
            [gbytbnxw]: 377ms
            [g4xvpwqa]: 357ms
            [hpd3hn75]: 107ms
            [huadamis]: 146ms
            [inxoy6ui]: 81ms
            [jbrse33y]: 306ms
            [kvj2xrmm]: 101ms
            [otedzi6b]: 244ms
            [pmitrhwg]: 292ms
            [qivwuhf6]: 371ms
            [vifimgiw]: 309ms
.................................................................

Mutable File Servermap Update Status

    Started: 14:58:58 04-Mar-2012
    Finished: 14:59:01 04-Mar-2012
    Storage Index: jlsl7mexe6eytco6lv7mz2rzda
    Helper?: No
    Progress: 100.0%
    Status: Finished

Update Results

    Timings:
        Total: 2.88s
            Initial Queries: 434ms
            Cumulative Verify: 0us
        Per-Server Response Times:
            [aty4re3a]: 755ms
            [baeq4skx]: 892ms
            [b55ww7wa]: 748ms
            [gbytbnxw]: 837ms
            [g4xvpwqa]: 1.47s
            [hpd3hn75]: 338ms
            [huadamis]: 704ms
            [inxoy6ui]: 543ms
            [jbrse33y]: 2.69s
            [kvj2xrmm]: 2.54s
            [pmitrhwg]: 1.04s
            [qivwuhf6]: 972ms


.................................................................


Mutable File Servermap Update Status

    Started: 13:10:25 04-Mar-2012
    Finished: 13:10:28 04-Mar-2012
    Storage Index: jlsl7mexe6eytco6lv7mz2rzda
    Helper?: No
    Progress: 100.0%
    Status: Finished

Update Results

    Timings:
        Total: 3.12s
            Initial Queries: 12ms
            Cumulative Verify: 0us
        Per-Server Response Times:
            [baeq4skx]: 251ms
            [b55ww7wa]: 184ms
            [gbytbnxw]: 333ms
            [g4xvpwqa]: 339ms
            [hpd3hn75]: 103ms
            [huadamis]: 299ms
            [inxoy6ui]: 3.11s
            [jbrse33y]: 270ms
            [kvj2xrmm]: 126ms
            [pmitrhwg]: 424ms
            [qivwuhf6]: 311ms

.................................................................


Mutable File Servermap Update Status

    Started: 13:10:28 04-Mar-2012
    Finished: 13:10:29 04-Mar-2012
    Storage Index: c7agg7guhpyfehrzt6v4ibco4i
    Helper?: No
    Progress: 100.0%
    Status: Finished

Update Results

    Timings:
        Total: 1.17s
            Initial Queries: 12ms
            Cumulative Verify: 0us
        Per-Server Response Times:
            [baeq4skx]: 252ms
            [b55ww7wa]: 194ms
            [gbytbnxw]: 311ms
            [g4xvpwqa]: 301ms
            [hpd3hn75]: 96ms
            [huadamis]: 127ms
            [inxoy6ui]: 798ms
            [jbrse33y]: 1.17s
            [kvj2xrmm]: 162ms
            [pmitrhwg]: 278ms
            [qivwuhf6]: 303ms

.................................................................