[volunteergrid2-l] How's the grid
Johannes Nix
Johannes.Nix at gmx.net
Tue Mar 6 07:14:26 UTC 2012
Hello,
> I am also experiencing poor grid performance. I uploaded my first
> files today (hooray!) which went about as I expected. Following the
> upload, however, when I try to list the Archives directory, the
> operation takes just over 17 minutes to complete:
I have been looking into this a while.
What seems to happen is that a few mapupdate MODE_READ executions
occasionally take a very long time. It's of course plausible that a
system can't response fast if it is trashing. I have not found,
however any further correlation with load on cerezal (inxoy6ui),
which is currently doing nothing else.
Another (weak) observation is that such long response times
seem to cluster in some way. Is is possible that there
is some non-obvious interdependency on the network?
Johannes
.................................................................
Mutable File Servermap Update Status
Started: 07:47:29 06-Mar-2012
Finished: 07:48:11 06-Mar-2012
Storage Index: jlsl7mexe6eytco6lv7mz2rzda
Helper?: No
Progress: 100.0%
Status: Finished
Update Results
Timings:
Total: 42 seconds
Initial Queries: 16ms
Cumulative Verify: 0us
Per-Server Response Times:
[baeq4skx]: 496ms
[b55ww7wa]: 190ms
[gbytbnxw]: 516ms
[g4xvpwqa]: 368ms
[hpd3hn75]: 269ms
[huadamis]: 355ms
[inxoy6ui]: 367ms
[jbrse33y]: 494ms
[kvj2xrmm]: 42 seconds
[otedzi6b]: 376ms
[pmitrhwg]: 1.08s
[qivwuhf6]: late(48 seconds)
[vifimgiw]: 435ms
.................................................................
Mutable File Servermap Update Status
Started: 00:55:51 06-Mar-2012
Finished: 00:55:52 06-Mar-2012
Storage Index: pu2rajxiq3v4jlzmuyzbxyhisa
Helper?: No
Progress: 100.0%
Status: Finished
Update Results
Timings:
Total: 388ms
Initial Queries: 15ms
Cumulative Verify: 0us
Per-Server Response Times:
[baeq4skx]: 304ms
[b55ww7wa]: 149ms
[gbytbnxw]: 377ms
[g4xvpwqa]: 357ms
[hpd3hn75]: 107ms
[huadamis]: 146ms
[inxoy6ui]: 81ms
[jbrse33y]: 306ms
[kvj2xrmm]: 101ms
[otedzi6b]: 244ms
[pmitrhwg]: 292ms
[qivwuhf6]: 371ms
[vifimgiw]: 309ms
.................................................................
Mutable File Servermap Update Status
Started: 14:58:58 04-Mar-2012
Finished: 14:59:01 04-Mar-2012
Storage Index: jlsl7mexe6eytco6lv7mz2rzda
Helper?: No
Progress: 100.0%
Status: Finished
Update Results
Timings:
Total: 2.88s
Initial Queries: 434ms
Cumulative Verify: 0us
Per-Server Response Times:
[aty4re3a]: 755ms
[baeq4skx]: 892ms
[b55ww7wa]: 748ms
[gbytbnxw]: 837ms
[g4xvpwqa]: 1.47s
[hpd3hn75]: 338ms
[huadamis]: 704ms
[inxoy6ui]: 543ms
[jbrse33y]: 2.69s
[kvj2xrmm]: 2.54s
[pmitrhwg]: 1.04s
[qivwuhf6]: 972ms
.................................................................
Mutable File Servermap Update Status
Started: 13:10:25 04-Mar-2012
Finished: 13:10:28 04-Mar-2012
Storage Index: jlsl7mexe6eytco6lv7mz2rzda
Helper?: No
Progress: 100.0%
Status: Finished
Update Results
Timings:
Total: 3.12s
Initial Queries: 12ms
Cumulative Verify: 0us
Per-Server Response Times:
[baeq4skx]: 251ms
[b55ww7wa]: 184ms
[gbytbnxw]: 333ms
[g4xvpwqa]: 339ms
[hpd3hn75]: 103ms
[huadamis]: 299ms
[inxoy6ui]: 3.11s
[jbrse33y]: 270ms
[kvj2xrmm]: 126ms
[pmitrhwg]: 424ms
[qivwuhf6]: 311ms
.................................................................
Mutable File Servermap Update Status
Started: 13:10:28 04-Mar-2012
Finished: 13:10:29 04-Mar-2012
Storage Index: c7agg7guhpyfehrzt6v4ibco4i
Helper?: No
Progress: 100.0%
Status: Finished
Update Results
Timings:
Total: 1.17s
Initial Queries: 12ms
Cumulative Verify: 0us
Per-Server Response Times:
[baeq4skx]: 252ms
[b55ww7wa]: 194ms
[gbytbnxw]: 311ms
[g4xvpwqa]: 301ms
[hpd3hn75]: 96ms
[huadamis]: 127ms
[inxoy6ui]: 798ms
[jbrse33y]: 1.17s
[kvj2xrmm]: 162ms
[pmitrhwg]: 278ms
[qivwuhf6]: 303ms
.................................................................
On Mon, 5 Mar 2012 21:18:27 -0700
Steve Dodson <steve.dodson at gmail.com> wrote:
> I retried this a couple of times tonight and my first listing again
> took 17 minutes...however, subsequent listings were about 1 second
> each (more in line with my expectations). Perhaps these times are
> more a function of a caching mechanism? Notably, during the 17
> minute execution, the cerezal node was not a part of the active
> operations listing - and all the nodes listed had respectable
> response times. I don't know what's going on, but it's not easily
> reproducible and I feel like the investigative method I'm using isn't
> at all helpful.
>
> $ time tahoe ls tahoe:Archives
> 2012-03-04_21:55:25Z
>
> real 17m23.789s
>
> $ time tahoe ls tahoe:Archives
> 2012-03-04_21:55:25Z
>
> real 0m1.151s
>
> $ time tahoe ls tahoe:
> Archives
> Latest
>
> real 0m0.688s
> $ time tahoe ls tahoe:Archives
> 2012-03-04_21:55:25Z
>
> real 0m1.047s
> $ time tahoe ls tahoe:Archives
> 2012-03-04_21:55:25Z
>
> real 0m1.158s
>
> On Mon, Mar 5, 2012 at 12:12 PM, Johannes Nix <Johannes.Nix at gmx.net>
> wrote:
>
> > Hello,
> >
> >
> > Steve wrote
> > > >
> > > > sdodson at hiro:~$ time tahoe ls tahoe:Archives
> > > > 2012-03-04_21:55:25Z
> > > >
> > > > real 17m14.812s
> > > > user 0m0.260s
> > > > sys 0m0.040s
> > > >
> > > > In looking at the "Mutable File Servermap Update Status" page
> > > > in the WUI, it appears that cerezal's node is problematic (6
> > > > seconds!) from my location:
> > > >
> >
> > [ ... ]
> >
> > > > [inxoy6ui]: 5.58s (cerezal)
> >
> > Hm. That is probably too long. It could, of course, be because of
> > swapping; however, local commands run smoothly. cerezal is currently
> > doing backup uploads, the storage node runs with a running set size
> > of 48 % RAM. The backup itself does not seem to consume much
> > resources, around 0 % CPU and 5 % RAM. However there could be some
> > interference with file system caches which on Linux use any RAM not
> > claimed by programs. What could also cause problems is NFS service
> > which I was using to upload things.
> >
> > To have an idea what is causing poor response times, I'll
> > switch off upload and NFS and would like to ask you to measure
> > response times again and send them to me.
> >
> > Is it possible that the upload slows thing down by
> > clogging the network connection too much?
> >
> > Johannes
> >
>
>
>
More information about the volunteergrid2-l
mailing list