[volunteergrid2-l] something is wrong with node "ianchov at gmail.com"

Fri Mar 9 22:48:02 UTC 2012

Hi Christoph,

Cerezal has configured the stats but does not appear neither.

Can you tell me what lines from twistd.log this mean:

###############################################
2012-03-07 18:22:00+0000 [-] Reconnector._failed
(furl=pb://kqyu52yzkipbt5ktyu6x56s6isi2khyc@dynafoo.dyndns.org:3917/h66p63tgqkfodqxpos3pt2e
mjziov6bp): [Failure instance: Traceback (failure with no frames):
<class 'foolscap.tokens.NegotiationError'>: no connection established
within client timeout
###############################################

Also, as I wrote on March 6th, the answer times of server nodes
are correlated with each other (which is not what I would expect).
Therefore, a long answer of one node _can_ be due to a problem on that
node,  but you cannot conclude that it
is so. It can, for example, be a problem with the client. 
Look at the list below at the entry from 14:58:58 04-Mar-2012,
you have four different nodes which have answer times of
more than one second. Maybe this occurs when saturating
networks links by downloads, I do not know.

Also, if a download fails after 2.5 hours, as far as I understand
this does not mean that the server did not respond so much
time, but that the connection was lost after that time (and
possible no other server was found). I've seen also that large
files might fail more frequently, but I do not know why.

- Johannes

> I have been looking into this a while.
> 
> What seems to happen is that a few mapupdate MODE_READ executions
> occasionally take a very long time. It's of course plausible that a
> system can't response fast if it is trashing. I have not found,
> however any further correlation with load on cerezal (inxoy6ui),
> which is currently doing nothing else.
> 
> Another (weak) observation is that such long response times
> seem to cluster in some way. Is is possible that there
> is some non-obvious interdependency on the network?
> 
> Johannes
> 
> 
> .................................................................
> Mutable File Servermap Update Status
> 
>     Started: 07:47:29 06-Mar-2012
>     Finished: 07:48:11 06-Mar-2012
>     Storage Index: jlsl7mexe6eytco6lv7mz2rzda
>     Helper?: No
>     Progress: 100.0%
>     Status: Finished
> 
> Update Results
> 
>     Timings:
>         Total: 42 seconds
>             Initial Queries: 16ms
>             Cumulative Verify: 0us
>         Per-Server Response Times:
>             [baeq4skx]: 496ms
>             [b55ww7wa]: 190ms
>             [gbytbnxw]: 516ms
>             [g4xvpwqa]: 368ms
>             [hpd3hn75]: 269ms
>             [huadamis]: 355ms
>             [inxoy6ui]: 367ms
>             [jbrse33y]: 494ms
>             [kvj2xrmm]: 42 seconds
>             [otedzi6b]: 376ms
>             [pmitrhwg]: 1.08s
>             [qivwuhf6]: late(48 seconds)
>             [vifimgiw]: 435ms
> .................................................................
> 
> Mutable File Servermap Update Status
> 
>     Started: 00:55:51 06-Mar-2012
>     Finished: 00:55:52 06-Mar-2012
>     Storage Index: pu2rajxiq3v4jlzmuyzbxyhisa
>     Helper?: No
>     Progress: 100.0%
>     Status: Finished
> 
> Update Results
> 
>     Timings:
>         Total: 388ms
>             Initial Queries: 15ms
>             Cumulative Verify: 0us
>         Per-Server Response Times:
>             [baeq4skx]: 304ms
>             [b55ww7wa]: 149ms
>             [gbytbnxw]: 377ms
>             [g4xvpwqa]: 357ms
>             [hpd3hn75]: 107ms
>             [huadamis]: 146ms
>             [inxoy6ui]: 81ms
>             [jbrse33y]: 306ms
>             [kvj2xrmm]: 101ms
>             [otedzi6b]: 244ms
>             [pmitrhwg]: 292ms
>             [qivwuhf6]: 371ms
>             [vifimgiw]: 309ms
> .................................................................
> 
> Mutable File Servermap Update Status
> 
>     Started: 14:58:58 04-Mar-2012
>     Finished: 14:59:01 04-Mar-2012
>     Storage Index: jlsl7mexe6eytco6lv7mz2rzda
>     Helper?: No
>     Progress: 100.0%
>     Status: Finished
> 
> Update Results
> 
>     Timings:
>         Total: 2.88s
>             Initial Queries: 434ms
>             Cumulative Verify: 0us
>         Per-Server Response Times:
>             [aty4re3a]: 755ms
>             [baeq4skx]: 892ms
>             [b55ww7wa]: 748ms
>             [gbytbnxw]: 837ms
>             [g4xvpwqa]: 1.47s
>             [hpd3hn75]: 338ms
>             [huadamis]: 704ms
>             [inxoy6ui]: 543ms
>             [jbrse33y]: 2.69s
>             [kvj2xrmm]: 2.54s
>             [pmitrhwg]: 1.04s
>             [qivwuhf6]: 972ms
> 
> 
> .................................................................
> 
> 
> Mutable File Servermap Update Status
> 
>     Started: 13:10:25 04-Mar-2012
>     Finished: 13:10:28 04-Mar-2012
>     Storage Index: jlsl7mexe6eytco6lv7mz2rzda
>     Helper?: No
>     Progress: 100.0%
>     Status: Finished
> 
> Update Results
> 
>     Timings:
>         Total: 3.12s
>             Initial Queries: 12ms
>             Cumulative Verify: 0us
>         Per-Server Response Times:
>             [baeq4skx]: 251ms
>             [b55ww7wa]: 184ms
>             [gbytbnxw]: 333ms
>             [g4xvpwqa]: 339ms
>             [hpd3hn75]: 103ms
>             [huadamis]: 299ms
>             [inxoy6ui]: 3.11s
>             [jbrse33y]: 270ms
>             [kvj2xrmm]: 126ms
>             [pmitrhwg]: 424ms
>             [qivwuhf6]: 311ms
> 
> .................................................................
> 
> 
> Mutable File Servermap Update Status
> 
>     Started: 13:10:28 04-Mar-2012
>     Finished: 13:10:29 04-Mar-2012
>     Storage Index: c7agg7guhpyfehrzt6v4ibco4i
>     Helper?: No
>     Progress: 100.0%
>     Status: Finished
> 
> Update Results
> 
>     Timings:
>         Total: 1.17s
>             Initial Queries: 12ms
>             Cumulative Verify: 0us
>         Per-Server Response Times:
>             [baeq4skx]: 252ms
>             [b55ww7wa]: 194ms
>             [gbytbnxw]: 311ms
>             [g4xvpwqa]: 301ms
>             [hpd3hn75]: 96ms
>             [huadamis]: 127ms
>             [inxoy6ui]: 798ms
>             [jbrse33y]: 1.17s
>             [kvj2xrmm]: 162ms
>             [pmitrhwg]: 278ms
>             [qivwuhf6]: 303ms
> 
> .................................................................
> 

On Fri, 09 Mar 2012 22:35:43 +0100
Christoph Langguth <christoph at rosenkeller.org> wrote:

> Am 09.03.2012 22:28, schrieb Iantcho Vassilev:
> > Ohh...i am sleeping...
> > i was trying vg-stats url and not vg2...
> 
> :-D
> 
> >
> > ANyway here is the twistd.log
> > 2012-03-09 07:23:13+0200 [-] Log opened.
> > 2012-03-09 07:23:13+0200 [-] twistd 10.1.0 (C:\Python26\python.exe
> > 2.6.6) starting up.
> > 2012-03-09 07:23:13+0200 [-] reactor class:
> > twisted.internet.selectreactor.SelectReactor.
> > 2012-03-09 07:23:13+0200 [-] foolscap.pb.Listener starting on 55392
> > 2012-03-09 07:23:13+0200 [-] nevow.appserver.NevowSite starting on
> > 3456 2012-03-09 07:23:13+0200 [-] Starting
> > factory<nevow.appserver.NevowSite instance at 0x0000000002708408>
> > 2012-03-09 07:23:13+0200 [-] My pid: 3912
> > 2012-03-09 07:23:13+0200 [-]
> > twisted.internet.protocol.DatagramProtocol starting on 57066
> > 2012-03-09 07:23:13+0200 [-] Starting protocol
> > <twisted.internet.protocol.DatagramProtocol instance at
> > 0x000000000270B488> 2012-03-09 07:23:13+0200 [-] (Port 57066 Closed)
> > 2012-03-09 07:23:13+0200 [-] Stopping protocol
> > <twisted.internet.protocol.DatagramProtocol instance at
> > 0x000000000270B488> 2012-03-09 19:23:47+0200 [-] Timing out client:
> > 0x000000000270B488> IPv4Address(TCP,
> > '127.0.0.1', 64018)
> >
> >
> > Why here is port 57066?
> No idea...
> 
> > And also i have a lots of incidents...? Actually how are they
> > intepred?
> >
> flogtool dump <incident-file>
> 
> On Ubuntu, you need package python-foolscap:
> root at bender:~# dpkg -S `which flogtool`
> python-foolscap: /usr/bin/flogtool
> 