cpu monitor sometimes shows negative numbers

[warner]

On some of our production machines, the cpu watcher is displaying weird numbers: maybe there's a small field that wraps.

'stats': {'cpu_monitor.15min_avg': 0.019444445629178814,
           'cpu_monitor.1min_avg': 0.0096666806870237352,
           'cpu_monitor.5min_avg': 0.0070999994301001535,
           'cpu_monitor.total': -273.11918400000002,

[warner]

incidentally, the -273 number is getting less negative over time (a little while later it was at -271)

[warner]

Zooko did some digging, and learned that the clock(3) syscall returns a signed 32-bit number (which counts microseconds), which means it will wrap after 72 minutes of 100% CPU. It is not clear how python will handle this (since python has no native signed datatypes).

So, task number one: write a program that burns 100% CPU and prints out the value of time.clock() every minute: see what happens after 72 minutes, and figure out how to deal with that.

Task number two: change src/allmydata/stats.py . It currently records time.clock() every minute and looks for 1/5/15 minute deltas later. It needs to do something more complicated: perhaps record the delta between one-minute samples (clipping negative jumps that occur at wraparound, or doing some clever math to compensate for the wraparound), and then sum the last 1/5/15 values.

[davidsarah]

Replying to warner:

Zooko did some digging, and learned that the clock(3) syscall returns a signed 32-bit number (which counts microseconds), which means it will wrap after 72 minutes of 100% CPU.

~36 minutes, no? (2³¹ / 1000000 / 60)

[exarkun]

This ticket is about operational visibility for operations that are no longer operational.