[tahoe-dev] running a stats gatherer

Jody Harris imhavoc at gmail.com
Tue Dec 22 21:52:59 PST 2009


That's very helpful. Thank you!

On a quick read-through I have a question I didn't see answered:

Can I run the stats gatherer on a machine running one or more tahoe nodes?
Is there a way to reconfigure the port (:3456) to avoid a collision with an
existing node? (I know that I can reassign the other nodes local port, but I
wanted to ask this question here.)

Thank you for putting this documentation up!


----
- Think carefully.
- Contra mundum - "Against the world" (St. Athanasius)
- Credo ut intelliga - "I believe that I may know" (St. Augustin of Hippo)


On Tue, Dec 22, 2009 at 10:27 PM, Brian Warner <warner at lothar.com> wrote:

> Jody Harris wrote:
> > I have looked all over the site, I've searched, I cannot find the
> > documentation on setting up a stats gathered.
>
> Sure. I just wrote up a lengthy note on Tahoe stats, the gatherer, and a
> few other topics, which (as of 30 seconds ago) now lives in the source
> tree in docs/stats.txt . I've attached a copy here. Please let me know
> if this answers your questions: if not, I'd like to update
> docs/stats.txt with additional information.
>
> cheers,
>  -Brian
>
> = Tahoe Statistics =
>
> Each Tahoe node collects and publishes statistics about its operations as
> it
> runs. These include counters of how many files have been uploaded and
> downloaded, CPU usage information, performance numbers like latency of
> storage server operations, and available disk space.
>
> The easiest way to see the stats for any given node is use the web
> interface.
> From the main "Welcome Page", follow the "Operational Statistics" link
> inside
> the small "This Client" box. If the welcome page lives at
> http://localhost:3456/, then the statistics page will live at
> http://localhost:3456/statistics . This presents a summary of the stats
> block, along with a copy of the raw counters. To obtain just the raw
> counters
> (in JSON format), use /statistics?t=json instead.
>
> = Statistics Categories =
>
> The stats dictionary contains two keys: 'counters' and 'stats'. 'counters'
> are strictly counters: they are reset to zero when the node is started, and
> grow upwards. 'stats' are non-incrementing values, used to measure the
> current state of various systems. Some stats are actually booleans,
> expressed
> as '1' for true and '0' for false (internal restrictions require all stats
> values to be numbers).
>
> Under both the 'counters' and 'stats' dictionaries, each individual stat
> has
> a key with a dot-separated name, breaking them up into groups like
> 'cpu_monitor' and 'storage_server'.
>
> The currently available stats (as of release 1.6.0 or so) are described
> here:
>
> counters.storage_server.*: this group counts inbound storage-server
>                           operations. They are not provided by client-only
>                           nodes which have been configured to not run a
>                           storage server (with [storage]enabled=false in
>                           tahoe.cfg)
>  allocate, write, close, abort: these are for immutable file uploads.
>                                 'allocate' is incremented when a client
> asks
>                                 if it can upload a share to the server.
>                                 'write' is incremented for each chunk of
>                                 data written. 'close' is incremented when
>                                 the share is finished. 'abort' is
>                                 incremented if the client abandons the
>                                 uploaed.
>  get, read: these are for immutable file downloads. 'get' is incremented
>             when a client asks if the server has a specific share. 'read'
> is
>             incremented for each chunk of data read.
>  readv, writev: these are for immutable file creation, publish, and
>                 retrieve. 'readv' is incremented each time a client reads
>                 part of a mutable share. 'writev' is incremented each time
> a
>                 client sends a modification request.
>  add-lease, renew, cancel: these are for share lease modifications.
>                            'add-lease' is incremented when an 'add-lease'
>                            operation is performed (which either adds a new
>                            lease or renews an existing lease). 'renew' is
>                            for the 'renew-lease' operation (which can only
>                            be used to renew an existing one). 'cancel' is
>                            used for the 'cancel-lease' operation.
>  bytes_freed: this counts how many bytes were freed when a 'cancel-lease'
>               operation removed the last lease from a share and the share
>               was thus deleted.
>  bytes_added: this counts how many bytes were consumed by immutable share
>               uploads. It is incremented at the same time as the 'close'
>               counter.
>
> stats.storage_server.*:
>  allocated: this counts how many bytes are currently 'allocated', which
>            tracks the space that will eventually be consumed by immutable
>            share upload operations. The stat is increased as soon as the
>            upload begins (at the same time the 'allocated' counter is
>            incremented), and goes back to zero when the 'close' or 'abort'
>            message is received (at which point the 'disk_used' stat should
>            incremented by the same amount).
>  disk_total
>  disk_used
>  disk_free_for_root
>  disk_free_for_nonroot
>  disk_avail
>  reserved_space: these all reflect disk-space usage policies and status.
>                 'disk_total' is the total size of disk where the storage
>                 server's BASEDIR/storage/shares directory lives, as
> reported
>                 by /bin/df or equivalent. 'disk_used',
> 'disk_free_for_root',
>                 and 'disk_free_for_nonroot' show related information.
>                 'reserved_space' reports the reservation configured by the
>                 tahoe.cfg [storage]reserved_space value. 'disk_avail'
>                 reports the remaining disk space available for the Tahoe
>                 server after subtracting reserved_space from disk_avail.
> All
>                 values are in bytes.
>  accepting_immutable_shares: this is '1' if the storage server is currently
>                             accepting uploads of immutable shares. It may
> be
>                             '0' if a server is disabled by configuration,
> or
>                             if the disk is full (i.e. disk_avail is less
>                             than reserved_space).
>  total_bucket_count: this counts the number of 'buckets' (i.e. unique
>                     storage-index values) currently managed by the storage
>                     server. It indicates roughly how many files are managed
>                     by the server.
>  latencies.*.*: these stats keep track of local disk latencies for
>                storage-server operations. A number of percentile values are
>                tracked for many operations. For example,
>                'storage_server.latencies.readv.50_0_percentile' records the
>                median response time for a 'readv' request. All values are
> in
>                seconds. These are recorded by the storage server, starting
>                from the time the request arrives (post-deserialization) and
>                ending when the response begins serialization. As such, they
>                are mostly useful for measuring disk speeds. The operations
>                tracked are the same as the counters.storage_server.*
> counter
>                values (allocate, write, close, get, read, add-lease, renew,
>                cancel, readv, writev). The percentile values tracked are:
>                mean, 01_0_percentile, 10_0_percentile, 50_0_percentile,
>                90_0_percentile, 95_0_percentile, 99_0_percentile,
>                99_9_percentile. (the last value, 99.9 percentile, means
> that
>                999 out of the last 1000 operations were faster than the
>                given number, and is the same threshold used by Amazon's
>                internal SLA, according to the Dynamo paper).
>
> counters.uploader.files_uploaded
> counters.uploader.bytes_uploaded
> counters.downloader.files_downloaded
> counters.downloader.bytes_downloaded
>
>  These count client activity: a Tahoe client will increment these when it
>  uploads or downloads an immutable file. 'files_uploaded' is incremented by
>  one for each operation, while 'bytes_uploaded' is incremented by the size
> of
>  the file.
>
> counters.mutable.files_published
> counters.mutable.bytes_published
> counters.mutable.files_retrieved
> counters.mutable.bytes_retrieved
>
>  These count client activity for mutable files. 'published' is the act of
>  changing an existing mutable file (or creating a brand-new mutable file).
>  'retrieved' is the act of reading its current contents.
>
> counters.chk_upload_helper.*
>
>  These count activity of the "Helper", which receives ciphertext from
> clients
>  and performs erasure-coding and share upload for files that are not
> already
>  in the grid. The code which implements these counters is in
>  src/allmydata/immutable/offloaded.py .
>
>  upload_requests: incremented each time a client asks to upload a file
>  upload_already_present: incremented when the file is already in the grid
>  upload_need_upload: incremented when the file is not already in the grid
>  resumes: incremented when the helper already has partial ciphertext for
>           the requested upload, indicating that the client is resuming an
>           earlier upload
>  fetched_bytes: this counts how many bytes of ciphertext have been fetched
>                 from uploading clients
>  encoded_bytes: this counts how many bytes of ciphertext have been
>                 encoded and turned into successfully-uploaded shares. If no
>                 uploads have failed or been abandoned, encoded_bytes should
>                 eventually equal fetched_bytes.
>
> stats.chk_upload_helper.*
>
>  These also track Helper activity:
>
>  active_uploads: how many files are currently being uploaded. 0 when idle.
>  incoming_count: how many cache files are present in the incoming/
> directory,
>                  which holds ciphertext files that are still being fetched
>                  from the client
>  incoming_size: total size of cache files in the incoming/ directory
>  incoming_size_old: total size of 'old' cache files (more than 48 hours)
>  encoding_count: how many cache files are present in the encoding/
> directory,
>                  which holds ciphertext files that are being encoded and
>                  uploaded
>  encoding_size: total size of cache files in the encoding/ directory
>  encoding_size_old: total size of 'old' cache files (more than 48 hours)
>
> stats.node.uptime: how many seconds since the node process was started
>
> stats.cpu_monitor.*:
>  .1min_avg, 5min_avg, 15min_avg: estimate of what percentage of system CPU
>                                  time was consumed by the node process,
> over
>                                  the given time interval. Expressed as a
>                                  float, 0.0 for 0%, 1.0 for 100%
>  .total: estimate of total number of CPU seconds consumed by node since
>          the process was started. Ticket #472 indicates that .total may
>          sometimes be negative due to wraparound of the kernel's counter.
>
> stats.load_monitor.*:
>  When enabled, the "load monitor" continually schedules a one-second
>  callback, and measures how late the response is. This estimates system
> load
>  (if the system is idle, the response should be on time). This is only
>  enabled if a stats-gatherer is configured.
>
>  .avg_load: average "load" value (seconds late) over the last minute
>  .max_load: maximum "load" value over the last minute
>
>
> = Running a Tahoe Stats-Gatherer Service =
>
> The "stats-gatherer" is a simple daemon that periodically collects stats
> from
> several tahoe nodes. It could be useful, e.g., in a production environment,
> where you want to monitor dozens of storage servers from a central
> management
> host.
>
> The stats gatherer listens on a network port using the same Foolscap
> connection library that Tahoe clients use to connect to storage servers.
> Tahoe nodes can be configured to connect to the stats gatherer and publish
> their stats on a periodic basis. (in fact, what happens is that nodes
> connect
> to the gatherer and offer it a second FURL which points back to the node's
> "stats port", which the gatherer then uses to pull stats on a periodic
> basis.
> The initial connection is flipped to allow the nodes to live behind NAT
> boxes, as long as the stats-gatherer has a reachable IP address)
>
> The stats-gatherer is created in the same fashion as regular tahoe client
> nodes and introducer nodes. Choose a base directory for the gatherer to
> live
> in (but do not create the directory). Then run:
>
>  tahoe create-stats-gatherer $BASEDIR
>
> and start it with "tahoe start $BASEDIR". Once running, the gatherer will
> write a FURL into $BASEDIR/stats_gatherer.furl .
>
> To configure a Tahoe client/server node to contact the stats gatherer, copy
> this FURL into the node's tahoe.cfg file, in a section named "[client]",
> under a key named "stats_gatherer.furl", like so:
>
>  [client]
>  stats_gatherer.furl = pb://
> qbo4ktl667zmtiuou6lwbjryli2brv6t at 192.168.0.8:49997/wxycb4kaexzskubjnauxeoptympyf45y
>
> or simply copy the stats_gatherer.furl file into the node's base directory
> (next to the tahoe.cfg file): it will be interpreted in the same way.
>
> Once running, the stats gatherer will create a standard python "pickle"
> file
> in $BASEDIR/stats.pickle . Once a minute, the gatherer will pull stats
> information from every connected node and write them into the pickle. The
> pickle will contain a dictionary, in which node identifiers (known as
> "tubid"
> strings) are the keys, and the values are a dict with 'timestamp',
> 'nickname', and 'stats' keys. d[tubid][stats] will contain the stats
> dictionary as made available at http://localhost:3456/statistics?t=json .
> The
> pickle file will only contain the most recent update from each node.
>
> Other tools can be built to examine these stats and render them into
> something useful. For example, a tool could sum the
> "storage_server.disk_avail' values from all servers to compute a
> total-disk-available number for the entire grid (however, the "disk
> watcher"
> daemon, in misc/spacetime/, is better suited for this specific task).
>
> = Using Munin To Graph Stats Values =
>
> The misc/munin/ directory contains various plugins to graph stats for Tahoe
> nodes. They are intended for use with the Munin system-management tool,
> which
> typically polls target systems every 5 minutes and produces a web page with
> graphs of various things over multiple time scales (last hour, last month,
> last year).
>
> Most of the plugins are designed to pull stats from a single Tahoe node,
> and
> are configured with the http://localhost:3456/statistics?t=json URL. The
> "tahoe_stats" plugin is designed to read from the pickle file created by
> the
> stats-gatherer. Some are to be used with the disk watcher, and a few (like
> tahoe_nodememory) are designed to watch the node processes directly (and
> must
> therefore run on the same host as the target node).
>
> Please see the docstrings at the beginning of each plugin for details, and
> the "tahoe-conf" file for notes about configuration and installing these
> plugins into a Munin environment.
>
> _______________________________________________
> tahoe-dev mailing list
> tahoe-dev at allmydata.org
> http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://allmydata.org/pipermail/tahoe-dev/attachments/20091222/96710e64/attachment-0001.htm 


More information about the tahoe-dev mailing list