document munin plugins and make them discoverable

[zooko]

allmydata.com maintains some munin plugins to help you monitor the health and performance of your Tahoe-LAFS nodes. You can find them in misc/munin. But how are users supposed to discover this fact, and how do they learn how to use these munin plugins?

[zooko]

Please use a more informative patch description, like the example patches shown here:

​http://tahoe-lafs.org/trac/tahoe-lafs/wiki/Patches

[freestorm]

Split munin plugins into 2 directories

[freestorm]

Please ignore munin_stats-gatherer.darcs.txt patch

[freestorm]

Is tahoe_estimate_files​ relevant?

I can't get it working with my config

I set:

node_dirs = [os.path.expanduser("/home/tahoe-storage/")]
N = 10
num_servers = 5

correction is always 0.0, so

files = unique_strings * (32*32/len(sections)) * correction

will always return files.value 0

[freestorm]

I need help from somebody to review the future README file for stats-gatherer plugins.

Please read and edit ​this wiki page
When file is going to be ready, I will create the patch.

[warner]

tahoe_estimate_files assumes an allmydata-scale grid, in which the number of servers is somewhat larger than the number of shares (N), such that each server will have either 1 or 0 shares. With your values of N=10 and num_servers=5, each server will have 2 shares, and we'd need a different equation. I think for your environment you'd want to set correction to 1.0 to get it right (if every file appears on every server, there's no undercounting to be corrected for, so the chance/no_chance/correction stuff is irrelevant).

[freestorm]

Thanks for your reply Warner,

I will modify tahoe_estimate_files​ to add plugin settings in tahoe-conf​ like other plugins

[freestorm]

Patch for Munin Stats-Gatherer plugins

[freestorm]

Patch ​munin_plugins_for_stats-gatherer.dpatch is ready for review.
It contain modifications for Munin Stats-Gatherer plugins only
Another patch will contain "local_plugins" modifications

[freestorm]

Munin plugin ​tahoe_overhead speak about a PHP script to collect total-deep-size from another server.

Somebody know where this PHP script can be found?

[secorp]

The last two attachments are the working copies of the php scripts. The checked in versions are in the www project on the old dev Allmydata box. We should really get this out into the public too ... I thought we had done this already but can't seem to find the trac instance we set up for it.

[freestorm]

Replying to secorp:

The last two attachments are the working copies of the php scripts. The checked in versions are in the www project on the old dev Allmydata box. We should really get this out into the public too ... I thought we had done this already but can't seem to find the trac instance we set up for it.

Thanks for the files secorp

I looked them, but to fully understand them, I need the DB structure, .inc/includes.php and .inc/ui/dbconnection.php.

Actually I will mark ​tahoe_overhead as boken, because we need to close #669 before continue with those PHP scripts.

[warner]

those PHP scripts are deeply obsolete, because they depended upon the DB that managed customer accounts at allmydata.com . I'd suggest adding a comment that says "to use this script, you will need some source of 'tahoe deep-size' results for all rootcaps that reference data in your grid. This tool was written for the allmydata.com environment, which maintained such a value."

[freestorm]

I wrote the README file for munin local plugins.

It would be nice if someone can take a look a this wiki ​page to correct my spelling errors.

As far as this page is good I will write the Patch.

[terrell]

i've looked over the linked page in comment:13 above and made some whitespace and formatting/spelling changes.

i'm not sure of the content itself - there are underscores at the end of two of the plugin names...

the two examples at the end use a plugin that isn't listed in the table.

[terrell]

Replying to freestorm:

I need help from somebody to review the future README file for stats-gatherer plugins.

Please read and edit ​this wiki page
When file is going to be ready, I will create the patch.

i've retouched this page as well.

[freestorm]

Replying to terrell:

i've looked over the linked page in comment:13 above and made some whitespace and formatting/spelling changes.

i'm not sure of the content itself - there are underscores at the end of two of the plugin names...

the two examples at the end use a plugin that isn't listed in the table.

Thanks for your help. I'm going to edit the two example at the end of file.

The underscore at the end of some plugin is normal.

But I'm not sure that we need to put the README in wiki format. The final README format will be plain text. I use the wiki to review those files before creating the patch

[freestorm]

Munin local plugins

[freestorm]

Patch ​munin_plugins_for_stats-gatherer.dpatch is ready for review.
It contain modifications for Munin Stats-Gatherer plugins only

Patch ​local_munin_plugins.darcs.patch is ready for review.
It contain modifications for Munin Local plugins.

[zooko]

We could consider including this in 1.8.1 because it doesn't touch core code and therefore can't introduce a regression.

[francois]

Hi Fred, I have reviewed your two patches, see my comments in-line.

Replying to freestorm:

Patch ​munin_plugins_for_stats-gatherer.dpatch is ready for review.

• ​line 100: These plugins need a Tahoe-LAFS Helper

This statement is wrong, a stats-gatherer is required, not a helper. I guess you can remove this line because this is already stated previously in the document.

• You should probably warn users in the NEWS file that tahoe_* plugins have been renamed tahoe_stats_*.

• It seems that there is weird indentation between lines ​773 and ​776.

Patch ​local_munin_plugins.darcs.patch is ready for review.
It contain modifications for Munin Local plugins.

• A small typo on ​line 27, Taoe-LAFS instead of Tahoe-LAFS

• The directory mentionned in ​line 38 should be probably be specified relative to the top directory, ie. contrib/stats-gatherer_plugins/ instead of stats-gatherer_plugins. Same comment for all directories.

That's all, thanks for updating this documentation, I'm sure that new users of the munin plugins are going to have much less trouble getting started ;)

[freestorm]

Replying to francois:

Hi François,

I have some questions about your comments.

Hi Fred, I have reviewed your two patches, see my comments in-line.

Replying to freestorm:

Patch ​munin_plugins_for_stats-gatherer.dpatch is ready for review.

• ​line 100: These plugins need a Tahoe-LAFS Helper

This statement is wrong, a stats-gatherer is required, not a helper. I guess you can remove this line because this is already stated previously in the document.

I wrote this because if you don't have a Tahoe-LAFS Helper, it make no sense to run these plugins.
The helpers plugins would return 0 or NULL if it not exist a Helper on the grid (I need to make further tests to be sure)

• You should probably warn users in the NEWS file that tahoe_* plugins have been renamed tahoe_stats_*.

• It seems that there is weird indentation between lines ​773 and ​776.

Patch ​local_munin_plugins.darcs.patch is ready for review.
It contain modifications for Munin Local plugins.

• A small typo on ​line 27, Taoe-LAFS instead of Tahoe-LAFS

• The directory mentionned in ​line 38 should be probably be specified relative to the top directory, ie. contrib/stats-gatherer_plugins/ instead of stats-gatherer_plugins. Same comment for all directories.

Need I refer all directories from $TAHOE_SOURCE?
In the current Patch, all directories are mentionned from these files:

• README
• stats-gatherer_plugins/README
• local_plugins/README

That's all, thanks for updating this documentation, I'm sure that new users of the munin plugins are going to have much less trouble getting started ;)

[francois]

Replying to freestorm:

I wrote this because if you don't have a Tahoe-LAFS Helper, it make no sense to run these plugins.
The helpers plugins would return 0 or NULL if it not exist a Helper on the grid (I need to make further tests to be sure)

Oops, my bad, my comment was indeed incorrect, please discard it.

Need I refer all directories from $TAHOE_SOURCE?
In the current Patch, all directories are mentionned from these files:

• README
• stats-gatherer_plugins/README
• local_plugins/README

Yes, it looks much clearer to me to always use the top directory as reference.