fix the definition of needs-rebalancing

[daira]

Split from #1784 (also see 1115#comment:27):

The value of needs-rebalancing is computed inconsistently between checker.py and filenode.py. In checker.py it is computed as:

# The file needs rebalancing if the set of servers that have at least
# one share is less than the number of uniquely-numbered shares
# available.
# TODO: this may be wrong, see ticket #1115 comment:27 and ticket #1784.
needs_rebalancing = bool(good_share_hosts < len(verifiedshares))

In filenode.py it is computed as

# TODO: this may be wrong, see ticket #1115 comment:27 and ticket #1784.
needs_rebalancing = bool(len(sm) >= verifycap.total_shares)

where len(sm) is equal to count-shares-good. I don't understand this latter definition at all, it looks completely wrong. The definition in checker.py is more subtly wrong because it credits servers that only have duplicated shares as contributing to existing balance. The correct definition should be something like 'iff the happiness count is less than the number of uniquely-numbered good shares available'.

I propose to change filenode.py and checker.py to be consistent, and to both use the happiness-based definition above.

[daira]

"comments and a caveat in webapi.rst indicating that the needs-rebalancing field may be computed incorrectly" were added in [b06f8cd8d03a6239/trunk].

[zooko]

I think this ticket is actually about "needs-repair" rather than about "needs-rebalancing". I'm not 100% sure of the definition of "needs-rebalancing", but I think of it as something like "The file doesn't need repair (i.e. its Happiness level is already above the requirement), but it would be better to move some of the shares around to other servers.

Possible reasons that moving shares might be useful even when it doesn't improve Happiness:

• to optimize for faster-finding of the shares by moving them to servers earlier in the permuted ring, or
• to move shares from servers that are full to servers that are mostly empty (load-balancing total storage)

[daira]

Replying to zooko:

I think this ticket is actually about "needs-repair" rather than about "needs-rebalancing". I'm not 100% sure of the definition of "needs-rebalancing", but I think of it as something like "The file doesn't need repair (i.e. its Happiness level is already above the requirement), but it would be better to move some of the shares around to other servers.

From docs/frontends/webapi.rst@0bebbe32#debugging-and-testing-features​:

  needs-rebalancing: (bool) This field is intended to be True iff
                     reliability could be improved for this file by
                     rebalancing, i.e. by moving some shares to other
                     servers. It may be incorrect in some cases for
                     Tahoe-LAFS up to and including v1.10, and its
                     precise definition is expected to change.

Reliability can be improved when the shares-of-happiness count is less than shares.total. (In some cases, adding more servers may be necessary for an improvement, but I think that's fine.)

[daira]

We decided to remove needs-rebalancing for 1.11.

[daira]

​​https://github.com/tahoe-lafs/tahoe-lafs/pull/74 implements this ticket (i.e. removes need-rebalancing) and #1784.

[remyroy]

I'll review this patch/pull request.

[remyroy]

Review:

Everything seems great except for the use of the somewhat confusing "Happiness Count" expression. Count is fine for something countable like the number of good shares but it seems wrong for this. I think it would be preferable to use something like "Happiness Level", "Happiness Metric" or "Servers-of-Happiness Level".

Reassigning to daira for patch changes.

[daira]

Zooko convinced me in the Tahoe hack hour to use "Happiness Level" (because it goes well with "happiness threshold").

[Daira Hopwood <daira@…>]

In 0ef593947755a8b81edc73d033219724268faf80/trunk:

Remove 'needs-rebalancing' and add 'count-happiness' to checker reports; repair tests. fixes #1784, #2105

Signed-off-by: Daira Hopwood <daira@…>

[Daira Hopwood <daira@…>]

In 0ef593947755a8b81edc73d033219724268faf80/trunk:

Remove 'needs-rebalancing' and add 'count-happiness' to checker reports; repair tests. fixes #1784, #2105

Signed-off-by: Daira Hopwood <daira@…>

[Daira Hopwood <daira@…>]

In 0ef593947755a8b81edc73d033219724268faf80/trunk:

Remove 'needs-rebalancing' and add 'count-happiness' to checker reports; repair tests. fixes #1784, #2105

Signed-off-by: Daira Hopwood <daira@…>

[Daira Hopwood <daira@…>]

In 0ef593947755a8b81edc73d033219724268faf80/trunk:

Remove 'needs-rebalancing' and add 'count-happiness' to checker reports; repair tests. fixes #1784, #2105

Signed-off-by: Daira Hopwood <daira@…>

[Daira Hopwood <daira@…>]

In 0ef593947755a8b81edc73d033219724268faf80/trunk:

Remove 'needs-rebalancing' and add 'count-happiness' to checker reports; repair tests. fixes #1784, #2105

Signed-off-by: Daira Hopwood <daira@…>