#541 new defect

remove foolscap 'reference'-token bug workaround in mutable publish

Reported by: warner Owned by:
Priority: major Milestone: 1.15.0
Component: code-mutable Version: 1.2.0
Keywords: backward-compatibility foolscap Cc:
Launchpad Bug:

Description (last modified by zooko)

To close this ticket, remove the workaround which was needed only to avoid a Foolscap bug which is long since fixed.


Foolscap 0.2.5 had a bug in which an inbound constraint (say aTupleConstraint) wouldn't accept a 'reference' token that pointed at a tuple. This would occur when the same tuple was sent multiple times in a single request.

The mutable-file publish code, when it is creating a brand-new mutable file, and when there are fewer than N servers (so multiple shares are sent to the same server), can trigger this case, because the same assert-that-there-are-no-shares test vector (a static tuple of numbers and strings) is used once per share. Python seems to compile the tuple once, so foolscap sees the same object appear in the outbound arguments multiple times, and uses the 'reference' token to preserve this (unintentional) relationship.

The workaround is to construct this tuple at runtime, basically changing:

 for x in something:
   testv = (0, 1, 'eq', '')
   stuff.append(testv)

into:

 for x in something:
   testv = tuple([0, 1, 'eq', ''])
   stuff.append(testv)

The constraint bug was fixed in foolscap-0.2.6, but since it's an inbound-side bug, what really matters is the version of foolscap on the other end of the wire, so to interoperate with older versions of tahoe (which may be using older versions of foolscap), we must avoid sending the 'reference' token at all. So this workaround (which was removed in f7b4c45d4651b25b because tahoe now requires a newer version of foolscap) needs to be reinstated.

In addition, I've seen another problem that relates to this code. A few days ago I started noticing failures in the buildbot 'speed test', which runs trunk code against a statically configured 'perfnet' (with a client node on prodcs2 and four storage nodes plus introducer on tahoebs1). Four nodes and ten shares means multiple shares per node, so the testv-and-readv-and-writev call gets multiple shares, which means a 'reference' token.

The storage server appears to still have problems with the 'reference' token, even though it's running foolscap-0.3.1 . Sometimes I see a Violation that claims getObject(reftoken) returned a non-tuple. Sometimes I see a "dangling reference" BananaError, which indicates that getObject() failed to find the reference. The BananaError causes the connection to be dropped, so any remaining writev calls fail with a DeadReferenceError.

I can provoke this behavior with a local trunk tree against the storage nodes on tahoebs1. I haven't been able to reproduce this with local storage nodes (running foolscap-0.3.1). If I reintroduce the tuple([0,1,'eq','']) workaround, the problem seems to go away.

If I revert the recent versioning changes (which tries to call 'get_version' on the storage server, and falls back to a default value if that fails), then the problem seems to go away. The tahoebs1 perfnet nodes are a few months old (they are running r2901), so they don't have get_version(). So I'm suspecting that the Violation that gets raised somehow messes things up, such that getObject() remembers some bit of bogus state, so later getObject() calls either get a bad value or no value at all.

I'm going to commit the tuple([0,1,'eq','']) workaround, since we need it anyways. But at some point we need to come back to this and examine it further, because I still don't understand the new problem.

The first step will be to add instrumentation to the storage servers, which means first bouncing one of the tahoebs1 nodes and see if it still has the problem. (running the same code locally didn't experience the problem; the only difference I can think of is that the tahoebs1 nodes have been running for two months, and might have all experienced something in that time to mess them up now, whereas my local nodes are younger).

Change History (15)

comment:1 Changed at 2008-11-25T23:04:57Z by warner

More notes on the second problem:

  • bouncing the storage server does *not* make the problem go away
  • adding instrumentation to TupleConstraint reveals that the "not a tuple" complaint is triggered by a list. Instead of getting a tuple, it got a list of [(0, sharedata)] (a list with one element, which is a tuple with two elements). That looks like a write vector.
    • in one test, that write vector was sent as object 79. A subsequent 'reference' token cited object 78. Maybe there's sometimes an off-by-one error?
    • further testing suggests that the setObject() counters are completely broken

comment:2 Changed at 2008-11-25T23:11:13Z by warner

Foolscap#104 created to track this one.

comment:3 Changed at 2008-11-26T00:05:01Z by warner

Ok, I think I have Foolscap #104 figured out: shared references that follow a Violation on the wire will be broken. I won't be likely to fix it until next week, since GC is more important, but by avoiding 'reference' tokens (i.e. avoiding shared references), Tahoe should be ok for now.

comment:4 Changed at 2010-03-24T22:47:18Z by davidsarah

  • Keywords backward-compatibility foolscap added

Which versions of Tahoe used foolscap 0.2.5, 0.2.6 and 0.3.1?

comment:5 Changed at 2010-03-25T06:14:33Z by zooko

Versions of Tahoe-LAFS don't require specific versions of foolscap, only "at least this version" versions of foolscap, so older versions of Tahoe-LAFS could have been paired with newer versions of foolscap. Here are the details:

  • current darcs head of Tahoe-LAFS requires foolscap >= 0.4.1, since: r3874 (2009-05-23)
  • before that it required foolscap >= 0.4.0, since r3870 (2009-05-22)
  • before that it required foolscap >= 0.3.1, since r2968 (2008-09-20)
  • before that it required foolscap >= 0.3.0, since r2825 (2008-08-05)
  • before that it required foolscap >= 0.2.9, since r2729 (2008-07-30)
  • before that it required foolscap >= 0.2.8, since r2666 (2008-06-09)
  • before that it required foolscap >= 0.2.5, since r2381 (2008-04-03)
  • before that it required foolscap >= 0.2.4, since r2062 (2008-02-05)

comment:6 Changed at 2010-03-26T01:46:37Z by zooko

Oh, but I didn't answer which versions of Tahoe-LAFS were involved. I get that information by looking at The Parade of Release Notes:

  • Tahoe-LAFS v1.6.1 r4244 (2010-02-24)
  • Tahoe-LAFS v1.6.0 r4220 (2010-02-02)
  • Tahoe-LAFS v1.5.0 r4037 (2009-08-02)
  • Tahoe-LAFS v1.4.1 r3853 (2009-04-14)
  • Tahoe-LAFS v1.3.0 r3620 (2009-02-14)
  • Tahoe-LAFS v1.2.0 r2789 (2008-07-22)
  • Tahoe-LAFS v1.1.0 r2693 (2008-06-11)
  • Tahoe-LAFS v1.0.0 r2348 (2008-03-26)

comment:7 Changed at 2010-08-15T06:11:20Z by zooko

I guess we can safely remove the work-around now.

comment:8 Changed at 2011-04-27T17:03:39Z by zooko

  • Milestone changed from undecided to 1.10.0

Well, let's leave it in for one more major release... :-)

comment:9 Changed at 2012-03-29T21:23:25Z by davidsarah

http://foolscap.lothar.com/trac/ticket/104 was fixed in foolscap 0.4, which was first required by Tahoe-LAFS r3870, released as Tahoe-LAFS v1.5.0.

relnotes.txt says:

Servers from this release can serve clients of all versions back to v1.0 and clients from this release can use servers of all versions back to v1.0.

which I believe is already incorrect, although I can't remember why not. Should we be making this claim when we don't test it?

comment:10 Changed at 2013-08-13T22:50:29Z by daira

  • Description modified (diff)
  • Milestone changed from 1.11.0 to 1.12.0

comment:11 Changed at 2015-08-25T17:41:42Z by zooko

  • Description modified (diff)
  • Summary changed from foolscap 'reference'-token bug workaround in mutable publish to remove foolscap 'reference'-token bug workaround in mutable publish

comment:12 Changed at 2015-08-25T20:20:06Z by daira

  • Milestone changed from 1.12.0 to 1.11.0

comment:13 Changed at 2016-03-22T05:02:52Z by warner

  • Milestone changed from 1.11.0 to 1.12.0

Milestone renamed

comment:14 Changed at 2016-06-28T18:20:37Z by warner

  • Milestone changed from 1.12.0 to 1.13.0

moving most tickets from 1.12 to 1.13 so we can release 1.12 with magic-folders

comment:15 Changed at 2020-06-30T14:45:13Z by exarkun

  • Milestone changed from 1.13.0 to 1.15.0

Moving open issues out of closed milestones.

Note: See TracTickets for help on using tickets.