KeyError in mutable download

[warner]

Zooko observed a KeyError exception raised during mutable-file retrieval, in #651 (specifically at git/src/allmydata/mutable/retrieve.py?rev=4061258c#L157). This is a bug in the retrieval code, probably triggered by running out of valid shares. Either the self.remaining_sharemap mapping should not be modified while download() is iterating over it, or more likely download() must be prepared for remaining_sharemap to change during the iteration.

[zooko]

It's really bothering me that mutable file upload and download behavior is so finicky, buggy, inefficient, hard to understand, different from immutable file upload and download behavior, etc. So I'm putting a bunch of tickets into the "1.8" Milestone. I am not, however, at this time, volunteering to work on these tickets, so it might be a mistake to put them into the 1.8 Milestone, but I really hope that someone else will volunteer or that I will decide to do it myself. :-)

[zooko]

If you like this ticket, you're probably the kind of weirdo who would like #474 (uncaught exception in mutable-retrieve: UCW between mapupdate and retrieve), #548 (mutable publish sends queries to servers that have already been asked), #540 (inappropriate "uncoordinated write error" after handling a server failure), #546 (mutable-file surprise shares raise inappropriate UCWE), #547 (mapupdate(MODE_WRITE) triggers on a false boundary), #548 (mutable publish sends queries to servers that have already been asked), #549 (MODE_WRITE mapupdate: maybe increase epsilon to handle large batches of new servers better). And we need more weirdos like you!

[davidsarah]

This seems unlikely to be fixed for 1.9.

[davidsarah]

The description of this ticket probably refers to the code as it was before the start of Kevan's MDMF patches, i.e. up to and including darcs revision 4772 / git revision 4061258c. The code changes a lot for MDMF and this bug might have been incidentally fixed -- in particular, there is no longer any indexing into self.remaining_sharemap, and all of the loops over its keys are done synchronously with the keys having been copied beforehand.

If that's correct, this ticket can be closed as "cannot reproduce".

(There was a vaguely similar bug in git/src/allmydata/mutable/publish.py that is easy to confuse with this one, but that was fixed in #1749.)

[davidsarah]

Editing description to refer to the right revision.