KeyError exception seen in log when doing a mutable publish on the pubgrid

[davidsarah]

See attached log. This happened when publishing a mutable file version using SFTP (with WinSCP client on Windows), but I don't think that's directly relevant. It may be relevant that there were only two servers connected to the grid at the time. The same error happened again on a retry.

The file in question was mutable.txt in ​this directory.

[davidsarah]

Log showing KeyError? exception

[davidsarah]

The two servers connected were "shiny" (etnv37dedl2kef5c5ny2obgftav2eelw) and the public gateway "soultcer-skulls" (sp26qyqclbwjv6lgqdxik5lfdlxrkpnz). "shiny" is my server and is only on-line when I am. The log is from "shiny".

[davidsarah]

I have also seen this (on the same file) with tahoe put, so it's definitely not SFTP-specific. The encoding parameters had been set to N = 2, k = 2, happy = 2 (but the #778 patches had not been applied).

[zooko]

It's really bothering me that mutable file upload and download behavior is so finicky, buggy, inefficient, hard to understand, different from immutable file upload and download behavior, etc. So I'm putting a bunch of tickets into the "1.8" Milestone. I am not, however, at this time, volunteering to work on these tickets, so it might be a mistake to put them into the 1.8 Milestone, but I really hope that someone else will volunteer or that I will decide to do it myself. :-)

[Brian Warner <warner@…>]

In 370e6f271e40945b:

SDMF: update filenode with correct k/N after Retrieve. Fixes #1510.

Without this, we get a regression when modifying a mutable file that was
created with more shares (larger N) than our current tahoe.cfg . The
modification attempt creates new versions of the (0,1,..,newN-1) shares, but
leaves the old versions of the (newN,..,oldN-1) shares alone (and throws a
assertion error in SDMFSlotWriteProxy.finish_publishing in the process).

The mixed versions that result (some shares with e.g. N=10, some with N=20,
such that both versions are recoverable) cause problems for the Publish code,
even before MDMF landed. Might be related to refs #1390 and refs #1042.

[davidsarah]

The bug in #1510 caused a situation where there were stored shares for an SDMF file with higher shnums than expected. When the resulting file is updated using version 1.8.2 (or earlier, presumably), this would cause a KeyError almost identical to the one in log.txt​. This situation is still inadequately tested, but note that it may cause different errors after 1.9 (specifically, after the MDMF changes were landed) than before 1.9.