#1042 new defect

KeyError exception seen in log when doing a mutable publish on the pubgrid

Reported by: davidsarah Owned by:
Priority: major Milestone: eventually
Component: code-mutable Version: 1.6.1
Keywords: mutable upload reliability Cc:
Launchpad Bug:


See attached log. This happened when publishing a mutable file version using SFTP (with WinSCP client on Windows), but I don't think that's directly relevant. It may be relevant that there were only two servers connected to the grid at the time. The same error happened again on a retry.

The file in question was mutable.txt in this directory.

Attachments (1)

log.txt (9.1 KB) - added by davidsarah at 2010-05-15T03:33:39Z.
Log showing KeyError? exception

Download all attachments as: .zip

Change History (7)

Changed at 2010-05-15T03:33:39Z by davidsarah

Log showing KeyError? exception

comment:1 Changed at 2010-05-15T03:37:45Z by davidsarah

The two servers connected were "shiny" (etnv37dedl2kef5c5ny2obgftav2eelw) and the public gateway "soultcer-skulls" (sp26qyqclbwjv6lgqdxik5lfdlxrkpnz). "shiny" is my server and is only on-line when I am. The log is from "shiny".

comment:2 Changed at 2010-05-19T21:01:23Z by davidsarah

I have also seen this (on the same file) with tahoe put, so it's definitely not SFTP-specific. The encoding parameters had been set to N = 2, k = 2, happy = 2 (but the #778 patches had not been applied).

comment:3 Changed at 2010-05-27T22:12:59Z by zooko

  • Milestone changed from undecided to 1.8.0

It's really bothering me that mutable file upload and download behavior is so finicky, buggy, inefficient, hard to understand, different from immutable file upload and download behavior, etc. So I'm putting a bunch of tickets into the "1.8" Milestone. I am not, however, at this time, volunteering to work on these tickets, so it might be a mistake to put them into the 1.8 Milestone, but I really hope that someone else will volunteer or that I will decide to do it myself. :-)

comment:4 Changed at 2010-08-14T06:45:17Z by zooko

  • Milestone changed from 1.8.0 to eventually

comment:5 Changed at 2011-08-27T22:51:05Z by Brian Warner <warner@…>

In 370e6f271e40945b:

SDMF: update filenode with correct k/N after Retrieve. Fixes #1510.

Without this, we get a regression when modifying a mutable file that was
created with more shares (larger N) than our current tahoe.cfg . The
modification attempt creates new versions of the (0,1,..,newN-1) shares, but
leaves the old versions of the (newN,..,oldN-1) shares alone (and throws a
assertion error in SDMFSlotWriteProxy.finish_publishing in the process).

The mixed versions that result (some shares with e.g. N=10, some with N=20,
such that both versions are recoverable) cause problems for the Publish code,
even before MDMF landed. Might be related to refs #1390 and refs #1042.

comment:6 Changed at 2011-08-27T23:24:51Z by davidsarah

The bug in #1510 caused a situation where there were stored shares for an SDMF file with higher shnums than expected. When the resulting file is updated using version 1.8.2 (or earlier, presumably), this would cause a KeyError almost identical to the one in log.txt. This situation is still inadequately tested, but note that it may cause different errors after 1.9 (specifically, after the MDMF changes were landed) than before 1.9.

Note: See TracTickets for help on using tickets.