#948 closed defect (fixed)

LiteralFileURI instance has no attribute 'storage_index'

Reported by: francois Owned by: bigpig
Priority: critical Milestone: 1.6.1
Component: code-dirnodes Version: 1.6.0
Keywords: preservation verify repair tahoe-backup usability error reliability regression tahoe-check wui test reviewed Cc:
Launchpad Bug:

Description

A daily cronjob is running 'tahoe deep-check --repair' on files which were created by 'tahoe backup'. It began to fail with the following exception sometimes around the implementation of immutable directories.

As soon as this failure occurs, the deep-check operation is aborted. This might be dangerous because the remaining files are not being correctly checked.

$ tahoe deep-check -v --repair --add-lease tahoe:
[...]
ERROR: AttributeError(LiteralFileURI instance has no attribute 'storage_index')
[Failure instance: Traceback: <type 'exceptions.AttributeError'>: LiteralFileURI instance has no attribute 'storage_index'
/usr/lib/python2.5/site-packages/twisted/internet/defer.py:285:unpause
/usr/lib/python2.5/site-packages/twisted/internet/defer.py:328:_runCallbacks
/home/francois/dev/tahoe/src/allmydata/dirnode.py:676:<lambda>
/home/francois/dev/tahoe/src/allmydata/dirnode.py:631:_deep_traverse_dirnode
--- <exception caught here> ---
/usr/lib/python2.5/site-packages/twisted/internet/defer.py:106:maybeDeferred
/home/francois/dev/tahoe/src/allmydata/web/directory.py:1114:add_node
/home/francois/dev/tahoe/src/allmydata/dirnode.py:349:get_storage_index
$ tahoe --version
allmydata-tahoe: 1.5.0-r4228, foolscap: 0.4.2, pycryptopp: 0.5.15, zfec: 1.4.5, Twisted: 8.1.0, Nevow: 0.9.33-r17222, zope.interface: 3.3.1, python: 2.5.2, platform: Linux-debian_5.0.4-i686-32bit, sqlite: 3.5.9, simplejson: 1.9.2, argparse: 0.9.1, pyOpenSSL: 0.7, pyutil: 1.3.34, zbase32: 1.1.1, setuptools: 0.6c12dev, pysqlite: 2.3.2

Attachments (12)

fix-948-darcspatch.txt (69.4 KB) - added by davidsarah at 2010-02-22T03:01:44Z.
Change direct accesses to an_uri.storage_index to calls to .get_storage_index() (fixes #948)
test-web-948-darcspatch.txt (56.9 KB) - added by davidsarah at 2010-02-22T03:02:34Z.
Additions to test_web.py for #948
all-948-darcspatch.txt (76.9 KB) - added by davidsarah at 2010-02-22T03:45:29Z.
Additional fix for abbrev_si, with test (includes previous two patches)
test-dirnode-lit.darcspatch.txt (10.4 KB) - added by zooko at 2010-02-22T05:37:15Z.
add tests of literal dirnodes (the current code already passes these tests) This patch replaces the previous patch by the same name, which was buggy.
test-dirnode-lit-2-darcspatch.txt (66.5 KB) - added by davidsarah at 2010-02-24T04:44:46Z.
dirnode: add tests of literal dirnodes (current and fix for #948)
More Info for DIR2-LIT directory.html (18.1 KB) - added by davidsarah at 2010-02-24T05:41:51Z.
Nevow traceback showing bug when following More Info link of a DIR2-LIT directory (http://127.0.0.1:3456/uri/URI%3ADIR2-LIT%3Agqytunj2onug64tufqzdcosvkjetutcjkq5gw4tvm5vwszdgnz5hgyzufqydulbshj5x2lbm/?t=info)
test-dirnode-lit-in-deep-check.darcspatch.txt (17.3 KB) - added by zooko at 2010-02-24T06:56:03Z.
add tests of literal dirnodes to test_deepcheck.py (the current trunk fails these tests and so does the current all-948-darcspatch.txt! Of course I'm not sure if it is the tests or the code that is wrong)
test-dirnode-lit-in-deep-check-2-darcspatch.txt (70.1 KB) - added by davidsarah at 2010-02-24T08:07:55Z.
directories: add DIR2-LIT directories to test_deepcheck.py (#948) -- fixes test bugs
additional-fixes-for-948-darcspatch.txt (58.4 KB) - added by davidsarah at 2010-02-24T08:16:09Z.
Additional fixes for DIR2-LIT More Info page and deep-check/manifest operations (#948)
updates-to-NEWS-darcspatch.txt (54.2 KB) - added by davidsarah at 2010-02-24T08:24:52Z.
Updates to NEWS relevant to this ticket and #577
test-dirnode-lit-in-wui-darcspatch.txt (56.5 KB) - added by davidsarah at 2010-02-25T04:28:35Z.
Additional test for DIR2-LIT directories in test_web.py, fixed version (#948). Corrects a mistake in the use of Deferreds.
all-darcspatch.txt (122.9 KB) - added by davidsarah at 2010-02-27T07:08:51Z.
All patches to be applied for #948

Download all attachments as: .zip

Change History (38)

comment:1 Changed at 2010-02-14T22:46:16Z by davidsarah

  • Component changed from code-mutable to code-dirnodes
  • Keywords error reliability added; integrity removed
  • Milestone changed from undecided to 1.7.0

get_storage_index on DirectoryNode is implemented like this:

    def get_storage_index(self):
        return self._uri._filenode_uri.storage_index

which makes unwarranted assumptions about attributes of URI objects. It should be

    def get_storage_index(self):
        return self._uri.get_storage_index()

(This delegates to get_storage_index on _DirectoryBaseURI, which delegates to get_storage_index on LiteralFileURI, which correctly returns None. The bug probably occurred as a result of trying to skip one of these delegation steps, violating abstraction.)

Please check whether this is sufficient to fix the problem -- I haven't checked whether tahoe check handles a storage index of None correctly, although it should if it handles LIT files.

comment:2 Changed at 2010-02-14T22:51:03Z by davidsarah

  • Keywords regression added

(I meant tahoe deep-check, not tahoe check.)

This is a regression relative to 1.5.0, since tahoe backup didn't use immutable directories then, and only immutable directories can wrap a LiteralFileURI.

comment:3 Changed at 2010-02-14T23:32:31Z by davidsarah

Looking at where get_storage_index, it seems that several other operations on empty immutable directories that are small enough to fit in a LIT file are likely to fail. Viewing them in the WUI certainly provokes the same error:

We should probably put out a point release (although there's time to fix other stuff and perhaps include #778.)

comment:4 Changed at 2010-02-15T00:09:13Z by davidsarah

  • Keywords tahoe-check wui added

comment:5 Changed at 2010-02-15T06:24:27Z by warner

In general, a storage_index of None is used to indicate a "non-distributed file", which the Checker ought to skip entirely (non-distributed files are always healthy). I'd guess that we used to have about 90% support for these (i.e. of the code that I wrote, I probably made it tolerate LITs about 90% of the time, and/or we have test coverage of this for about 90% of the code paths). But it sounds like something changed recently.

comment:6 Changed at 2010-02-15T19:25:09Z by davidsarah

  • Milestone changed from 1.7.0 to 1.6.1

comment:7 Changed at 2010-02-15T19:28:39Z by davidsarah

  • Keywords test added

As far as I can see, we have no tests for DIR2-LIT URIs, which is why this escaped notice in the 1.6.0 release cycle.

comment:8 Changed at 2010-02-21T02:50:19Z by zooko

I'm writing tests.

comment:9 Changed at 2010-02-22T02:07:32Z by davidsarah

Zooko is writing tests for test_dirnode.py. I'll write them for test_web.py, and write the actual fix. Note that there are other cases -- lots in test code and a few in non-test code -- where we directly access .storage_index on an URI object rather than calling .get_storage_index().

Changed at 2010-02-22T03:01:44Z by davidsarah

Change direct accesses to an_uri.storage_index to calls to .get_storage_index() (fixes #948)

Changed at 2010-02-22T03:02:34Z by davidsarah

Additions to test_web.py for #948

Changed at 2010-02-22T03:45:29Z by davidsarah

Additional fix for abbrev_si, with test (includes previous two patches)

Changed at 2010-02-22T05:37:15Z by zooko

add tests of literal dirnodes (the current code already passes these tests) This patch replaces the previous patch by the same name, which was buggy.

comment:10 Changed at 2010-02-22T05:45:57Z by zooko

We should add a test of deepcheck (where this issue was first detected). I think all we need to do is add a DIR2:LIT to DeepCheckWebGood.set_up_tree().

comment:11 Changed at 2010-02-23T04:29:04Z by davidsarah

DeepCheckWebGood?.set_up_tree: agreed. I also want to check the WUI output for a DIR2:LIT, which is in test_web.py.

Changed at 2010-02-24T04:44:46Z by davidsarah

dirnode: add tests of literal dirnodes (current and fix for #948)

Changed at 2010-02-24T05:41:51Z by davidsarah

Nevow traceback showing bug when following More Info link of a DIR2-LIT directory (http://127.0.0.1:3456/uri/URI%3ADIR2-LIT%3Agqytunj2onug64tufqzdcosvkjetutcjkq5gw4tvm5vwszdgnz5hgyzufqydulbshj5x2lbm/?t=info)

Changed at 2010-02-24T06:56:03Z by zooko

add tests of literal dirnodes to test_deepcheck.py (the current trunk fails these tests and so does the current all-948-darcspatch.txt! Of course I'm not sure if it is the tests or the code that is wrong)

Changed at 2010-02-24T08:07:55Z by davidsarah

directories: add DIR2-LIT directories to test_deepcheck.py (#948) -- fixes test bugs

Changed at 2010-02-24T08:16:09Z by davidsarah

Additional fixes for DIR2-LIT More Info page and deep-check/manifest operations (#948)

comment:12 Changed at 2010-02-24T08:21:07Z by davidsarah

  • Keywords review-needed added

The patch bundles that need review are:

  • all-948-darcspatch.txt
  • test-dirnode-lit-2-darcspatch.txt
  • test-dirnode-lit-in-deep-check-2-darcspatch.txt
  • additional-fixes-for-948-darcspatch.txt

This does not include the test of WUI output, which is currently failing (the functionality seems to work, so I think it's a test problem -- will investigate tomorrow).

Changed at 2010-02-24T08:24:52Z by davidsarah

Updates to NEWS relevant to this ticket and #577

comment:13 follow-up: Changed at 2010-02-24T14:51:29Z by zooko

I think we need to add to some of these tests -- at least to test_deepcheck.py -- a large immutable directory. Large enough so that it isn't a LIT.

comment:14 in reply to: ↑ 13 Changed at 2010-02-25T03:02:16Z by davidsarah

Replying to zooko:

I think we need to add to some of these tests -- at least to test_deepcheck.py -- a large immutable directory. Large enough so that it isn't a LIT.

I agree that would be useful, but strictly speaking this ticket is about DIR2-LITs, so let's get these patches reviewed first.

Changed at 2010-02-25T04:28:35Z by davidsarah

Additional test for DIR2-LIT directories in test_web.py, fixed version (#948). Corrects a mistake in the use of Deferreds.

comment:15 Changed at 2010-02-27T03:26:59Z by zooko

  • Owner set to zooko
  • Status changed from new to assigned

Okay I'm reviewing

  • test-dirnode-lit-in-wui-darcspatch.txt
  • updates-to-NEWS-darcspatch.txt
  • all-948-darcspatch.txt
  • test-dirnode-lit-2-darcspatch.txt
  • test-dirnode-lit-in-deep-check-2-darcspatch.txt
  • additional-fixes-for-948-darcspatch.txt

Is that the right set to review?

comment:16 Changed at 2010-02-27T04:01:06Z by zooko

I reviewed test-dirnode-lit-in-wui-darcspatch.txt and it looks good.

comment:17 follow-up: Changed at 2010-02-27T04:03:41Z by zooko

comment:19 Changed at 2010-02-27T04:24:29Z by zooko

Okay, now I've reviewed test-dirnode-lit-in-deep-check-2-darcspatch.txt and approved it. Technically, it would be better if someone else reviewed test-dirnode-lit-in-deep-check-2-darcspatch.txt since I'm part author of it, but Brian's not around on IRC right now and I'm hoping to announce v1.6.1 tomorrow! :-) If Kevan or anyone else is reading this and wants to read over test-dirnode-lit-in-deep-check-2-darcspatch.txt and look for goofs, that would be great. It will probably take only a few minutes, especially if you don't mind being uncertain about exactly what all the tests do and just look for things that look suspiciously like mistakes to you. If you develop a confident understanding of what the tests actually do here, all the better!

comment:20 Changed at 2010-02-27T04:27:24Z by zooko

Okay, I've reviewed test-dirnode-lit-2-darcspatch.txt and I like it. Again, I'm not the perfect reviewer since I'm the author of most of this one, so if you are reading this and you can double-check that patch, that would be great.

comment:21 Changed at 2010-02-27T05:47:55Z by zooko

Okay, I've reviewed all-948-darcspatch.txt and could find nothing in it to change!

comment:22 Changed at 2010-02-27T06:06:13Z by zooko

Okay I'm looking at additional-fixes-for-948-darcspatch.txt, which is the last patch from the set of six patches that constitute the fix for this ticket (per comment:15). For this one I have some questions:

  • Is this correct:
    +            self.root_storage_index_s = "<none>"  # is this correct?
    

(Elsewhere we use either "" or "<LIT>" to be the string encoding of a null storage index.)

  • Should we add tests of the changes in this patch, mainly the changes to emit "" when a storage index is None, or there already unit tests that fail when those changes are not applied?
  • The construction base32.b2a(m.origin_si or "") feels slightly weird to me because it feels like a type error—we really mean that the expression should evaluate to "" not that the argument to b2a() should be "". However, the alternative is bigger and uglier: m.origin_si and base32.b2a(m.origin_si) or "". Maybe we should define a function:
    def b2a_or_empty(si):
        if si:
            return base32.b2a(si)
        else:
            return ""
    

comment:23 Changed at 2010-02-27T06:06:35Z by zooko

  • Keywords review-needed removed
  • Owner changed from zooko to davidsarah
  • Status changed from assigned to new

Changed at 2010-02-27T07:08:51Z by davidsarah

All patches to be applied for #948

comment:24 Changed at 2010-02-27T07:12:50Z by davidsarah

  • Keywords review-needed added
  • Owner changed from davidsarah to bigpig

comment:25 Changed at 2010-02-27T07:34:34Z by davidsarah

  • Keywords reviewed added; review-needed removed

all-darcspatch.txt includes a change to avoid base32.b2a(... or ""), although not using a separate function. This change has been reviewed by bigpig.

The changes to emit "" when the storage index is None are tested by test_deepcheck.py. We decided not to change "<none>" for 1.6.1.

Note: See TracTickets for help on using tickets.