Opened at 2012-02-16T18:04:45Z

Closed at 2012-05-17T00:24:03Z

Last modified at 2016-06-28T18:17:14Z

#1678 closed defect (fixed)

S3 backend: either handle or avoid truncated get_bucket responses

Reported by: davidsarah Owned by: davidsarah
Priority: critical Milestone: 1.14.0
Component: code-storage Version: 1.9.0-s3branch
Keywords: s3-backend 410 Cc:
Launchpad Bug:

Description (last modified by daira)

The GET Bucket AWS call may return a truncated response, by default after 1000 objects ​(doc). Currently we don't take that into account (actually I forgot that we didn't :-( ), which might be causing some of the 410 Gone errors.

In the meantime, here is a patch to log this case as WEIRD, so that it will trigger an incident.

Attachments (4)

1678-incident-on-truncate.darcs.patch​ (71.6 KB) - added by davidsarah at 2012-02-16T18:15:24Z.
S3 backend: make truncated GET Bucket responses trigger an incident. Does not include tests. refs #1678 [This depends on the patch for #1589 due to an import in s3_common.py needed by both.]
1678-incident-on-truncate-v2.darcs.patch​ (48.8 KB) - added by davidsarah at 2012-02-16T18:49:39Z.
S3 backend: make truncated GET Bucket responses trigger an incident. Does not include tests. refs #1678 [This depends on the patch for #1589 due to an import in s3_common.py needed by both.]
1678-incident-on-truncate-v3.darcs.patch​ (63.1 KB) - added by davidsarah at 2012-02-20T18:03:49Z.
S3 backend: make truncated GET Bucket responses trigger an incident. Includes tests and patches for #1589.
s3-implement-prefix-queries.darcs.patch​ (138.8 KB) - added by davidsarah at 2012-03-09T00:15:56Z.
Implementation of prefix queries, for information only (doesn't fix the problem yet). Depends on txaws 0.2.1.post4, diff from 0.2.1 at ​https://leastauthority.com/static/patches/txAWS-0.2.1-to-post4.diff

Download all attachments as: .zip

Change History (18)

comment:1 Changed at 2012-02-16T18:04:53Z by davidsarah

  • Owner set to davidsarah
  • Status changed from new to assigned

Changed at 2012-02-16T18:15:24Z by davidsarah

S3 backend: make truncated GET Bucket responses trigger an incident. Does not include tests. refs #1678 [This depends on the patch for #1589 due to an import in s3_common.py needed by both.]

comment:2 Changed at 2012-02-16T18:17:52Z by davidsarah

For some reason that patchfile doesn't include the change I recorded. Maybe a side-effect of using --ask-deps. Will fix.

Changed at 2012-02-16T18:49:39Z by davidsarah

S3 backend: make truncated GET Bucket responses trigger an incident. Does not include tests. refs #1678 [This depends on the patch for #1589 due to an import in s3_common.py needed by both.]

comment:3 Changed at 2012-02-16T18:51:34Z by davidsarah

1678-incident-on-truncate-v2.darcs.patch​ (recorded without --ask-deps) seems to include the change. Odd.

comment:4 Changed at 2012-02-17T14:27:03Z by zooko

I review attachment:1678-incident-on-truncate-v2.darcs.patch​ . I saw no error, but as David-Sarah mentioned, it needs a test.

Changed at 2012-02-20T18:03:49Z by davidsarah

S3 backend: make truncated GET Bucket responses trigger an incident. Includes tests and patches for #1589.

comment:5 Changed at 2012-02-20T18:04:51Z by davidsarah

  • Keywords review-needed added
  • Owner changed from davidsarah to zooko
  • Status changed from assigned to new

comment:6 Changed at 2012-02-20T18:41:43Z by zooko

  • Keywords reviewed added; review-needed removed

Okay, I reviewed the added tests in attachment:1678-incident-on-truncate-v3.darcs.patch​ and saw no problem!

comment:7 Changed at 2012-02-20T18:41:54Z by zooko

  • Owner changed from zooko to davidsarah

comment:8 Changed at 2012-03-05T20:24:22Z by davidsarah

  • Priority changed from major to critical

On secorp's LAE storage server, a 500 error (#1590) occurred after a sequence of truncated responses. With a bit of luck, the 500 might be a side effect of the truncated responses so that fixing the latter will also fix #1590.

Last edited at 2012-03-05T20:24:47Z by davidsarah (previous) (diff)

Changed at 2012-03-09T00:15:56Z by davidsarah

Implementation of prefix queries, for information only (doesn't fix the problem yet). Depends on txaws 0.2.1.post4, diff from 0.2.1 at ​https://leastauthority.com/static/patches/txAWS-0.2.1-to-post4.diff

comment:9 Changed at 2012-03-09T00:27:08Z by davidsarah

I implemented prefix queries (so we no longer list all objects in the bucket and filter, which needed to be fixed anyway). That seems to be working, and has given a measurable performance improvement of ~0.44 seconds per DHYB on secorp's server, but, it didn't stop the truncated responses as I expected. We no longer get lots of truncated responses with at or near 1000 objects, but we do still get occasional truncated responses with 0 or 1 objects. This makes no sense and is contrary to the S3 API documentation. Frustrating.

Version 0, edited at 2012-03-09T00:27:08Z by davidsarah (next)

comment:10 Changed at 2012-03-09T05:37:54Z by davidsarah

Actually the patch is basically correct. It appeared not to be because we were incorrectly reporting all queries as truncated. (I thought that BucketListing.is_truncated was a boolean rather than a string, and the string "false" is truthy. Down with implicit conversions!)

However, it turns out that truncated queries are not the cause of #1590 :-(

I'll post an updated patch for this ticket, fixing the incorrect detection of truncated queries, tomorrow.

comment:11 Changed at 2012-05-17T00:24:03Z by davidsarah

  • Keywords reviewed removed
  • Resolution set to fixed
  • Status changed from new to closed

Fixed mainly in [5634/ticket999-S3-backend].

comment:12 Changed at 2014-11-27T04:08:46Z by daira

  • Description modified (diff)
  • Milestone changed from soon to 1.12.0

comment:13 Changed at 2016-03-22T05:02:25Z by warner

  • Milestone changed from 1.12.0 to 1.13.0

Milestone renamed

comment:14 Changed at 2016-06-28T18:17:14Z by warner

  • Milestone changed from 1.13.0 to 1.14.0

renaming milestone

Note: See TracTickets for help on using tickets.