#1678 closed defect (fixed)
S3 backend: either handle or avoid truncated get_bucket responses
| Reported by: | davidsarah | Owned by: | davidsarah |
|---|---|---|---|
| Priority: | critical | Milestone: | 1.14.0 |
| Component: | code-storage | Version: | 1.9.0-s3branch |
| Keywords: | s3-backend 410 | Cc: | |
| Launchpad Bug: |
Description (last modified by daira)
The GET Bucket AWS call may return a truncated response, by default after 1000 objects (doc). Currently we don't take that into account (actually I forgot that we didn't :-( ), which might be causing some of the 410 Gone errors.
In the meantime, here is a patch to log this case as WEIRD, so that it will trigger an incident.
Attachments (4)
Change History (18)
comment:1 Changed at 2012-02-16T18:04:53Z by davidsarah
- Owner set to davidsarah
- Status changed from new to assigned
Changed at 2012-02-16T18:15:24Z by davidsarah
comment:2 Changed at 2012-02-16T18:17:52Z by davidsarah
For some reason that patchfile doesn't include the change I recorded. Maybe a side-effect of using --ask-deps. Will fix.
Changed at 2012-02-16T18:49:39Z by davidsarah
comment:3 Changed at 2012-02-16T18:51:34Z by davidsarah
1678-incident-on-truncate-v2.darcs.patch (recorded without --ask-deps) seems to include the change. Odd.
comment:4 Changed at 2012-02-17T14:27:03Z by zooko
I review attachment:1678-incident-on-truncate-v2.darcs.patch . I saw no error, but as David-Sarah mentioned, it needs a test.
Changed at 2012-02-20T18:03:49Z by davidsarah
S3 backend: make truncated GET Bucket responses trigger an incident. Includes tests and patches for #1589.
comment:5 Changed at 2012-02-20T18:04:51Z by davidsarah
- Keywords review-needed added
- Owner changed from davidsarah to zooko
- Status changed from assigned to new
comment:6 Changed at 2012-02-20T18:41:43Z by zooko
- Keywords reviewed added; review-needed removed
Okay, I reviewed the added tests in attachment:1678-incident-on-truncate-v3.darcs.patch and saw no problem!
comment:7 Changed at 2012-02-20T18:41:54Z by zooko
- Owner changed from zooko to davidsarah
comment:8 Changed at 2012-03-05T20:24:22Z by davidsarah
- Priority changed from major to critical
On secorp's LAE storage server, a 500 error (#1590) occurred after a sequence of truncated responses. With a bit of luck, the 500 might be a side effect of the truncated responses so that fixing the latter will also #1590.
Changed at 2012-03-09T00:15:56Z by davidsarah
Implementation of prefix queries, for information only (doesn't fix the problem yet). Depends on txaws 0.2.1.post4, diff from 0.2.1 at https://leastauthority.com/static/patches/txAWS-0.2.1-to-post4.diff
comment:9 Changed at 2012-03-09T00:27:08Z by davidsarah
I implemented prefix queries (so we no longer list all objects in the bucket and filter them, which is something that needed to be fixed anyway). That change seems to be working, and has given a measurable performance improvement of ~0.44 seconds per DHYB on secorp's server, but, it didn't stop the truncated responses as I expected. We no longer get lots of truncated responses with at or near 1000 objects, but we do still get occasional truncated responses with 0 or 1 objects. This makes no sense and is contrary to the S3 API documentation. Frustrating.
comment:10 Changed at 2012-03-09T05:37:54Z by davidsarah
Actually the patch is basically correct. It appeared not to be because we were incorrectly reporting all queries as truncated. (I thought that BucketListing.is_truncated was a boolean rather than a string, and the string "false" is truthy. Down with implicit conversions!)
However, it turns out that truncated queries are not the cause of #1590 :-(
I'll post an updated patch for this ticket, fixing the incorrect detection of truncated queries, tomorrow.
comment:11 Changed at 2012-05-17T00:24:03Z by davidsarah
- Keywords reviewed removed
- Resolution set to fixed
- Status changed from new to closed
Fixed mainly in [5634/ticket999-S3-backend].
comment:12 Changed at 2014-11-27T04:08:46Z by daira
- Description modified (diff)
- Milestone changed from soon to 1.12.0
comment:13 Changed at 2016-03-22T05:02:25Z by warner
- Milestone changed from 1.12.0 to 1.13.0
Milestone renamed
comment:14 Changed at 2016-06-28T18:17:14Z by warner
- Milestone changed from 1.13.0 to 1.14.0
renaming milestone
S3 backend: make truncated GET Bucket responses trigger an incident. Does not include tests. refs #1678 [This depends on the patch for #1589 due to an import in s3_common.py needed by both.]