Opened at 2015-12-10T15:27:42Z
Closed at 2016-02-08T13:29:27Z
#2635 closed defect (fixed)
Magic Folder: periodically do a full uploader scan
Reported by: | daira | Owned by: | daira |
---|---|---|---|
Priority: | normal | Milestone: | undecided |
Component: | code-frontend-magic-folder | Version: | n/a |
Keywords: | magic-folder reliability upload | Cc: | |
Launchpad Bug: |
Description
This would help to compensate for:
- missed events, due to unreliability of the filesystem notification mechanism or queue overflows (#1430),
- failed uploads.
Change History (22)
comment:1 Changed at 2015-12-14T15:59:22Z by daira
- Owner changed from daira to dawuud
comment:2 Changed at 2015-12-15T16:25:23Z by dawuud
in my dev branch i taught the uploader's _scan method to use the deque instead of calling _process directly. this breaks all our magic-folder tests... so the next is to make the tests pass by firing the hook deferred when a directory and all it's children are done being uploaded.
https://github.com/david415/tahoe-lafs/tree/2635.periodic-uploader-scan.0
comment:3 Changed at 2015-12-16T15:08:02Z by dawuud
after teaching the uploader to use the deque to process directories i've been trying to get the tests to pass... and i'm not sure how to do it but i pushed my latest WIP comment anyways.
https://github.com/david415/tahoe-lafs/tree/2635.periodic-uploader-scan.0
maybe daira or meejah could help me with this?
comment:4 Changed at 2015-12-16T15:08:17Z by dawuud
- Owner changed from dawuud to daira
comment:5 Changed at 2015-12-16T16:04:13Z by daira
The current design is wrong because it assumes that _scan finishes processing all of the descendant directories under the directory it is called with, when actually it only adds callbacks to scan all of its direct children. This affects both the initial scan triggered by start_scanning (which assumes that all desendants have been removed from the _pending set), and the intent of the tests that move subtrees.
comment:6 Changed at 2015-12-17T11:06:16Z by dawuud
It's almost as if we want nested DeferredList?`s representing nested directories such that we can determine when a given directory and all it's children have been uploaded.
comment:7 Changed at 2015-12-17T16:23:19Z by dawuud
here's the dev branch from our pairing session today: https://github.com/david415/tahoe-lafs/tree/2635.periodic-uploader-scan.1
comment:8 Changed at 2015-12-17T21:50:20Z by dawuud
ok... all the tests are passing again with this commit: https://github.com/david415/tahoe-lafs/commit/4610a7f97d4823548129b4d483d7e50e4c5c18ee
next i need to periodic scan... should be easy.
comment:9 Changed at 2015-12-17T23:23:27Z by dawuud
here's a naive periodic full scan: https://github.com/david415/tahoe-lafs/commit/d1eef67f7632ee96d8e3e30b1c8b5f9d084d8973
actually i think it's an open question how we should determine the full scan is complete and we should therefore perform another full scan now because our duration has elapsed.
just think how bad it would be if the full scan didn't complete before the next one started!
perhaps we can just check the pending set every X minutes and if it's empty then do the full scan?
comment:10 Changed at 2015-12-18T00:37:03Z by dawuud
ok i made it only do the full scan if the pending set is empty. i also added a unit test... it's a MockTest? similar to test_move_tree but upon file creation inside the magicfolder dir we do not emit a fake inotify event. instead we wait for the periodic scan and upload... however i am trying to use clock advances to speed up this test and it's not yet passing.
comment:11 Changed at 2015-12-21T17:57:49Z by daira
- Status changed from new to assigned
Reviewing progress so far.
comment:12 Changed at 2015-12-21T22:30:43Z by daira
comment:13 Changed at 2015-12-22T16:14:30Z by dawuud
work in progress from out pairing session: https://github.com/david415/tahoe-lafs/tree/2635.periodic-uploader-scan.2
includes minor fixes and more debug print statements
comment:14 Changed at 2016-01-01T00:39:49Z by daira
The current rebased version of this branch is https://github.com/tahoe-lafs/tahoe-lafs/tree/2635.periodic-uploader-scan.3.
comment:15 Changed at 2016-01-25T14:52:59Z by dawuud
the current state of this ticket is that we cannot get the new unit test to pass.
comment:16 Changed at 2016-01-25T15:21:40Z by dawuud
comment:17 Changed at 2016-01-27T13:00:23Z by dawuud
here's the latest dev branch from the pairing session with Daira: https://github.com/david415/tahoe-lafs/tree/2635.periodic-uploader-scan.4
we did not get the unit test passing... there wasn't enough time. we thought we had the solution but seeing is believing ;-p i tried adding a callback to the upload_d deferred to do another clock advance but the test still hangs.
comment:18 Changed at 2016-01-27T16:04:00Z by daira
We fixed this in the pairing today :-)
comment:19 Changed at 2016-01-27T16:06:59Z by dawuud
Fixed in this dev branch: https://github.com/david415/tahoe-lafs/tree/2635.periodic-uploader-scan.4
here's the pull request into stable.10: https://github.com/tahoe-lafs/tahoe-lafs/pull/231/files
if we want to squash history then i can do that... but anyway we can use this as a place for code review.
comment:20 Changed at 2016-01-27T16:08:21Z by dawuud
- Status changed from assigned to new
comment:21 Changed at 2016-02-03T11:34:05Z by dawuud
daira fixed the periodic uploader unit test: https://github.com/david415/tahoe-lafs/tree/2635.periodic-uploader-scan.4
i think we can review and then merge this into our latest stable branch soon.
comment:22 Changed at 2016-02-08T13:29:27Z by dawuud
- Resolution set to fixed
- Status changed from new to closed
it looks like we can close this ticket because the code has already been merged into our stable.12 branch;
Work in progress on branch 2635.periodic-uploader-scan.0 (I hadn't got very far with it).