Magic Folder: periodically do a full uploader scan

[daira]

This would help to compensate for:

• missed events, due to unreliability of the filesystem notification mechanism or queue overflows (#1430),
• failed uploads.

[daira]

Work in progress on branch 2635.periodic-uploader-scan.0 (I hadn't got very far with it).

[dawuud]

in my dev branch i taught the uploader's _scan method to use the deque instead of calling _process directly. this breaks all our magic-folder tests... so the next is to make the tests pass by firing the hook deferred when a directory and all it's children are done being uploaded.

​https://github.com/david415/tahoe-lafs/tree/2635.periodic-uploader-scan.0

[dawuud]

after teaching the uploader to use the deque to process directories i've been trying to get the tests to pass... and i'm not sure how to do it but i pushed my latest WIP comment anyways.

​https://github.com/david415/tahoe-lafs/tree/2635.periodic-uploader-scan.0

maybe daira or meejah could help me with this?

[daira]

The current design is wrong because it assumes that _scan finishes processing all of the descendant directories under the directory it is called with, when actually it only adds callbacks to scan all of its direct children. This affects both the initial scan triggered by start_scanning (which assumes that all desendants have been removed from the _pending set), and the intent of the tests that move subtrees.

[dawuud]

It's almost as if we want nested DeferredList?`s representing nested directories such that we can determine when a given directory and all it's children have been uploaded.

[dawuud]

here's the dev branch from our pairing session today: ​https://github.com/david415/tahoe-lafs/tree/2635.periodic-uploader-scan.1

[dawuud]

ok... all the tests are passing again with this commit: ​https://github.com/david415/tahoe-lafs/commit/4610a7f97d4823548129b4d483d7e50e4c5c18ee

next i need to periodic scan... should be easy.

[dawuud]

here's a naive periodic full scan: ​https://github.com/david415/tahoe-lafs/commit/d1eef67f7632ee96d8e3e30b1c8b5f9d084d8973

actually i think it's an open question how we should determine the full scan is complete and we should therefore perform another full scan now because our duration has elapsed.

just think how bad it would be if the full scan didn't complete before the next one started!

perhaps we can just check the pending set every X minutes and if it's empty then do the full scan?

[dawuud]

ok i made it only do the full scan if the pending set is empty. i also added a unit test... it's a MockTest? similar to test_move_tree but upon file creation inside the magicfolder dir we do not emit a fake inotify event. instead we wait for the periodic scan and upload... however i am trying to use clock advances to speed up this test and it's not yet passing.

[daira]

Reviewing progress so far.

[daira]

Rebased to ​https://github.com/tahoe-lafs/tahoe-lafs/commits/2635.periodic-uploader-scan.2.

[dawuud]

work in progress from out pairing session: ​https://github.com/david415/tahoe-lafs/tree/2635.periodic-uploader-scan.2

includes minor fixes and more debug print statements

[daira]

The current rebased version of this branch is ​https://github.com/tahoe-lafs/tahoe-lafs/tree/2635.periodic-uploader-scan.3. It doesn't include ​https://github.com/david415/tahoe-lafs/commit/b729c9ae5ee70eca36cc6b4c304f1b0a808d137d but you can cherry-pick that.

[dawuud]

the current state of this ticket is that we cannot get the new unit test to pass.

[dawuud]

​https://github.com/david415/tahoe-lafs/tree/2635.periodic-uploader-scan.3

[dawuud]

here's the latest dev branch from the pairing session with Daira: ​https://github.com/david415/tahoe-lafs/tree/2635.periodic-uploader-scan.4

we did not get the unit test passing... there wasn't enough time. we thought we had the solution but seeing is believing ;-p i tried adding a callback to the upload_d deferred to do another clock advance but the test still hangs.

[daira]

We fixed this in the pairing today :-)

[dawuud]

Fixed in this dev branch: ​https://github.com/david415/tahoe-lafs/tree/2635.periodic-uploader-scan.4

here's the pull request into stable.10: ​https://github.com/tahoe-lafs/tahoe-lafs/pull/231/files

if we want to squash history then i can do that... but anyway we can use this as a place for code review.

[dawuud]

daira fixed the periodic uploader unit test: ​https://github.com/david415/tahoe-lafs/tree/2635.periodic-uploader-scan.4

i think we can review and then merge this into our latest stable branch soon.

[dawuud]

it looks like we can close this ticket because the code has already been merged into our stable.12 branch;

​https://github.com/tahoe-lafs/tahoe-lafs/pull/237/files