magic-folder: implement download retry

[dawuud]

The downloader should retry if the operation fails... this can happen because of lossy networks such as wifi causing the tcp connections to storage servers to drop. Foolscap handles reconnection with exponential backoff... however the application layer still needs to retry... otherwise the user will experience our software as brittle.

[dawuud]

i started to write an experimental test for the upload retry here in this dev branch: ​https://github.com/david415/tahoe-lafs/tree/2537.upload-download-retry.0

hmm but maybe the nonetwork grid isn't appropriate for testing connectivity retry?

[daira]

Note that this is not part of the OTF grant (milestones 1-6).

[daira]

See also #2420.

[dawuud]

I've changed this ticket to only be concerned with retrying downloads because #2635 already deals with uploading file objets that previously failed to get uploaded.

[dawuud]

the next step is to simply write unit tests for download failure and retry. that is... we should already be retrying the download the next time we perform a remote dmd scan.

[dawuud]

i tried writing a test... although i'm still stuck on the part where i create error... my test hangs... probably i've used the hook mixin deferred with incorrect ordering or something like that.

​https://github.com/david415/tahoe-lafs/tree/2537.download-retry.0

[dawuud]

hey i made slight progress here... and got Alice and Bob sharing a file... and then after that I had some trouble producing a grid failure for Bob's download of the file that Alice just uploaded. We want to delete the shares, corrupt the share, or shutdown the storage servers such that it causes the download to fail.

​https://github.com/david415/tahoe-lafs/tree/2537.download-retry.0

[dawuud]

here's a pull request available for a casual code review: ​https://github.com/tahoe-lafs/tahoe-lafs/pull/238/files

although it doesn't at the moment work or pass.

[daira]

Replying to dawuud:

here's a pull request available for a casual code review: ​https://github.com/tahoe-lafs/tahoe-lafs/pull/238/files

although it doesn't at the moment work or pass.

There was a shallow bug (assuming that d.addCallback returns self), and also the branch included too much history from a previous magic-folder-stable branch, so I closed the PR. @dawuud will push a new 2537.download-retry.2 branch.

[dawuud]

i'm still not able to get the test to pass ​https://github.com/david415/tahoe-lafs/tree/2537.download-retry.2

it's true that we fixed a shallow bug... but there were other problems with the test. today i've experimented a bit with rewriting it. basically we want a test wherein:

• alice uploads a file
• grid failure
• bob fails to download file
• bob later rescans and successfully downloads file

[dawuud]

here's the resulting dev branch from our pairing session: ​https://github.com/david415/tahoe-lafs/tree/2537.download-retry.3

the test works but daira pointed out that there is likely a race condition and we should investigate a better way to make the download fail...

[daira]

Current branch is ​https://github.com/tahoe-lafs/tahoe-lafs/tree/2537.download-retry.4

[dawuud]

this unit test and feature now work properly thanks to Daira and Meejah. here's the latest: ​https://github.com/tahoe-lafs/tahoe-lafs/commits/2537.download-retry.5

I presume we shall not close this ticket until we have merged Meejah's inline-callback-deque changeset into a stable branch. Once we've done that we can merge in this unit test and close this ticket as well as ​https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2412

[daira]

Fixed on 2438.magic-folder-stable.12.