#1440 closed enhancement (fixed)

drop-upload: don't perform redundant uploads when a file is quickly modified three or more times

Reported by: davidsarah Owned by: daira
Priority: major Milestone: 1.12.0
Component: code-frontend-magic-folder Version: 1.8.2
Keywords: drop-upload performance otf-magic-folder-objective2 Cc:
Launchpad Bug:

Description

When a file is uploaded by the drop-upload frontend, events that would cause further uploads of the same file are queued. This queue only needs to record one event for a given file, e.g. if a file is uploading and gets two more modification events before the first upload completes, the middle event can be dropped.

Change History (16)

comment:1 Changed at 2011-07-25T12:27:11Z by davidsarah

  • Type changed from defect to enhancement

comment:2 Changed at 2011-07-25T12:27:20Z by davidsarah

  • Component changed from unknown to code-frontend

comment:3 Changed at 2014-04-15T01:11:02Z by daira

See also #2220.

comment:4 Changed at 2014-12-02T19:47:11Z by warner

  • Component changed from code-frontend to code-frontend-drop-upload

comment:5 Changed at 2015-03-17T22:15:04Z by daira

  • Owner changed from davidsarah to daira
  • Status changed from new to assigned

comment:6 Changed at 2015-04-02T14:55:19Z by daira

  • Keywords otf-magic-folder-objective2 added

comment:7 follow-up: Changed at 2015-04-07T22:25:25Z by dawuud

I thought of a solution. Would something along these lines be acceptable? :

We use an explicit dequeue implementation and we utilize a helping hash map to track currently queued files/dirs. Each time we add a file to the queue we first check if that file is already in the hashmap. We only add a file to the queue when it is not already in the hashmap. Likewise we must remove it from the hashmap once that queue item has been processed.

comment:8 in reply to: ↑ 7 Changed at 2015-04-10T17:07:55Z by daira

Replying to dawuud:

We use an explicit dequeue implementation and we utilize a helping hash map to track currently queued files/dirs. Each time we add a file to the queue we first check if that file is already in the hashmap. We only add a file to the queue when it is not already in the hashmap. Likewise we must remove it from the hashmap once that queue item has been processed.

Yes, this is basically the same as the pending_delay code in allmydata/windows/inotify.py. We should add something similar to drop_upload.py, and then we can probably remove the Windows-specific implementation.

Last edited at 2015-04-10T17:09:34Z by daira (previous) (diff)

comment:9 Changed at 2015-04-10T23:29:21Z by dawuud

ok here's some code that does that: https://github.com/david415/tahoe-lafs/tree/david-1440-1

although this code doesn't work because i branched from my code for ticket #1449

comment:10 Changed at 2015-04-12T22:33:33Z by daira

  • Milestone changed from undecided to 1.11.0

comment:11 Changed at 2015-04-14T07:22:56Z by dawuud

here's my latest... i've used the explicit deque from the other branch and i've added a pending set of files... this allows deduplication: https://github.com/david415/tahoe-lafs/tree/dropupload-redundant-uploads-1

I had to comment out part of the drop uploader unit test to get it to pass. We need a unit test for this deduplicating of upload events.

comment:12 Changed at 2015-04-14T21:17:36Z by dawuud

OK I pushed more code into the same branch. I worked on the unit test for deduplicating uploads but so far only the mock test works ;-( Maybe Daira can help me figure this out?

Version 0, edited at 2015-04-14T21:17:36Z by dawuud (next)

comment:14 Changed at 2015-05-02T16:42:30Z by daira

  • Resolution set to fixed
  • Status changed from assigned to closed

comment:15 Changed at 2015-05-02T16:44:06Z by daira

Closing this and using ticket #2406 for any further review comments.

comment:16 Changed at 2016-03-22T05:02:52Z by warner

  • Milestone changed from 1.11.0 to 1.12.0

Milestone renamed

Note: See TracTickets for help on using tickets.