#1430 closed defect (wontfix)

Magic Folder on Linux: handle inotify queue overflow correctly

Reported by: davidsarah Owned by: daira
Priority: normal Milestone: eventually
Component: code-frontend-magic-folder Version: 1.8.2
Keywords: inotify reliability magic-folder Cc:
Launchpad Bug:

Description (last modified by daira)

The inotify API has a maximum number of queued events, controlled by /proc/sys/fs/inotify/max_queued_events. This could affect the reliability of the Magic Folder uploader in detecting all file-change events when a large number of files in the local directory are modified. In principle it could detect this using the IN_Q_OVERFLOW event, and rescan all files in the directory to see whether they were last-modified since the timestamp of the previous event (assuming that we trust the last-modified timestamps).

Change History (13)

comment:1 Changed at 2011-07-31T16:15:13Z by davidsarah

  • Priority changed from major to minor

The default value in /proc/sys/fs/inotify/max_queued_events on my system was 16384, so a queue overflow is fairly unlikely to happen in practice.

comment:2 Changed at 2011-07-31T17:21:32Z by davidsarah

Also, in the current code (see the second patch in http://tahoe-lafs.org/trac/tahoe-lafs/attachment/ticket/1429/drop-upload-4.darcs.patch), there is a much faster handler that pops events from the inotify queue and immediately pushes them onto foolscap's eventual send queue, which is limited only by memory. So the rate of filesystem events would have to be faster than twisted.internet.inotify and this handler can keep up with, rather than faster than the actual uploads.

comment:3 follow-up: Changed at 2011-07-31T20:20:19Z by zooko

So maybe we should close this ticket as "fixed" ? :-) Or perhaps try to detect if a substantial number of pending inotify events have piled up and emit a warning or error message?

comment:4 in reply to: ↑ 3 Changed at 2011-08-01T00:37:52Z by davidsarah

Replying to zooko:

So maybe we should close this ticket as "fixed" ? :-)

It's not fixed, it's just unlikely to happen. I don't see any problem with leaving the ticket open for 1.9.

comment:5 Changed at 2014-12-02T19:52:47Z by warner

  • Component changed from code-frontend to code-frontend-drop-upload

comment:6 Changed at 2015-03-17T22:15:25Z by daira

  • Owner changed from davidsarah to daira
  • Status changed from new to assigned

comment:7 Changed at 2015-06-01T16:11:09Z by daira

  • Keywords magic-folder added

Add magic-folder keyword to all drop-upload tickets.

comment:8 Changed at 2015-10-29T02:29:12Z by daira

  • Description modified (diff)
  • Summary changed from drop-upload: handle inotify queue overflow correctly to Magic Folder on Linux: handle inotify queue overflow correctly

comment:9 Changed at 2015-10-29T02:29:36Z by daira

  • Keywords reliability added; drop-upload removed

comment:10 Changed at 2015-10-29T02:29:52Z by daira

  • Priority changed from minor to normal

comment:11 Changed at 2015-12-03T18:09:38Z by daira

This does not block merging Magic Folder to trunk.

comment:12 Changed at 2016-02-15T15:48:17Z by daira

Twisted does not pass on the IN_Q_OVERFLOW event, so we cannot handle that event: https://twistedmatrix.com/trac/ticket/6722

Last edited at 2016-02-15T15:51:52Z by daira (previous) (diff)

comment:13 Changed at 2016-02-15T15:56:04Z by daira

  • Resolution set to wontfix
  • Status changed from assigned to closed

I'm wontfixing this because we now have the periodic uploader (#2420), so there is no risk of this causing file changes to be permanently ignored.

Note: See TracTickets for help on using tickets.