[tahoe-lafs-trac-stream] [tahoe-lafs] #1885: cloud backend: redundant reads of chunks from cloud when downloading large files (was: cloud backend: redundant reads of chunks from cloud when downloading large MDMF file)

[tahoe-lafs]

#1885: cloud backend: redundant reads of chunks from cloud when downloading large
files
-------------------------+-------------------------------------------------
     Reporter:           |      Owner:
  davidsarah             |     Status:  new
         Type:  defect   |  Milestone:  1.11.0
     Priority:  normal   |    Version:  1.9.2
    Component:  code-    |   Keywords:  cloud-backend cache download
  storage                |  performance
   Resolution:           |
Launchpad Bug:           |
-------------------------+-------------------------------------------------
Changes (by daira):

 * keywords:  cloud-backend cache download performance mdmf => cloud-backend
     cache download performance


Old description:

> I uploaded a 7.7 MiB video as an MDMF file using the cloud backend on S3
> (as of [https://github.com/davidsarah/tahoe-
> lafs/commit/022796fb7813c2f42d668a0ee3de9abae869deb5 1819-cloud-
> merge/022796fb]), and then downloaded it. From {{{flogtool tail}}}ing the
> storage server, I saw that it was reading the same chunks multiple times
> during the download. That suggests that the chunk cache is not operating
> well enough.
>
> The file was being downloaded by playing it as a video in Chromium; I
> don't think that makes a difference.

New description:

 I uploaded a 7.7 MiB video as an MDMF file using the cloud backend on S3
 (as of [https://github.com/davidsarah/tahoe-
 lafs/commit/022796fb7813c2f42d668a0ee3de9abae869deb5 1819-cloud-
 merge/022796fb]), and then downloaded it. From {{{flogtool tail}}}ing the
 storage server, I saw that it was reading the same chunks multiple times
 during the download. That suggests that the chunk cache is not operating
 well enough.

 The file was being downloaded by playing it as a video in Chromium; I
 don't think that makes a difference.

 Update: this also applies to immutable files if they are large enough.

--

Comment:

 I changed {{{ChunkCache}}} to use a true LRU replacement policy, and that
 seems to have fixed this problem. (LRU is not often used because keeping
 track of ages can be inefficient for a large cache, but here we only need
 a cache of a few elements. In practice 5 chunks seems to be sufficient for
 the sizes of files I've tested; will investigate whether it's enough for
 larger files later.)

-- 
Ticket URL: <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1885#comment:5>
tahoe-lafs <https://tahoe-lafs.org>
secure decentralized storage