cloud backend: redundant reads of chunks from cloud when downloading large files

[davidsarah]

I uploaded a 7.7 MiB video as an MDMF file using the cloud backend on S3 (as of ​1819-cloud-merge/022796fb), and then downloaded it. From flogtool tailing the storage server, I saw that it was reading the same chunks multiple times during the download. That suggests that the chunk cache is not operating well enough.

The file was being downloaded by playing it as a video in Chromium; I don't think that makes a difference.

Update: this also applies to immutable files if they are large enough.

[davidsarah]

During the upload and download, the server memory usage didn't go above 50 MiB according to the statmover graph.

[davidsarah]

Same behaviour for a straight download, rather than playing a video. Each chunk seems to get read 5 times, and the first chunk (containing the header) many more times.

[daira]

I changed ChunkCache to use a true LRU replacement policy, and that seems to have fixed this problem. (LRU is not often used because keeping track of ages can be inefficient for a large cache, but here we only need a cache of a few elements. In practice 5 chunks seems to be sufficient for the sizes of files I've tested; will investigate whether it's enough for larger files later.)