Opened at 2019-01-25T16:15:04Z

Closed at 2019-07-02T12:53:10Z

#2963 closed defect (fixed)

Fedora package installation is unreliable and spuriously fails CI jobs

Reported by: exarkun Owned by: exarkun
Priority: normal Milestone: undecided
Component: dev-infrastructure Version: 1.12.1
Keywords: Cc:
Launchpad Bug:

Description

For example, seen on <​https://circleci.com/gh/tahoe-lafs/tahoe-lafs/2136>, yum install --assumeyes git eventually fails like: 

The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
Error: Error downloading packages:
  Status code: 503 for https://mirrors.fedoraproject.org/metalink?repo=updates-released-f28&arch=x86_64
Exited with code 1

Change History (3)

comment:1 Changed at 2019-01-25T16:15:10Z by exarkun

  • Owner set to exarkun
  • Status changed from new to assigned

comment:2 Changed at 2019-01-31T00:48:07Z by exarkun

Another way it fails: 

[MIRROR] kernel-headers-4.20.3-200.fc29.x86_64.rpm: Curl error (28): Timeout was reached for http://csc.mcs.sdsmt.edu/fedora/linux/updates/29/Everything/x86_64/Packages/k/kernel-headers-4.20.3-200.fc29.x86_64.rpm [Connection timed out after 30002 milliseconds]
[MIRROR] kernel-headers-4.20.3-200.fc29.x86_64.rpm: Status code: 404 for https://mirror.umd.edu/fedora/linux/updates/29/Everything/x86_64/Packages/k/kernel-headers-4.20.3-200.fc29.x86_64.rpm
[MIRROR] kernel-headers-4.20.3-200.fc29.x86_64.rpm: Status code: 404 for http://mirror.umd.edu/fedora/linux/updates/29/Everything/x86_64/Packages/k/kernel-headers-4.20.3-200.fc29.x86_64.rpm
[MIRROR] kernel-headers-4.20.3-200.fc29.x86_64.rpm: Status code: 404 for http://pubmirror2.math.uh.edu/fedora-buffet/fedora/linux/updates/29/Everything/x86_64/Packages/k/kernel-headers-4.20.3-200.fc29.x86_64.rpm
[MIRROR] kernel-headers-4.20.3-200.fc29.x86_64.rpm: Status code: 404 for https://pubmirror2.math.uh.edu/fedora-buffet/fedora/linux/updates/29/Everything/x86_64/Packages/k/kernel-headers-4.20.3-200.fc29.x86_64.rpm
[MIRROR] kernel-headers-4.20.3-200.fc29.x86_64.rpm: Status code: 404 for http://repo.ialab.dsu.edu/fedora/linux/updates/29/Everything/x86_64/Packages/k/kernel-headers-4.20.3-200.fc29.x86_64.rpm
[MIRROR] kernel-headers-4.20.3-200.fc29.x86_64.rpm: Status code: 404 for http://mirrors.mit.edu/fedora/linux/updates/29/Everything/x86_64/Packages/k/kernel-headers-4.20.3-200.fc29.x86_64.rpm
[MIRROR] kernel-headers-4.20.3-200.fc29.x86_64.rpm: Status code: 404 for http://mirror.math.princeton.edu/pub/fedora/linux/updates/29/Everything/x86_64/Packages/k/kernel-headers-4.20.3-200.fc29.x86_64.rpm
[MIRROR] kernel-headers-4.20.3-200.fc29.x86_64.rpm: Status code: 404 for https://mirror.vcu.edu/pub/gnu_linux/fedora/updates/29/Everything/x86_64/Packages/k/kernel-headers-4.20.3-200.fc29.x86_64.rpm
[MIRROR] kernel-headers-4.20.3-200.fc29.x86_64.rpm: Status code: 404 for http://mirror.vcu.edu/pub/gnu_linux/fedora/updates/29/Everything/x86_64/Packages/k/kernel-headers-4.20.3-200.fc29.x86_64.rpm
[MIRROR] kernel-headers-4.20.3-200.fc29.x86_64.rpm: Status code: 404 for http://mirror.cs.pitt.edu/fedora/linux/updates/29/Everything/x86_64/Packages/k/kernel-headers-4.20.3-200.fc29.x86_64.rpm
[MIRROR] kernel-headers-4.20.3-200.fc29.x86_64.rpm: Status code: 404 for http://mirror.cc.vt.edu/pub/fedora/linux/updates/29/Everything/x86_64/Packages/k/kernel-headers-4.20.3-200.fc29.x86_64.rpm
[MIRROR] kernel-headers-4.20.3-200.fc29.x86_64.rpm: Status code: 404 for http://archive.linux.duke.edu/pub/fedora/linux/updates/29/Everything/x86_64/Packages/k/kernel-headers-4.20.3-200.fc29.x86_64.rpm
[MIRROR] kernel-headers-4.20.3-200.fc29.x86_64.rpm: Status code: 404 for http://mirror.uoregon.edu/fedora/linux/updates/29/Everything/x86_64/Packages/k/kernel-headers-4.20.3-200.fc29.x86_64.rpm
[MIRROR] kernel-headers-4.20.3-200.fc29.x86_64.rpm: Status code: 404 for http://kdeforge2.unl.edu/mirrors/fedora/linux/updates/29/Everything/x86_64/Packages/k/kernel-headers-4.20.3-200.fc29.x86_64.rpm
[MIRROR] kernel-headers-4.20.3-200.fc29.x86_64.rpm: Status code: 404 for http://mirrors.oit.uci.edu/fedora/linux/updates/29/Everything/x86_64/Packages/k/kernel-headers-4.20.3-200.fc29.x86_64.rpm
[MIRROR] kernel-headers-4.20.3-200.fc29.x86_64.rpm: Status code: 404 for http://mirror.seas.harvard.edu/fedora/linux/updates/29/Everything/x86_64/Packages/k/kernel-headers-4.20.3-200.fc29.x86_64.rpm
[MIRROR] kernel-headers-4.20.3-200.fc29.x86_64.rpm: Status code: 404 for https://pubmirror1.math.uh.edu/fedora-buffet/fedora/linux/updates/29/Everything/x86_64/Packages/k/kernel-headers-4.20.3-200.fc29.x86_64.rpm
[MIRROR] kernel-headers-4.20.3-200.fc29.x86_64.rpm: Status code: 404 for http://pubmirror1.math.uh.edu/fedora-buffet/fedora/linux/updates/29/Everything/x86_64/Packages/k/kernel-headers-4.20.3-200.fc29.x86_64.rpm
[MIRROR] kernel-headers-4.20.3-200.fc29.x86_64.rpm: Status code: 404 for http://mirrors.kernel.org/fedora/updates/29/Everything/x86_64/Packages/k/kernel-headers-4.20.3-200.fc29.x86_64.rpm
[MIRROR] kernel-headers-4.20.3-200.fc29.x86_64.rpm: Status code: 404 for http://mirrors.syringanetworks.net/fedora/linux/updates/29/Everything/x86_64/Packages/k/kernel-headers-4.20.3-200.fc29.x86_64.rpm
[MIRROR] kernel-headers-4.20.3-200.fc29.x86_64.rpm: Status code: 404 for http://mirror.clarkson.edu/fedora/linux/updates/29/Everything/x86_64/Packages/k/kernel-headers-4.20.3-200.fc29.x86_64.rpm
[MIRROR] kernel-headers-4.20.3-200.fc29.x86_64.rpm: Status code: 404 for https://mirror.clarkson.edu/fedora/linux/updates/29/Everything/x86_64/Packages/k/kernel-headers-4.20.3-200.fc29.x86_64.rpm
[MIRROR] kernel-headers-4.20.3-200.fc29.x86_64.rpm: Status code: 404 for https://sjc.edge.kernel.org/fedora-buffet/fedora/linux/updates/29/Everything/x86_64/Packages/k/kernel-headers-4.20.3-200.fc29.x86_64.rpm
[MIRROR] kernel-headers-4.20.3-200.fc29.x86_64.rpm: Status code: 404 for http://fedora.mirror.constant.com/fedora/linux/updates/29/Everything/x86_64/Packages/k/kernel-headers-4.20.3-200.fc29.x86_64.rpm
[MIRROR] kernel-headers-4.20.3-200.fc29.x86_64.rpm: Status code: 404 for https://fedora.mirror.constant.com/fedora/linux/updates/29/Everything/x86_64/Packages/k/kernel-headers-4.20.3-200.fc29.x86_64.rpm
[MIRROR] kernel-headers-4.20.3-200.fc29.x86_64.rpm: Status code: 404 for https://ewr.edge.kernel.org/fedora-buffet/fedora/linux/updates/29/Everything/x86_64/Packages/k/kernel-headers-4.20.3-200.fc29.x86_64.rpm
[MIRROR] kernel-headers-4.20.3-200.fc29.x86_64.rpm: Status code: 404 for http://mirror.cogentco.com/pub/linux/fedora/linux/updates/29/Everything/x86_64/Packages/k/kernel-headers-4.20.3-200.fc29.x86_64.rpm
[MIRROR] kernel-headers-4.20.3-200.fc29.x86_64.rpm: Status code: 404 for http://fedora.mirrors.pair.com/linux/updates/29/Everything/x86_64/Packages/k/kernel-headers-4.20.3-200.fc29.x86_64.rpm
[MIRROR] kernel-headers-4.20.3-200.fc29.x86_64.rpm: Status code: 404 for https://mirror.steadfastnet.com/fedora/updates/29/Everything/x86_64/Packages/k/kernel-headers-4.20.3-200.fc29.x86_64.rpm
[MIRROR] kernel-headers-4.20.3-200.fc29.x86_64.rpm: Status code: 404 for http://mirror.steadfastnet.com/fedora/updates/29/Everything/x86_64/Packages/k/kernel-headers-4.20.3-200.fc29.x86_64.rpm
[MIRROR] kernel-headers-4.20.3-200.fc29.x86_64.rpm: Status code: 404 for http://mirror.web-ster.com/fedora/updates/29/Everything/x86_64/Packages/k/kernel-headers-4.20.3-200.fc29.x86_64.rpm
[MIRROR] kernel-headers-4.20.3-200.fc29.x86_64.rpm: Status code: 404 for https://mirror.us-midwest-1.nexcess.net/fedora/updates/29/Everything/x86_64/Packages/k/kernel-headers-4.20.3-200.fc29.x86_64.rpm
[MIRROR] kernel-headers-4.20.3-200.fc29.x86_64.rpm: Status code: 404 for http://mirror.us-midwest-1.nexcess.net/fedora/updates/29/Everything/x86_64/Packages/k/kernel-headers-4.20.3-200.fc29.x86_64.rpm
[MIRROR] kernel-headers-4.20.3-200.fc29.x86_64.rpm: Status code: 404 for http://mirror.mrjester.net/fedora/linux/updates/29/Everything/x86_64/Packages/k/kernel-headers-4.20.3-200.fc29.x86_64.rpm
[MIRROR] kernel-headers-4.20.3-200.fc29.x86_64.rpm: Status code: 404 for https://mirror.mrjester.net/fedora/linux/updates/29/Everything/x86_64/Packages/k/kernel-headers-4.20.3-200.fc29.x86_64.rpm
[MIRROR] kernel-headers-4.20.3-200.fc29.x86_64.rpm: Status code: 404 for http://mirror.metrocast.net/fedora/linux/updates/29/Everything/x86_64/Packages/k/kernel-headers-4.20.3-200.fc29.x86_64.rpm
[MIRROR] kernel-headers-4.20.3-200.fc29.x86_64.rpm: Status code: 404 for http://repo.atlantic.net/fedora/linux/updates/29/Everything/x86_64/Packages/k/kernel-headers-4.20.3-200.fc29.x86_64.rpm
[MIRROR] kernel-headers-4.20.3-200.fc29.x86_64.rpm: Status code: 404 for http://mirror.datto.com/fedora/primary/updates/29/Everything/x86_64/Packages/k/kernel-headers-4.20.3-200.fc29.x86_64.rpm
[MIRROR] kernel-headers-4.20.3-200.fc29.x86_64.rpm: Status code: 404 for http://mirror.sjc02.svwh.net/fedora/updates/29/Everything/x86_64/Packages/k/kernel-headers-4.20.3-200.fc29.x86_64.rpm
[MIRROR] kernel-headers-4.20.3-200.fc29.x86_64.rpm: Status code: 404 for http://fedora.mirror.lstn.net/updates/29/Everything/x86_64/Packages/k/kernel-headers-4.20.3-200.fc29.x86_64.rpm
[MIRROR] kernel-headers-4.20.3-200.fc29.x86_64.rpm: Status code: 404 for http://mirror.stavlor.net/pub/fedora/linux/updates/29/Everything/x86_64/Packages/k/kernel-headers-4.20.3-200.fc29.x86_64.rpm
[MIRROR] kernel-headers-4.20.3-200.fc29.x86_64.rpm: Status code: 404 for http://mirror.liberty.edu/mirror/fedora/linux/updates/29/Everything/x86_64/Packages/k/kernel-headers-4.20.3-200.fc29.x86_64.rpm
[MIRROR] kernel-headers-4.20.3-200.fc29.x86_64.rpm: Status code: 404 for https://mirrors.xmission.com/fedora/linux/updates/29/Everything/x86_64/Packages/k/kernel-headers-4.20.3-200.fc29.x86_64.rpm
[MIRROR] kernel-headers-4.20.3-200.fc29.x86_64.rpm: Status code: 404 for http://mirrors.xmission.com/fedora/linux/updates/29/Everything/x86_64/Packages/k/kernel-headers-4.20.3-200.fc29.x86_64.rpm
[MIRROR] kernel-headers-4.20.3-200.fc29.x86_64.rpm: Status code: 404 for http://ftp.cse.buffalo.edu/pub/fedora/linux/updates/29/Everything/x86_64/Packages/k/kernel-headers-4.20.3-200.fc29.x86_64.rpm
[MIRROR] kernel-headers-4.20.3-200.fc29.x86_64.rpm: Status code: 404 for http://mirror.prgmr.com/pub/fedora/linux/updates/29/Everything/x86_64/Packages/k/kernel-headers-4.20.3-200.fc29.x86_64.rpm
[MIRROR] kernel-headers-4.20.3-200.fc29.x86_64.rpm: Status code: 404 for https://dl.fedoraproject.org/pub/fedora/linux/updates/29/Everything/x86_64/Packages/k/kernel-headers-4.20.3-200.fc29.x86_64.rpm
[MIRROR] kernel-headers-4.20.3-200.fc29.x86_64.rpm: Status code: 404 for http://dl.fedoraproject.org/pub/fedora/linux/updates/29/Everything/x86_64/Packages/k/kernel-headers-4.20.3-200.fc29.x86_64.rpm
[MIRROR] kernel-headers-4.20.3-200.fc29.x86_64.rpm: Status code: 404 for http://fedora.mirror.iweb.com/linux/updates/29/Everything/x86_64/Packages/k/kernel-headers-4.20.3-200.fc29.x86_64.rpm
[MIRROR] kernel-headers-4.20.3-200.fc29.x86_64.rpm: Status code: 404 for http://mirror.csclub.uwaterloo.ca/fedora/linux/updates/29/Everything/x86_64/Packages/k/kernel-headers-4.20.3-200.fc29.x86_64.rpm
[MIRROR] kernel-headers-4.20.3-200.fc29.x86_64.rpm: Status code: 404 for https://mirror.csclub.uwaterloo.ca/fedora/linux/updates/29/Everything/x86_64/Packages/k/kernel-headers-4.20.3-200.fc29.x86_64.rpm
[MIRROR] kernel-headers-4.20.3-200.fc29.x86_64.rpm: Status code: 404 for http://mirror.its.dal.ca/pub/fedora/linux/updates/29/Everything/x86_64/Packages/k/kernel-headers-4.20.3-200.fc29.x86_64.rpm
[FAILED] kernel-headers-4.20.3-200.fc29.x86_64.rpm: No more mirrors to try - All mirrors were already tried without success

The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
Error: Error downloading packages:
  Cannot download Packages/k/kernel-headers-4.20.3-200.fc29.x86_64.rpm: All mirrors were tried
Exited with code 1

​https://circleci.com/gh/tahoe-lafs/tahoe-lafs/2265

comment:3 Changed at 2019-07-02T12:53:10Z by exarkun

  • Resolution set to fixed
  • Status changed from assigned to closed

In 325c522d7c73065e3f7e10518a9785b822d1405a CircleCI config changed to pre-build images with everything already installed. This can still happen during image builds but those aren't in the path for test runs. No doubt there is room for improvement (reporting about image build failures? image build retries? tiered image promotion?). Nevertheless, this isn't going to bother developers day to day anymore so I'm calling it fixed.

Note: See TracTickets for help on using tickets.