[tahoe-lafs-trac-stream] [tahoe-lafs] #2049: Decide where "packaging tests" should live.

[tahoe-lafs]

#2049: Decide where "packaging tests" should live.
-------------------------------------------------+-------------------------
 Reporter:  nejucomo                             |          Owner:  daira
     Type:  task                                 |         Status:  new
 Priority:  normal                               |      Milestone:
Component:  unknown                              |  undecided
 Keywords:  packaging dev-infrastructure         |        Version:  1.10.0
  buildbot pypi                                  |  Launchpad Bug:
-------------------------------------------------+-------------------------
 '''Task:''' Define a policy about where packaging tests live, where
 buildbot configuration lives, then specify that policy on the wiki.

 '''Details:'''

 We'd benefit from "packaging tests" which are different from unittests.
 The latter target a prepared installation or repository checkout, whereas
 packaging tests have these distinctions:

 '''Inputs:''' Packaging tests may target as input:

  * ''build tests'' - a fresh repository checkout in order to test creation
 of distribution formats -- ''e.g.'' `sdist`, `linux distro packages`, sumo
 tarballs, etc...
  * ''installation tests'' - a locally prepared distribution format --
 ''e.g.'' `sdist`, `.deb`, etc...
  * ''distribution tests'' - the network -- ''e.g.'' a PyPI package name
 and `http(s)` connections to python packaging mirrors; a debian package
 name and connections to debian repositories; etc...

 '''Staging or Production''': We might like to run the same tests against
 unreleased versions and distinctly against "the live internet".

  * ''Staging tests'' might rely on a local PyPI mirror to ensure that ''if
 we were'' to distribute sdists to the real PyPI, ''then'' users will
 probably successfully be able to install them.
  * ''Production tests'' rely on the actual distribution that live users
 use.  Failures of these tests let us know when our released packages are
 unavailable to users, and can help identify why.

 '''Brittleness''': sensitivity to transient conditions:

  * Packaging tests start from either a repository clone or an sdist and
 should behave fairly deterministically without networking.
  * Installation tests start from local packaging files, and should behave
 deterministically, barring system state issues.
  * Distributions tests necessarily use the internet and will give false
 positives for transient errors.  (''Note:'' Errors due to transient
 outages are important information!  Suppose `tahoe-lafs` install relies on
 `some.third-party.package.host.example.com` and that site goes down.  We
 want to know that people are now unable to install `tahoe-lafs` through
 specific channels.)


 We may want different packaging tests to live in different places
 depending on these axes.  Some of these tests may not require buildbot,
 and/or may not be developed or maintained by us at all!  For example,
 tests which verify live PyPI packages may be run by PyPI itself.  We
 should still account for this in an "official policy".

 We probably want test infrastructure operators to have some flexibility to
 only run some categories of tests.  For example, a debian test box would
 not want to run windows installation tests, or some operators may prefer
 to limit networking and only run non-distribution tests.

-- 
Ticket URL: <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2049>
tahoe-lafs <https://tahoe-lafs.org>
secure decentralized storage