[tahoe-lafs-trac-stream] [Tahoe-LAFS] #2049: Decide where "packaging tests" should live.

[Tahoe-LAFS]

#2049: Decide where "packaging tests" should live.
-------------------------+-------------------------------------------------
     Reporter:           |      Owner:  nejucomo
  nejucomo               |     Status:  new
         Type:  task     |  Milestone:  soon (release n/a)
     Priority:  normal   |    Version:  1.10.0
    Component:  dev-     |   Keywords:  packaging dev-infrastructure
  infrastructure         |  buildbot pypi openitp-packaging
   Resolution:           |
Launchpad Bug:           |
-------------------------+-------------------------------------------------

Comment (by Zancas):

 Replying to [ticket:2049 nejucomo]:
 > '''Task:''' Define a policy about where packaging tests live, where
 buildbot configuration lives, then specify that policy on the wiki.
 >
 > '''Details:'''
 >
 > We'd benefit from "packaging tests" which are different from unittests.
 The latter target a prepared installation or repository checkout, whereas
 packaging tests have these distinctions:
 >
 > '''Inputs:''' Packaging tests may target as input:
 >
 >  * ''build tests'' - a fresh repository checkout in order to test
 creation of distribution formats -- ''e.g.'' `sdist`, `linux distro
 packages`, sumo tarballs, etc...
 >  * ''installation tests'' - a locally prepared distribution format --
 ''e.g.'' `sdist`, `.deb`, etc...
 >  * ''distribution tests'' - the network -- ''e.g.'' a PyPI package name
 and `http(s)` connections to python packaging mirrors; a debian package
 name and connections to debian repositories; etc...
 >
 > '''Staging or Production''': We might like to run the same tests against
 unreleased versions and distinctly against "the live internet".
 >
 >  * ''Staging tests'' might rely on a local PyPI mirror to ensure that
 ''if we were'' to distribute sdists to the real PyPI, ''then'' users will
 probably successfully be able to install them.
 >  * ''Production tests'' rely on the actual distribution that live users
 use.  Failures of these tests let us know when our released packages are
 unavailable to users, and can help identify why.
 >
 > '''Brittleness''': sensitivity to transient conditions:
 >
 >  * Packaging tests start from either a repository clone or an sdist and
 should behave fairly deterministically without networking.
 >  * Installation tests start from local packaging files, and should
 behave deterministically, barring system state issues.
 >  * Distributions tests necessarily use the internet and will give false
 positives for transient errors.  (''Note:'' Errors due to transient
 outages are important information!  Suppose `tahoe-lafs` install relies on
 `some.third-party.package.host.example.com` and that site goes down.  We
 want to know that people are now unable to install `tahoe-lafs` through
 specific channels.)
 >
 >
 > We may want different packaging tests to live in different places
 depending on these axes.  Some of these tests may not require buildbot,
 and/or may not be developed or maintained by us at all!  For example,
 tests which verify live PyPI packages may be run by PyPI itself.  We
 should still account for this in an "official policy".
 >
 > We probably want test infrastructure operators to have some flexibility
 to only run some categories of tests.  For example, a debian test box
 would not want to run windows installation tests, or some operators may
 prefer to limit networking and only run non-distribution tests.


 Here're some tweaks to the test type taxonomy outlined above.

 1.  I think the name "network distribution tests" is easier to understand,
 perhaps because it's more specific, than "distribution tests".  I believe
 this because when I added the "network " before instances of "distribution
 test" I found the text easier to understand.

 2.  If I Understand Correctly (IIUC) the word "Packaging" which is the
 first word after the first bullet point in the "Brittleness" section is a
 substitution error, and should be the word "Build".

 3.  It seems to me that "Brittleness" is a criteria, like "Inputs", by
 which one can categorize test "packaging test" types, whereas "Staging or
 Production" is a discriminator that applies more agnostically to all
 tests.  Therefore I propose that the description will have a more
 intuitive flow if the order of the sections:  "Brittleness" and "Staging
 or Production" are swapped.  Also it might be nice to provide some other
 indicator that the "Staging or Production" section is not quite the same
 kind of section as the other two, e.g. an interposed sentence, or making
 both "Inputs" and "Brittleness" be subsections of a "Test Type" section.

 4.  I propose that we replace the term "Brittleness" with the word
 "Robustness".

--
Ticket URL: <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2049#comment:21>
Tahoe-LAFS <https://Tahoe-LAFS.org>
secure decentralized storage