mini-Summit report, day 2

[Brian Warner]

On 7/2/14 2:50 PM, Greg Troxel wrote:

> Hopefully this has a way to avoid doing any fetching of dependencies,
> and failing instead, and is no less cross unfriendly than python has
> to be to start with. (From packaging system view) tahoe really doesn't
> seem all that special - it's just a bunch of python code with a list
> of dependencies, so it seems symptomatic of larger brokenness in the
> python world that this is necessary...

Indeed :). Python's current package-installation tools don't make it
easy to categorically prevent all fetching, but I think we can work
around that.

The "safe" thing to do involves a setup phase:

1: identify the transitive closure of modules needed to run tahoe, by
   doing an unsafe "pip install tahoe" and seeing what it fetches.
2: set aside the tarballs used during that recursive install
3: inspect that code, somehow, and record the hashes of the tarballs
4: be prepared to repeat this process each time we want to update
   anything

Then we change the build process to:

1: create a new clean virtualenv, to isolate us from site-installed
   packages
2: use "pip install --no-deps --download" to fetch tarballs, or record
   the URLs from the setup phase, or just include all the tarballs in a
   SUMO build.
3: if we downloaded tarballs, check their hashes
4: "pip install --no-deps" on each tarball, into the virtualenv
5: run tahoe's "setup.py develop --no-deps", into the virtualenv
6: hope that nothing we touched has a setup_requires=

1 is either the first instruction we give to the developer who wants to
run Tahoe from source, or it's included in a tool (or setup.py command,
or Makefile target) that we encourage developers to use. Peep is
basically somebody else's project to do steps 2-4.

To deal with 6, the best approach is to get setuptools to honor some
flag, so that "pip install --no-deps" will fail rather than install any
setup_requires=. Maybe "setup.py install" could learn --no-deps or
--no-download or --no-setup-requires. Setuptools is actually being
maintained these days, so I think it's actually feasible to get this
fixed upstream.

As a workaround, instead of using "pip install --no-deps TARBALL", we
could manually unpack the tarball and monkeypatch its setup.py before
running it (to disable setuptool's ability to download anything), or
write a setup.cfg which does something like set allow_hosts= to an empty
string (which might disable downloads, we'd have to test it).

None of this is specific to Tahoe, but Tahoe is among the few projects
with users who care deeply about not just installing random code found
on the internet :).


cheers,
 -Brian