pypiwin32 not picked up in Windows wheel install

[crd]

I just installed tahoe-lafs 1.11.0 on Windows 10 and ran into a minor problem in that it did not automatically pick up the pypiwin32 package:

Microsoft Windows [Version 10.0.10586]
(c) 2015 Microsoft Corporation. All rights reserved.

C:\Users\crd>\Python27\Scripts\virtualenv tahoe-lafs
New python executable in C:\Users\crd\tahoe-lafs\Scripts\python.exe
Installing setuptools, pip, wheel...done.

C:\Users\crd>tahoe-lafs\Scripts\activate

(tahoe-lafs) C:\Users\crd>pip list
pip (8.1.1)
setuptools (20.4)
wheel (0.29.0)

(tahoe-lafs) C:\Users\crd>pip install --find-links=https://tahoe-lafs.org/deps/ tahoe-lafs
Collecting tahoe-lafs
  Downloading tahoe_lafs-1.11.0-py2-none-any.whl (1.1MB)
    100% |################################| 1.1MB 522kB/s
Requirement already satisfied (use --upgrade to upgrade): setuptools>=11.3 in c:\users\crd\tahoe-lafs\lib\site-packages (from tahoe-lafs)
Collecting simplejson>=1.4 (from tahoe-lafs)
  Downloading simplejson-3.8.2-cp27-cp27m-win_amd64.whl (66kB)
    100% |################################| 71kB 1.7MB/s
Collecting service-identity (from tahoe-lafs)
  Downloading service_identity-16.0.0-py2.py3-none-any.whl
Collecting zope.interface!=3.6.3,!=3.6.4,>=3.6.0 (from tahoe-lafs)
  Downloading https://tahoe-lafs.org/deps/zope.interface-4.1.3-cp27-cp27m-win_amd64.whl (130kB)
    100% |################################| 133kB 497kB/s
Collecting pyasn1>=0.1.8 (from tahoe-lafs)
  Downloading pyasn1-0.1.9-py2.py3-none-any.whl
Collecting foolscap>=0.10.1 (from tahoe-lafs)
  Downloading foolscap-0.11.0-py2-none-any.whl (290kB)
    100% |################################| 296kB 1.2MB/s
Collecting pycrypto!=2.2,!=2.4,>=2.1.0 (from tahoe-lafs)
  Downloading https://tahoe-lafs.org/deps/pycrypto-2.6.1-cp27-cp27m-win_amd64.whl (478kB)
    100% |################################| 481kB 860kB/s
Collecting pyOpenSSL>=0.14 (from tahoe-lafs)
  Downloading pyOpenSSL-16.0.0-py2.py3-none-any.whl (45kB)
    100% |################################| 51kB 1.8MB/s
Collecting characteristic>=14.0.0 (from tahoe-lafs)
  Downloading characteristic-14.3.0-py2.py3-none-any.whl
Collecting zfec>=1.1.0 (from tahoe-lafs)
  Downloading https://tahoe-lafs.org/deps/zfec-1.4.24-cp27-cp27m-win_amd64.whl (70kB)
    100% |################################| 71kB 669kB/s
Collecting pycryptopp>=0.6.0 (from tahoe-lafs)
  Downloading https://tahoe-lafs.org/deps/pycryptopp-0.7.1.869544967005693312591928092448767568728501330214-cp27-cp27m-win_amd64.whl (1.5MB)
    100% |################################| 1.5MB 380kB/s
Collecting pyasn1-modules>=0.0.5 (from tahoe-lafs)
  Downloading pyasn1_modules-0.0.8-py2.py3-none-any.whl
Collecting Nevow>=0.11.1 (from tahoe-lafs)
  Downloading Nevow-0.13.0-py2-none-any.whl (448kB)
    100% |################################| 450kB 898kB/s
Collecting Twisted[tls]>=15.1.0 (from tahoe-lafs)
  Downloading Twisted-16.0.0-cp27-none-win_amd64.whl (3.2MB)
    100% |################################| 3.2MB 215kB/s
Collecting attrs (from service-identity->tahoe-lafs)
  Downloading attrs-15.2.0-py2.py3-none-any.whl
Collecting cryptography>=1.3 (from pyOpenSSL>=0.14->tahoe-lafs)
  Downloading cryptography-1.3.1-cp27-none-win_amd64.whl (1.3MB)
    100% |################################| 1.3MB 451kB/s
Collecting six>=1.5.2 (from pyOpenSSL>=0.14->tahoe-lafs)
  Downloading six-1.10.0-py2.py3-none-any.whl
Collecting pyutil>=1.3.19 (from zfec>=1.1.0->tahoe-lafs)
  Downloading https://tahoe-lafs.org/deps/pyutil-2.0.0-py2-none-any.whl (180kB)
    100% |################################| 184kB 747kB/s
Collecting idna>=0.6 (from Twisted[tls]>=15.1.0->tahoe-lafs)
  Downloading idna-2.1-py2-none-any.whl (54kB)
    100% |################################| 61kB 1.8MB/s
Collecting enum34 (from cryptography>=1.3->pyOpenSSL>=0.14->tahoe-lafs)
  Downloading https://tahoe-lafs.org/deps/enum34-1.1.2-cp27-none-any.whl (57kB)
    100% |################################| 61kB 516kB/s
Collecting ipaddress (from cryptography>=1.3->pyOpenSSL>=0.14->tahoe-lafs)
  Downloading ipaddress-1.0.16-py27-none-any.whl
Collecting cffi>=1.4.1 (from cryptography>=1.3->pyOpenSSL>=0.14->tahoe-lafs)
  Downloading cffi-1.5.2-cp27-none-win_amd64.whl (150kB)
    100% |################################| 153kB 1.5MB/s
Collecting zbase32>=1.0 (from pyutil>=1.3.19->zfec>=1.1.0->tahoe-lafs)
  Downloading https://tahoe-lafs.org/deps/zbase32-1.1.5-py2-none-any.whl
Collecting pycparser (from cffi>=1.4.1->cryptography>=1.3->pyOpenSSL>=0.14->tahoe-lafs)
  Downloading https://tahoe-lafs.org/deps/pycparser-2.14-cp27-none-any.whl (188kB)
    100% |################################| 194kB 1.2MB/s
Installing collected packages: simplejson, enum34, ipaddress, pyasn1, six, idna, pycparser, cffi, cryptography, pyOpenSSL, pyasn1-modules, attrs, service-identity, zope.interface, Twisted, foolscap, pycrypto, characteristic, zbase32, pyutil, zfec, pycryptopp, Nevow, tahoe-lafs
Successfully installed Nevow-0.13.0 Twisted-16.0.0 attrs-15.2.0 cffi-1.5.2 characteristic-14.3.0 cryptography-1.3.1 enum34-1.1.2 foolscap-0.11.0 idna-2.1 ipaddress-1.0.16 pyOpenSSL-16.0.0 pyasn1-0.1.9 pyasn1-modules-0.0.8 pycparser-2.14 pycrypto-2.6.1 pycryptopp-0.7.1.869544967005693312591928092448767568728501330214 pyutil-2.0.0 service-identity-16.0.0 simplejson-3.8.2 six-1.10.0 tahoe-lafs-1.11.0 zbase32-1.1.5 zfec-1.4.24 zope.interface-4.1.3

(tahoe-lafs) C:\Users\crd>tahoe --version
Traceback (most recent call last):
  File "c:\python27\Lib\runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "c:\python27\Lib\runpy.py", line 72, in _run_code
    exec code in run_globals
  File "C:\Users\crd\tahoe-lafs\Scripts\tahoe.exe\__main__.py", line 5, in <module>
  File "c:\users\crd\tahoe-lafs\lib\site-packages\allmydata\__init__.py", line 412, in <module>
    _vers_and_locs_list, _cross_check_errors = get_package_versions_and_locations()
  File "c:\users\crd\tahoe-lafs\lib\site-packages\allmydata\__init__.py", line 222, in get_package_versions_and_locations
    for p in pkg_resources.require(install_requires)])
  File "c:\users\crd\tahoe-lafs\lib\site-packages\pkg_resources\__init__.py", line 943, in require
    needed = self.resolve(parse_requirements(requirements))
  File "c:\users\crd\tahoe-lafs\lib\site-packages\pkg_resources\__init__.py", line 830, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'pypiwin32' distribution was not found and is required by the application

(tahoe-lafs) C:\Users\crd>
(tahoe-lafs) C:\Users\crd>pip install --find-links=https://tahoe-lafs.org/deps/ pypiwin32
Collecting pypiwin32
  Downloading pypiwin32-219-cp27-none-win_amd64.whl (7.3MB)
    100% |################################| 7.3MB 98kB/s
Installing collected packages: pypiwin32
Successfully installed pypiwin32-219

(tahoe-lafs) C:\Users\crd>tahoe --version
tahoe-lafs: 1.11.0 [master: 04a3e7993f70ac87c208d61f4387a59f5f419367]
foolscap: 0.11.0
pycryptopp: 0.7.1.869544967005693312591928092448767568728501330214
zfec: 1.4.24
Twisted: 16.0.0
Nevow: 0.13.0
zope.interface: unknown
python: 2.7.11
platform: Windows-10-10.0.10586
pyOpenSSL: 16.0.0
OpenSSL: 1.0.2g [ 1 Mar 2016]
simplejson: 3.8.2
pycrypto: 2.6.1
pyasn1: 0.1.9
service-identity: 16.0.0
characteristic: 14.3.0
pyasn1-modules: 0.0.8
cryptography: 1.3.1
cffi: 1.5.2
six: 1.10.0
enum34: 1.1.2
pycparser: 2.14
setuptools: 20.4
pypiwin32: 219
attrs: 15.2.0 [according to pkg_resources]
ipaddress: 1.0.16 [according to pkg_resources]
idna: 2.1 [according to pkg_resources]

[warner]

Huh. That looks like tahoe didn't express it's need for pypiwin32 the first time it was asked (during the install), but did express it the second time (at runtime).

Oh.. it's because I uploaded tahoe_lafs-1.11.0-py2-none-any.whl , and the wheel was built on a unix box. That wheel doesn't express the dependency: only a source tarball has the conditional "am I on windows? then ask for pypiwin32" code.

I need to delete that wheel.

[warner]

I've deleted the wheel from pypi. Could you try a new installation?

If you see it trying to use that same wheel again, it may be in your HTTP cache (~\AppData\Local\pip\cache). If so, just delete that directory.

[warner]

crd reported on IRC that this fixed the problem.

Changing the focus of this ticket: to be consistent about not building wheels (which is a shame), or find a way to make wheels that won't be used on windows where the dependencies aren't sufficient. Could/should we use a tahoe-lafs[windows] target?

[warner]

The right fix is to use a platform-specific "extra", with a setup.py syntax like:

      extras_require={
          ':sys_platform=="win32"': ["pypiwin32"],

[warner]

​https://github.com/tahoe-lafs/tahoe-lafs/pull/388

[Brian Warner <warner@…>]

In a06cf2e/trunk:

express pypiwin32 dep via a setup.py extra

Previously this looked at sys.platform to decide what dependencies to
include. The problem with that approach is that wheels built on a unix box
won't work on windows (and vice versa), when the (pure-python) Tahoe wheels
aren't supposed to be platform-dependent.

setup.py provides a syntax to express this properly, so wheels created on
either platform will include the pypiwin32 dependency in the metadata, but
marked as only being relevant when installing on a win32 platform.

closes ticket:2763