#2392 closed defect (fixed)

depend on `Twisted[windows]` (or something) to get pypiwin32 installed

Reported by: zooko Owned by: glyph
Priority: normal Milestone: undecided
Component: packaging Version: 1.10.0
Keywords: review Cc:
Launchpad Bug:

Description (last modified by warner)

https://pypi.python.org/pypi/pypiwin32

This is a fork (actually a very small fork, a.k.a. a toothpick) of pywin32, produced by Glyph in order to fix the problem of pywin32 not being pip-installable:

http://sourceforge.net/p/pywin32/bugs/669/

Depending on this would allow us to use newer versions of Twisted, which would remove the problem with comment:8:ticket:2066. I think this would allow us to close #2066.

(note: as of March-2016, we *do* have a platform-conditional install_requires= dependency on pypiwin32, and this ticket is about finding a better way to express that dependency)

Change History (16)

comment:1 Changed at 2015-03-19T20:03:01Z by zooko

  • Keywords design-review-needed added
  • Owner set to daira

comment:2 Changed at 2015-03-19T23:36:57Z by daira

Can zetuptoolz use wheels? I think it is forked from a version of setuptools that predates wheels.

comment:3 Changed at 2015-03-20T15:52:45Z by daira

  • Owner changed from daira to zooko

comment:4 Changed at 2015-07-21T20:10:06Z by zooko

glyph: could you build a .egg for Windows of pypiwin32?

comment:5 Changed at 2015-07-21T20:10:18Z by zooko

  • Owner changed from zooko to glyph

comment:6 Changed at 2015-07-21T23:08:39Z by glyph

I don't think so. My initial attempts produce errors, although I need to do some diagnosis as to why (it may be a local development environment setup problem).

I'm also reluctant to upload them. Eggs require easy_install, and easy_install doesn't verify SSL certificates, so this would be facilitating the arbitrary execution of unverified code I've been trying so hard to avoid in Python-packaging-land. I take it there are still some difficulties dealing with wheels in tahoe? I will keep trying, and I'll upload them if there's really no other way to do this in Tahoe.

comment:7 Changed at 2015-07-21T23:10:00Z by glyph

Ideally, you would just add a dependency on twisted[windows_platform]. But that will add a dependency on pyOpenSSL - are you still trying to avoid that? I wonder if there's some other way to break down our extras to give you the matrix Tahoe wants.

comment:8 Changed at 2016-03-25T16:59:21Z by warner

twisted[windows_SOMETHING] is how I'd prefer to do this. We're no longer trying to avoid pyopenssl, in fact because of a setuptools bug/lacking-feature, I think we need twisted[tls,windows_SOMETHING]. (Foolscap requires twisted[tls], and Twisted requires foolscap, but since setuptools will see Tahoe's twisted first, it will install twisted (without TLS), and then when it gets to foolscap later, it ignores the [tls], and we miss out on service-identity, and things break).

At present (Twisted-16.0.0), twisted[windows_platform] would also pull in the following _PLATFORM_INDEPENDENT features:

  • [tls] : that ok
  • [conch] : that's ok too
  • [soap]: um, I'd like to avoid it: 760KB of wheels, wstools, defusedxml, and docutils
  • [serial]: probably small, but completely unrelated to Tahoe

Also, because I suspect the same setuptools bug would cause problems if we Tahoe had two separate Twisted dependencies with different extras (e.g. Twisted[tls], Twisted[windows_platform]), the conditional logic in our setup.py/_auto_deps.py would need to be a bit grotty. So if it doesn't sound like Twisted is likely to change its windows dependency any time soon, I think we can continue to just conditionally depend on pypiwin32 when setup.py is run on windows.

BTW, we're all about wheels now. The existing pypiwin32 wheels should suit us just fine.

If you *were* going to re-slice the Twisted extras, maybe I'd suggest just a Twisted[windows] extra, that only pulls in what's necessary to run the reactor on windows (pypiwin32). In fact I'm not entirely sure what the windows_platform and osx_platform extras are for. E.g. on OS-X it would seem appropriate for us for depend on Twisted[osx], just in case there's some special magic that's necessary, but the current Twisted[osx_platform] includes pyobjc, which is gigantic, not available as wheels, takes something like an hour to compile, and not something we need (yet.. if/when we manage a non-web-based GUI, it may become necessary).

comment:9 Changed at 2016-03-25T17:09:45Z by warner

  • Description modified (diff)
  • Summary changed from add dependency on pypiwin32 to satisfy Twisted on Windows to depend on `Twisted[windows]` (or something) to get pypiwin32 installed

Changing the title of this ticket, since we actually added the dependency on pypiwin32 a few weeks ago (as part of switching to pip, in #1582).

comment:10 Changed at 2016-03-25T17:26:58Z by glyph

If you *were* going to re-slice the Twisted extras ...

It seems like this should really be in a ticket on Twisted :).

comment:11 Changed at 2016-03-25T17:32:42Z by glyph

the current Twisted[osx_platform] includes pyobjc, which is gigantic, not available as wheels, takes something like an hour to compile, and not something we need (yet.. if/when we manage a non-web-based GUI, it may become necessary).

This is actually necessary for doing anything that talks to OS X platform APIs, even those that have nothing to do with the GUI, like CoreLocation?, and CoreWLAN, and SCNetworkReachability.

However, Twisted's dependency on all of pyobjc is somewhat mistaken; we could get away with just pyobjc-core and pyobjc-framework-Cocoa which should be substantially faster to install.

In the meanwhile, for developers that do a lot with tahoe, if they're hitting this problem now, one way to jump-start your local wheel cache is to make a temporary virtualenv, and then do:

pip install pyobjc-core
pip install pyobjc

as two separate commands; compiling pyobjc's frameworks is much, much faster if you already have an importable pyobjc-core in your environment. Once you've done this once, pip install pyobjc will be plenty fast since you'll have local wheels.

comment:12 Changed at 2018-08-06T17:50:53Z by exarkun

  • Keywords design-review-needed removed

Dropping the design-review-needed keyword as there is no discernible design here for review.

comment:13 Changed at 2020-01-17T16:12:55Z by exarkun

The limitations of setuptools, pip, "extras", etc make it unlikely anything significant is going to change on this front any time soon.

The one thing that should likely be changed is that "pywin32" now ships wheels and "pypiwin32" is no longer relevant (and also increasingly out of date, one supposes - although if it works "out of date" might be a good thing rather than a bad thing). Also, Twisted now depends on "pywin32" on Windows so Tahoe-LAFS currently depends on "pypiwin32" *and" pywin32" on Windows. Oops. That's easy enough to fix, though (at least until Twisted changes its "windows_platform" extra again; sigh, duplication).

comment:15 Changed at 2020-01-20T19:09:37Z by GitHub <noreply@…>

  • Resolution set to fixed
  • Status changed from new to closed

In 5bd84895/trunk:

Merge pull request #685 from tahoe-lafs/2392.pywin32

Replace pypiwin32 with pywin32

Fixes: ticket:2392

comment:16 Changed at 2020-01-20T19:20:37Z by glyph

🎉❗️

Note: See TracTickets for help on using tickets.