depend on `Twisted[windows]` (or something) to get pypiwin32 installed

[zooko]

​https://pypi.python.org/pypi/pypiwin32

This is a fork (actually a very small fork, a.k.a. a toothpick) of pywin32, produced by Glyph in order to fix the problem of pywin32 not being pip-installable:

​http://sourceforge.net/p/pywin32/bugs/669/

Depending on this would allow us to use newer versions of Twisted, which would remove the problem with comment:8:ticket:2066. I think this would allow us to close #2066.

(note: as of March-2016, we *do* have a platform-conditional install_requires= dependency on pypiwin32, and this ticket is about finding a better way to express that dependency)

[daira]

Can zetuptoolz use wheels? I think it is forked from a version of setuptools that predates wheels.

[zooko]

glyph: could you build a .egg for Windows of pypiwin32?

[glyph]

I don't think so. My initial attempts produce errors, although I need to do some diagnosis as to why (it may be a local development environment setup problem).

I'm also reluctant to upload them. Eggs require easy_install, and easy_install doesn't verify SSL certificates, so this would be facilitating the arbitrary execution of unverified code I've been trying so hard to avoid in Python-packaging-land. I take it there are still some difficulties dealing with wheels in tahoe? I will keep trying, and I'll upload them if there's really no other way to do this in Tahoe.

[glyph]

Ideally, you would just add a dependency on twisted[windows_platform]. But that will add a dependency on pyOpenSSL - are you still trying to avoid that? I wonder if there's some other way to break down our extras to give you the matrix Tahoe wants.

[warner]

twisted[windows_SOMETHING] is how I'd prefer to do this. We're no longer trying to avoid pyopenssl, in fact because of a setuptools bug/lacking-feature, I think we need twisted[tls,windows_SOMETHING]. (Foolscap requires twisted[tls], and Twisted requires foolscap, but since setuptools will see Tahoe's twisted first, it will install twisted (without TLS), and then when it gets to foolscap later, it ignores the [tls], and we miss out on service-identity, and things break).

At present (Twisted-16.0.0), twisted[windows_platform] would also pull in the following _PLATFORM_INDEPENDENT features:

• [tls] : that ok
• [conch] : that's ok too
• [soap]: um, I'd like to avoid it: 760KB of wheels, wstools, defusedxml, and docutils
• [serial]: probably small, but completely unrelated to Tahoe

Also, because I suspect the same setuptools bug would cause problems if we Tahoe had two separate Twisted dependencies with different extras (e.g. Twisted[tls], Twisted[windows_platform]), the conditional logic in our setup.py/_auto_deps.py would need to be a bit grotty. So if it doesn't sound like Twisted is likely to change its windows dependency any time soon, I think we can continue to just conditionally depend on pypiwin32 when setup.py is run on windows.

BTW, we're all about wheels now. The existing pypiwin32 wheels should suit us just fine.

If you *were* going to re-slice the Twisted extras, maybe I'd suggest just a Twisted[windows] extra, that only pulls in what's necessary to run the reactor on windows (pypiwin32). In fact I'm not entirely sure what the windows_platform and osx_platform extras are for. E.g. on OS-X it would seem appropriate for us for depend on Twisted[osx], just in case there's some special magic that's necessary, but the current Twisted[osx_platform] includes pyobjc, which is gigantic, not available as wheels, takes something like an hour to compile, and not something we need (yet.. if/when we manage a non-web-based GUI, it may become necessary).

[warner]

Changing the title of this ticket, since we actually added the dependency on pypiwin32 a few weeks ago (as part of switching to pip, in #1582).

[glyph]

If you *were* going to re-slice the Twisted extras ...

It seems like this should really be in a ticket on Twisted :).

[glyph]

the current Twisted[osx_platform] includes pyobjc, which is gigantic, not available as wheels, takes something like an hour to compile, and not something we need (yet.. if/when we manage a non-web-based GUI, it may become necessary).

This is actually necessary for doing anything that talks to OS X platform APIs, even those that have nothing to do with the GUI, like CoreLocation?, and CoreWLAN, and SCNetworkReachability.

However, Twisted's dependency on all of pyobjc is somewhat mistaken; we could get away with just pyobjc-core and pyobjc-framework-Cocoa which should be substantially faster to install.

In the meanwhile, for developers that do a lot with tahoe, if they're hitting this problem now, one way to jump-start your local wheel cache is to make a temporary virtualenv, and then do:

pip install pyobjc-core
pip install pyobjc

as two separate commands; compiling pyobjc's frameworks is much, much faster if you already have an importable pyobjc-core in your environment. Once you've done this once, pip install pyobjc will be plenty fast since you'll have local wheels.

[exarkun]

Dropping the design-review-needed keyword as there is no discernible design here for review.

[exarkun]

The limitations of setuptools, pip, "extras", etc make it unlikely anything significant is going to change on this front any time soon.

The one thing that should likely be changed is that "pywin32" now ships wheels and "pypiwin32" is no longer relevant (and also increasingly out of date, one supposes - although if it works "out of date" might be a good thing rather than a bad thing). Also, Twisted now depends on "pywin32" on Windows so Tahoe-LAFS currently depends on "pypiwin32" *and" pywin32" on Windows. Oops. That's easy enough to fix, though (at least until Twisted changes its "windows_platform" extra again; sigh, duplication).

[exarkun]

​https://github.com/tahoe-lafs/tahoe-lafs/pull/685

[GitHub <noreply@…>]

In 5bd84895/trunk:

Merge pull request #685 from tahoe-lafs/2392.pywin32

Replace pypiwin32 with pywin32

Fixes: ticket:2392

[glyph]

🎉❗️