port to Python 3

[zooko]

It might be nice if Tahoe-LAFS worked on Python 3. In my opinion, the ​2to3 tool, which is a translator from Python 2 source code to Python 3 source code is a non-starter. I don't want that tool to be invoked in our tool chain, and I don't see how we could even accept patches containing source code emitted by that tool, since that code would probably not be understood or maintained properly. On the other hand, the ​six tool — which is a library containing abstractions that are implemented differently by Python 2 and Python 3 — seems like a fine way to go.

So: if there is anyone out there who wants to port Tahoe-LAFS to Python 3, and you want your patches accepted into the mainline Tahoe-LAFS project, please start with "six"!

This Milestone is in "undecided", because it isn't clear to me if it will ever get done or anyone will ever really care. ☹

[daira]

To get an incomplete list of warnings about incompatibilities with Python 3, do:

bin/tahoe @python -3 @tahoe debug trial 2>&1 |grep '^/' |sort |uniq >warnings.txt

I'll attach the results.

[daira]

python -3 warnings for Tahoe-LAFS test suite

[daira]

​https://github.com/tahoe-lafs/tahoe-lafs/commit/2065-python3-strawman

[daira]

That branch was obtained by running a patched version of python-modernize and applying some manual fixes. It's just to play with and is quite a way off being actually useful.

[daira]

I probably won't have time to work on this for a while. If anyone wants to help, the most useful thing would probably be to fix some of the ​bugs I filed against python-modernize.

[zooko]

This looks like a better library than six, for our purposes: ​https://pypi.python.org/pypi/nine

[daira]

I don't agree that nine is a better approach than six, for a large existing Python 2 codebase. It's designed for newly written Python 3 codebases that also need to work on Python 2.

I've been working on-and-off at my fork of python-modernize at ​https://github.com/daira/python-modernize.

[daira]

Replying to daira:

I don't agree that nine is a better approach than six, for a large existing Python 2 codebase. It's designed for newly written Python 3 codebases that also need to work on Python 2.

Besides, they're not mutually exclusive; you can run python-modernize to get the code closer to Python 3, and then potentially switch to nine after any needed manual refactoring.

[daira]

OTOH some of the things nine does are eminently sensible. Kind of makes me want to write a sevenpointfive library ;-)

[daira]

I'm just putting this here as a possible alternative to six: ​http://python-future.org. It might be too complicated. (Also, pyOpenSSL depends on cryptography which depends on six, and we don't really want to have a dependency on two different Python-2/3-compatibility libraries if we can help it.)

[zooko]

python-future comes with a automatic (or at least mostly-automatic?) converter from Python2 to future: ​http://python-future.org/automatic_conversion.html

Here's a question for Daira or anyone else to chime in with their opinion: suppose we could wave a magic wand and have Tahoe-LAFS completely ported to six, nine, or future. Which of those three languages is the best programming language? Assume that you will then have to maintain and extend Tahoe-LAFS in that programming language for the next 10 years.

So the question is: what's the better programming language: ​six, ​nine, or ​future?

Anyone reading this probably doesn't need to be reminded, but what makes a programming language good is that it facilitates auditability (a.k.a. code-review a.k.a. readability) and testability.

[daira]

Replying to zooko:

Here's a question for Daira or anyone else to chime in with their opinion: suppose we could wave a magic wand and have Tahoe-LAFS completely ported to six, nine, or future. Which of those three languages is the best programming language?

I see little difference between them for long-term maintenance. There wouldn't be very many calls to them compared to the overall size of the code base.

[daira]

(That's actually one of the big objections I have to Python 3 -- it changed a bunch of backward-incompatible things that are utterly and profoundly unimportant to the maintainability of a code base.)

[zooko]

Yeah, I think another good option at this point is just stick with Python2. Having looked at six, nine, and future's documentation, it seems like future has the most detailed and organized documentation. Nine's is sparse and it says to "use the source", so I looked at the source and it is just one small source file, which is both good (less code is always better) and possibly bad (does this mean that six and future are offering more things that I would miss if I were using nine?).

Anyway, I guess I'm pretty ambivalent about the whole thing, and think maybe we should just close this ticket as "wontfix".

[daira]

I've been working on-and-off (mostly off) on my fork of python-modernize to get it closer to being able to process Tahoe-LAFS without breaking stuff. I don't want to close this ticket right now.

[zooko]

Replying to daira:

I've been working on-and-off (mostly off) on my fork of python-modernize to get it closer to being able to process Tahoe-LAFS without breaking stuff. I don't want to close this ticket right now.

Okay! Also maybe you should check out the modernizer automation in the "future" project as an alternative to python-modernize.

[daira]

Stage 1 of that modernizer automation uses lib2to3, which is the same thing python-modernize uses. (If the future project has improved some of those or added fixes, those changes should be pushed upstream, or if not then I'll apply them to my fork if I agree with them.)

I don't want to use stage 2 of their modernizer because I think my argument in comment:10 is compelling:

Also, pyOpenSSL depends on cryptography which depends on six, and we don't really want to have a dependency on two different Python-2/3-compatibility libraries if we can help it.

[zooko]

​This python-dev letter by Nick Coghlan says:

I think python-future provides a better "subset language" experience than modernize+six does, and provides a 3->common subset converter in addition to a 2->common subset (see ​http://python-future.org/faq.html#relationship-between-python-future-and-other-compatibility-tools for more on the difference between the two approaches).

I looked at the ​http://python-future.org/faq.html#what-is-the-relationship-between-future-and-six and it does make future sound like a nicer language than six.

[daira]

"Everything you did not want to know about Unicode in Python 3"

​http://lucumr.pocoo.org/2014/5/12/everything-about-unicode/

[tlhonmey]

Do note that Python 2.7 EOL is slated for 2020... So there's not a lot of time left before security fixes stop...

[richardARPANET]

I may be able to do this porting work. However can someone explain why the milestone states Python2 support should be retained? ​https://tahoe-lafs.org/trac/tahoe-lafs/milestone/Support%20Python%203

The vast majority of Python projects are dropping Python 2 support completely leading up to the end of this year before the deadline date.

[exarkun]

Hi richardARPANET,

Thanks for your interest. There has been a bunch of activity on this porting effort recently so regardless of anything else I suggest you coordinate with some of the others who are doing that work. These include meejah, heartsucker, and pythoncoder (which you may find on #tahoe-lafs on freenode at various times). The mailing list is probably also a good way to coordinate if you prefer.

Regarding the point that Python 2 support be retained, the motivation for this is that we don't feel we can manage a monolithic porting effort. That is, if Python 2 support is not retained then either:

(a) there is one huge PR porting all of tahoe-lafs at once that we feel would never be reviewable.

(b) there are many small PRs and between the time the first and last are merged, the code is broken (some of it is Python 2 only, some Python 3 only, you can never run it) and we feel this would be too disruptived of other development.

I hope that clears things up. Thanks again for your interest. Please let me know if any of the above is unclear or if you have other ideas, comments, questions, etc. (comments here are fine, so is use of the mailing list or irc, whatever you prefer).

[exarkun]

Also, I've edited the milestone description to try to explain the motivation for the Python 2/3 approach. Note that it may be that once the port is done, Python 2 support is almost immediately dropped - it will have served its purpose at that point and, as you rightly point out, upstream support of CPython 2.7 is almost at an end.

[richardARPANET]

@exarkun Thanks for clarifying the approach, I'll sync up with the other people working on this one.

[exarkun]

Let's just use the milestone to track the big-picture goal and use tickets for fine-grained tasks that can easily be resolved with a modest PR.