#2065 closed enhancement (duplicate)

port to Python 3

Reported by: zooko Owned by:
Priority: normal Milestone: Support Python 3
Component: code Version: 1.10.0
Keywords: Python3 Cc:
Launchpad Bug:

Description

It might be nice if Tahoe-LAFS worked on Python 3. In my opinion, the 2to3 tool, which is a translator from Python 2 source code to Python 3 source code is a non-starter. I don't want that tool to be invoked in our tool chain, and I don't see how we could even accept patches containing source code emitted by that tool, since that code would probably not be understood or maintained properly. On the other hand, the six tool — which is a library containing abstractions that are implemented differently by Python 2 and Python 3 — seems like a fine way to go.

So: if there is anyone out there who wants to port Tahoe-LAFS to Python 3, and you want your patches accepted into the mainline Tahoe-LAFS project, please start with "six"!

This Milestone is in "undecided", because it isn't clear to me if it will ever get done or anyone will ever really care. ☹

Attachments (1)

warnings.txt (19.2 KB) - added by daira at 2013-08-29T01:58:58Z.
python -3 warnings for Tahoe-LAFS test suite

Download all attachments as: .zip

Change History (27)

comment:1 Changed at 2013-08-27T15:05:43Z by daira

To get an incomplete list of warnings about incompatibilities with Python 3, do:

bin/tahoe @python -3 @tahoe debug trial 2>&1 |grep '^/' |sort |uniq >warnings.txt

I'll attach the results.

Changed at 2013-08-29T01:58:58Z by daira

python -3 warnings for Tahoe-LAFS test suite

comment:3 Changed at 2013-08-30T03:21:11Z by daira

That branch was obtained by running a patched version of python-modernize and applying some manual fixes. It's just to play with and is quite a way off being actually useful.

Last edited at 2013-08-30T03:23:17Z by daira (previous) (diff)

comment:4 Changed at 2013-08-30T03:23:57Z by daira

  • Owner set to daira
  • Status changed from new to assigned

comment:5 Changed at 2013-09-01T11:38:33Z by daira

  • Owner daira deleted
  • Status changed from assigned to new

I probably won't have time to work on this for a while. If anyone wants to help, the most useful thing would probably be to fix some of the bugs I filed against python-modernize.

comment:6 Changed at 2013-12-30T21:55:36Z by zooko

This looks like a better library than six, for our purposes: https://pypi.python.org/pypi/nine

comment:7 follow-up: Changed at 2013-12-30T22:18:58Z by daira

I don't agree that nine is a better approach than six, for a large existing Python 2 codebase. It's designed for newly written Python 3 codebases that also need to work on Python 2.

I've been working on-and-off at my fork of python-modernize at https://github.com/daira/python-modernize.

comment:8 in reply to: ↑ 7 Changed at 2013-12-30T22:21:07Z by daira

Replying to daira:

I don't agree that nine is a better approach than six, for a large existing Python 2 codebase. It's designed for newly written Python 3 codebases that also need to work on Python 2.

Besides, they're not mutually exclusive; you can run python-modernize to get the code closer to Python 3, and then potentially switch to nine after any needed manual refactoring.

comment:9 Changed at 2013-12-30T22:29:22Z by daira

OTOH some of the things nine does are eminently sensible. Kind of makes me want to write a sevenpointfive library ;-)

comment:10 Changed at 2014-04-28T16:35:31Z by daira

I'm just putting this here as another possible alternative to six: http://python-future.org. It might be too complicated. (Also, pyOpenSSL depends on cryptography which depends on six, and we don't really want to have a dependency on two different Python-2/3-compatibility libraries if we can help it.)

Last edited at 2014-04-28T16:36:29Z by daira (previous) (diff)

comment:11 follow-up: Changed at 2014-05-06T14:45:51Z by zooko

python-future comes with a automatic (or at least mostly-automatic?) converter from Python2 to future: http://python-future.org/automatic_conversion.html

Here's a question for Daira or anyone else to chime in with their opinion: suppose we could wave a magic wand and have Tahoe-LAFS completely ported to six, nine, or future. Which of those three languages is the best programming language? Assume that you will then have to maintain and extend Tahoe-LAFS in that programming language for the next 10 years.

So the question is: what's the better programming language: six, nine, or future?

Anyone reading this probably doesn't need to be reminded, but what makes a programming language good is that it facilitates auditability (a.k.a. code-review a.k.a. readability) and testability.

Last edited at 2014-05-06T14:47:16Z by zooko (previous) (diff)

comment:12 in reply to: ↑ 11 Changed at 2014-05-06T17:58:15Z by daira

Replying to zooko:

Here's a question for Daira or anyone else to chime in with their opinion: suppose we could wave a magic wand and have Tahoe-LAFS completely ported to six, nine, or future. Which of those three languages is the best programming language?

I see little difference between them for long-term maintenance. There wouldn't be very many calls to them compared to the overall size of the code base.

comment:13 Changed at 2014-05-06T18:03:06Z by daira

(That's actually one of the big objections I have to Python 3 -- it changed a bunch of backward-incompatible things that are utterly and profoundly unimportant to the maintainability of a code base.)

comment:14 Changed at 2014-05-06T19:14:37Z by zooko

Yeah, I think another good option at this point is just stick with Python2. Having looked at six, nine, and future's documentation, it seems like future has the most detailed and organized documentation. Nine's is sparse and it says to "use the source", so I looked at the source and it is just one small source file, which is both good (less code is always better) and possibly bad (does this mean that six and future are offering more things that I would miss if I were using nine?).

Anyway, I guess I'm pretty ambivalent about the whole thing, and think maybe we should just close this ticket as "wontfix".

comment:15 follow-up: Changed at 2014-05-08T12:11:11Z by daira

I've been working on-and-off (mostly off) on my fork of python-modernize to get it closer to being able to process Tahoe-LAFS without breaking stuff. I don't want to close this ticket right now.

comment:16 in reply to: ↑ 15 Changed at 2014-05-12T17:22:56Z by zooko

Replying to daira:

I've been working on-and-off (mostly off) on my fork of python-modernize to get it closer to being able to process Tahoe-LAFS without breaking stuff. I don't want to close this ticket right now.

Okay! Also maybe you should check out the modernizer automation in the "future" project as an alternative to python-modernize.

comment:17 Changed at 2014-05-12T17:53:27Z by daira

Stage 1 of that modernizer automation uses lib2to3, which is the same thing python-modernize uses. (If the future project has improved some of those or added fixes, those changes should be pushed upstream, or if not then I'll apply them to my fork if I agree with them.)

I don't want to use stage 2 of their modernizer because I think my argument in comment:10 is compelling:

Also, pyOpenSSL depends on cryptography which depends on six, and we don't really want to have a dependency on two different Python-2/3-compatibility libraries if we can help it.

comment:18 Changed at 2014-05-29T15:20:16Z by zooko

  • Type changed from defect to enhancement

This python-dev letter by Nick Coghlan says:

I think python-future provides a better "subset language" experience than modernize+six does, and provides a 3->common subset converter in addition to a 2->common subset (see http://python-future.org/faq.html#relationship-between-python-future-and-other-compatibility-tools for more on the difference between the two approaches).

I looked at the http://python-future.org/faq.html#what-is-the-relationship-between-future-and-six and it does make future sound like a nicer language than six.

comment:19 Changed at 2014-05-30T11:33:02Z by daira

"Everything you did not want to know about Unicode in Python 3"

http://lucumr.pocoo.org/2014/5/12/everything-about-unicode/

comment:20 Changed at 2019-01-02T19:56:56Z by tlhonmey

Do note that Python 2.7 EOL is slated for 2020... So there's not a lot of time left before security fixes stop...

comment:21 Changed at 2019-03-15T16:19:58Z by exarkun

  • Milestone changed from undecided to Support Python 3

comment:22 Changed at 2019-03-28T16:11:21Z by richardARPANET

I may be able to do this porting work. However can someone explain why the milestone states Python2 support should be retained? https://tahoe-lafs.org/trac/tahoe-lafs/milestone/Support%20Python%203

The vast majority of Python projects are dropping Python 2 support completely leading up to the end of this year before the deadline date.

comment:23 Changed at 2019-03-28T16:35:18Z by exarkun

Hi richardARPANET,

Thanks for your interest. There has been a bunch of activity on this porting effort recently so regardless of anything else I suggest you coordinate with some of the others who are doing that work. These include meejah, heartsucker, and pythoncoder (which you may find on #tahoe-lafs on freenode at various times). The mailing list is probably also a good way to coordinate if you prefer.

Regarding the point that Python 2 support be retained, the motivation for this is that we don't feel we can manage a monolithic porting effort. That is, if Python 2 support is not retained then either:

(a) there is one huge PR porting all of tahoe-lafs at once that we feel would never be reviewable.

(b) there are many small PRs and between the time the first and last are merged, the code is broken (some of it is Python 2 only, some Python 3 only, you can never run it) and we feel this would be too disruptived of other development.

I hope that clears things up. Thanks again for your interest. Please let me know if any of the above is unclear or if you have other ideas, comments, questions, etc. (comments here are fine, so is use of the mailing list or irc, whatever you prefer).

comment:24 Changed at 2019-03-28T16:40:10Z by exarkun

Also, I've edited the milestone description to try to explain the motivation for the Python 2/3 approach. Note that it may be that once the port is done, Python 2 support is almost immediately dropped - it will have served its purpose at that point and, as you rightly point out, upstream support of CPython 2.7 is almost at an end.

comment:25 Changed at 2019-03-29T08:21:30Z by richardARPANET

@exarkun Thanks for clarifying the approach, I'll sync up with the other people working on this one.

comment:26 Changed at 2019-09-13T12:47:06Z by exarkun

  • Resolution set to duplicate
  • Status changed from new to closed

Let's just use the milestone to track the big-picture goal and use tickets for fine-grained tasks that can easily be resolved with a modest PR.

Note: See TracTickets for help on using tickets.