drop support for Python < 2.6

[zooko]

Let's drop support for versions of Python < 2.6. According to distrowatch's tables about the versions of packages that come with distros, this would exclude Tahoe-LAFS 1.10 from being usable with the included Python on the following versions of major distros:

distro	excluded version	excluded version release date	oldest supported[*] version	oldest supported[*] release date	Python in oldest supported[*]	notes	next version	Python in next version
​Fedora	10	2008-11-25	15	2011-05-24	2.7		rawhide will be 17	2.7.2
​Ubuntu	8.10	2008-10-30	10.04 LTS	2010-04-29	2.6		will be 12.04 precise	2.7.2
​Debian	5.0	2009-02-15	6.0	2011-02-06	2.6		unstable sid will be 7.0	2.7.2
​RedHat	5.7	2011-07-21	5.7	2011-07-21	2.4	I'm told that people tend to install their own newer version of Python when they want to use newer Python programs on older RHEL	(current, not next) 6.2	2.6.6

Among other benefits, dropping support for Pythons older than 2.6 would allow us to remove the dependency on simplejson and the (complex imperatively decided) dependency on pysqlite.

See comment:11 -- apparently Python ≥ 2.6 is already ubiquitous in the *BSD world.

[*] that is, the oldest OS version supported by the OS distributors and still receiving security patches.

[davidsarah]

I don't understand the "oldest supported" columns for FreeBSD and RedHat, if they're supposed to be referencing the "included" Python.

Another thing that could be dropped is the SHA-256 bindings in pycryptopp, since hashlib has SHA-256 since Python 2.5.

[zooko]

Twisted is planning to drop support for Python ≤ 2.5 after their next release, which means sometime this year or so:

​http://twistedmatrix.com/trac/ticket/5553

[zooko]

Replying to davidsarah:

I don't understand the "oldest supported" columns for FreeBSD and RedHat, if they're supposed to be referencing the "included" Python.

I changed the title of the "oldest supported" column to "oldest supported version of the operating system". Does it make sense now?

[gdt]

As far as I can tell the FreeBSD column doesn't quite make sense. I'm a little fuzzy, but I think python isn't part of the base system, but is instead in "ports", which is similar to NetBSD. I think the culture is to run relatively recent ports even on older base systems, so my guess is that dropping <=2.5 will not be a significant issue for FreeBSD users.

The chart above doesn't have NetBSD, but python 2.6 has been default in pkgsrc for probably 2 years if not more, and the next quarterly release of pkgsrc (April) will have 2.7 as the default. So dropping support for <=2.5 is fine from that viewpoint also.

[davidsarah]

Replying to zooko:

Replying to davidsarah:

I don't understand the "oldest supported" columns for FreeBSD and RedHat, if they're supposed to be referencing the "included" Python.

I changed the title of the "oldest supported" column to "oldest supported version of the operating system". Does it make sense now?

No, that wasn't what I was confused about. I was confused as to how it is possible for an entry in the first column, i.e. "excluded version of the operating system", to be greater than or equal to the "oldest supported version of the operating system" -- i.e. FreeBSD 7.2 >= FreeBSD 7.1 and RedHat 5.7 >= RedHat 5.7.

[zooko]

Replying to davidsarah:

No, that wasn't what I was confused about. I was confused as to how it is possible for an entry in the first column, i.e. "excluded version of the operating system", to be greater than or equal to the "oldest supported version of the operating system" -- i.e. FreeBSD 7.2 >= FreeBSD 7.1 and RedHat 5.7 >= RedHat 5.7.

I'm still not sure I understand what you were/are confused about. "excluded version of the operating system" is supposed to be the newest release of the operating system that doesn't have Python ≥ 2.6. "oldest supported version of the operating system" is supposed to be the oldest release of the operating system that is supported *by the operating system distributors*, i.e. the oldest release of the operating system that still gets security patches. Does that help? I apologize for the confusing wording.

[zooko]

Replying to gdt:

As far as I can tell the FreeBSD column doesn't quite make sense. I'm a little fuzzy, but I think python isn't part of the base system, but is instead in "ports", which is similar to NetBSD. I think the culture is to run relatively recent ports even on older base systems, so my guess is that dropping <=2.5 will not be a significant issue for FreeBSD users.

I guess I was following distrowatch on that. So you're saying that even users of FreeBSD older than 7.3, such as FreeBSD 7.2 or 7.1 (or even 6?) will be able to deploy a Python 2.6-requiring Tahoe-LAFS without installing a local Python?

The chart above doesn't have NetBSD, but python 2.6 has been default in pkgsrc for probably 2 years if not more, and the next quarterly release of pkgsrc (April) will have 2.7 as the default. So dropping support for <=2.5 is fine from that viewpoint also.

According to ​distrowatch, the NetBSD entry would look like this:

distro	excluded version	excluded version release date	oldest supported version of the operating system	oldest supported release date	Python in oldest supported	notes	next version	Python in next version
​NetBSD	5.0	2009-04-29	4.0	2007-12-19	2.5.2		pkgsrc? NetBSD-CURRENT? will be 6.0?	2.7.x

But apparently distrowatch, or my way of reading it, is inaccurate. Could you, gdt, please update the table to reflect NetBSD? Thanks!

[davidsarah]

sqlite bindings are in Python 2.5, and so is hashlib.sha256. When we've had buildslave failures for unsupported syntax, they've usually been for 2.4.x and not 2.5. So is the simplejson dependency the only reason to drop support for 2.5?

If so, note that the json module in the stdlib is not identical to simplejson. According to ​http://stackoverflow.com/questions/712791/json-and-simplejson-module-differences-in-python, the stdlib version is less frequently updated and slower (although I would be more concerned about security-relevant bugfixes than speed).

I agree we should definitely drop support for 2.4.x.

zooko wrote:

"oldest supported version of the operating system" is supposed to be the oldest release of the operating system that is supported *by the operating system distributors*, i.e. the oldest release of the operating system that still gets security patches.

Oh, ok, that wasn't at all obvious.

[gdt]

distrowatch is confused; NetBSD simply does not include python (at all, and never has). In NetBSD culture, one use pkgsrc to get extras from among the 6000 or so other things that are not part of the operating system and are in pkgsrc. FreeBSD is basically the same way, except I can't speak as authoritatively about it.

On NetBSD, one basically needs a version of pkgsrc (stable ones are released quarterly) that supports the version of the OS one is running (meaning that most packages will build on that OS version), and then it's easy to use whatever is in that pkgsrc version. So people on NetBSD can have python 2.6 or 2.7 very easily, and 2.6 is what's been normal for a while. In NetBSD, at least the stable branch of the last two releases are generally supported by pkgsrc. 6.0 isn't out, so anyone with 4.0, 5.0, 5.1. etc. will be able to have python 2.6 or 2.7 without difficulty, from the current 2011Q4 stable branch, or the April 2012Q1 branch.

But, the notion that "NetBSD 4.0 has python 2.5.2" is confused. What they might mean is that a stable version of pkgsrc associated in time with the release of NetBSD 4.0 had python 2.5.2, but that doesn't matter now.

On FreeBSD, I believe the culture is that one upgrades from 7.x to 7.x+1 quite easily, and you're running old/not-really-maintained-code if you don't. So sure, people need to build python, but that's done automatically when building tha tahoe-lafs port (which either does exist or ought to).

Also, on FreeBSD (or Dragonfly, OpenBSD, Mac, Linux, Solaris, AIX, etc) one can use pkgsrc.

I suspect the problem is that distrowatch either doesn't get it that all the world does not work like GNU/Linux, or is trying to map the world into that model.

But all in all, my belief is that dropping 2.5 is a non-event for the BSD world.

I haven't updated the table, because I don't think it makes sense for other than Linux-style distributions and I don't know how to do it sensibly. There's no Windows row, which would have the same issues, as Windows doesn't include python either.

[warner]

I found some of my old notes for what we could clean up if we required a newer python version. If we dropped 2.4 and required >=2.5, we could:

• replace DictOfSets with collections.defaultdict(set)
• replace pycryptopp.hash.SHA256 with hashlib.sha256
• replace pysqlite2|sqlite3 with stdlib sqlite3
• switch to PEP328 relative imports

And then >=2.6 would get us stdlib json, and could replace our LRU cache (unless we already got rid of it?) with collections.deque(maxlen=). It might even be possible to get a codebase that works with py2.6 and py3.3 simultaneously, since 2.6 adds b"", print-as-function (guarded by __future__), and "except FooError as e:" (although, of course, we kind of have to wait for our dependencies to move to py3 first).

[davidsarah]

Require Python 2.5. Includes simplifications resulting from being able to use sqlite3 from the standard library. This also drops sqlite3 from the set of versions and paths we report.

[davidsarah]

Note that it's only dropping support for Python 2.4.x that is in milestone 1.10.

[davidsarah]

Since we now require Python 2.5, we can use os.SEEK_END. (Apply on top of previous patch.)

[davidsarah]

Zooko read these patches and approved of them, so I committed them (only on trunk, not for 1.9.2): 0fc196ea5fa4ad2f, a1a1b5bf8a6e1a09, 4ddcde309454179a

Please open separate tickets for any other simplifications that can be made, or if we want to also drop 2.5. (From comment:16, only the sqlite change has been done.)

[davidsarah]

By the way, dropping 2.6 would not allow eliminating the dependency on simplejson; that was added to the stdlib in 2.7.

[davidsarah]

Oh, I'd missed comment:4 and ​http://twistedmatrix.com/trac/ticket/5553. So there is still a strong incentive to drop support for 2.5, so that we don't have to impose a conditional "Twisted <= 12.1" requirement in _auto_deps.py for Python 2.5.

[warner]

Replying to davidsarah:

By the way, requiring 2.6 would not allow eliminating the dependency on simplejson; that was added to the stdlib in 2.7.

​http://docs.python.org/library/json.html#module-json says that json was added in python2.6 . Some minor features were added in the py2.7 version, but I don't think they'd be useful to us.

[zooko]

According to my notes from the original description in this ticket, plus gdt's comments in comment:11, there aren't many people who require Python 2.5. We should post to the tahoe-dev list asking someone to speak up if they use Python 2.5. I'll bet nobody does!

[david-sarah@…]

In b82cddef7b7f5eee:

setup.py and bin/tahoe-script.template: the error when we try to use Python 3 should give the correct minimum Python version (now 2.5). refs #1658

[david-sarah <david-sarah@…>]

In b82cddef7b7f5eee:

setup.py and bin/tahoe-script.template: the error when we try to use Python 3 should give the correct minimum Python version (now 2.5). refs #1658

[davidsarah]

In src/allmydata/test/test_upload.py​:

# copied from python docs because itertools.combinations was added in
# python 2.6 and we support >= 2.4.
def combinations(iterable, r):

[david-sarah@…]

In 5923/cloud-backend:

[rebased for cloud-backend] setup.py and bin/tahoe-script.template: the error when we try to use Python 3 should give the correct minimum Python version (now 2.5). refs #1658

[davidsarah]

Fixed in ​7008ffa5.

I realized that the patch fixing #1775 depends on a construct added in Python 2.6 ("except Exception as e"). I think we had already decided to drop Python 2.5 for Tahoe-LAFS v1.10.