Historically, Debian hasn't particularly objected to packaging obsolete versions of programming languages without upstream support.
I doubt anyone is checking for potential security problems in the Algol 68 and Fortran 77 implementations that Debian ships, and I don't think the people using those packages are particularly inconvenienced by that.
It seems a shame that the social pressure to persuade people to port their code to Python 3 means that Debian is going to have weaker support for 10-year-old Python than 40-year-old Fortran.
In particular, there are ongoing efforts to try to make it the normal thing for scientists to make the programs they ran on their data available so that their results can be reproduced; aggressively dropping older programming language implementations rather gets in the way of that.
What? This isn't about "languages". It's about software!
Algol 68 and Fortran 77 may have stale (but maintained) compilers or interpreters in the package repository. Starting very soon - Python 2 will have an entire set of unmaintained runtime and libraries in the package repository. You know - actual, offically, unmaintained software! Unmaintained software that other packages, including Calibre in this example, further build on. Of course they're throwing this out.
If you're concerned about language support there are multiple solutions like:
* Port your stuff to Python 3 and enjoy first class support
* Running Debian Buster
* Running CentOS 8, which will probably profit from RHEL 8's Python 2 support-with-an-asterisk until 2029 [1].
* Installing it manually on your system
* Installing it manually in a container
This is how it feels to run legacy software - it's bloody inconvenient. But that's not a problem Debian needs to solve for anyone.
Fortran 77 doesn't have a stale compiler; modern Fortran compilers support Fortran back to the beginning of time. Almost all of the weirdnesses are limited to the front-end, so it's not that difficult to support.
Python 2 is only unmaintained because the Python foundation is refusing to let anyone "officially" take over support. There are thousands of programs in Debian with less core developers than Python 2 working just fine.
In the case, the name is utility and not an artifact. I would argue that having the same name is necessary to even having a workalike, at least for a system command and a symlink.
As long time Python user, my view of the PSF is that they could mature a little bit. Their ambivalent hostility towards PyPy and Jython, the lack of portable standard library, and now with how they are continuing to mismanage the 2to3 transition. None of this is how adults should behave.
That's true, but it might make packaging tricky -- lots of the old Python 2 software people want to run will have '#!/usr/bin/env python' hard-wired in, and having to fix those would be annoying.
Upgrading the OS is also annoying, so why not just stay in one place for next 5, 10, 50? years. If you don't upgrade your machine it probably will continue to work, python 2 won't disappear.
I wish PSF would pull the plug from python 2 much sooner so we wouldn't have to deal with whiners for 10 f...ing years. The problem with python 3 wasn't the incompatibility, the problem was keeping python 2 alive for so long. Other languages usually gave only 1 year and everyone moved, because they didn't want to stay behind.
Python 3 came out in December 2008. But for a lot of packages, it wasn't realistic to migrate until it was possible to support both Pythons from the same codebase. And that wasn't realistic until 2.7/3.3, which was 2012.
At that point the rising tide started and packages with no dependencies began doing dual support. But it was many years before there was enough coverage that application level packages like django could consider migrating.
So yes, the cutoff could have been somewhat sooner, but if the PSF had attempted to force a transition in 2009, it would have been a death knell for Python. Not even numpy was available for Python 3 until 2011!
Yeah, I started using Python 3 immediately. By biggest pain point was probably print going from a statement to a function, which while I agree it was the right move, I still have muscle memory that wants to use the statement. There are a few other pain points, like Unicode encoding, csv line endings. But, what prevented me from moving earlier was lack of 3rd party support. Things like DB and MQ access...
The PSF actively discouraged folks from writing packages that were compatible with both. They wanted a hard schism and even tacit support for 2 was to reject the wishes of the church.
The original proposal from GvR for dual support in libraries was to release separate source, where the Py3 version would be the generated output of running source transformation tools like 2to3 against the Py2 source:
No they don’t. Step functions usually signify the change in evolutionary epochs. Meaning any time the change is too great, it literally means starting over.
Python barely survived 2to3, it should serve as a warning to every other system that thinks it can do a stop the world upgrade.
My C89 code, C++98 code and Fortran 77 code all still compile on a modern Ubuntu, and none of these are expected to break any time soon. The compilers just support multiple versions.
I'd strongly suggest watching "What is C++" [0] from this year's cppcon. There's a lot of good content, but the relevant bit is that c especially (remember, c++98 is younger than python2!) values backwards compatibility very highly. This is sometimes useful, but it has costs. Performance costs, safety costs, and especially ergonomic/complexity costs for future development (the kitchen-sink-ness of modern cpp).
People aren't wrong to make different tradeoffs than you. Especially given that strict backwards compatibility is impossible (especially in a language like c/++, people will rely on things that aren't intentionally public, because they can introspect your bytes).
Compiler changes break code all the time, it's just almost always semantic and not syntactic. I find this amusing, since the syntactic changes are usually easier to detect and fix.
And python2 still works, it just won't get new features.
> And python2 still works, it just won't get new features.
Probably more importantly, it wont get security fixes.
Yeah, Python 2 will still work. Ita not like there's a kill switch or anything. It just wont receive any official or supported updates. Effectively, Py2 will be abandonware.
I have been extremely critical of the P2 to P3 drama, but to be totally fair, things haven't been as smooth as we like to remember in C++ land. While the c++11 standard (and successors) are highly compatible with c++98, most compilers at the turn of the millennium had questionable conformance and each compiler upgrade to a more recent and strict version was a lot of pain (I still remember the pain when GCC started enforcing two phase lookup).
I agree things could be better in c++, but never was never anything like python 3, redefining / , or removing printf. Also, usually once you fixed your code, it still worked in older compilers (although not always).
While I generally agree with your comment, Python generally gives easily accessible migration paths. Regarding your division example, you could import from __future__ integer_division (maybe it was another name?) Since I think Python 2.5 before it became the default in, I think, 2.7.
I wish the wallrus operator was available as a future import, bit it's not.
Also, Python never had printf. It had a print statement; print became a function in Python 3. Yes, its a syntax change, but its for the better. Having print as a first class statement expands its utility and versatility.
Removing printf was an example of a change C++ could make of similar effect (they could argue you should be using cout, so you don't need it). Sorry it was unclear
I don't understand people complaining about the removal of the print statement. Switching that to a function call is the easiest part of porting to Python 3. You don't even have to do it yourself, just run 2to3. If you're complaining about that I feel like you're just looking for something to complain about.
You are right. The print statement is easy to complain about. My main issue was all the strings/unicode changes, which were hard to track down as often code would continue to work until a non-unicode input came in, when it would error. Debugging all those issues took ages.
This is a good point, because of the dynamic nature of python, the incompatibility is not always visible at compile time (parse type?), but can happen at run time and only rarely.
Those languages also don’t have massive runtime libraries with thousands of potential bugs that may even be security holes. They are not comparable in the slightest.
> Ruby broke applications and it wasn't even major version release I think it was 1.8 to 1.9.
It was a major version release; Ruby, through 1.8, used an version numbering scheme similar to the old Linux version numbering scheme; odd second digits were development major versions, even second digits were stable major versions.
1.9 was a slight break because it was a major release but, IIRC, seen as stable.
Isn't Debian literally the one distro where that shouldn't be a problem?
You create a package called "constrictor2.7", and set it up to provide /usr/bin/python using update-alternatives. Then #!/usr/bin/python and #!/usr/bin/env python both work.
Starting next year, I'd expect most other platforms will not include python2 in the default distribution, so the default "python" can either point to python3 or nothing at all.
Isn't this article an indication that Debian intends to? I'd also guess there's a pretty good chance that distros based on Debian (e.g. Ubuntu) will follow that switch
As far as I know Debian intends to have no plain `python` on the default PATH once the python2 packages are removed, so that it will be up to the system administrator to decide where that will point.
Package maintainers have to deal with worse things than that. Also, major distributions migrated their core packages to Python 3 a long time ago. Even if Python 2 was supported for longer, the Python 2 age is definitely over for system programming.
Sure, but those symlinks have to be consistently created and maintained, possibly on multiple new and existing systems.
If you already have infrastructure that does similar things, it's probably not much of an issue, but I can see missing or misdirected symlinks being a common problem. Maintaining the symlink in the distribution (Debian or whatever) reduces the chance of errors.
If there is any authentic popular demand for such a thing at all, I have no doubt that somebody in the python 2 community will see fit to publish a script that creates the appropriate symlinks.
You can fork it, rename it, and make it provide python2 binary via both Debian alternatives and package-provides, then ask for inclusion in upstream. There's nothing that stops you and it's done for other languages as well. For example in C you have GCC and llvm providing cc.
* Everybody knows they should have moved to Python 3 for over three major versions of Debian stable.
* Debian is not committed to keep legacy software running in perpituity in new versions.
* For Debian Buster they have already committed to supporting Python 2 until at least 2022 - and probably 2024 for LTS. They're already putting in the commitment!
* The next version of Debian stable will probably be released mid 2021 - long after Python 2's EOL of Jan 1.
* RHEL's Python 2 support amounts to life support at best.
* Keeping around legacy software takes precious volunteer time - which is all Debian has.
> they have already committed to supporting Python 2 until at least 2022 - and probably 2024 for LTS. They're already putting in the commitment!
This is the entire point of a stable distro. To pick some software, stabilize it a specific api, and keep it supported. Upstream need-not support that specific version of a release, for reference, see the kernel.
Huh? I'm just saying it's not unreasonable to support software downstream that upstream doesn't. That's the entire point.
I don't care what Debian does at this point, too old for practical use.
> should support software they aren't interested in.
You mean like pestering all the other package maintainers to support python3 when they're not interested in doing that? That goes both ways. Nobody wanted python3, expect PSF. This is the continued backlash.
"Why should they do that?" is the better question. The tenets of Debian include stability. Having packages that may break at any time with no recourse is not stable.
Nothing about this is aggressive. And having maintained compilers for ancient languages is really really different from having an unmaintained, obsolete runtime/interpreter/ecosystem.
And nothing is stopping you from installing Python 2. All scientists I know rely on anaconda to manage their Python environment, and couldn't care less what the system python does. I really _don't_ want to rely on system python for reproducing my old results.
On the flipside: how useful is your code really if it's not reproducible with a modern Python version? From 3.3 to 3.8, there should be perfect backwards compatibility?
For packages, I can understand it, sure. So include a requirements.txt, and just spin up a 3.8 virtualenv to reproduce, no?
You mean like the recent issue that affected over a hundred papers because they assumed Python's glob module returned files in a sorted fashion and they depended upon the order the files were processed?
Well, that's kinda what I was saying. How useful is your code really, if it depends on the order in a list of files in a fragile way that's hard to fix if the order changes?
Probably very useful. Any big codebase is going to have lots of implicit dependencies by accident. It requires active development to stay compatible, and it's okay to not be perfect.
Backwards compatibility is there, but minor changes and patches might mean different things for certain users. One can never be sure without sifting throug changelogs and maintaining test suites. And that is for stuff you know you are looking for.
"In particular, there are ongoing efforts to try to make it the normal thing for scientists to make the programs they ran on their data available so that their results can be reproduced; aggressively dropping older programming language implementations rather gets in the way of that."
But if this is important to you then depending on the OS to provide your python seems like a bad idea, independent of the python2/python3 issues. Wouldn't it be much better to use something like conda / the anaconda distribution, so that you can specify precise versions of each library for your environment?
Having said that, I don't know how anaconda is going to handle the EOL of python2. So I guess the question is: if someone wants to be able to reliably reproduce a python2 environment with specific set of libraries, what is best way to do that?
> Having said that, I don't know how anaconda is going to handle the EOL of python2. So I guess the question is: if someone wants to be able to reliably reproduce a python2 environment with specific set of libraries, what is best way to do that?
What's to handle? Old versions of python2 packages are still in the online repos. The EOL means that there won't be any new releases to python. All of the old releases will still work though.
Recently I had to get some old Rails 1.8 apps running so I could get some screenshots for an upcoming project. Docker containers are great for this. It took a while to get everything working, but now I can export those container images and always have a version of these old projects I can run on a container runtime. I'll never use them for production of course, but they're good historically.
As others have stated, those are old implementations, but I'm sure Debain will still accept security patches for them. Python2 is about to lose major support. You cannot just let rotting dependencies stick around in your tree. You're not going to have a good time.
If people really want that stuff, they can pack it into a Docker container, or Snap/Flatpack/whatever garbage for old GUI applications.
I guess in 10 years time, when dockerhub goes away, and most of the packaging scripts have broken online dependencies, a bunch of those docker images are going to be really hard to rebuild...
Developers and scientists, in my experience, prefer to get the latest Python wheels from the Internet rather than reply on distributions to curate their packages. The consumers of distribution-provided packages is... other distribution-provided packages.
I have seen multiple people post to Ycombinator claiming that porting to Python3 is always trivial and easily done. And, yeah, sure, for a 380-line script written after Python3 came out, porting the code so it runs the same both in Python2 and Python3 is trivial.
However, for an over 6000-line script written in the mid-2000s, before Python3 even existed, porting to Python3 is non trivial. Trust me, I tried.
First, the syntax sugar stuff was easy enough to fix: Do a simple search and replace so that all cases of stuff like
print "foo"
Become
print("foo")
I had to do something similar with the way this script calls raise(). Not to mention adding a hack so Python2 uses “xrange” while Python3 uses “range”. After doing all of this, the code still does not run in Python3. As far as I can tell, the issue is that Python3 does implicit float-to-int conversions differently than Python2; the code will not raise an error in Python3, but it will silently run differently, until the state is different enough that the Python3 version raises an error while the Python2 version works as expected.
At this point, I just threw my hands in the air and gave up.
No one has claimed porting to python 3 is easy in this thread. That said, if you start by "search and replace", then you are probably doing it wrong [0].
Hacker News gets some of the smartest people willing to jump into a comment section, but its also full of uninformed idiots talking out of their ass and trying to sound authoritative. I won’t defend them, but complaining about something someone else posted in another thread isn’t gonna get you too much sympathy either.
OP changed their comment to clarify they were referring to HN discussions historically and not just this thread. I can't edit my comment now, and I apologize for the confusion. I would encourage you to review the YC comment guidelines [0] and engage in more thoughtful discussion in the future.
>Python3 does implicit float-to-int conversions differently
I don't think so? Might be floating point division.
> In Python 3.0, 5 / 2 will return 2.5 and 5 // 2 will return 2. The former is floating point division, and the latter is floor division, sometimes also called integer
division.
I think an automated tool like `2to3` is probably a better bet for doing that kind of conversion. It should be a mostly automatic process for most scripts.
I believe 2to3 hasn't been the entirely recommended way for a while - modernize
(dual compatibility via the six module) and futurize (more magic to make as many python3 idioms work in 2 as possible) let you move towards python 3 without breaking compatibility from python 2, and give you ways of working in both versions at once - 2to3 isn't even mentioned on https://docs.python.org/3/howto/pyporting.html.
That said, syntax conversion is usually only a small part of the problem - unicode handling can be more problematic, especially if you've never spent much time thinking about or working with encoding/decoding.
It's definitely _much_ easier if you have a good grasp of both versions of the language and the changes that were made, and it isn't helped by the pervasive myth that "It's just brackets on print statements".
We recently converted a ~900kloc open source codebase to dual python 2/3 with many scattered interested parties and approached it in roughly a phased way - Using CI to prevent backsliding of syntax converted portions of the codebase and getting the tests running (and failing) in python3 early were crucial. (To add to this - a very _early_ step was getting the future imports in place - every file had future absolute_import, division imported at a very early stage. For division, you can also start python2 with -Qnew and it'll force all division to "modern" style).
Six is just trading dependency on py2 for dependency on a library that wouldn't otherwise be needed if you just made a code base forward compatible. It is more sensible to gradually work on getting 2to3 to do a clean conversion and then doing a final conversion when the py2 code is ready to be EOLed.
I'm reading "getting 2to3 to do a clean conversion" as having a python2 codebase but then running 2to3 to build a "python3" version before deployment. If I recall, that was touted as the approach in the early days of python3 and conversions. I haven't seen it used or talked about much, recently, although I'm aware that later tools like pasteurize attempt to do the reverse (but haven't tried them).
I worry that doing this would basically cause much of the pain of maintaining separate codebases - inevitable bugs that affect one or the other, with the problem of not being able to directly write in one of them, and getting to where you have to check both for new code. Plus, it doesn't cover everything (or can't - e.g. ambiguous cases were removed), and has a few places where it makes mistakes (have seen but cannot recall) - so you end up having to write cross-compatible code for some things yourself anyway.
I feel that extra dependencies, especially commonly known, understood and reasonably well documented ones, is an extremely small price to pay.
We decided early on for our large project that developing separately and having a version shear where we "switched" to python3 would probably be chaos. The slow and gentle way, you can make changes and adjust the other developers at a more reasonable pace, the point at which you drop python2 is merely a decision with an empty changelog (at which point you can start stripping stuff out).
You are arguing for a hard cutover, which is how we got into this mess in the first place. By using a forward or backward shim, one can run on both. 2to3 is a CASE tool, it requires manual fixup, so it cannot be used to maintain a dual version codebase.
I read their argument as, getting the codebase to the point at which running 2to3 completes the transformation. I personally remain unconvinced that that code wouldn't require the equivalent of a compatibility library anyway.
This is a hard cut over from Python2 to 3. This presupposes that both upstream and downstream deps are on Python3. On which magical day does everyone decide "to have their port finished" ?
If it is the same work to to support both 2 and 3 as jettison support for 2 in favor of three. Then supporting both so the system can actually upgrade is far preferable. Applications and libraries not used in isolation. To jettison support while the rest of the dependency graph is flux is analogous to gimbal lock, you can't just get here from there. Without `six` the whole Python ecosystem would have bifurcated and Python2 would still be the dominant Python.
2to3 does NOT handle the change in division behavior. There is no automated way to do so because python2 only does floor division sometimes and it's decided at runtime based on the arguments being ints or not.
That works for writing new code in Python 2.7. It would also have worked for going the recommended route of gradually migrating toward Python 3 compatibility while remaining on Python 2, _provided_ you're working with an organized project that's amenable to converting on a file-by-file basis.
I can see where it falls flat on its face when you're trying to migrate a monolithic 6000-line script. But, IMO, that example falls flat on its face when used as a rhetorical device. If you've got a monolithic 6000-line script with (presumably) no test coverage, you've made your own bed, and I don't think that the inherent difficulties of maintaining something like that should be blamed exclusively on the 2->3 transition.
In my experience in the business world, it is far too common to find a company’s critical infrastructure running on stuff more messy than this particular 6000-line monstrosity (which, for the record, I am not the original author of, and, yes, I have added a few test cases for code I have added to it like my own RNG).
I once was getting paid double my previous salary to maintain Visual Basic code which makes this 6000-line script look like well structured fully tested object oriented code. When my boss came in to the room and said the project was cancelled, I let out a sigh of relief that I would never have to look at that code again.
In terms of Python2, it will be maintained until at least 2024 because it’s part of CentOS/RHEL 7. I think it will be maintained until 2029 because I saw that package in CentOS 8, but I do not recall if it’s a core package with a guaranteed 2029 EOL.
Python 2 is not a core package in rhel8. You can get it via an extra collection that will be supported for the life of rhel7.
So using rhel8 won't be a solution. Rhel developers are probably eager to drop Python 2 support as soon as possible. Which is when rhel7 goes EOL. Of course, it wouldn't particularly surprise me if customer pressure makes them support it for longer.
Many people aren't even bothered to try. The company I work for still writes new stuff in Python _2.7.3_ - not even a recent release of 2.7. Some of their stuff will be difficult to convert, but a lot of it really is trivial. They just don't care to put any effort into it until things start to break down.
I can understand why people get frustrated by situations like this and are tempted to call out programs still running Python 2 and those who advocate for keeping it.
I'm a huge fan of some of the features introduced by Python 3, so I switched and am not interested in going back. However, I think there is room for other interpreters to step in and continue supporting Python 2 even if the PSF wont.
Maybe they tried, and stuff broke in weird and unexpected ways, so they gave up. For example, the first time we upgraded from that to (I think) .8, lots of strange errors started coming in from our webapp. Apparently kombu, one of the dependencies of celery, was importing an underscore-function from the standard library, which no longer existed in the new version.
There are occasional cases where, with some Python libraries, I need to use a “hidden” _function to do my job. The one time I had to do that, I made sure to contact the authors to let them know what I did and why:
If I were in that situation, I would make a reasonable effort to keep scripts as polyglots: Make sure they both run in Python 2 and in Python 3. In my experience, doing so is pretty easy to do when writing new code from scratch with Python 3 compatibility in mind.
The “no one is maintaining Python 2” issue is an example of open source economics in action: Everyone wants Python2 to be maintained, but no one is stepping up to plate to pay for it to be maintained, so it’s not getting maintained.
Nobody is saying otherwise, but maintaining EOL software is also hard. The Debian project does have any able developers volunteering to do it for Python 2 in Debian's next release. That's all that there is to it.
It's so vitally important to write tests. Years ago I started doing full tests suites, even for my open source, work, once it got mature enough I felt comfortable with the patterns I established. Then I'd add tests before developing new features.
Unit tests help make upgrading so much easier. I was once on a huge Scala codebase that needed to go up from Spray.io to Akka-HTTP. Tests help save me on tons of potential bugs in migrating over. The few things that got through, I could write more tests for.
On another open source project I maintain, my functional tests helped me migrate from Scala 2.11 to 2.12 along with all my dependencies with relatively low friction.
If you're porting your Python2 to Python3 stuff now, it's never too late to start writing some tests. Write them over the weird division stuff you've found. Make sure they run in Py2 and fail in Py3 so you can isolate what it is. Even a couple of small tests in a large code base that only cover stuff you're finding now is better than no tests at all.
There is no sure-er mark of a Python novice than the "2 to 3 conversion is trivial" myth. Python has a long history, there are lots of versions of 2, there are complicated data processing changes from all versions of 2 to all versions of 3. Not every conversion is hard, but they can be an enormous amount of work.
Python doesn't implicitly conver floats to in ints. As others have commented, there have been changes to how division is handled in that int / int may result in a float. But, to the beat of my knowledge, there's no implicit float -> int conversion. Maybe. Some libs convert floats to into, but its not the norm.
That actually isn't a function call, despite the appearance. Python 3 raise is still a statement, not a function (unlike print). So this
raise(ValueError, "some text")
does not do what you think; the proper way in Python 3 to spell what you probably expect is
raise ValueError("some text")
Here is a transcript of a shell session showing the difference:
Python 3.5.2 (default, Oct 8 2019, 13:06:37)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> raise ValueError("some text")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: some text
>>> raise(ValueError, "some text")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: exceptions must derive from BaseException
Note that the exception you actually get the second time isn't even ValueError, it's TypeError; it's being raised because "some text" is a string, not an instance of an exception class derived from BaseException.
In other words, you should not be treating raise in Python 3 the way print is treated. The 2to3 tool treats it properly.
Well, yes, porting large projects to Py3 is not trivial.
That's why they had, what, around 5 years to do it since the 2.7 EOL was extended to 2020?
I think most of the problems you encountered would be solved by the six library, btw.
But the most annoying bugs were those were some Python 2 quirk was "just working" (like comparing strings to integers in Python 2) or the conversion of ints and floats and then it correctly broke on Python 3.
Using the six library is one way to make a large project being written from the ground up run in both Python2 and Python3.
For this particular project, I would had needed to get the code to run in both Python3 and an ancient version of Python2 made back when boost.python was still a thing, so six would be out of the question.
I ported a website written in 2009 from Python 2 to Python 3, which was around 20,000 lines of code. It wasn't "trivial", but it took me (a single developer) a little less than a week. Most of the difficulty wasn't code changes, but upgrading dependencies.
Recently I had to port some code from the legacy version of an internal DSL to our newer version. I just wrote a primitive parser that would understand 98% percent of the code (in a safe way) and lowered that into the new DSL. Worked like a charm, and I didn't have to rewrite ~2000000 lines (a colleague tried pattern matching, but that was unusable for complex expressions).
I wonder if/why there is no such ~~thing~~ [edit] effort for python2/3?
If you're only accepting programs that P2 accepts, you can even get away with being lazy and "understanding" more than P2.
This isn't from experience (I have yet to do a conversion myself), but one thing I've heard is that the str/unicode change revealed that a lot of codebases didn't understand the difference in py2, were using the wrong one at API boundaries, and required a lot of manual work to figure out what it should be for each one so py3 could handle it.
You hopefully mean after spending 1 minute running py2to3 right? I mean I get that some people feel that it’s tedious or difficult to convert if they spend ages manually changing thousands of lines of code in a completely automatable fashion. Which is why it’s nice that you don’t have to actually do that. You just run py2to3 and the focus on differences that matter.
The primary choices for distribution are to either demand maintainers that they updates their scripts, or to take up maintaining the python 2.7 code base themselves, a code base which is written in C and mostly before 2000.
Is your code in Debian right now, or was your comment a general question in regard to the status of python 2.7?
Out of curiosity, how is this issue handled for languages like Algol or Fortran? The gfortran codebase was also written significantly in C and mostly before 2000.
The Python 2 -> 3 transition has been unusually bad.
In most programming languages, a change to the language is practically always backwards-compatible. The update may add new capabilities, but the old capabilities and syntax remain in place. If the language changes semantics, it's usually in bizarre edge cases of the language. If a basic data type was a mistake, for example, a new data type is added & people are encouraged to use it instead... but the old one keeps working. If an existing data type or method should be renamed, the new name is provided & the old name keeps working for a long time so that people can incrementally update.
In short, the normal expectation in programming language updates is that existing code will keep working, and that updates can be done slowly and incrementally. That's historically been true for Python as well; updates typically didn't break much. In the few cases where there were significant backwards-incompatible changes, languages typically give 10+ years of warning.
The Python 2 -> 3 transition has instead been an example in how to do things badly. Pretty much no Python 2-only code will work without change; you have to change print to print(), division's semantics have changed, string semantics have changed, common names have changed (xrange to range), and so on. The original automated transition tool originally envisioned could never work reliably, because Python's dynamic nature ensures that there's not enough information to work without help. So the demands on people to make changes were unusually large for a language update.
A related problem was that there was never an implementation that supported Python2 and Python3 at the same time. That meant that if you depended on 100 Python2 libraries, you had to wait for all of them to transition before you could transition. If 99.99% of the libraries you depend on are available in Python3, you must use Python2. You can easily create C or Fortran programs that call on code written long ago unchanged; that is not true for Python3.
So in the Python3 transition all code had to be modified simultaneously across the ecosystem, and tools couldn't do it well.
Eventually the Python3 folks made enough changes to Python3 so that it started becoming practical to write code that worked on both Python2 and Python3. That should have happened from the beginning.
The only language I can think of that handled a transition worse is Visual Basic. When "Visual Basic for .NET" came out, it was so grossly incompatible with the previous Visual Basic that it was called "Visual Fred". In numbers I saw long ago, about 1/3 stayed with the old unsupported Visual Basic, 1/3 changed to a completely different language (that was less likely to abuse them), and only 1/3 of the previous Visual Basic users transitioned to Visual Basic for .NET.
I think that Python3, all by itself, is a fine language. The problem in this case is that the developers of Python made it unnecessarily hard to transition to Python3. That was both unfortunate and unnecessary.
Perl 6 botched the transition so badly they're not even going to call themselves Perl anymore, and Perl has basically fallen off the map for new projects. The Python 2->3 move was rough but the language seems to be thriving despite that.
Perl 6 is really a different language from Perl 5, so I think the new name is what should have happened from the beginning.
But I basically agree with you that using the name "Perl 6" for Raku led to most people abandoning Perl 5. Hopefully people will think more carefully about transition in the future. But too many technologists ignore history, so I don't have high hopes.
Write some tests around the problematic functionality before attempting the final port, it should help in this situation. Should be broken into multiple files as well.
The code base in question is not one I wrote; it is one I decided to maintain years after the original author lost interest in it. [1]
This is not too different from what I have seen in multiple jobs, where the new hire doesn’t get to write all the code they use from scratch, but instead has to maintain someone else’s mess. Often times, management and market conditions demand more from programmers than they have time to properly implement with test cases, well structured code, good comments, etc.
Technical debt is a big problem in code used by businesses, and the technical debt this particular 6000-line script has is a lot less than some stuff I have seen and have been paid to maintain.
The fact of the matter is that most open source code out there is quite a bit cleaner and maintainable than the code which is behind firewalls, running important businesses.
[1] Edit: To the original author’s credit, this particular code base is an incredibly popular and successful map generator for Civilization 4, with some 40 thousand downloads; it got messy because the author was more interested in adding new features requested by users than in keeping the code base clean. It is a single script because that made it a lot easier for users to install. And, newer versions of the same generator are written in Lua and not Python because that is what newer versions of Civilization use for scripting.
A) Not your job to police this site, B) just because your feelings were somehow hurt about code you didn't even write, doesn't actually mean I violated commenting guidelines, and C) maybe you ought to read the guidelines yourself, considering one of them specifically says not to comment when you think a post is inappropriate.
You want a "good critical comment"? Here:
* Try breaking your code up into more than one file. There are other/better ways of releasing Python besides "one file".
* Lay off the code commenting, it's vastly overdone here.
* Use version control tools to version your code, not comments.
* Python convention is to use underscore_case, not camelCase.
* Variable/function/class names should be descriptive, not random letters/numbers/incomprehensible abbreviations.
* Delete commented out code -- it's stored in version control, you're not going to lose it.
* The "initialize" func is a terrible concept and represents a fundamental misunderstanding of how Python classes work. This whole misadventure is a cautionary tale in, "Why a light architecture pass matters even in 'throwaway' scripts."
* Add tests.
* The massive if/else chains can be replaced in a number of different, more readable and therefore superior, ways.
That's some of what I saw in 2 minutes of review. This code is in very poor condition, and if you think the state of this code is an indictment on how hard it is to convert python from 2 to 3, you're more than just a little mistaken. You would need to pay me substantial amounts of money to spend more of my time going deeper into what's wrong with this code, as there is a lot and it is not generally enjoyable to work with code so thoroughly broken as this.
The script (somehow) works. That matters. However, this is not a good representation of why migrating from Python2 to Python3 is hard; this code is in no shape to be used to represent anything other than how not to write Python.
Could have been said more kindly, but the point is accurate. This reads like C code that happens to be written in Python. There is a lot of fiddly bit-twiddling operations.
The main issue — besides not using 2to3 — is that / in Python3 is interpreted as floating point division, whereas it used to be integer division if operands were integers. Replacing / by // in the relevant code blocks might help? Or getting another implementation of the RNG?
I'm running Calibre on Python 3 (via an Arch package provided by one of the people working on the port). It and mercurial/tortoisehg and the only remaining reasons Python 2 is on my system. All of these things will be on Python 3 soon enough.
Why doesn't Debian just ship the unreleased Python 3 port of Calibre? It's not like Debian restricts themselves to only formal upstream releases, they have never hesitated to backport patches to unreleased versions. If lacking Python 3 support is such a massive bug, wouldn't using patches that have not been officially released, that fix the bug, be acceptable? Debian testing isn't Debian stable, there is not a reasonable expectation that the code shipped is perfect.
Mercurial also works on Python 3, though it's only a beta. I could understand not shipping it because version control bugs can be serious, but hey, someone's gotta test the beta. I've used it and it not encountered any bugs, but went back to py2 because tortoisehg won't work with Python 3 mercurial yet (and then bitbucket dropped mercurial support so now I'm moving everything to git, grumble grumble, probably will sadly stop using mercurial altogether within the next year).
It's definitely possible for Debian to package unofficial releases out of the upstream VCS. You can sometimes tell that's what happened because it'll have something like ~git20191005.f8ab33 in the Debian package's version number.
It's basically up to the maintainer either way. The freeze for the next Debian release is at least months away (no date set yet), so it's not too urgent, and waiting for upstream is still reasonable at this point.
Here's a recent post from the Debian maintainer: https://www.preining.info/blog/2019/10/state-of-calibre-in-d.... The tl;dr is that most/all of the work is done but he's holding off updating unstable to Python 3 because he doesn't want to break external plugins. He also mentions that he can't update to 4.0 or higher because Debian still hasn't updated Qt to 5.12. That seems slightly inaccurate as it's in experimental (https://packages.debian.org/experimental/qt5-default); I think the maintainer could do an upload of a Calibre 4.x using Qt 5.12 & Python 3 to experimental. But experimental is not in common use regardless.
One can sense the frustration dripping from the thread. This feels like pretty aggressive communication ("frankly, I don't care", etc.)... it could've been a misunderstanding about what "removal from testing" means... In particular since the packages remain available in unstable, the argument raised about users paying the price isn't really valid.
I think you're overreading it. Wanting to remove python2 dependancy now with the next Debian release as target (probably 2 year from now) seem reasonable to me. The aggressivity you perceive is to me just being firm. Refuse and push back any exceptions. The work has to be done, end of story.
Personnally, I approve, otherwise it'll spiral down in nitpicking and special cases. Would not be the first time with Debian.
> If the package has still many users (popcon >= 300), or is needed to build another package which cannot be removed, document that by adding the "py2keep" user tag
So they are not even that firm. Overall, it seems very well handled.
I agree it's the right call, my remark is only about the communication style (e.g. "frankly, I don't care") and how it comes off to an outsider to the community.
Can someone link the list of packages that needed to be ported?
I get the impression from this thread that newly ported packages won't be included in the next testing release, but hey, there's always unstable.
Has anybody interest in forking the python project and continuing to maintain 2.7 until forever? And by "maintain" I mean no new features, just bug fixes and performance improvements. Maybe if we find enough volunteers we can pull that off.
There is still much useful and valuable legacy software out there that many people use which is getting abandoned for no other reason than some people (like the authors of calibre) are not very fond of the direction python is going. Others can't upgrade to 3.x because you have standard libraries especially in the academic sector that people literally put their whole phd into and that can not "easly" be upgraded just like that because of high complexity, lack of staff and time.
> Maintenance-mode fork of the officially-deprecated Python 2 interpreter and standard library. Bugs fixed, security holes patched, and maybe some performance improvements, but no changes to the syntax or semantics (for that see the Tauthon project, unaffiliated.)
- - -
Sometime between now and the end of the year I'm going to fork Py2 and set up build jobs to run the test suites. (Maybe later today, it's not hard, it just takes a few spare hours.)
Going forward I plan to work on things like improvements to the docs, curating additional packages and libraries (additional to the standard lib), and language tooling (I'd like to keep things like Nuitka, Snakefood, etc. working.) I also want to run fuzzers and other code quality tools over it all.
Before it's too late, I'd like to humbly suggest a name change. Even as a Blackadder fan, the name sounds terrible at a glance, especially for such a (potentially) high profile project. Naming isn't everything, of course, but it has more significance than we consciously realize. It can subconsciously impact decisions in favour of or against the project, as irrational as that is.
I appreciate the sentiment but it's deliberate. I don't want to become high profile. The Python folks have made it pretty clear that they don't want continuity of Python 2 to "dilute the brand" of Python-as-Python-3, so I wanted to wave a big "Don't take this too seriously!" flag over this fork. (Also for the BBC, eh? It would be a little on the nose to call it "Blackadder". Or "Baldrick".)
> No special pleading. No whining. No excuses.
> No buts and no exceptions. No "middle ground".
This sounds strong to me. Not to be provocative since I am not familiar with the mailing list, I would like to ask is it common to communicate this way in this Debian community? Is it effective to push things moving forward?
I think it's perfectly reasonable. Package maintainers are stomping their feet and huffing about making a change that is in everyone's best interest. Why shouldn't the distribution set strong guidelines and enforce them?
Too much is often sacrificed at the altar of backward compatibility. Move forward and leave the laggards behind.
I will never forget how Debian hijacked the real ffmpeg with their hobbled version until some point later [1]. Pretty disgusting. While I unfortunately use this distro fairly solely, I will move to other options at some point. Debian may be free but often it proves to be too expensive.
How did a project like Babel not evolve to compile python2 into (horrific code) that runs with python2 compatible semantics on the python3 runtime ...? Is that not possible for some reason?
But the thing with Babel is that it only adds features to JavaScript, it doesn't translate between two disjoint languages. I'd assume that has an effect on its viability as a project.
Python 2 and 3 are not compatible via syntax translation in the same way that many new JS features can be transpiled to old JS. In Python 3, strings work differently in subtle ways that force you to be more careful, and a source code translator doesn't have any reasonable way to know whether a variable is a string anyway. You also see this sort of thing in JS: Babel never tried to implement Proxy because it really needs runtime support and can't be implemented by syntax translation.
It’s not possible in general because you can only do a syntactic translation of python. To preserve the meaning of the program you would have to statically know something about the data types used. This is one reason why Dropbox, Facebook, and google have developed their own python type checkers.
gclient (necessary for building chromium, among other things) got an "early version" working with Python3 just a few weeks ago. Apparently this stuff is still being ported.
> Originally meaning pictograph, the word emoji comes from Japanese e (絵, "picture") + moji (文字, "character"); the resemblance to the English words emotion and emoticon is purely coincidental.https://en.wikipedia.org/wiki/Emoji
A package with no active developer gets removed unless someone takes up the role of maintaining it.
Since python software foundation will stop developing it there is no upstream support after 2020. In theory a new upstream for python 2 could pop up, and there is speculation that Redhat might do it, but Debian do not want to become the new upstream for python 2. If they included python 2 in next release they would have to support it for as long as next release get support. What will happen instead is that they will keep supporting python 2 in older releases but when those support periods are over then its over.
If they're going to do it, for the sake of the community they should announce that soon. Even if they don't plan on taking up the responsibility until the middle or end of 2020 it might be something that distros and companies would consider when making their plans.
Is there any evidence that people are regularly checking whether software has been abandoned, and those packages removed? It seems more plausible to me that this was an excuse to replace the default 'python' with Python 3.
The normal thing in Debian is for a maintainer who wants to step down to orphan the package so that other maintainers get the chance to pick it up.
It seems perfectly reasonable for the Debian Python team to decide they no longer want Python 2 getting in their way. But I think they should have done more to check whether someone else wants to take over.
There's nothing terribly wrong with packaging mature software that doesn't have an active upstream. Realistically nobody cares about any remaining bugs in Python 2 being fixed; the work involved would basically be making sure it still compiles with newer versions of GCC.
Yes, I think whoever ends up taking over Python 2 maintenance would be well advised to stub out the https support immediately, and maybe a bit later do the same for all support for network requests.
In particular, you don't want to have to worry about building against openssl.
I think the vast majority of cases where people have a good reason to carry on using Python 2 won't need that.
Do I understand correctly that you're asking why a distribution would remove a package at all, if it currently works just fine?
The answer is mainly security issues and compatibility bugs. If a package is abandoned, it is typically removed. Someone needs to write updates both to fix newly found security problems and general issues with other new packages. Perhaps we'll switch from the init system systemd to systemf, and some package depended on that. Then it won't work in a new version anymore. Like someone else commented, it's nothing but work.
I'm not sure why it is removed before any such issues arise or become known. My best guess is that it should prevent the removal from becoming super urgent once a vulnerability becomes known. If it is still used by anyone, they should either become the maintainer or (in this case the more realistic option than becoming a maintainer for python2) update the code to python3.
Not sure why you're being downvoted though, it seems like a legitimate question: "it works, why bother even removing it?"
The reason why it's removed before any such issues arise or become known is because Debian can't remove it afterwards - if they have included it in a release, they have committed to ensure that it's supported until the EOL of that release; and if noone upstream is supporting it, then it would fall upon them to provide these fixes.
You can't simply remove packages from a release after it's shipped; either your release includes that package completely (including a commit to support them for the whole lifetime of your release) or not at all.
Yes, I was under the impression that distros would keep providing a lot of old packages that no one was looking at much anymore, until they actually started to cause problems.
Including it in the distro means people will use it. Since it's unmaintained, that basically assures security vulnerabilities. Avoiding this situation is one of the main reasons people use distros.
Python 2 will not be maintained upstream beyond January 2020[1]. Therefore Debian will not include Python 2 in its next release since Python 2 will be already be EOL on the day of that next Debian release.
Otherwise, who will fix the Debian Python packages when the next CVE comes out?
This isn't strictly true. The final cPython release (2.7.18) is slated for April next year [0], and PyPy has no such plans to drop support [1].
I don't honestly think Python 2 is going anywhere for a while yet (judging from some of the commercial codebases I've seen at least). I sometimes wonder if this mess would have dragged on quite so long if there'd been better features in Python 3.
> The final cPython release (2.7.18) is slated for April next year [0]...
Fair enough, but there is still no commitment from upstream for most of the supported release of the next Debian release, so that is moot.
> ...PyPy has no such plans to drop support
This isn't relevant to the CPython package dependencies which are the issue at matter for Debian on this matter. They haven't explicitly said CPython because it's implicit the way that Debian's Python packaging is structured.
Python 2 is obsolete. It won't have upstream support anymore when bullseye is released, which means that Debian has to do all the maintenance by themselves. Even if there are no updates, that still means fixing security vulnerabilities and keeping it compatible with changes in the environment (i.e. new packaging policy, new compilers, new hardening features, etc.). There's nothing to gain but work by keeping it.
Javascript: Hey you kids: quit doing cartwheels on the heads of 20 friends who are also all doing cartwheels twenty levels deep. No cartwheels more than 3 levels deep or someone's going to get hurt! You've been told over and over now.
Python: Hey you kids: sit in this cubicle for the entire summer and do 20,000 long division problems just to keep the school's lights on. And don't get too encouraged because there's no way in hell we can finish all the long division before the beginning of the school year. Some of your classrooms will be dark. p.s. our principal got burned out and quit.
I never actually read the whole story behind this. To me, it looks like the original author of leftpad was in the right - radically changing a package can break someone's build. Maybe he went a bit too far with his next action, but if NPM don't care about its end-users enough to ensure packages remain relatively stable (within the limited control they have), why should an individual contributor?
I doubt anyone is checking for potential security problems in the Algol 68 and Fortran 77 implementations that Debian ships, and I don't think the people using those packages are particularly inconvenienced by that.
It seems a shame that the social pressure to persuade people to port their code to Python 3 means that Debian is going to have weaker support for 10-year-old Python than 40-year-old Fortran.
In particular, there are ongoing efforts to try to make it the normal thing for scientists to make the programs they ran on their data available so that their results can be reproduced; aggressively dropping older programming language implementations rather gets in the way of that.
Algol 68 and Fortran 77 may have stale (but maintained) compilers or interpreters in the package repository. Starting very soon - Python 2 will have an entire set of unmaintained runtime and libraries in the package repository. You know - actual, offically, unmaintained software! Unmaintained software that other packages, including Calibre in this example, further build on. Of course they're throwing this out.
If you're concerned about language support there are multiple solutions like:
* Port your stuff to Python 3 and enjoy first class support
* Running Debian Buster
* Running CentOS 8, which will probably profit from RHEL 8's Python 2 support-with-an-asterisk until 2029 [1].
* Installing it manually on your system
* Installing it manually in a container
This is how it feels to run legacy software - it's bloody inconvenient. But that's not a problem Debian needs to solve for anyone.
[1]: https://developers.redhat.com/blog/2018/11/14/python-in-rhel...
They can't stop someone from releasing something that runs just like Python 2 and has a different name.
As long time Python user, my view of the PSF is that they could mature a little bit. Their ambivalent hostility towards PyPy and Jython, the lack of portable standard library, and now with how they are continuing to mismanage the 2to3 transition. None of this is how adults should behave.
I wish PSF would pull the plug from python 2 much sooner so we wouldn't have to deal with whiners for 10 f...ing years. The problem with python 3 wasn't the incompatibility, the problem was keeping python 2 alive for so long. Other languages usually gave only 1 year and everyone moved, because they didn't want to stay behind.
Python 3 came out in December 2008. But for a lot of packages, it wasn't realistic to migrate until it was possible to support both Pythons from the same codebase. And that wasn't realistic until 2.7/3.3, which was 2012.
At that point the rising tide started and packages with no dependencies began doing dual support. But it was many years before there was enough coverage that application level packages like django could consider migrating.
So yes, the cutoff could have been somewhat sooner, but if the PSF had attempted to force a transition in 2009, it would have been a death knell for Python. Not even numpy was available for Python 3 until 2011!
https://www.python.org/dev/peps/pep-3000/#compatibility-and-...
Not hard to see why this didn't get a lot of traction.
Python barely survived 2to3, it should serve as a warning to every other system that thinks it can do a stop the world upgrade.
My C89 code, C++98 code and Fortran 77 code all still compile on a modern Ubuntu, and none of these are expected to break any time soon. The compilers just support multiple versions.
People aren't wrong to make different tradeoffs than you. Especially given that strict backwards compatibility is impossible (especially in a language like c/++, people will rely on things that aren't intentionally public, because they can introspect your bytes).
Compiler changes break code all the time, it's just almost always semantic and not syntactic. I find this amusing, since the syntactic changes are usually easier to detect and fix.
And python2 still works, it just won't get new features.
[0]: https://youtu.be/LJh5QCV4wDg
Probably more importantly, it wont get security fixes.
Yeah, Python 2 will still work. Ita not like there's a kill switch or anything. It just wont receive any official or supported updates. Effectively, Py2 will be abandonware.
I wish the wallrus operator was available as a future import, bit it's not.
Also, Python never had printf. It had a print statement; print became a function in Python 3. Yes, its a syntax change, but its for the better. Having print as a first class statement expands its utility and versatility.
You need very good test coverage.
Tcl 8.3 to 8.4 when they added unicode support as well.
I believe PHP did similar thing.
It was a major version release; Ruby, through 1.8, used an version numbering scheme similar to the old Linux version numbering scheme; odd second digits were development major versions, even second digits were stable major versions.
1.9 was a slight break because it was a major release but, IIRC, seen as stable.
You create a package called "constrictor2.7", and set it up to provide /usr/bin/python using update-alternatives. Then #!/usr/bin/python and #!/usr/bin/env python both work.
See eg https://lists.debian.org/debian-python/2018/05/msg00063.html from the Debian Python maintainer.
(That's a year and a half old, but I haven't seen a message go past suggesting they've changed their minds.)
If you already have infrastructure that does similar things, it's probably not much of an issue, but I can see missing or misdirected symlinks being a common problem. Maintaining the symlink in the distribution (Debian or whatever) reduces the chance of errors.
(Cynical? Moi?)
* Everybody knows they should have moved to Python 3 for over three major versions of Debian stable.
* Debian is not committed to keep legacy software running in perpituity in new versions.
* For Debian Buster they have already committed to supporting Python 2 until at least 2022 - and probably 2024 for LTS. They're already putting in the commitment!
* The next version of Debian stable will probably be released mid 2021 - long after Python 2's EOL of Jan 1.
* RHEL's Python 2 support amounts to life support at best.
* Keeping around legacy software takes precious volunteer time - which is all Debian has.
This is the entire point of a stable distro. To pick some software, stabilize it a specific api, and keep it supported. Upstream need-not support that specific version of a release, for reference, see the kernel.
I don't care what Debian does at this point, too old for practical use.
> should support software they aren't interested in.
You mean like pestering all the other package maintainers to support python3 when they're not interested in doing that? That goes both ways. Nobody wanted python3, expect PSF. This is the continued backlash.
And nothing is stopping you from installing Python 2. All scientists I know rely on anaconda to manage their Python environment, and couldn't care less what the system python does. I really _don't_ want to rely on system python for reproducing my old results.
For packages, I can understand it, sure. So include a requirements.txt, and just spin up a 3.8 virtualenv to reproduce, no?
But if this is important to you then depending on the OS to provide your python seems like a bad idea, independent of the python2/python3 issues. Wouldn't it be much better to use something like conda / the anaconda distribution, so that you can specify precise versions of each library for your environment?
Having said that, I don't know how anaconda is going to handle the EOL of python2. So I guess the question is: if someone wants to be able to reliably reproduce a python2 environment with specific set of libraries, what is best way to do that?
What's to handle? Old versions of python2 packages are still in the online repos. The EOL means that there won't be any new releases to python. All of the old releases will still work though.
As others have stated, those are old implementations, but I'm sure Debain will still accept security patches for them. Python2 is about to lose major support. You cannot just let rotting dependencies stick around in your tree. You're not going to have a good time.
If people really want that stuff, they can pack it into a Docker container, or Snap/Flatpack/whatever garbage for old GUI applications.
However, for an over 6000-line script written in the mid-2000s, before Python3 even existed, porting to Python3 is non trivial. Trust me, I tried.
First, the syntax sugar stuff was easy enough to fix: Do a simple search and replace so that all cases of stuff like
Become I had to do something similar with the way this script calls raise(). Not to mention adding a hack so Python2 uses “xrange” while Python3 uses “range”. After doing all of this, the code still does not run in Python3. As far as I can tell, the issue is that Python3 does implicit float-to-int conversions differently than Python2; the code will not raise an error in Python3, but it will silently run differently, until the state is different enough that the Python3 version raises an error while the Python2 version works as expected.At this point, I just threw my hands in the air and gave up.
For reference, the offending code is here:
https://github.com/samboy/misc-civ4-mapscripts/blob/master/T...
[0]: https://docs.python.org/2/library/2to3.html
[0]: https://news.ycombinator.com/newsguidelines.html
I don't think so? Might be floating point division.
> In Python 3.0, 5 / 2 will return 2.5 and 5 // 2 will return 2. The former is floating point division, and the latter is floor division, sometimes also called integer division.
https://stackoverflow.com/questions/183853/what-is-the-diffe...
I think an automated tool like `2to3` is probably a better bet for doing that kind of conversion. It should be a mostly automatic process for most scripts.
That said, syntax conversion is usually only a small part of the problem - unicode handling can be more problematic, especially if you've never spent much time thinking about or working with encoding/decoding.
It's definitely _much_ easier if you have a good grasp of both versions of the language and the changes that were made, and it isn't helped by the pervasive myth that "It's just brackets on print statements".
We recently converted a ~900kloc open source codebase to dual python 2/3 with many scattered interested parties and approached it in roughly a phased way - Using CI to prevent backsliding of syntax converted portions of the codebase and getting the tests running (and failing) in python3 early were crucial. (To add to this - a very _early_ step was getting the future imports in place - every file had future absolute_import, division imported at a very early stage. For division, you can also start python2 with -Qnew and it'll force all division to "modern" style).
I worry that doing this would basically cause much of the pain of maintaining separate codebases - inevitable bugs that affect one or the other, with the problem of not being able to directly write in one of them, and getting to where you have to check both for new code. Plus, it doesn't cover everything (or can't - e.g. ambiguous cases were removed), and has a few places where it makes mistakes (have seen but cannot recall) - so you end up having to write cross-compatible code for some things yourself anyway.
I feel that extra dependencies, especially commonly known, understood and reasonably well documented ones, is an extremely small price to pay.
We decided early on for our large project that developing separately and having a version shear where we "switched" to python3 would probably be chaos. The slow and gentle way, you can make changes and adjust the other developers at a more reasonable pace, the point at which you drop python2 is merely a decision with an empty changelog (at which point you can start stripping stuff out).
And, do hard cutovers ever work well?
If it is the same work to to support both 2 and 3 as jettison support for 2 in favor of three. Then supporting both so the system can actually upgrade is far preferable. Applications and libraries not used in isolation. To jettison support while the rest of the dependency graph is flux is analogous to gimbal lock, you can't just get here from there. Without `six` the whole Python ecosystem would have bifurcated and Python2 would still be the dominant Python.
I can see where it falls flat on its face when you're trying to migrate a monolithic 6000-line script. But, IMO, that example falls flat on its face when used as a rhetorical device. If you've got a monolithic 6000-line script with (presumably) no test coverage, you've made your own bed, and I don't think that the inherent difficulties of maintaining something like that should be blamed exclusively on the 2->3 transition.
I once was getting paid double my previous salary to maintain Visual Basic code which makes this 6000-line script look like well structured fully tested object oriented code. When my boss came in to the room and said the project was cancelled, I let out a sigh of relief that I would never have to look at that code again.
In terms of Python2, it will be maintained until at least 2024 because it’s part of CentOS/RHEL 7. I think it will be maintained until 2029 because I saw that package in CentOS 8, but I do not recall if it’s a core package with a guaranteed 2029 EOL.
So using rhel8 won't be a solution. Rhel developers are probably eager to drop Python 2 support as soon as possible. Which is when rhel7 goes EOL. Of course, it wouldn't particularly surprise me if customer pressure makes them support it for longer.
Many people aren't even bothered to try. The company I work for still writes new stuff in Python _2.7.3_ - not even a recent release of 2.7. Some of their stuff will be difficult to convert, but a lot of it really is trivial. They just don't care to put any effort into it until things start to break down.
I can understand why people get frustrated by situations like this and are tempted to call out programs still running Python 2 and those who advocate for keeping it.
I'm a huge fan of some of the features introduced by Python 3, so I switched and am not interested in going back. However, I think there is room for other interpreters to step in and continue supporting Python 2 even if the PSF wont.
Familiar number.
Maybe they tried, and stuff broke in weird and unexpected ways, so they gave up. For example, the first time we upgraded from that to (I think) .8, lots of strange errors started coming in from our webapp. Apparently kombu, one of the dependencies of celery, was importing an underscore-function from the standard library, which no longer existed in the new version.
However, I'm painfully aware that it does not apply to most of our apps and it does not justify using an outdated interpreter for writing new apps.
https://github.com/cobrateam/splinter/issues/591
The “no one is maintaining Python 2” issue is an example of open source economics in action: Everyone wants Python2 to be maintained, but no one is stepping up to plate to pay for it to be maintained, so it’s not getting maintained.
Unit tests help make upgrading so much easier. I was once on a huge Scala codebase that needed to go up from Spray.io to Akka-HTTP. Tests help save me on tons of potential bugs in migrating over. The few things that got through, I could write more tests for.
On another open source project I maintain, my functional tests helped me migrate from Scala 2.11 to 2.12 along with all my dependencies with relatively low friction.
If you're porting your Python2 to Python3 stuff now, it's never too late to start writing some tests. Write them over the weird division stuff you've found. Make sure they run in Py2 and fail in Py3 so you can isolate what it is. Even a couple of small tests in a large code base that only cover stuff you're finding now is better than no tests at all.
You can also write custom transforms with 2to3 but I've found https://pybowler.io is a decent wrapper. I haven't used it for 2to3 conversion though.
That actually isn't a function call, despite the appearance. Python 3 raise is still a statement, not a function (unlike print). So this
does not do what you think; the proper way in Python 3 to spell what you probably expect is Here is a transcript of a shell session showing the difference: Note that the exception you actually get the second time isn't even ValueError, it's TypeError; it's being raised because "some text" is a string, not an instance of an exception class derived from BaseException.In other words, you should not be treating raise in Python 3 the way print is treated. The 2to3 tool treats it properly.
That's why they had, what, around 5 years to do it since the 2.7 EOL was extended to 2020?
I think most of the problems you encountered would be solved by the six library, btw.
But the most annoying bugs were those were some Python 2 quirk was "just working" (like comparing strings to integers in Python 2) or the conversion of ints and floats and then it correctly broke on Python 3.
For this particular project, I would had needed to get the code to run in both Python3 and an ancient version of Python2 made back when boost.python was still a thing, so six would be out of the question.
I wonder if/why there is no such ~~thing~~ [edit] effort for python2/3?
If you're only accepting programs that P2 accepts, you can even get away with being lazy and "understanding" more than P2.
I wonder if these are symptoms of duck typing?
Edit: 2to3 is what I'm talking about https://docs.python.org/2/library/2to3.html
You hopefully mean after spending 1 minute running py2to3 right? I mean I get that some people feel that it’s tedious or difficult to convert if they spend ages manually changing thousands of lines of code in a completely automatable fashion. Which is why it’s nice that you don’t have to actually do that. You just run py2to3 and the focus on differences that matter.
Is your code in Debian right now, or was your comment a general question in regard to the status of python 2.7?
In most programming languages, a change to the language is practically always backwards-compatible. The update may add new capabilities, but the old capabilities and syntax remain in place. If the language changes semantics, it's usually in bizarre edge cases of the language. If a basic data type was a mistake, for example, a new data type is added & people are encouraged to use it instead... but the old one keeps working. If an existing data type or method should be renamed, the new name is provided & the old name keeps working for a long time so that people can incrementally update.
In short, the normal expectation in programming language updates is that existing code will keep working, and that updates can be done slowly and incrementally. That's historically been true for Python as well; updates typically didn't break much. In the few cases where there were significant backwards-incompatible changes, languages typically give 10+ years of warning.
The Python 2 -> 3 transition has instead been an example in how to do things badly. Pretty much no Python 2-only code will work without change; you have to change print to print(), division's semantics have changed, string semantics have changed, common names have changed (xrange to range), and so on. The original automated transition tool originally envisioned could never work reliably, because Python's dynamic nature ensures that there's not enough information to work without help. So the demands on people to make changes were unusually large for a language update.
A related problem was that there was never an implementation that supported Python2 and Python3 at the same time. That meant that if you depended on 100 Python2 libraries, you had to wait for all of them to transition before you could transition. If 99.99% of the libraries you depend on are available in Python3, you must use Python2. You can easily create C or Fortran programs that call on code written long ago unchanged; that is not true for Python3.
So in the Python3 transition all code had to be modified simultaneously across the ecosystem, and tools couldn't do it well.
Eventually the Python3 folks made enough changes to Python3 so that it started becoming practical to write code that worked on both Python2 and Python3. That should have happened from the beginning.
The only language I can think of that handled a transition worse is Visual Basic. When "Visual Basic for .NET" came out, it was so grossly incompatible with the previous Visual Basic that it was called "Visual Fred". In numbers I saw long ago, about 1/3 stayed with the old unsupported Visual Basic, 1/3 changed to a completely different language (that was less likely to abuse them), and only 1/3 of the previous Visual Basic users transitioned to Visual Basic for .NET.
I think that Python3, all by itself, is a fine language. The problem in this case is that the developers of Python made it unnecessarily hard to transition to Python3. That was both unfortunate and unnecessary.
But I basically agree with you that using the name "Perl 6" for Raku led to most people abandoning Perl 5. Hopefully people will think more carefully about transition in the future. But too many technologists ignore history, so I don't have high hopes.
Write some tests around the problematic functionality before attempting the final port, it should help in this situation. Should be broken into multiple files as well.
This is not too different from what I have seen in multiple jobs, where the new hire doesn’t get to write all the code they use from scratch, but instead has to maintain someone else’s mess. Often times, management and market conditions demand more from programmers than they have time to properly implement with test cases, well structured code, good comments, etc.
Technical debt is a big problem in code used by businesses, and the technical debt this particular 6000-line script has is a lot less than some stuff I have seen and have been paid to maintain.
The fact of the matter is that most open source code out there is quite a bit cleaner and maintainable than the code which is behind firewalls, running important businesses.
[1] Edit: To the original author’s credit, this particular code base is an incredibly popular and successful map generator for Civilization 4, with some 40 thousand downloads; it got messy because the author was more interested in adding new features requested by users than in keeping the code base clean. It is a single script because that made it a lot easier for users to install. And, newer versions of the same generator are written in Lua and not Python because that is what newer versions of Civilization use for scripting.
You want a "good critical comment"? Here:
* Try breaking your code up into more than one file. There are other/better ways of releasing Python besides "one file".
* Lay off the code commenting, it's vastly overdone here.
* Use version control tools to version your code, not comments.
* Python convention is to use underscore_case, not camelCase.
* Variable/function/class names should be descriptive, not random letters/numbers/incomprehensible abbreviations.
* Delete commented out code -- it's stored in version control, you're not going to lose it.
* The "initialize" func is a terrible concept and represents a fundamental misunderstanding of how Python classes work. This whole misadventure is a cautionary tale in, "Why a light architecture pass matters even in 'throwaway' scripts."
* Add tests.
* The massive if/else chains can be replaced in a number of different, more readable and therefore superior, ways.
That's some of what I saw in 2 minutes of review. This code is in very poor condition, and if you think the state of this code is an indictment on how hard it is to convert python from 2 to 3, you're more than just a little mistaken. You would need to pay me substantial amounts of money to spend more of my time going deeper into what's wrong with this code, as there is a lot and it is not generally enjoyable to work with code so thoroughly broken as this.
The script (somehow) works. That matters. However, this is not a good representation of why migrating from Python2 to Python3 is hard; this code is in no shape to be used to represent anything other than how not to write Python.
The main issue — besides not using 2to3 — is that / in Python3 is interpreted as floating point division, whereas it used to be integer division if operands were integers. Replacing / by // in the relevant code blocks might help? Or getting another implementation of the RNG?
Why doesn't Debian just ship the unreleased Python 3 port of Calibre? It's not like Debian restricts themselves to only formal upstream releases, they have never hesitated to backport patches to unreleased versions. If lacking Python 3 support is such a massive bug, wouldn't using patches that have not been officially released, that fix the bug, be acceptable? Debian testing isn't Debian stable, there is not a reasonable expectation that the code shipped is perfect.
Mercurial also works on Python 3, though it's only a beta. I could understand not shipping it because version control bugs can be serious, but hey, someone's gotta test the beta. I've used it and it not encountered any bugs, but went back to py2 because tortoisehg won't work with Python 3 mercurial yet (and then bitbucket dropped mercurial support so now I'm moving everything to git, grumble grumble, probably will sadly stop using mercurial altogether within the next year).
In this particular case, the Debian maintainer seems to prefer to wait for upstream to release an official Python3-supporting version of Calibre though: https://github.com/norbusan/calibre-debian/blob/master/debia...
It's basically up to the maintainer either way. The freeze for the next Debian release is at least months away (no date set yet), so it's not too urgent, and waiting for upstream is still reasonable at this point.
Here's a recent post from the Debian maintainer: https://www.preining.info/blog/2019/10/state-of-calibre-in-d.... The tl;dr is that most/all of the work is done but he's holding off updating unstable to Python 3 because he doesn't want to break external plugins. He also mentions that he can't update to 4.0 or higher because Debian still hasn't updated Qt to 5.12. That seems slightly inaccurate as it's in experimental (https://packages.debian.org/experimental/qt5-default); I think the maintainer could do an upload of a Calibre 4.x using Qt 5.12 & Python 3 to experimental. But experimental is not in common use regardless.
The post seems to be an extension of https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=936270#87
Personnally, I approve, otherwise it'll spiral down in nitpicking and special cases. Would not be the first time with Debian.
> If the package has still many users (popcon >= 300), or is needed to build another package which cannot be removed, document that by adding the "py2keep" user tag
So they are not even that firm. Overall, it seems very well handled.
No Python 3 Port: https://bugs.debian.org/cgi-bin/pkgreport.cgi?tag=py3noport;...
There is still much useful and valuable legacy software out there that many people use which is getting abandoned for no other reason than some people (like the authors of calibre) are not very fond of the direction python is going. Others can't upgrade to 3.x because you have standard libraries especially in the academic sector that people literally put their whole phd into and that can not "easly" be upgraded just like that because of high complexity, lack of staff and time.
It's vaporware at the moment, but uh: https://git.sr.ht/~sforman/Bladders
> Maintenance-mode fork of the officially-deprecated Python 2 interpreter and standard library. Bugs fixed, security holes patched, and maybe some performance improvements, but no changes to the syntax or semantics (for that see the Tauthon project, unaffiliated.)
- - -
Sometime between now and the end of the year I'm going to fork Py2 and set up build jobs to run the test suites. (Maybe later today, it's not hard, it just takes a few spare hours.)
Going forward I plan to work on things like improvements to the docs, curating additional packages and libraries (additional to the standard lib), and language tooling (I'd like to keep things like Nuitka, Snakefood, etc. working.) I also want to run fuzzers and other code quality tools over it all.
This sounds strong to me. Not to be provocative since I am not familiar with the mailing list, I would like to ask is it common to communicate this way in this Debian community? Is it effective to push things moving forward?
Too much is often sacrificed at the altar of backward compatibility. Move forward and leave the laggards behind.
Edit: reference.
[1] https://stackoverflow.com/questions/9477115/what-are-the-dif...
But the thing with Babel is that it only adds features to JavaScript, it doesn't translate between two disjoint languages. I'd assume that has an effect on its viability as a project.
Think of the lost productivity that has resulted.
People complain about JS all the time, but JS hasn’t broken existing code and has improved.
JS supports ‘let’ declarations and variables named ‘let’, python could have improved without breaking everything.
https://groups.google.com/a/chromium.org/forum/#!topic/chrom...
Why wouldn't you consider it an emoji?
> Originally meaning pictograph, the word emoji comes from Japanese e (絵, "picture") + moji (文字, "character"); the resemblance to the English words emotion and emoticon is purely coincidental. https://en.wikipedia.org/wiki/Emoji
Since python software foundation will stop developing it there is no upstream support after 2020. In theory a new upstream for python 2 could pop up, and there is speculation that Redhat might do it, but Debian do not want to become the new upstream for python 2. If they included python 2 in next release they would have to support it for as long as next release get support. What will happen instead is that they will keep supporting python 2 in older releases but when those support periods are over then its over.
If they're going to do it, for the sake of the community they should announce that soon. Even if they don't plan on taking up the responsibility until the middle or end of 2020 it might be something that distros and companies would consider when making their plans.
It seems perfectly reasonable for the Debian Python team to decide they no longer want Python 2 getting in their way. But I think they should have done more to check whether someone else wants to take over.
There's nothing terribly wrong with packaging mature software that doesn't have an active upstream. Realistically nobody cares about any remaining bugs in Python 2 being fixed; the work involved would basically be making sure it still compiles with newer versions of GCC.
[1]: https://www.cvedetails.com/vulnerability-list/vendor_id-1021...
In particular, you don't want to have to worry about building against openssl.
I think the vast majority of cases where people have a good reason to carry on using Python 2 won't need that.
The answer is mainly security issues and compatibility bugs. If a package is abandoned, it is typically removed. Someone needs to write updates both to fix newly found security problems and general issues with other new packages. Perhaps we'll switch from the init system systemd to systemf, and some package depended on that. Then it won't work in a new version anymore. Like someone else commented, it's nothing but work.
I'm not sure why it is removed before any such issues arise or become known. My best guess is that it should prevent the removal from becoming super urgent once a vulnerability becomes known. If it is still used by anyone, they should either become the maintainer or (in this case the more realistic option than becoming a maintainer for python2) update the code to python3.
Not sure why you're being downvoted though, it seems like a legitimate question: "it works, why bother even removing it?"
You can't simply remove packages from a release after it's shipped; either your release includes that package completely (including a commit to support them for the whole lifetime of your release) or not at all.
Otherwise, who will fix the Debian Python packages when the next CVE comes out?
[1] https://www.python.org/dev/peps/pep-0373/
I don't honestly think Python 2 is going anywhere for a while yet (judging from some of the commercial codebases I've seen at least). I sometimes wonder if this mess would have dragged on quite so long if there'd been better features in Python 3.
[0] - https://www.python.org/dev/peps/pep-0373/#maintenance-releas...
[1] - https://doc.pypy.org/en/latest/faq.html#how-long-will-pypy-s...
Fair enough, but there is still no commitment from upstream for most of the supported release of the next Debian release, so that is moot.
> ...PyPy has no such plans to drop support
This isn't relevant to the CPython package dependencies which are the issue at matter for Debian on this matter. They haven't explicitly said CPython because it's implicit the way that Debian's Python packaging is structured.
There are still quite a few people running VB6 programs.
Nothing helps this; there will always be laggards.
If it weren't for deadlines, progress would never get made.
https://pythonclock.org/
Python: Hey you kids: sit in this cubicle for the entire summer and do 20,000 long division problems just to keep the school's lights on. And don't get too encouraged because there's no way in hell we can finish all the long division before the beginning of the school year. Some of your classrooms will be dark. p.s. our principal got burned out and quit.