As an aside, why doesn’t the Python standard library extend/replace features with code from successful packages like Requests? Tried it and it didn’t work? Too much bloat? Already got too much on the to-do list?
This may somewhat explain why lots of successful packages are not in the stdlib: putting them in effectively killed future development until a few years ago. But today that argument is inconsistently applied and I'm not sure if it's a rule worth keeping.
Pypi have thrown out the downloads counter—a huge misservice to coders. Like I got all day to figure out the best libs for ten different features which I only need in passing, so my primary concern is to not pick complete garbage.
So, my solution to that now is to look up Github pages for the libs and choose the one with most stars. As much as I dislike Github for its occasional typical proprietary behavior, Gitlab doesn't help in this case.
A better approach might be to add a core of basic requests-like features built from the stdlib’s existing resources. That would be beneficial to many users and if they need more then there’s always Requests.
I'm not entirely sure what you're looking for, but have you tried https://www.enthought.com/product/enthought-deployment-manag... ?
Edit: example
$ edm envs create tester36 --version 3.6
$ edm shell -e tester36
(tester36) $ edm install ipython matplotlib pyqt> As an aside, why doesn’t the Python standard library extend/replace features with code from successful packages like Requests?
It is possible (ie. asyncio was separate package). It is slow process though.
https://python.libhunt.com/ Not exactly curation but it does have rated libs for lots of categories.
That would be my guess. I would also add that getting through the process of adding a third-party set of modules to the standard library can take quite a while.
Years ago I was in contact with the author of Nuitka, who was very excited to share his work thus far. During his presentation, Guido kept huffing and making snide comments under his breath. All because he disagrees with the premise behind Nuitka.
I like Python, and can appreciate his work and contributions. That said, I can't help think the community can become less toxic without him as BDFL.
[0] https://mail.python.org/pipermail/python-committers/2018-Jul...
I know he gave up the title, but as he personifies Python, I imagine the influx of requests for his attention may not have subsided much.
A core skill of any open-source project maintainer is recognizing when you're burnt out enough that your continued presence isn't helping the community and is just getting in the way of others, and stepping aside. If you can't do this, you're not a good maintainer. I'm happy to have sympathy for your personal problems, but you're still failing at your job. Here, for instance, is a short but effective way to do it: https://mail.mozilla.org/pipermail/rust-dev/2013-August/0054... (That was almost two years before Rust reached 1.0, and the language hasn't suffered one bit for it.)
And part of Amber's point is that core Python development is taking on too many burdens for no actual benefit. By bundling so much in the standard library and expecting the standard library to be usable for real work without installing additional packages, the core team (and Guido in particular) increases their own workload, impedes the ability of others to contribute, and produces worse results for end users than they would otherwise.
let's hope everything settles smoothly :)
- A language that is simply less active or whose users feel less ability to have a say in the language's development is going to be both less social and less dramatic, but also carries a high chance that existing users are only there for legacy reasons and are always considering moving to another language better suited for their purposes.
- A language run by a corporation is less "social," and has the drama play out in business negotiations, closed-door committee meetings, and lawsuits instead of on blogs and public mailing lists and (in this case) closed-door meetings with an expectation of public reporting.
I'm not aware of any drama in the ASP community, but that probably also had something to do with many users having picked Python for their next project instead of ASP. I'm aware of some drama in the Clojure community, the conclusion of which seems to have been that Clojure is their corporate sponsor's language and if it doesn't work for you you should find something else.
(Though in nothing on the level of some in open source).
In Python's defense, this was not obvious at the time; my understanding is Rust came to this approach by looking at the experience of Python and other languages. When Python's standard library was first being written, there were no easy package managers for any language, and the normal thing to do for installing dependencies in e.g. C was to grab random tarballs and figure out how to build and deploy them yourself. So avoiding that process made perfect sense.
Obviously it doesn't apply to everyone, and it certainly doesn't apply to most startups or open source developers, but I spent most of the last 20 years working in environments where you have to get permission for every third party library you bring on to the network. Many networks were essentially "airgapped", so it's not like you could just ignore the rules. The bureaucratic process alone meant that we preferred large bundles like Anaconda or Qt. Trying to use Cargo as it is typically used and documented would be a complete non-starter.
So Rust can get away more easily without having support for things like command line parsing in the standard library, because it isn't really trying to support situations where it would be inconvenient to give your program its own project directory and Cargo.toml and all.
Wasn't perl's CPAN developed around that time period (mid '90s) ?
Edit: based off of all the replies below, everyone understands the validity of what I'm trying to say, but also have fortunately pointed out my admittedly grevious error of not knowing you can just import hashmap from stdlib. The extra step is pretty minimal and not a problem. I'm hoping this is covered in the Rust book.
https://doc.rust-lang.org/std/collections/struct.HashMap.htm... https://doc.rust-lang.org/std/collections/struct.BTreeMap.ht...
That being said the point still stands, I remember specifically feeling it at the lack of included regex. I overall prefer the slim std lib of rust over the massive python though. Especially with the community being pretty good about nominating de facto standard packages like for regex.
https://doc.rust-lang.org/std/collections/struct.HashMap.htm...
Had I been using Java or Visual Basic or even C++ proirly maybe I wouldn't be so impressed as I were.
I think the mistake Python is doing is messing with it's simplicity with decorations, halfass lambdas and stuff. As a newbie you could understand Python code, while eg. C++ templates were magic. You need to know more Python to understand Python code nowadays.
There are other new stuff that might be confusing to beginners (like our mighty walrus operator), but those two examples have been there for effectively forever.
Tkinter needs to go...There is very little reason except for the legacy ones, why it needs to be there still...
If you're worried about people not RTFM when using your projects, you could always start your scripts with the standard try/import/except wrapper around the required package, and tell them to run pip install when it's not found--or (and this is a terrible idea) run it in a subprocess for them.
Being able to add items to a canvas, attach callbacks to them - without tracking them yourself, setting up the infrastructure for indexes, bounding boxes, all that stuff - I haven't found anything else as easy as Tkinter's Canvas.
If there is something better, I'd love to hear about it.
https://cairographics.org/pycairo/
You do have a good point about Tk's canvas easy input handling though, which is something Cairo doesn't do. It's just an immediate mode drawing api like PostScript's rendering model or html canvas 2d context.
You can pass a pycairo context into C++ Python extensions, and they can go to town with it quite efficiently.
https://github.com/SimHacker/micropolis/tree/master/Micropol...
https://github.com/SimHacker/micropolis/blob/master/Micropol...
There's always going to be a good debate about which batteries should be included. I'd almost put numpy ahead of Tkinter, but maybe that reflects my bias for scientific programming.
Coming from Visual Basic, and similar things (HyperCard, Turbo Pascal), being able to run a program with GUI on both Windows and Linux with no code changes was nothing short of amazing to me. So maybe that's one way to draw the line with batteries, namely to make it so a beginner can get started with Python, with no regrets about which OS they're using. This requires some out-of-box hardware abstraction, where they keyboard, mouse, and display, all effectively count as hardware.
Taking tkinter out of the standard library won't stop IDLE being built with it, if they chose to do so.
Amber's suggestion seems to be in the same direction (though perhaps not as extreme.)
[0] https://www.slideshare.net/mobile/hsbt/gemification-for-ruby...
The problem is that I cannot count on being able to install new software in many environments.
If I fight the battle to get centralized IT to install Python, I now have a guaranteed set of standard libraries as well. I'm never going to get permission to install anything other than default. Ever.
Consequently, the standard libraries need to be very complete and very useful.
And, while people seem to love the Rust approach to libraries, I'm not necessarily a fan. Far too many times I have pulled a library that is "obviously" something that a language should consider to be "standard library" and gotten bitten because it was broken. Only VERY core libraries in Rust are guaranteed to work across multiple architectures and OS's.
I think Rust is probably doing the right thing for Rust as "batteries included" is NOT one of its tenets. However, that doesn't make it right for everybody else.
Can you explain this more? What kind of place do you work? I've had some experience with large, bureaucratic companies, but nothing ever so far as "you can't install any other libraries."
You could install packages with `pip install --user` to have them install under your home directory.
The approach Ruby is taking with gemification and default and bundled gems for standard libraries is equivalent to the traditional standard library if you can't install updates, but superior in other cases.
In this case, even If you can't upgrade, you still get the same libraries but perhaps older versions.
In Rust, libstd is mainly for interfacing with the compiler and providing interoperability between packages (crates). The wider crate ecosystem is the real standard library, since external crates are as easy to use as the standard library.
For example, the libstd doesn't even have support for random number generation. There's a rand crate, which is now on 6th major breaking version. That's perfectly fine, because multiple versions can coexist in one program, and every user can upgrade (or not) at their own pace. And the crate was able to refine its interface six times, instead of being stuck with the first try forever.
As to why this is the case, I think maybe this is enabled by Go's backward's compatibility focus and encouragement to upgrade early and often, and the community's focus to utilize small interfaces sometimes from the stdlib itself, like io.Reader and io.Writer, or http.Handler. Added to that w.r.t. using the latest and greatest, most Go users frequently are using the latest version of Go even in production (per the go experience surveys).
I am sure it also helps that Google pays people to develop, maintain, and improve the stdlib.
The example of random number generators is a good one, too. There are a lot of applications that require (reproducable!) PRNG sequences and sometimes you have to share PRNGs between modules. Now, looking at the rand crate, I see that the prng part of it was recently mucked around with. If I have two 3rd party modules that I require to share a PRNG that I control (say, noise generators for procedural textures), I cannot compose them if one of them uses the old and one of them uses the new version of the library.
I don't know how many hours I've spent battling Psych errors because of this very thing, but it's way too many. Calling the gem something, anything else, would solve the issue.
It's great that they're unbundling a lot of things, but there's still some serious friction between external and internalized versions of these gems.
For Ruby, EventMachine sub-universe is really in bad shape. EventMachine is creaky and old. Event-aware packages are in short supply and are usually woefully out of date, unmaintained.
The reasons are similar, it's a constant drag on core compiler development to need to support various batteries included that most core contributors aren't going to care about, so it's easier to tell people "use CPAN".
There was even talk of "distros" for the interpreter. Where the core bits would be similar to what Linux is, and all the batteries would be provide as collections of add-on packages.
Strangely enough these efforts seem to stop at OS distributors. They really seem to like to install just the one "compiler", and wouldn't stand for a project like Perl or Python telling them "we mean for you to distribute the core compiler plus these 100 packages, because that's what forms our 'language'". "Strangely" because you'd think they'd be the best positioned to make easy work of packaging up such a thing, and it shouldn't in principle make a difference if you need to install 100 RPMs / APTs by default.
When did you have to use cpan in a modern system? Compare that to how many times you had to use pip.
Now, if you use a crappy OS or distro (or god forbid, some container built by you have no idea who on top of nobody knows what) then yeah, you are bound to do the leg work yourself, but you will be doing that regardless of the language/subsystem you are trying to use in that case.
Not to mention that it is the only way to do things professionally. For example, if you must have a system that parses XML but for company policy is not allowed to have even the means of performing a network request. With python you either have both xml and an http library and whatever else included and you will either have to do a special package with a striped down python+xml only or get a corporate exception. While on other languages you can install only the xml parser component package and your code will run happily and be compliant with company policy.
Well yeah. A sizable amount of new software is still being written in Python. But when I use Perl software (besides my custom scripts), it's always stuff that's old enough that the distribution is carrying packages for it.
If you disagree, please name a significant new software written in Perl that was released in the last, say, 5 years.
...wouldn't the company policy involve removing the means of performing a network request from the computer, making the notional capabilities of the software irrelevant?
Python will let you drive network requests through the OS. It's just that you wouldn't normally want to.
I mean that a significant use people get out of Python and Perl is that they aren't bare-bones like say Scheme or Lua where the standard library is really spartan.
It allows you to write useful code that works on the lowest common denominator of "just OS Perl or Python". Whether that's some random version on whatever Linux distro, or *BSD or Solaris or whatever without needing to write your own getopt library or whatever.
Which is why the "let's ship a bare-bones compiler and have people use CPAN or PyPi" is contentious. In theory it shouldn't matter, and for a lot of shops who install hundreds of packages it doesn't, but it does for people who target stdlib-only, which is a big use-case. Particularly since the people who have that use-case are drawn to these languages.
I've been using built-in environment isolation tools such as virtualenv for ages but have recently switched over to using miniconda for all things python. Among other things it has amazing support across all three major OS's, and I happen to be dealing with all three at any given time. Whether one uses miniconda, pipenv, virtualenv, or anything else like it, as far as I am concerned the days of ever using the system python are over. I will always create my own personal "distro" on the fly with full control over the python version and every add-on package.
You don’t have any Python scripts in your bin folder?
Not a million miles from the modularisation that Java has been going through.
But I think a lot of the value of a large standard library is that it makes it possible to write more programs without needing that first third-party dependency.
This is particularly good if you're using Python as a piece of glue inside something that isn't principally a Python project. It's easy to imagine a Python script doing a little bit of code generation in the build system of some larger project that wants to parse an XML file.
It's also a nice API for dealing with XML.
If only that were the norm amongst long-tail python users. Heck I don't do it; I have the anaconda distribution installed on Windows and when I need to do a bit of data analysis hope I have the correct version of packages installed.
Making this core to the python workflow (bundling virtualenv? updating all docs to say "Set up a virtualenv first"?) is the first required change, before thinking about unbundling stdlib
And it is a totally miserable experience on Windows, every single time.
But, I do wonder whether it needs to grow. Python is in a stage where adoption of new std library features is inherently slow, not only in third-party libraries like twisted, but also in applications, even if they use newer Python versions.
What kills Python here is that it is most commonly bundled with the linux distribution or the OS (true also on mac). This reduces cycle times drastically. Compare this to newr language platforms that people like to install in newer versions on older platforms quite regularly.
Some recent additions would be fine additions at an early stage of language development, but surely not for Python.
It doesn't kill it, quite the contrary, rather makes it ubiquitous. If you want another version you just install virtualenv. It's the same with Perl. We use the Perl version shipped with the distribution (openSuSE) and deploy to that. It's older but it's stable and it works. On our dev environment (mac) we have the same version with all of the modules installed in plenv. We also chose a framework with as little deoendencies as possible (Mojolicious). It looks like it was a great choice.
Asyncio is in the stdlib so that we have an official lib and API. The main benefit is that most people now, when looking for async, are not wondering about twisted or gevent or tornado. Most just go asyncio. Most dev efforts go to asyncio. It's the end of the great async war. Is it perfect ? No. And I don't care. It's one thing less to worry about. For those who know what they are doing, you can still choose and pip install twisted, but most people don't, and that's solved. Before that, just choosing the lib was a nighmare, as basically it's a definitive call. Out it on pypi, even with a "stdlib" tag, we go back to the 200X era. And it was not fun.
And the goal for having things like xml/sqlite/ssl without installing anything makes python very useful in a load of situations where you can't install stuff. Sometime you are offline. Sometime you are in a restricted env. Sometime you are not on your machine. Sometime your security protocol is hell. Don't assume people use Python as we do, from our comfortable dev laptop driven by the knowledge of our craft. Python is used in banks, by scientists, in schools, by kids, by poor people in the third world, by geographers and pentesters. The python user base is incredibly diverse, it's why it's so popular: it fits a lot of use cases.
So I see the benefit of having a side version of official modules we can pip install that can move faster. I see the benefit of cleaning the stdlib of old stuff, like the wave module, Template or @static.
But I'm glad I don't have anything to install to generate a uuid or unzip stuff. I'm glad I don't have to worry about twisted anymore (depiste that I did write a book on the topic !).
Also, pip install is NOT simple when you learn the language. I have to spend some time in the classroom, even with adult professionals, to explain the various subtleties of site-packages, import path, py -x on windows, python-pip on linux, -m, virtualenv, header files, etc. before my students become autonomous with it. Without a teachers, this turn into months of bad practices and frustrations.
You'd have to fix that first, way, way before moving stuff to pypi. I do think it should be high priority actually: it affects way more than pip.
If you're in the flow and trying to hack together something, the last thing you need is to lose all momentum to pick a date time library. I've had this issue tons of times with Node and Rust, where I'm not up to date with the current meta and my 30 minute hack job is interrupted 5 minutes in by having to google which library should I use to do an HTTP request. (I've actually lost interest in whatever I was doing a few times because of this.)
Python's stdlib is nobody's favourite, but when you start to get to its limits, you're probably past your flow state, you've written most of the logic and you can spend some time to replace http.client with requests because the latter is much better.
On a tangent note, I've been trying to find another scripting language to replace Python because I'm not a fan of it anymore (I won't get into it right now), and considering what I just wrote, there's not much that can replace it, as most languages have a bare-bones standard library and if you're not up to date with the current best library to do X, you'll never achieve great productivity.
Have you considered ruby?
Wait... what ? No way ! Some of us do you use Python to process wav files. If anything, I'd like this module to be updated, not removed.
It was pitched first as a common low level async loop for other applications like Twisted and Tornado.
Then people started using it directly and the keywords were added.
It's great for the people who think the way asyncio does, others are now forced to use it. I find all of Twisted, Go and Jane Street's Async easier to use.
Perhaps Python is just the wrong language for me.
Explain? I’ve hated the slow adoption of 3.x from 2.x, and generally how terrible it is to have apps that are 2.x on your 3.x system, and would like to know more about how that happened.
I maintain a Python 2 & 3 compatible project that has no external dependencies.
In some cases of common OS-induced pain, I'd say "do whatever 3 does" in 2, since that'll make migration easier in the long run. (But I understand that can be hard, and I think my responses to your examples below even demonstrate that to be hard.)
To your specific pain points:
> Unicode stdio?
Mostly, `io` should handle this in both 3/2. You might need to help it get the right encoding in 2.
> Unicode file paths
This is going to be a mess in any language, because file paths really aren't text. On nix, they're byte strings that don't have nuls in them. Hopefully* they're encoded according to LANG, and hopefully LANG is a UTF-8 variant, but it isn't required, and it isn't required that two users on the same system use compatible LANGs, so you get a Tower of Babel. I really wish OSs would just start enforcing a. Unicode filenames, and b. no newlines in filenames; those two alone would make life so much easier.
Hopefully you've seen os.fsencode / fsdecode, but alas those aren't in 2, so I'm not sure they really help you. Often one is not really munging paths that much, and can just pass through whatever value/type you get, but it does happen, of course. (E.g., adding or removing extensions)
> Unicode sys.argv
This is also a pain point, since again, the underlying type in nix is a byte string without nuls. I'd hope it decodes w/ the LANG encoding, but since the user could easily tab-complete a filename, fsencode/decode might be more appropriate. I think I'd say "do whatever 3 does".
1 Jan 2020 is nearly here. Forget about 2 / assume UTF-8 in 2 and don't support anything else?
>
I feel like Python 3 completely wrecked strings instead of making them better.*A clear separation of text and binary is needed in the long run, and makes other operations much clearer and saner. The pain you're feeling is introduced from the OS not having the same clarity.
No. No on just so many levels.
It finally fixed it by introducing two (or three if you count bytearray) types for completely different semantics of data.
Python 2 was just a mess whenever you left ascii.
After A TON OF PAIN I decided to put the time in to watch the talk "Pragmatic Unicode, or, How do I stop the pain?"[0] by Ned Batchelder and it all just clicked. Now, I make unicode sandwiches like a boss.
fwiw, six can easily be vendored into a project to avoid the technical external dependency. that is how we manage it for kafka-python.
there are a few modules that are simply at different locations but have the same API
if PY3:
from http import client as httplib
else:
import httplib
PY3 = sys.version_info >= (3, 0)
PY2 = sys.version_info < (3, 0)
PY26 = sys.version_info >= (2, 6) and sys.version_info < (2, 7)
isinstance(<maybe_string>, basestring if PY2 else str)
# Python 2.6 doesn't properly UTF-8 encode syslog messages, so it needs
# to be performed in a custom formatter.
formatter_class = UnicodeLoggingFormatter if PY26 else logging.FormatterAs the maintainer of WebOb/Pyramid/Waitress we have a compat module that contains all of the changes/renames/functions to help with the Python2/3 compatibility and all tests run across both platforms.
Are the functions borrowed heavily from six? Yes, but we don't need to vendor all of six.
To be fair, one of our three dependencies does in fact depend on six, but we don't use six anywhere in our code. This dependency of ours is used only in one specific area of the codebase, and in fact I (the person who added the dependency) wish that we didn't have to use it at all.
Amber is nothing short of an open source hero, having brought Twisted, one of the best open source projects in the world, to new heights. Her insights are as important as anyone in the python community, and after six consecutive PyCons sprinting at the Twisted table (including literally in a chair with Amber to my left and Glyph to my right earlier this month), I consider Amber's voice to be one of the truest and clearest among the leadership of the language into the future.
Amber and Guido are both beautiful human beings.
In the dispute that is the topic of this blog post, Amber is basically totally right. Moreover, the distinction has less to do with any kind of nagging python 2 holdover than this article suggests. The standard lib's role as a place where code goes to die is a view that is widely held and accurate for many cases.
The following question went unanswered during the Steering Council Q&A:
"Every feature request has a constituency of people who want it. Is there a constituency for conservatism and minimalism?"
...and that's really what this whole thing is about.
I agree with a lot of her arguments. I never understood why tkinter was included (I could see why it might have been added years ago, but before Python3 came along it felt obsolete and unused). I've had to support old versions of Python along with their bugs, and it sucks.
But languages are about choices. Python's syntax and decision to use whitespace is a strong choice--so is their stdlib. I've been on projects that saw performance problems with etree, moved to lxml, then had to go back to etree because of missing features. I've spent a lot of time looking over arrow, dateutil, and moment because datetime seemed inadequate. But I like that there is a thoughtful default that serves many needs. A lot of these examples, like requests, are built right on top of stdlibs--so both would be needed even if it was shipped along with stdlib. I'm ok if code goes there to die because I would hope that due diligence was taken when it was included.
I kind of agree with Guido for a lot of this. I can't wait to move from Python 2 and have looked a lot of the new stdlib and looked for backports. I've looked at twisted and alternatives and I'm so happy something is built in even if its far from perfect.
When it comes to constructive criticism, I think Amber did a good job with her criticism but can do better at the constructive front. Her problem statement was spot on and I agree that the direction she proposed is a good one.
However, to separate the standard library from the core is probably even more dramatic than the Python 2 to 3 migration. Is that what the community can afford at the moment? What's needed to make the transition? What's the opportunity costs? i.e. what other developments we can do for a bigger impact? What are the pros and cons?
And it wasn't the only thing at the language summit that proposed expanding that approach: there was a discussion of carrying time zone updates in the same way, by shipping something with the interpreter that works but allowing updates from PyPI. http://pyfound.blogspot.com/2019/05/paul-ganssle-time-zones-...
So I think the development community / target audience at the language summit already understands the pros and cons of the suggested approach and the technical route to get there. (For an end user, my guess is the experience will be that anything in the Python 3.x standard library today will still be in the Python 3.x standard library, but you'll have to `pip install` a newer version if you want more features, and you get the benefit of being able to `pip install` something from the standard library where you previously couldn't.)
What do we know? Software configuration management is a heinous problem.
Python gets it more correct than most.
Let us rejoice, be patient, and respect everyone's good-faith efforts.
Amber wants to remove asyncio from the standard library. That's minimalism but not conservatism.
Also note that one way of interpreting her proposal is "generalizing the 'ensurepip' model" to keep things already in the standard library in the standard library, but move feature development externally and make it easy to upgrade packages in the standard library. (Then Twisted, which is also installed externally, can simply depend on a fixed version of a standard-library package.)
Perhaps, but Guido's response was shitty here (and purposefully refusing to get the point), whether one think Amber was right or not.
I deal constantly with the shitshot that is Python packaging (and lack-luster default packages) and I happen to think she is 100% on the spot.
And what if the standard library just gets dated? Take Node for instance. The fs module has a whole bunch of outdated callback based functions. Sure, you can wrap them in promisify, but it sucks that we have these outdated functions stuck around forever.
There's definitely tradeoffs with package/dependency multiplication, but I don't think standard libraries are as clear cut as people make them out to be.
That depends on what you are writing.
For an application that is deployed stand-alone, you'll likely fat-package it with python and all the libraries. In case of a security issue, you create a new version of your application that bundles the fixed python.
For an application that is deployed on the system python (more typical on Linux), it's the system admin's task to update the system python.
As the user of the stdlib, it really depends on what the security issue is. For most users, if they can verify they're not using module X, then there's no problem and no rush for them to upgrade. In a language like Python that's very dynamic, you also might be able to download a hotfix .py file provided by the language maintainers (or any other party) as an alternative to upgrading, or as something for the interim while you wait for the upgrade to get released, backported and released for older major versions, and available for the deploy environment -- this might even take the form of an iptables rule, depending on the issue. In short my point is that there are alternatives besides bumping one release's release version and hoping people upgrade.
> Except, users don't upgrade stuff.
Actually users do. Not all users, but your sweeping statement isn't true of all users either. Experience and reading reports tell me that many if not most users actually upgrade, especially when there's a significant security issue and especially when upgrading is easy. 2.7.14 to 2.7.15 was a problem for no one.
For the users who don't upgrade, they're not likely to upgrade a third party / split off thing any more than the core language.
There's also another party of users worth mentioning, those that don't upgrade specific things because past upgrades have been a terrible experience. This is a reputation issue.
> And what if the standard library just gets dated? Take Node for instance. The fs module has a whole bunch of outdated callback based functions. Sure, you can wrap them in promisify, but it sucks that we have these outdated functions stuck around forever.
Sucks for whom? I've been out of the Node ecosystem for several years, is there a pfs module that uses promises (either built-in or as an external lib)? If so users can use that if they want. Does it suck for the maintainer? I don't see how if those functions are small and don't really need any maintenance.
What would suck is if all the existing programs that wrote some fs-using code a long time ago and haven't had to touch it now suddenly need to go update it to satisfy people's fashions.
You might be interested in this talk that makes the case to stop breaking your API especially if you're requiring more and providing less. https://www.youtube.com/watch?v=oyLBGkS5ICk
Sure, some people definitely upgrade. But there's a big difference between requiring someone to actively hear and heed your warning and having them automatically upgrade whenever they create a newer project. Or even just prompting them to upgrade like NPM does for package vulnerabilities.
> For the users who don't upgrade, they're not likely to upgrade a third party / split off thing any more than the core language.
That's not the point. If your standard library is a package, any new project will automatically download the latest version of the standard library. If I have a vulnerability in an npm package, I simply push a new version and people will automatically download that new version when they create a new project. If there is a vulnerability in the Python standard library, users will continue to have that vulnerability until they actively patch it.
> Sucks for whom? I've been out of the Node ecosystem for several years, is there a pfs module that uses promises (either built-in or as an external lib)? If so users can use that if they want. Does it suck for the maintainer? I don't see how if those functions are small and don't really need any maintenance.
Not a canonical one. They're starting to add a new promise based API, but now we have two different APIs doing the same thing. What if we come up with a newer way to do async? Do we add yet another API?
> What would suck is if all the existing programs that wrote some fs-using code a long time ago and haven't had to touch it now suddenly need to go update it to satisfy people's fashions.
That's a bit of a straw man. With packages, you can still support multiple versions. It's just that fs would no longer be tied to Node. You could make a newer fs package, fs 2.0 or whatever, that uses promises. The older one would still be supported, of course, but it wouldn't be stuck in Node.
Perhaps I'm misreading this, but isn't that the purpose of package managers? I can run an `npm install foo` and update it at will.
Sure you are going to need to redo code anyways from scratch eventually. But, programmers like I mentioned, which is surprisingly a huge chunk, make problems worse for themselves throughout the lifetime of the code by being short sighted and stamping their approval on code their too lazy to rewrite because their boss doesn't know any better.
For speed, pypy is the way to go.
I may have had to clean up in the aftermath a couple of times.
Well, yeah, of course. It's expensive to reinvent the wheel.
Recently started using Go a little, and it does feel better but it just takes longer for me to write (probably because it's newer to me too). At work, people want things quickly. I'd love to be able to spend my time writing things properly in Go, but it's just not as quick and easy as Python is.
If an explanation boosts your ego, there is a decent chance it is flawed or incomplete.
I have to develop enterprise software that runs across a wide range of platforms, and being able to take advantage of the fact that Python is pre-installed on all of these systems with its standard library is a godsend.
https://www.google.com/search?q=site:golang.org/pkg/+"frozen...
Third party package problems will be an issue at some point but that’s more due to them be so focused on minimalism they don’t have clear guidance on setting up and maintaining packages.
Why? Not only Go. Clojure as well. Clojure's standard library is incredibly consistent and stable. Documentation pretty dry though. That's the only complain I can think of.
I think any discussion about NPM or JS packaging compared to other package managers needs to discuss those things as orthogonal, largely unrelated concepts. Otherwise everyone just picks a favorite punching bag (e.g. left-pad) and talks past each other.
https://groups.google.com/forum/?hl=en#!topic/nodejs/erDWyS4...
NPM installs (and downloads?) a copy of each package for each project. This is the wrong thing to do.
I'm cautiously hopeful on the direction that .Net Core is taking, with the major pieces of the old framework modularized as nuget packages.
People leave all the time during talks, and I've been to talks where most people left as soon as Q&A started.
Perl has a much better set of modules that extend standard functionality, which considering how much flack Perl gets for being hard to read, is rather funny. Rather than every new feature being its own independent project, most of the useful modules inherit a parent and follow the same convention, leading to very simple and easy to use extensions. And Perl Core isn't all that great, but it does have some batteries included, and everything else is extended easily and in a more standard manner by CPAN.
Sure, I've found some gems on CPAN, but, having worked on both Perl, Python, and Java at reasonable scale for awhile, I cannot understand all the praise CPAN gets. It's the worst-quality scripting language package ecosystem out there. Even NPM does a better job, and some things about NPM are awful. CPAN might have been the first/only/best package manager for a get-shit-done scripting language at some point, but not any more.
Separately, I agree about modules which extend language functionality (e.g. class systems, async programming, runtime typing) specifically. Perl does pretty well in that area. While many of those language-extension modules really don't play well with any other metaprogramming tools being installed in the project, I don't imagine that any alternatives in other languages do, either. My main beef above is with "simple" (read: not pervasive semantics changes) modules like IPC utilities, HTTP clients, or loggers that don't know how to stay in their lanes.
Search engines are a "cool" technology that have become the de facto way to find what you're looking for. But if there's a lot of content related to what you're looking for, they can suck.
Go to PyPI and search for "semantic version". 10,000+ projects for "semantic version" found. As you go through page after page of different modules related to versioning, the one module you won't find immediately is Versio (https://pypi.org/project/Versio/), a well-documented and useful module which I ended up using. I have no idea how I found this module, but it certainly wasn't from PyPI's search engine.
Now go to CPAN (really metacpan) and search for "semantic version". Yes, you're still looking at thousands of results - but wait! There are only two modules here that look useful: Version::Dotted::Semantic, and SemVer. And the description comes straight from the docs' README, rather than being a short uninformative blurb. The first module, Version::Dotted::Semantic, is inheriting a separate module, Version::Dotted, and adding some extra functionality. Not only does the search page give more information about the module, but the hierarchy makes it easier to find (and later extend) useful modules in an intuitive way. Since the base module's functionality is boring, generic, and simple, it's less likely that people will make 20 different versions of it, so it'll be reused more often and thus remain stable for a long time.
A lot of CPAN's module names have sprawled over time and gotten less useful, but there's still a general convention that you name your module as a hierarchy of what it does (even if it's kind of verbose) and make small, reusable modules, rather than giant modules that are hard to extend. Not all modules measure up to this standard, and there's definitely room to improve, but I think Python modules could benefit greatly from a system like this.
As far as curation goes, PyPI is often filled with cruft. While searching for Jenkins packages, you will come across lots of entries like this: https://pypi.org/project/jenkins2api/. The homepage leads to a GitHub 404, it's only ever had one release, and it has no documentation. This project should probably not have been listed on the main search page, or at least sorted well down the list by default with intelligent filters and marked accordingly. (The "date last updated" and "trending" sorting just results in having virtually no Jenkins-related modules in the results at all)
> Van Rossum argued instead that if the Twisted team wants the ecosystem to evolve, they should stop supporting older Python versions and force users to upgrade. Brown acknowledged this point, but said half of Twisted users are still on Python 2 and it is difficult to abandon them. The debate at this point became personal for Van Rossum, and he left angrily.
The end of life date is already set: January 1, 2020.
It's open source so I don't know what you're looking for in terms of a formal handover. Yes, I bet Red Hat and others will continue to maintain their own versions past that date.
To ensure things move along: pip has been printing highly-visible "python 2.7 will deprecate soon" warnings for a couple months or so now.
This is actually an interesting paradox to be in, and one that Linus Torvalds recently commented on. His focus, like Guido’s, is the user and even fixing a bug can break the user.
That's how you get an inconsistent mess that never evolves. There's something called semver, increase the version number and do the fix / refactors / radical redesign / whatever. People will see that you've went from version 1.0 to 87.3 in one year and they may choose not to use your thing because you're moving too fast for them, but that's life...
I think this goes to her point of Twisted wanting to support really old versions of Python and they would be a lot more comfortable not supporting really old versions of packages.
Guido should stop acting like a child. Listening to people, hearing them out - even when it’s uncomfortable - is the mark of a good leader.
I tell new PMs “this is the best job in the world 90% of the time but the other 10% is eating shit with a smile”
There is always a new person coming with a new idea. It's exhausting, because it's required to make the language evolve, but it's also a new opportunity to screw things up evertime.
And if he had let most people got their way during the last 2 decades, python would have ended just meh.
Of course, everytime a new debate starts, everybody thinks that this time, just this time, he is wrong and they are right. I did too. We are all part of it.
I get the reaction. There is a limit to what a person can take, and it's why he stepped down as a bdfl.
But seing your baby and your reputation at stake is hard.
The language itself is inferior in expressiveness and performance to almost any other modern language. The only reason anyone uses it anymore is the network effects of the library are very strong, especially is fields relating to ML and data science.
Did asyncio module feel bloated? It certainly did. It seems like every module from subprocess to networking to io is crammed into it.
On the other hand, did it get the job done without resorting to any packages or threading? Yep, and that is pretty powerful and rare.
I think bloatedness of the stdlib isn't actually a practical problem. It's just an inelegance that you kind of have to learn to tolerate.
However brown has solid good points.
The brilliant selling point of python is the massive standard lib. If the quality of the libraries fall, then python's use as a tool drops dramatically.
One of Node's massive failures is that is has no standard lib.
I've been astounded how few non-trivial packages are actually in a consumable state, and how many seemingly-simple packages have N dependencies that pull in M more. By and large it's a zombie wasteland of cruft.
I wouldn't really trust only-crowdsourced ratings, but I think that might be a nice component for npms.io to include, perhaps. Stackoverflow answers, for example, seem to be directionally correct if you sort by upvote count.
If a package is really niche, it may not make sense to put in the integration work. But for a package that is used by a significant majority in a general application - why would you want to keep it separate if it is so much better?
I am ignoring human interaction here - there are probably of dozens of answers to that question if you count personal motivations.
Seems like a false dilemma to me. The core team could still maintain "blessed" packages that don't ship with the default installation.
> why would you want to keep it separate if it is so much better?
This is addressed in the article, most of the 5th paragraph is dedicated to it.
Put another way, the whole Python universe from my point of view has become a cautionary tale about breaking changes. Given Python's popularity, this might be an unpopular opinion, but I have yet to find someone who loves Python who still loves it as much when they discover other newer languages (I'm sure you exist, I just haven't met you!).
Python is having its time in the sun really because it is a default install for most unix distributions, so even people stuck in government labs can use it because 2.7 is already installed. Even apt depends on it via the debian software-properties package, so it isn't going anywhere any time soon.
The real question is how many people would use Python if it was as little known as, say, Elixir.
https://pythonclock.org/ has Py2 reaching EOL in 7 months. Realistically I'd say the time passed years ago.
py3 is over ten years old now. We're not talking about some new unstable piece of kit, I'd imagine that a large percentage, perhaps even 50%, of the HN audience started their career after the transition had already started.
I think there is in general nothing wrong with a language ecosystem where key parts of the whole platform are in well-maintained third-party libraries rather than the standard library. Which is also something Amber Brown is saying here.
I’m not sure if Python’s ideal solution is to reduce stdlib and have endorsed packages in PyPI, but it would be an improvement over the current process.
I remember one of Go's core devs voicing some of the same concerns regarding the SMTP-library [0] a while back.
I've never liked asyncio, while Twisted felt natural to me. So I would agree that an inferior solution has been pushed heavily in the stdlib and also to the syntax level.
Moving the entire stdlib to PyPI is of course entirely foolish and would destroy Python.
In certain situations, I've been working with a python interpreter on a RHEL machine where pip was not installed (and I was not allowed to install it as well as make other modification: the machine was owned by the client and I had to work with what I had available).
- having some basic functionality in the core libraries was a godsend because I could work with that, even though it was not "ergonomic"
- not being ergonomic, it was a "poor experience" (and certainly not optimized or anything nice to see).
This heading is the essential problem in innovation writ large: some giant can ignore you and squash you without any effort at all, without even considering your existence.
Right now the equilibrium in this tug-of-war is that a certain set of functionality comes by default in the python standard library and everything else is just a package that you can install.
Obviously from a dev point of view it's a hassle to have to decide which of two or more packages for X is best, the pythonic way would be that there should be one and only one package for X. Of course at the cutting edge there have to be competing packages because there needs to be room for innovation.
But obviously not everything can be in the python standard library.
Not really sure what the solution is, but maybe there should be tiers of packages, with "Tier 1" being standard library, "Tier 2" having some kind of official stamp that it has been security audited to a certain standard, that Python has some control over who gets to modify it and why etc. Then maybe "Tier 3" could cover everything else, i.e. any bob random can go make a package on PyPi and it's Tier 3.
In addition, the process of going from Tier 3 to Tier 2 would give people a chance to winnow libraries down to one way of doing each thing at the Tier 2 level.
This might not be realistic but it's what my gut is telling me. C&C welcome.
PHP has a similar issue to this. The people writing C were not using the language. The best example is PDO. A lot of C was written, but it was essentially abandonware because the PHP users could not make any changes without getting the C maintainers to both agree and have the time.
Force the upgrade already. Code still on 2.7 if still useful can be upgraded.
My "favourite" library quirk. socket.fromfd is only available on "Unix" on Python 2.X, that was fixed in Python 3.X
The worse offender being the logging library. It's the least pythonic thing in the whole std library (ok maybe ABC is worse, but oh well)