undefined | Better HN

0 pointskalefranz7y ago0 comments

> Is this a situation like grsecurity vs Torvalds?

Ha! Conda and Anaconda, Inc. are _not_ like grsecurity. Starting with the fact the Conda is BSD-licensed.

Recounting some of the history will probably provide the context you're looking for.

It's my understanding that the birth of Anaconda (the distribution) and Conda goes back to this statement by Guido regarding package building and installing at a PyData summit in 2012:

> It really sounds like you guys' needs are so unusual compared to the larger python community that you're just better off building your own.

https://www.youtube.com/watch?v=QjXJLVINsSA&t=59m10s

This predated PEP 427 (Wheel Binary Package Format 1.0).

Of course both conda and pip/wheels have evolved immensely since 2012, and their evolution has been guided by different constraints. Elsewhere in this thread I spoke about how pip/wheels/PyPA packaging has as it's primary target the site-packages directory, and then branches out from there when necessary. Conda's primary target is the prefix root. PyPA's mandate is to build, install, and manage python packages, which makes something like 'pip install python3' out of scope. Things like 'pip install postgresql' and 'pip install r' are _for sure_ out of scope. Conda has the luxury of being able to install and manage all packages, not just python packages.

Regarding the creep of wheels toward conda packages, and especially the static embedding within wheels of compiled "system" libraries, I question whether it's actually to the benefit of python _users_. No doubt the wheel "binary" format is a huge improvement over eggs for pure-python packages. But manylinux wheels are often built going beyond just compiled python extensions. Rather than describing these "system" dependencies in metadata, and then having pip ensure these system dependencies are present before installing the package, these wheels now implicitly cross the python-only line that PyPA in other cases holds. There are real consequences for users, with the result being that it pushes failure mechanisms back rather forward.

So, because of the differences in scope, my guess is that the python community would think it inappropriate to make Conda an official PEP. Conda will probably someday install wheels (at least pure-python ones); maybe someday pip can reach out to conda or apt-get or yum to ask for non-python dependencies to be installed.

> and make conda-forge part of the Python Foundation (instead of a private company)

Conda-forge is actually a community-driven organization completely independent of Anaconda, Inc.

0 comments

craigds7y ago

Binary wheels are a nightmare in terms of knowing what you have installed. We disabled them in our builds when we discovered a few packages had vendored old, terribly-insecure static versions of libxml2 (and who knows what else) which silently replaced using the system version (which received OS security updates). We only found out when we hit one of the behavioural quirks of that version and spent ages investigating it with gdb.

If you're using manylinux wheels you're probably using static libs you have no idea you're using. That way lies madness.

I havent used conda so maybe this doesn't apply there somehow.

RayDonnelly7y ago

We are a rolling Software Distribution and providing up to date software is one of our main goals.

Another is building that software with good security flags, see: https://www.anaconda.com/blog/developer-blog/improved-securi...

We also keep track of CVEs in our software and actively look for patches (e.g. pycrypto is dead now but Debian maintains patches to fix reported CVEs) or write our own (though usually to fix build-system bugs rather than security issues).

But yes, static linking and leaving software building to non-experts using whatever tools they like (without studying anything to do with low-level binary security or how to achieve that), statically linking insecure (some time to become old) libraries is far from ideal.

Anaconda Distribution strongly prefers dynamic linking and shared package dependencies so we can update to address critical security issues without needing to rebuild significant portions of our stack.

sandGorgon7y ago

> my guess is that the python community would think it inappropriate to make Conda an official PEP.

will you guys try ? I really, really, really hope you do.

j / k navigate · click thread line to collapse