pip's resolving algorithm is not sound. If your Python projects are really simple it seems to work but as your projects get more complex the failure rate creeps up over time. You might
pip install
something and have it fail and then go back to zero and restart and have it work but at some point that will fail. conda has a correct resolving algorithm but the packages are out of date and add about as many quality problems as they fix.
I worked at a place where the engineering manager was absolutely exasperated with the problems we were having with building and deploying AI/ML software in Python. I had figured out pretty much all the problems after about nine months and had developed a 'wheelhouse' procedure for building our system reliably, but it was too late.
Not long after I sketched out a system that was a lot like uv but it was written in Python and thus had problems with maintaining its own stable Python enivronment (e.g. poetry seems to trash itself every six months or so.)
Writing uv in Rust was genius because it eliminates that problem of the system having a stable surface to stand on instead of pipping itself into oblivion, never mind that it is much faster than my system would have been. (My system had the extra feature that it used http range requests to extract the metadata from wheel files before pypi started letting you download the metadata directly.)
I didn't go forward with developing it because I argued with a lot of people who, like you, thought it was "the perfect being the enemy of the good" when it was really "the incorrect being the enemy of the correct." I'd worked on plenty of projects where I was right about the technology and wrong about the politics and I am so happy that uv has saved the Python community from itself.
May I introduce you to our lord and saviour, Nix and it's most holy child nixpkgs! With only a small tithing of your sanity and ability to Interop with any other dependency management you can free yourself of all dependency woes forever!
Except python builders in nixpkgs are really brain damaged because of the writers ways they inject search path which for example breaks if you try to execute a separate python interpreter assuming same library environment...
Within the holy church of Nix the sect of Python is troubled one, it can however be tamed into use via vast tomes of scripture. Sadly these times can only be written by those you have truly given their mind and body over to the almighty Nix.
Nix is really the best experience I've had with Python package management but only if all the dependencies are already in nixpkgs. If you want to quickly try something off github it's usually a pain in the ass.
Ugh, I hate writing this but that's where docker and microservices comes to the rescue. It's a pain in the butt and inefficient to run but if you don't care about the overhead (and if you do care, why are you still using Python?), it works.
My experience was that docker was a tool data scientists would use to speedrun the process of finding broken Pythons. For instance we'd inexplicably find a Python had Hungarian as the default charset, etc.
If you can write a Dockerfile to install something you can write a bash script. Circa 2006 I was running web servers on both Linux and Windows with hundreds of web sites on them with various databases, etc. It really was simple then as "configure a filesystem path" and "configure a database connection" and I had scripts that could create a site in 30 seconds or so.
Sure today you might have five or six different databases for a site but it's not that different in my mind. Having way too many different versions of things installed is a vice, not a virtue.
> If you can write a Dockerfile to install something you can write a bash script.
Docker is great for making sure that magic bash script that brings the system up actually works again on someone else’s computer or after a big upgrade on your dev machine or whatever.
So many custom build scripts I’ve run into over the years have some kind of unstated dependency on the initial system they were written on, or explicit dependencies on something tricky to install, and as such are really annoying to diagnose later on, especially if they make significant system changes.
Docker is strictly better than a folder full of bash scripts and a Readme.txt. I would have loved having it when I had to operate servers like that with tons of websites running on them. So much nicer to be able to manage dependency upgrades per-site rather than server-wide, invariably causing something to quietly break on one of 200 sites.
Unspoken libc dependencies are my favorite. Granted, you need to wait a few years after launching the project to feel that pain, but once you’re there, the experience is… unforgettable.
Second best are OpenSSL dependencies. I sincerely hope I won’t have to deal with that again.
Unfortunately sometimes you get to host things not written by you, or which exist for a long time and thus there's a lot of history involved that prevents nice and tidy.
My first production use of kubernetes started out because we put in the entirety of what we had to migrate to new hosting into spreadsheet, with columns for various parts of stack used by the websites, and figured we would go insane trying to pack it up - or we would lose the contract because we would be as expensive as the last company.
Could we package it nicely without docker? Yes, but the effort to package it in docker was smaller than packaging it in a way where it wouldn't conflict on a single host, because the simple script becomes way harder when you need to handle multiple versions of the same package, something that most distro do not support at all (these days I think we could have done it with NixOS, but that's a different kettle of deranged fishes)
And then the complexity of managing the stack was quickly made easier by turning each site into separate artifact (docker container) handled by k8s manifests (especially when it came to dealing with about 1000 domains across those apps).
So, theoretically discipline is enough, practical world is much dirtier though.
> If you can write a Dockerfile to install something you can write a bash script.
The trick isn't installing things, it's uninstalling them. Docker container is isolated in ways your bash script equivalent is not - particularly when first developing it, when you're bound to make an occasional mistake.
My understanding is that UTF-8 is the world's charset and that reasonable Hungarians would use that (e.g. I sure don't use us-ascii or iso-latin-1 if I can at all help it. I mean my "better half" reads 中文 so I don't have to and having it all in UTF-8 makes it easy) The other mystery is how the data sci's found it.
> You might `pip install` something and have it fail and then go back to zero and restart and have it work but at some point that will fail.
Can you give a concrete example, starting from a fresh venv, that causes a failure that shouldn't happen?
> but it was written in Python and thus had problems with maintaining its own stable Python enivronment
All it has to do is create an environment for itself upon installation which is compatible with its own code, and be written with the capability of installing into other environments (which basically just requires knowing what version of Python it uses and the appropriate paths - the platform and ABI can be assumed to match the tool, because it's running on the same machine).
This is fundamentally what uv is doing, implicitly, by not needing a Python environment to run.
But it's also what the tool I'm developing, Paper, is going to do explicitly.
What's more, you can simulate it just fine with Pip. Of course, that doesn't solve the issues you had with Pip, but it demonstrates that "maintaining its own stable Python environment" is just not a problem.
>Writing uv in Rust was genius because it eliminates that problem of the system having a stable surface to stand on instead of pipping itself into oblivion, never mind that it is much faster than my system would have been.
From what I can tell, the speed mainly comes from algorithmic issues, caching etc. Pip is just slow above and beyond anything Python forces on it.
An example. On my system, creating a new venv from scratch with Pip included (which loads Pip from within its own vendored wheel, which then runs in order to bootstrap itself into the venv) takes just over 3 seconds. Making a new venv without Pip, then asking a separate copy of Pip to install an already downloaded Pip wheel would be about 1.7 seconds. But making that venv and using the actual internal installation logic of Pip (which has been extracted by Pip developer Pradyun Gedam as https://github.com/pypa/installer ) would take about 0.25 seconds. (There's no command-line API for this; in my test environment I just put the `installer` code side by side with a driver script, which is copied from my development work on Paper.) It presumably could be faster still.
I honestly have no idea what Pip is doing the rest of that time. It only needs to unzip an archive and move some files around and perform trivial edits to others.
> (My system had the extra feature that it used http range requests to extract the metadata from wheel files before pypi started letting you download the metadata directly.)
Pip has had this feature for a long time (and it's still there - I think to support legacy projects without wheels, because I think the JSON API won't be able to provide the data since PyPI doesn't build the source packages). It's why the PyPI server supports range requests in the first place.
> I'd worked on plenty of projects where I was right about the technology and wrong about the politics and I am so happy that uv has saved the Python community from itself.
The community's politics are indeed awful. But Rust (or any other language outside of Python) is not needed to solve the problem.
It occurs to me later: `installer` isn't compiling the .py files to .pyc, which probably accounts for the time difference. This can normally be done on demand (or suppressed entirely) but Pip wants to do it up front. Bleh. "Installing" from already-unpacked files would still be much faster.
I worked at a place where the engineering manager was absolutely exasperated with the problems we were having with building and deploying AI/ML software in Python. I had figured out pretty much all the problems after about nine months and had developed a 'wheelhouse' procedure for building our system reliably, but it was too late.
Not long after I sketched out a system that was a lot like uv but it was written in Python and thus had problems with maintaining its own stable Python enivronment (e.g. poetry seems to trash itself every six months or so.)
Writing uv in Rust was genius because it eliminates that problem of the system having a stable surface to stand on instead of pipping itself into oblivion, never mind that it is much faster than my system would have been. (My system had the extra feature that it used http range requests to extract the metadata from wheel files before pypi started letting you download the metadata directly.)
I didn't go forward with developing it because I argued with a lot of people who, like you, thought it was "the perfect being the enemy of the good" when it was really "the incorrect being the enemy of the correct." I'd worked on plenty of projects where I was right about the technology and wrong about the politics and I am so happy that uv has saved the Python community from itself.