Managing virtualenvs with dynamic libraries on NixOS (Part 3)
In the previous articles of this series it has been shown what challenges virtualenv installations on NixOS have and how we automated our approach in our application deployments. This part explains why a Nix-based approach would be desirable and what is keeping us from doing it right now.
A nixified world
To give a little more context, as somebody who comes from Nix, I like the way how packages are built with Nix:
all dependencies, both build & runtime dependencies are declared explicitly, so it's relatively straightforward to spot vulnerable packages in the dependency tree.
When a vulnerability gets patched, it will trigger a rebuild of all dependant packages.
- Nix allows to patch packages within the tree, usually with overlays. This allows to change any dependency inside the tree.
- Ideally, everything would be built from source. That doesn't necessarily mean long compile times though since packages get built by a CI frequently and then pushed to a binary cache. If Nix needs a package and the exact package definition (i.e. the package with the same name, source & dependencies) exists in a binary cache, it will be downloaded from there instead of built.
A better overview over the benefits of Nix-based builds can be found in the manual.
Now, there's in fact a neat tool called poetry2nix which allows to package Python programs using poetry with Nix. With this tool, it's not necessary to define all dependencies by hand, the tool will automatically generate packages from what's inside the lockfile, poetry.lock.
It re-uses the Python package-set from nixpkgs and adjusts source & versions to what is in poetry.lock. E.g. psycopg2 is already packaged in nixpkgs, so poetry2nix will re-use the package definition and adjust source & version to what is in poetry.lock. That way, e.g. libpq.so is correctly found. If psycopg2 wouldn't be a part of nixpkgs, this wouldn't work out. For those packages, there are manual fixups in poetry2nix.
Now, upon throwing poetry2nix against a few real-world applications, I encountered that it's still significantly harder to use than the approach with prebuilt wheels. I identified the following reasons for this:
- This breaks for older applications with older dependencies since downgrading these dependencies would break build-tools that don't get downgraded. To give a concrete example:
- a test application I used required pluggy at 0.13 since the lockfile was a little dated.
when using latest nixpkgs with latest poetry2nix, some packages will use hatchling, a Python build tool, during their package build which depends on pluggy>=1.0.
The modification from poetry.lock subsequently caused hatchling to break.
See also nix-community/poetry2nix#1616.
Using an older nixpkgs or older poetry2nix also isn't a real option since there's a high chance that other fixes are missing.
As mentioned above, there are a lot of packages where manual intervention is needed to actually get a package to build.
Related to this: if a package changes its non-Python dependencies and the newer version is packaged in nixpkgs, then poetry2nix builds using the old version will fail to build since the wrong dependencies are used within the Nix package.
- Some dependencies, especially test libraries, are just missing from package metadata. This is fine for wheel installs, but breaks source-builds.
This can still be a realistic option if the interpreter and all Python dependencies are kept up-to-date and some of Nix's benefits are explicitly needed.
But at the end of the day we rather want to run customer applications than fixing up Python dependencies. There's especially one promising development that may make this easier in the future: PEP725 which adds fields to pyproject.toml to specify non-Python dependencies such as build-tools or libraries to link against on runtime.
These dependencies are specified using the PURL URls.
This will need a mapping from PURL identifiers to packages in nixpkgs. Also, there's still a risk that if a package requires a very old version of e.g. OpenSSL, it won't work with whatever is packaged in nixpkgs. IMHO, this is a point where one clearly should have to write their own overlays, but it's still not obvious where to draw a line between "reasonable version to request" and "having to write an overlay".
