xlibsWrapper yak

December 27, 2022

During one of the raids on stale nixpkgs packages I found Xaw3d package which was present in two forms in nixpkgs: Xaw3d and xorg.libXaw3d. Both were based on libXaw3d Xorg package of the same version and used different definitions of the package. xorg.libXaw3d was used by a single broken timedoctor package. Thus, it was easy to just delete both:

Duplicate packages are very confusing. New namespaces like xorg.* make it very easy to introduce subtly different packages.

A while later I attempted to patch xorg.libXfont2 package via an override locally:

final: prev: {
  xorg = prev.xorg.overrideScope' (f: p: {
    libXfont2 = p.libXfont2.overrideAttrs (oa: {
      patches = (oa.patches or []) ++ [
        ../libXfont2/nofollow.patch
      ];
    });
  });
}

That was supposed to work. It didn’t: pulseaudio expression became invalid and complained about missing xorg.xlibsWrapper attribute. Despite the fact I seemingly did not touch it in my expression.

Here is the xorg attribute definition:

   # somewhere in pkgs/top-level/all-packages.nix
   # ...
   xorg = recurseIntoAttrs ((lib.callPackageWith __splicedPackages ../servers/x11/xorg {
   }).overrideScope' (lib.callPackageWith __splicedPackages ../servers/x11/xorg/overrides.nix {
     inherit (darwin.apple_sdk.frameworks) ApplicationServices Carbon Cocoa;
     inherit (darwin.apple_sdk.libs) Xplugin;
     inherit (buildPackages.darwin) bootstrap_cmds;
     udev = if stdenv.isLinux then udev else null;
     libdrm = if stdenv.isLinux then libdrm else null;
     abiCompat = config.xorg.abiCompat # `config` because we have no `xorg.override`
       or (if stdenv.isDarwin then "1.18" else null); # 1.19 needs fixing on Darwin
   }) // { inherit xlibsWrapper; } );

Note that unusual // { inherit xlibsWrapper; } last minute attribute addition. It injects xlibsWrapper attribute after xorg was fully constructed with all it’s overrides. Attempt to redefine xorg attribute (as I did in an overlay above) throws xlibsWrapper attribute away. It has to be added back explicitly. I would say it’s unexpected.

More interestingly not every package used xorg.xlibsWrapper form to refer to the thing! Most packages used xlibsWrapper directly. Thus, I removed xorg.xlibsWrapper in favor of a global one in PR#194044. The fix had to adjust only 5 packages (gnome2.libgnomeui, pulseaudio, ted,vlc,xine-ui) and turned xorg into a proper overridable scope:

--- a/pkgs/top-level/all-packages.nix
+++ b/pkgs/top-level/all-packages.nix
@@ -24104,21 +24104,21 @@ with pkgs;
   # have created a cycle.
   xorg = recurseIntoAttrs ((lib.callPackageWith __splicedPackages ../servers/x11/xorg {
   }).overrideScope' (lib.callPackageWith __splicedPackages ../servers/x11/xorg/overrides.nix {
     inherit (darwin.apple_sdk.frameworks) ApplicationServices Carbon Cocoa;
     inherit (darwin.apple_sdk.libs) Xplugin;
     inherit (buildPackages.darwin) bootstrap_cmds;
     udev = if stdenv.isLinux then udev else null;
     libdrm = if stdenv.isLinux then libdrm else null;
     abiCompat = config.xorg.abiCompat # `config` because we have no `xorg.override`
       or (if stdenv.isDarwin then "1.18" else null); # 1.19 needs fixing on Darwin
-  }) // { inherit xlibsWrapper; } );
+  }));

So much better now. And then I looked at xlibsWrapper definition out of curiosity:

  # somewhere in pkgs/top-level/all-packages.nix
  # ...
  # Avoid using this. It isn't really a wrapper anymore, but we keep the name.
  xlibsWrapper = callPackage ../development/libraries/xlibs-wrapper {
    packages = [
      freetype fontconfig xorg.xorgproto xorg.libX11 xorg.libXt
      xorg.libXft xorg.libXext xorg.libSM xorg.libICE
    ];
  };
# somewhere in pkgs/development/libraries/xlibs-wrapper/default.nix
{lib, stdenv, packages}:

stdenv.mkDerivation {
  name = "xlibs-wrapper";

  dontBuild = true;

  installPhase = "mkdir -p $out";
  unpackPhase = "sourceRoot=.";

  propagatedBuildInputs = packages;

  preferLocalBuild = true;

  # For compatability with XFree86.
  passthru.buildClientLibs = true;

  meta = {
    platforms = lib.platforms.unix;
  };
}

The above says that:

xlibsWrapper is a remnant of the far past when Xorg became modular (around 7.0 version in 2005) where it’s single tarball release was split into ~200 smaller packages. nixpkgs kept this transitional package to ease porting to the new set of packages.

To delete xlibsWrapper completely I filed PR#194054 to track and update packages to set of actually used libraries one by one. 3 months (and 80 PRs) later I can finally say that all users were moved to finer grained set of libraries!

diffoscope helped me a lot while working on the transition: if binaries before and after the change look the same then chances are high I did not break any users.

Another yak became a bit less hairy.

There is still a lot to do: I think it is time to remove xorg.* attribute set and merge individual packages back into the top-level attribute set. This will turn xorg.* into normal packages and will allow updating them just like anything else. I filed Issue#207978 to track the effort.

Parting words

nixpkgs is full of bits with accidental complexity like xlibsWrapper or autogenerated set of xorg packages. Or long deprecated packages like llvm-5 (it still has a few reverse dependencies believe it or not). They are usually easy to fix, just take some effort to sort out all the numerous users.

Fun fact: it takes about 3 months of slow pace to sort out 80 packages. Something like 1 package a day. Sounds like a good estimate of large-scale changes.

Have fun!