Ski 1.4.0 is out!

August 20, 2022

TL;DR: ski-1.4.0 is available for download!

What’s new

I’ll paste NEWS entry as is:

It's a new fork based on lates 1.3.2 version available. It's new home
is https://github.com/trofi/ski. There are many changes since 1.1.0.
We'll list a few here since 1.3.2:

** https://github.com/trofi/ski is a new Ski home.

** Build is fixed to work with modern glibc, binutils and linux kernel
headers versions.

** More linux syscalls are emulated to at least get basic printf() to
work. More work needs to be done before Ski user emulation can run real
world programs.

** Dropped outdated ski.spec from tarball.

** Dropped ski-config from installed files. Ski does not provide a
stable library interface.

** Dropped dynamic hook loading infrastructure support. Ski no longer
tries to load and execute any libraries via 'SKIHOOK_PATH'. LTDL is
not required anymore.

** libski is not installed any more and is considered an internal
implementation detail of Ski.

While preparing this release I had a lot of fun fiddling with autoconf over past few hours. I hope I did not break Ski too much to and it still builds for others.

Ski status

Ski supports guest linux kernels up to 4.19. Anything later does not yet work as HP_SIM mode was removed from upstream kernel. 4.19 kernel support is great though: you can run gdb, networking, gcc and many other conventional tools right in the guest VM.

User mode emulation on the other hand is very weak. The “Hello World” does run when built against modern glibc, but I expect many basic things (like bash) to be still broken. It’s good enough for proof of concept, but expect to fix many bugs if you want to run a real application.

Text xterm interface (ski/bski) seems to work great. motif (xski) and gtk (gski) interfaces look completely broken. I fixed their compilation, but they render something unusable. I’m not sure if it was ever functional enough to be usable. If I get a bit more confidence in that I’ll probably just remove xski / gski in next release.

More words

Previous Ski release was out in February 2008, 14 years ago. A few things changed in the world since then. Incomplete list:

Sounds bad, eh?

Fear not! Things are not as grim as they might look. Lack of real hardware access happens to be a good motivation for me to clear the dust off good old Ski. Lack of new CPUs means that it’s not a moving target anymore (not that it ever did :) and we can just build a perfect (ahem) emulator.

I still have about 20 minor kernel patches to upstream and would like to get an emulator to support latest kernels to test these patches.

Pre-built OS images

A few months ago someone asked me if I had OS images left for Ski to run. I did not have them and spent a few minutes recreating them. The result is here:

It has an userspace with gcc-11.3.0. Not too old.

Nixpkgs attempt at building OS images

Gentoo’s support for large scale cross-compilation is a bit hairy. I wondered if I could make OS image building based on nixpkgs to reuse all my local hacks I use to tinker with the system.

A few days ago matoro asked me to send the fix matoro did for ITC precision detection upstream. I took it as an opportunity to bring up an equivalent test environment on NixOS locally (and to find out how ia64 timers really work in linux).

Initially I built just an ia64-unknown-linux-gnu cross-compiler. It took 3 lines of nixpkgs code:

lib/systems/examples.nix:  ia64 = { config = "ia64-unknown-linux-gnu"; };
lib/systems/inspect.nix:    isIa64         = { cpu = { family = "ia64"; }; };
lib/systems/parse.nix:    ia64     = { bits = 64; significantByte = littleEndian; family = "ia64"; };

That (and a few small one-off tweaks) allowed me to build vast majority of packages for ia64 with one command: bash, strace, mc, busybox, gcc, iproute2, tmux, git, util-linux, gdb. re2c building example:

$ nix build -f. pkgsCross.ia64.re2c

$ file result/bin/re2c
result/bin/re2c: ELF 64-bit LSB executable, IA-64, version 1 (SYSV), dynamically linked,
  interpreter /nix/store/...-glibc-ia64-unknown-linux-gnu-2.35-163/lib/ld-linux-ia64.so.2,
  for GNU/Linux 2.6.32, not stripped

Kernel was slightly harder to get running as it required 2 CONFIG_ options to be disabled to get something bootable:

# lib/systems/platforms.nix:

  ia64 = {
    linux-kernel = {
      name = "ia64";
      target = "vmlinuz";
      autoModules = false;
      baseConfig = "sim_defconfig";
      extraConfig = ''
        # Disable for gcc bug: https://gcc.gnu.org/PR106617
        LIBFC n

        # otherwise lerken panics in SKI
        SERIAL_8250 n
      '';
    };
  };

Here the main bit is sim_defconfig for HP_SIM mode and a SERIAL_8250 n workaround to prevent the kernel from crashing. Oh, and it has to be a 4.19 kernel (5.0 removed HP_SIM mode), thus the command to build the kernel is: nix build -f. pkgsCross.ia64.linuxPackages_4_19.kernel.

As I was foolish enough to use very fresh gcc I encountered a gcc bug: https://gcc.gnu.org/PR106617.

The above was good enogh to put together minimal runnable OS image. I wondered if I could also generate full rootfs as a .nix expression to later use it as is for making an ext2 image. The trick is to grab full closure of dependencies and put it in a single directory. tpw_rules pointed me at a closureInfo magic function which allows doing exactly that kind of things.

Just one rsync call is enough to build full rootfs! Here is my .nix expression to get a virtual machine with busybox, re2c, ski run script and /init script to boot a full VM:

# $ cat busybox-and-re2c.nix
{ pkgs ? import ~/n {}
, pkgsCross ? pkgs.pkgsCross.ia64
, targetKernel ? pkgsCross.linuxPackages_4_19.kernel
}:

rec {
  init-script = pkgs.writeScriptBin "init" (''
    #!${pkgsCross.busybox}/bin/busybox sh

    # expose both packages in the PATH
    export PATH=${pkgsCross.re2c}/bin:${pkgsCross.busybox}/bin

    # prepare file hierarchy
    mkdir -p /dev /proc
    mount -t proc proc /proc
    mount -t devtmpfs devtmpfs /dev

    # run the shell
    exec ${pkgsCross.busybox}/bin/busybox sh
  '');

  rootfs = pkgs.runCommand "rootfs" { closure = pkgs.closureInfo { rootPaths = [ init-script ]; }; } ''
    mkdir -p $out
    cd $out

    # Copy full closure locally
    ${pkgs.rsync}/bin/rsync -arv --files-from=$closure/store-paths / .

    # build convenience symlinks:
    ln -s ${init-script}/bin/init init
  '';

  run-script = pkgs.writeScriptBin "run-ia64-ski" ''
    #!${pkgs.bash}/bin/bash
    bski=${pkgs.ski}/bin/bski
    bootloader=${targetKernel.dev}/bootloader
    vmlinux=${targetKernel.dev}/vmlinux
    rootfs=${rootfs}
    image_size=1G
    image=./sdc

    set -e

    if [ ! -e "$image" ]; then
        echo "Populating '$image' with '$rootfs'..."
        fallocate -l "$image_size" "$image"
        mke2fs -d ${rootfs}/ "$image"
    fi

    $bski $bootloader $vmlinux root=/dev/sda simscsi=./sd init=/init rw TERM=xterm "$@"
  '';
}

We define 3 build targets above:

The whole thing can be built and ran in a single command as:

$ $(nix-build busybox-and-re2c.nix -A run-script)/bin/run-ia64-ski

loading /nix/store/jiqn22d0yp80vvysib16xjnb8w6j7c1v-linux-ia64-unknown-linux-gnu-4.19.255-dev/vmlinux...
starting kernel...
Linux version 4.19.255 (nixbld@localhost) (gcc version 13.0.0 20220814 (experimental) (GCC)) #2 SMP Fri Aug 19 07:53:12 UTC 2022
EFI v1.00 by Hewlett-Packard:
...
Run /init as init process
sh: can't access tty; job control turned off
/ # random: crng init done

/ # mount
/dev/root on / type ext4 (rw,relatime)
proc on /proc type proc (rw,relatime)
devtmpfs on /dev type devtmpfs (rw,relatime,size=53184k,nr_inodes=831,mode=755)

/ # ls /
dev         init        lost+found  nix         proc

/ # ls -l /
total 20
drwxr-xr-x    3 0        0             2300 Aug 20 00:29 dev
lrwxrwxrwx    1 0        0               57 Jan  1  1970 init -> /nix/store/ri7n3q59q042ddgzdanrsg5l4jrs85b8-init/bin/init
drwx------    2 0        0            16384 Aug 20 00:29 lost+found
dr-xr-xr-x    3 0        0             4096 Jan  1  1970 nix
dr-xr-xr-x   92 0        0                0 Aug 20 00:29 proc

/ # du -hs /nix/store/*
4.7M    /nix/store/1mrb68yaa7cn2x8lzq13vgd4gy6icz2i-re2c-ia64-unknown-linux-gnu-3.0
12.0K   /nix/store/ri7n3q59q042ddgzdanrsg5l4jrs85b8-init
31.9M   /nix/store/rr5mdcdw22ab7k78i6z6laywxdflhsj9-glibc-ia64-unknown-linux-gnu-2.35-163
6.3M    /nix/store/x83rlqm5ww1d6ggdkxlv48l0rzb0gwjc-ia64-unknown-linux-gnu-stage-final-gcc-13.0.0-lib
2.7M    /nix/store/xhkazsq7p21kj6853vfv7w244bbfa854-busybox-ia64-unknown-linux-gnu-1.35.0

/ # re2c --version
re2c 3.0

Just 44MB uncompressed! We can shrink it down to 9MB by compressing it:

$ xz -9 sdc
$ LANG=C ls -lh
-rw-r--r-- 1 slyfox users 8.6M Aug 20 08:02 sdc.xz

That’s even smaller than uncompressed kernel, which is around 21MB:

$ ls -lh /nix/store/jiqn22d0yp80vvysib16xjnb8w6j7c1v-linux-ia64-unknown-linux-gnu-4.19.255-dev/vmlinux |unnix
-r-xr-xr-x 2 root root 21M Jan  1  1970 /<<NIX>>/linux-ia64-unknown-linux-gnu-4.19.255-dev/vmlinux

You can also inspect rootfs from the host side without having to deal with filesystem image or booting an OS:

$ nix-build busybox-and-re2c.nix -A rootfs
/nix/store/11ig1jh624l5dniilcp2p8s2zjigrz44-rootfs

$ ls -1 result/nix/store/
1mrb68yaa7cn2x8lzq13vgd4gy6icz2i-re2c-ia64-unknown-linux-gnu-3.0
ri7n3q59q042ddgzdanrsg5l4jrs85b8-init
rr5mdcdw22ab7k78i6z6laywxdflhsj9-glibc-ia64-unknown-linux-gnu-2.35-163
x83rlqm5ww1d6ggdkxlv48l0rzb0gwjc-ia64-unknown-linux-gnu-stage-final-gcc-13.0.0-lib
xhkazsq7p21kj6853vfv7w244bbfa854-busybox-ia64-unknown-linux-gnu-1.35.0

That alone allowed me to fix a bunch of packages in nixpkgs (example upstream mc fix).

It’s such a fun toy to play with!

HP_SIM mode limitations

To do a full system emulation Ski relies on HP_SIM mode of kernel build. It’s very different from the real ia64 kernel you build for a real device. And different from KVM! A few points on why HP_SIM is special:

Possible next steps for Ski

The above are not fundamental limitations. I think it would not be hard to get rid of custom device code and switch over to virtio-mmio entirely. I imagine it would be something like:

It will require a bit of code on Ski side to handle virtio queues. Maybe we can borrow parts of qemu as is? Otherwise implementing them does not sound too hard.

Lack of buses is fine as well. VMs for other architectures seem to do it for ages and rely on device trees or manual kernel config options. We can do there same here. It would be useful though to get closer to real ia64 for interface discovery eventually.

Lack of platform initialization code should not be a big problem either: ia64 paltform is defined in terms of PAL and SAL calls which are very similar (at least in spirit) to current HP_SIM hypercalls.

Parting words

Ski is still alive! Try 1.4.0 :)

ia64 code can still be emulated and be debugged with Ski. Both can still expose generic bugs in various projects.

Ski is a great example of project on how little one needs to add to guest linux kernel and to hypervisor to boot the kernel and get userspace running there.

Instruction emulation is a large part of Ski. I still don’t quite get it. But I also never had a reason for deeper investigation. Maybe one day. Meanwhile it just works!

Have fun!