Design for the build system

Inspired by buildsome, the awesome build system, we can outline what a new build system would look like. The main technology is FUSE, as suggested in this issue, because it's the only reasonable way to globally hook filesystem access. We mount a basic pass-through FUSE file system over / and then just run whatever commands we want. FUSE traps all requests and then we can get the pid of the process using fuse_req_ctx or fuse_get_context which then through /proc gives us more useful information like process start time (pid + start time is unique), process command line (/proc/PID/cmdline), process environment (/proc/PID/environ), and process working directory (/proc/PID/cwd - this last is only valid before the process changes it, but the first thing any program does is read its executable files so we can trap it early). We also can get the parent process id, to recover the tree of subprocesses.

But just running commands isn't enough, we also have to handle rebuilds. When re-executing a process we must first roll back all the changes it and its sub-processes made, in order to ensure a clean build. To ensure an isolated build, we must also hide all the changes that were made by other processes that run after the re-executed process. These two steps should suffice for most purposes to enable deterministic builds, but of course more work can be done.

Next we introduce caching; as opposed to rebuilding, caching means that we can skip the build in the first place by downloading the pre-built result instead. For this we need a content-addressed store for all of the build files etc. and to store the metadata on which commands use which files. Also we need a special tool to run commands with, which checks if the command is cached before running it.

Finally we have to solve the problem of dependency hell. To summarize this is the situation where you have a library that gets updated but an application that depends on the library breaks on the update. The easiest solution is to do like Nix and install libraries into their own hashed directories, so that the new library goes in a new directory while the old directory sticks around and has the application still use it. But this breaks compatibility with a lot of software because the directory paths have to be hard-coded into the application somehow. A better solution is to use filesystem traps: a process that accesses the application files has a special label applied to it, so that when it later on accesses the library the label makes it see the old version of the library rather than the new library. This might(?) be implementable as another FUSE filesystem but an easier method is to just replace all executables with setcap CAP_SYS_ADMIN wrappers that mount an overlayfs on top and then exec the process.