Continuous delivery and distribution

Software development suffers from lag; every change must be made, then compiled, then tested in few dozen ways, before the next change can be made. The faster this process completes, the faster development can move forward. Non-incremental development, where changes are made without testing or compiling, leads to slower development times overall due to the numerous regressions that are introduced and have to be tracked down later.

Nix encourages incremental development by making it easy to download and use software; it creates a uniform interface for adding libraries and other dependencies, in the .nix file. But it also slows down development because the build process itself is non-incremental; it has to pull down every source file every single build which adds significant overhead (1.5 minutes for a simple Java app). Part of this is necessary overhead; in a cluster, a build file has to be replicated among the machines. And dependencies have to be expire every so often to ensure they don't get stale. Nix's insistence on knowing the hash of every source dependency before it's fetched means that you can't implement automatic updates without updating your Nix files automatically. But the whole point of Nix is to be a human-readable and human-editable description of your configuration; updating them automatically makes them just another intermediate file format.

A cardinal rule of source control is to never check generated files into the repository; Nixpkgs has been violating this rule, with the consequent ballooning of repository size as a result. Hence the need to start over with a newer, cleaner distro - solve both the incremental build problem, via a new package manager and build tool, and the repository problem, via a more principled approach to dependency management.

The place to start is a small prototype to prove that it works; I only program in Haskell these days. For monitoring file dependencies we can use hs-watchman, although it needs to be updated for 2017. For logging we can use GZipped JSON as the file format and the pipes-zlib library. For process tracing we can re-use the command infrastructure from Shake.