A new build system

A survey of build systems, and their advantages:

• Make - installed everywhere
• Ninja - portable format
• Shake - parallelism, unchanging files, monadic dependencies
• Pluto - dynamic dependencies
• Tup - fast rebuilds with a filesystem monitor
• Nix - configuration management, isolated builds

There are no easily found build systems that combine all these advantages. Why not? Probably due to the difficulty of switching projects over to new build systems; most people are happy to let sleeping dogs lie, even if in this case they are really hissing snakes that are still biting and constricting developers.

With no existing build system, the solution is to write a new build system. Relevant xkcd:

But the problem is in fact that there is no standard for build systems, just people (companies) doing whatever they feel like. When a committee comes along, develops USB-C, Unicode, etc., in practice it seems that everyone does end up switching. In software, there's an even stronger standardization process: the best open-source implementation usually ends up becoming the de-facto standard, just because everyone else is too lazy or too stupid to reimplement it. Look at LAMP and its history, or more recently Wordpress.


The closest candidate to a de-facto standard for build systems is Make; it has a large user community. But it's missing all the features of the other systems. Can we extend make? Probably not, look at the source, e.g. the dependencies file and main.c and job.c. It's heavily process-based and GNU-centric, the deferred string processing and loading of commands makes everything really slow, and then of course it's in C, which isn't exactly my language of choice. Also it doesn't support parallelism on Windows (maybe?).  OTOH a new implementation maintaining backwards compatibility with make is definitely possible - for example Mozilla has been using PyMake since 2011. Python has been much more successful as a scripting language than GNU Make's built in scripting language of Guile. And porting even a large project isn't that hard, the Tup guy got halfway there already. The hard part is changing over tooling / automation / infrastructure which assumes the use of Make. But as Tup mentions, make is flawed in the way it's most commonly used (overly recursive, incomplete DAG) and also slow because of how it processes files. And it doesn't use a database, all the cool build systems have a database. So all in all, Make is dead for all standards-making purposes; nobody uses it anymore in their new projects, except GNU. And autoconf can generate config files, so there's no need to replace the whole GNU toolchain entirely. Just replacing automake should be sufficient.

Requirements for a new tool in priority order, from 2007:

1. Dependency Management (we have many many libraries)
1. Search Paths 
2. Variants
2. Maintainability
3. Cost of training for day-to-day tasks (40+ developers)
4. Portability (but Linux is a must)
1. Windows, MSYS
5. Speed (A distant fifth; anything would be better than recursive make.)

But of course everyone has their own priorities. The only real priority is avoiding yet another Make DSL.