Monday, February 5, 2018

Nix pros and cons

Pros
  • Sandboxed builds
  • Cross-compile support
  • Run-time dependency information via hash-scanning
    • not unique, e.g. NuTyX (cards) scans dependencies in the Elf header. 
  • Multiple software versions don't conflict 
  • Non-destructive updates
  • Automated builds/tests with Hydra
  • Single config file for NixOS
Cons
  • Non-FHS layout
  • Repeated dependency information
  • Slow to evaluate
  • Manual version bumps, cluttering VCS

Friday, November 3, 2017

Continuations and type systems

How do we get a programming language which combines all the type features we have seen? Linear types, dependent types, subtypes, continuations rather than functions, jumbo types,and so on?

First we must recall the purpose of a type system: at least at a low level, types are calling conventions. The type of a function, module, etc. is its interface, its API. A function may have multiple types and hence multiple calling conventions; but at runtime presumably only a few of these will be used. For efficiency reasons it makes sense to have only one true type and construct all the other types as wrappers, but this is by no means necessary.

When we have a C type, for example int main(int argc, char **argv), this is actually a very strong type; the C standard and implementation specify many details of memory layout, stack usage, etc. that are assumed to hold. The interface with the kernel is a cleaner type; execve simply loads the proper memory layout and jumps to the starting address. To represent this API we must use continuations instead of functions, as they never return a value. After looking at IL we might denote this as the type ¬kernel. Expanding the type kernel out it might look more like ¬(stack pointer -> {argc, argv, env}), at least on Linux. Continuations map nicely to the power of assembly language, where jumping to an unknown code segment gives up all control. If a C 'function' calls exit() it will never return. In IL, there is also a 0 type, the type of 'enterable locations'. From the rules the only values of type 0 look like (λ x.F) T, where F is another value of type 0, and the only values of type ¬T look like λ x.(E1 E2). It's not clear what this is useful for.

At a high level, though, the idea of contracts remains. We have properties of our programs that we wish to remain true, like memory safety. Linear types are one method of enforcing these; we reject programs that don't use resources linearly. Dependent types are another form of contract, allowing us to write pre- and post- conditions on values.

Following the IL paper, and its related cousin TACC, we can define types of continuations - roughly, the type of a continuation is the argument layout it expects to have. We extend our data types to the Jumbo Connectives of arbitrary records and sum types (the Pi type can be ignored, continuations are more powerful than functions as an intermediate language). And also we have to generalize negation to delimited continuations, following Zeilberger and a more recent paper. We can implement these on LLVM in two ways: first, by translating them away into standard function calls, and second by using a special calling convention, jump-with-argument.

How do we combine linear types with continuations? Following this paper we might introduce multiple types of continuations, e.g. the paper's lowered negation ↓¬ and boxed negation !¬. Then linear function types translate as !¬(τ1 & ↓¬τ2) while normal function types translate as !¬(τ1 & !¬τ2). But this hides the Rig (semiring) structure of the context. Maybe instead we could use a numbered continuation, k¬, where k is from our Rig. Then adding dependent types is easy, and subtypes are just polarizing all the information flow. Simple!

Papers by Appel (1, 2 , 3, 4) describe a closure strategy analysis, wherein known functions are compiled using registers but unknown functions use the heap. They do not use the stack because it inefficiently holds on to values even after they are not being used; deallocation would require making holes in the stack. But of course nothing in life is free; as soon as one gives up the stack, one needs it again for interfacing with other (C) code.

Thursday, November 2, 2017

Incremental build farms

Build farms are expensive to setup and maintain. Or are they? Amazon gives away free instance-years, and services such as Travis and Appveyor build open-source projects for free. Can these services be used to bootstrap a Linux distribution?

The Nix answer to these questions is to use a binary cache - a server in the middle that stores cryptographically-signed binaries. It can start as simple as a single node, with a Nix store, then expand to use Amazon S3 or another large-scale filesystem.
But why start there? We could instead start with a distributed filesystem, like Bittorrent's DHT or IPFS. Then sharing files is as easy as sharing the root location hash, and all we need is a simple GitHub Pages site that gives out the root hash for everyone to synchronize on.

The issue of trust still remains though. Who builds what? How does the hash get calculated? Is there a cryptocoin (filecoin)?

Another approach starts from a single moderately-powerful system; Gentoo is a source-based Linux distribution that can usually be compiled and installed overnight. Speeding up Gentoo (re)compilations is a worthy goal. But then how do we go from one system to a network of systems sharing binaries? Gentoo (portage) actually supports binary packages, but in practice this seems like it is never used publicly; instead it has a centralized system of mirrors for distfiles. This is probably due to culture and limitations in compatibility of binary packages; it seems like Gentoo is usually set up to compile packages for the specific CPU it's running on, as a side-effect of too many USE flags ("Gentoo is for ricers"). But doing some sane defaults like Arch shouldn't be too hard, and proper dependency management should alleviate all the compatibility / configurability concerns.

Tuesday, October 31, 2017

Continuous delivery and distribution

Software development suffers from lag; every change must be made, then compiled, then tested in few dozen ways, before the next change can be made. The faster this process completes, the faster development can move forward. Non-incremental development, where changes are made without testing or compiling, leads to slower development times overall due to the numerous regressions that are introduced and have to be tracked down later.

Nix encourages incremental development by making it easy to download and use software; it creates a uniform interface for adding libraries and other dependencies, in the .nix file. But it also slows down development because the build process itself is non-incremental; it has to pull down every source file every single build which adds significant overhead (1.5 minutes for a simple Java app). Part of this is necessary overhead; in a cluster, a build file has to be replicated among the machines. And dependencies have to be expire every so often to ensure they don't get stale. Nix's insistence on knowing the hash of every source dependency before it's fetched means that you can't implement automatic updates without updating your Nix files automatically. But the whole point of Nix is to be a human-readable and human-editable description of your configuration; updating them automatically makes them just another intermediate file format.

A cardinal rule of source control is to never check generated files into the repository; Nixpkgs has been violating this rule, with the consequent ballooning of repository size as a result. Hence the need to start over with a newer, cleaner distro - solve both the incremental build problem, via a new package manager and build tool, and the repository problem, via a more principled approach to dependency management.

The place to start is a small prototype to prove that it works; I only program in Haskell these days. For monitoring file dependencies we can use hs-watchman, although it needs to be updated for 2017. For logging we can use GZipped JSON as the file format and the pipes-zlib library. For process tracing we can re-use the command infrastructure from Shake.

Type system features

Subtyping and open data types

One interesting feature of type systems is extensibility; we have the typical object-oriented inheritance, but this is actually too narrow for a proper type system. We instead have the notion of an extended conversion, a function from -> to. Useful examples start from integer conversions, but the most interesting example is adding more constructors to a datatype; A | B is a subtype of A | B | C. We can also do it with product types, by removing elements; (A,B,C) is a subtype of (A,B) is a subtype of A. It makes more sense with labels and records though; {foo: A, bar: B} is a subtype of {foo: A}. This is the closest it gets to inheritance in a type system. Implementations are few, unfortunately; it's primarily Stephen Dolan's work on BRICK (blog archived) and MLSub. BRICK is notable for supporting equirecursive and regular types, so that records do not need a newtype to destructure or bind, similar to dynamically typed languages such as Python, and for compiling to LLVM (no GC, unfortunately). MLSub is just a boring extension with lots of proofs of correctness in the thesis and the coined terms of "biunification" and "bisubstitution". He basically tries to convince you that his subtyping algorithm is almost as simple to implement as Hindley-Milner; which honestly I agree with.

The most interesting part of subtyping is that it explicitly exposes the idea of polarization or duality; for example a function arrow (A -> B) is contravariant in A and covariant in B. When we type check we can divide the type variables into two classes: the pos-variables (lower bounds) and the neg-variables (upper bounds). In fact we usually need both; for mutable variables, arrays, etc. we need both in and out parameters (what can be stored vs what can be retrieved). For example the parameter type for a function that accepts a mutable array of Ints and only reads from it will be MutArray[store=\bot, retrieve=Int], where the store is a lower bound and retrieve is an upper bound.

Dependent types

Once you have subtyping, what point is there in distinguishing values and types? Letting them be used in place almost interchangeably requires some complicated compilation tricks, and an interactive interpreter, but Idris has shown that it works reasonably well in a practical programming language. But dependent types can be extended further.

Insanely dependent types let you define dependent pairs (Σ A (B a), Idris term a ** B a) using a dependent product (Π, (a : A) -> B a). There's some trickery involved in the implementation, see the source for details. The short answer is you need extra scope to reference all the values, and a careful evaluation strategy to avoid infinite loops. The related idea of circular type signatures allows indexing datatypes by values of the same datatype; for example a matrix indexed by a vector, while defining a vector as a single-dimension matrix of length n. (Their notion of matrix is n-dimensional so really it's a tensor). Also interesting from that reddit thread is the idea of making values (terms) a subtype/instance of their type, so for example 1 is a subtype of Integer.

Richard Eisenberg is working on adding them to Haskell, although only the normal form of dependent types.

Linear Types

We could borrow type constructors from Girard's logic: ! and ?, respectively weakening a type to be usable more than once and to not be used at all. There remains the linear arrow ⊸. The usual intuitionistic arrow A -> B then desugars as !A ⊸ B. But this cannot be the final answer, because in IL we decompose A -> B as ¬(A & ¬B); functions do not really exist. Following this paper it seems like it makes more sense to just add multiplicities to the constructors, 0 1 and ω for erasable, linear, and ordinary types respectively. A note here mentions extending ω to having multiple types.

We could also use monads or their more general cousins of Arrows, but why?

Tweag says they're adding them to GHC. although it's hard to say.

Higher-order Types

This paper seems interesting and useful, shouldn't be too hard to add.

Extended type classes

Haskell's type classes have been wildly successful; from an implementation perspective, they amount to adding a constraint-solving engine that discovers and writes in new terms. But I think type classes should be able to add new type information too; unfortunately my efforts in that direction have been unsuccessful so far. It seems like a simple idea though; the set of type class instances in scope is fixed, so if only one instance unifies with the types in scope then it must be the instance to use (otherwise the term must be resolved elsewhere). This is to be distinguished from GHC's current implementation, which has a more restrictive condition requiring matching instead of unification, where matching is defined to add no new information. The question is how to propagate the information from unification in the constraint solver back to the type checker at large; I don't really know enough GHC internals to diagnose the errors in the test suite I ran into.

Type-directed name resolution

One of the most annoying parts of Haskell is that record constructor names must be globally unique since they count as functions. This has been worked on here, but the problem also shows up with type classes (particularly RebindableSyntax) and just in general with similarly-named or similarly-performing functions. What would be ideal IMO is if Haskell supported ad-hoc overloading, so you could just use the name as many times as you wanted and the compiler disambiguated by type. This would also get rid of the 'import Data.Map as M, Data.Set as S' boilerplate. But this would require intermingling the renaming and type-checking phases of GHC, so unlikely to happen.