A previous post described a data type of functional linear maps. As Andy Gill pointed out, we had a heck of a time trying to get good performance. This note describes a new representation that is very simple and much more efficient. It’s terribly obvious in retrospect but took me a good while to stumble onto.

The Haskell module described here is part of the vector-space library (version 0.5 or later) and requires ghc version 6.10 or better (for associated types).

Edits:

• 2008-11-09: Changed remarks about versions. The vector-space version 0.5 depends on ghc 6.10.
• 2008-10-21: Fixed the vector-space library link in the teaser.

Linear maps

Semantically, a linear map is a function f :: a -> b such that, for all scalar values s and “vectors” u, v :: a, the following properties hold:

By repeated application of these properties,

Taking the u_i as basis vectors, this form implies that a linear function is determined by its behavior on any basis of its domain type.

Therefore, a linear function can be represented simply as a function from a basis, using the representation described in Vector space bases via type families.

type u :-* v = Basis u -> v

The semantic function converts from (u :-* v) to (u -> v). It decomposes a source vector into its coordinates, applies the basis function to basis representations, and linearly combines the results.

lapply :: ( VectorSpace u, VectorSpace v
          , Scalar u ~ Scalar v, HasBasis u ) =>
          (u :-* v) -> (u -> v)
lapply lm = \ u -> sumV [s *^ lm b | (b,s) <- decompose u]

or

lapply lm = linearCombo . fmap (first lm) . decompose

The reverse function is easier. Convert a function f, presumed linear, to a linear map representation:

linear :: (VectorSpace u, VectorSpace v, HasBasis u) =>
          (u -> v) -> (u :-* v)

It suffices to apply f to basis values:

linear f = f . basisValue

Memoization

The idea of the linear map representation is to reconstruct an entire (linear) function out of just a few samples. In other words, we can make a very small sampling of function’s domain, and re-use those values in order to compute the function’s value at all domain values. As implemented above, however, this trick makes function application more expensive, not less. If lm = linear f, then each use of lapply lm can apply f to the value of every basis element, and then linearly combine results.

A simple trick fixes this efficiency problem: memoize the linear map. We could do the memoization privately, e.g.,

linear f = memo (f . basisValue)

If lm = linear f, then no matter how many times lapply lm is applied, the function f can only get applied as many times as the dimension of the domain of f.

However, there are several other ways to make linear maps, and it would be easy to forget to memoize each combining form. So, instead of the function representation above, I ensure that the function be memoized by representing it as a memo trie.

type u :-* v = Basis u :->: v

The conversion functions linear and lapply need just a little tweaking. Split memo into its definition untrie . trie, and then move the second phase (untrie) into lapply. We’ll also have to add HasTrie constraints:

linear :: ( VectorSpace u, VectorSpace v
          , HasBasis u, HasTrie (Basis u) ) =>
          (u -> v) -> (u :-* v)
linear f = trie (f . basisValue)
 
lapply :: ( VectorSpace u, VectorSpace v, Scalar u ~ Scalar v
          , HasBasis u, HasTrie (Basis u) ) =>
          (u :-* v) -> (u -> v)
lapply lm = linearCombo . fmap (first (untrie lm)) . decompose

Now we can build up linear maps conveniently and efficiently by using the operations on memo tries shown in Composing memo tries. For instance, suppose that h is a linear function of two arguments (linear in both, not it each) and m and n are two linear maps. Then liftA2 h m n is the linear function that applies h to the results of m and n.

lapply (liftA2 h m n) a = h (lapply m a) (lapply n a)

Exploiting the applicative functor instance for functions, we get another formulation:

lapply (liftA2 h m n) = liftA2 h (lapply m) (lapply n)

In other words, the meaning of a liftA2 is the liftA2 of the meanings, as discussed in Simplifying semantics with type class morphisms.

A basis B of a vector space V is a subset B of V, such that the elements of B are linearly independent and span V. That is to say, every element (vector) of V is a linear combination of elements of B, and no element of B is a linear combination of the other elements of B. Moreover, every basis determines a unique decomposition of any member of V into coordinates relative to B.

This post gives a simple Haskell implementation for a canonical basis of a vector space, and a means of decomposing vectors into coordinates. It uses indexed type families (associated types), and is quite general, despite its simplicity.

The Haskell module described here is part of the vector-space library (version 0.4 or later), which available on Hackage and a darcs repository. See the wiki page, interface documentation, and source code. The library version described below (0.5 or later) relies on ghc 6.10.

Edits:

• 2008-11-09: Tweaked comment above about version.
• 2008-02-09: just fiddling around

Additive groups and vector spaces

We’ll need a bit of preliminary before jumping into basis types.

An additive group has addition operation, with an identity (zero) and an additive inverse:

class AdditiveGroup v where
  (^+^)   :: v -> v -> v
  zeroV   :: v
  negateV :: v -> v

A vector space is an additive group with scalar multiplication, so it also has an associated type of scalars:

class AdditiveGroup v => VectorSpace v where
  type Scalar v :: *
  (*^) :: Scalar v -> v -> v

This associated type could instead by written as a functional dependency (fundep). In fact, the type family implementation in ghc 6.9 is not quite up to working with the code in this post, so the library contains a (6.9-friendly) version together with as a fundep (Data.VectorSpace) and the version given in this post (Data.AVectorSpace). Sometime after ghc-6.10 is released, I will retire the fundep version and rename AVectorSpace to VectorSpace. Similarly, the library temporarily contains Data.ABasis.

Basis types

Since Haskell doesn’t have subtyping, we can’t represent a basis type directly as a subset. Instead, for an arbitrary vector space v, represent the (canonical) basis as an associated type, Basis v, and a function that interprets a basis representation as a vector.

class VectorSpace v => HasBasis v where
  type Basis v :: *
  basisValue   :: Basis v -> v

Another method extracts coordinates (coefficients) for a vector. Each coordinate is associated with a basis element.

  decompose    :: v -> [(Basis v, Scalar v)]

We can also reverse the process, recomposing into a vector, by linearly combining the basis elements:

linearCombo :: VectorSpace v => [(v,Scalar v)] -> v
linearCombo ps = sumV [s *^ v | (v,s) <- ps]
 
recompose :: HasBasis v => [(Basis v, Scalar v)] -> v
recompose = linearCombo . fmap (first basisValue)

The defining property is

recompose . decompose == id

Primitive bases

Any scalar field is also a vector space over itself. For instance,

instance AdditiveGroup Double where
  zeroV   = 0.0
  (^+^)   = (+)
  negateV = negate
 
instance VectorSpace Double where
  type Scalar Double = Double
  (*^) = (*)

The canonical basis of a one-dimensional space has only one element, namely 1, which can be represented with no information.

instance HasBasis Double where
  type Basis Double = ()
  basisValue ()     = 1
  decompose s       = [((),s)]

Composing bases

Pairs of additive groups form additive groups:

instance (AdditiveGroup u,AdditiveGroup v)
      => AdditiveGroup (u,v) where
  zeroV             = (zeroV,zeroV)
  (u,v) ^+^ (u',v') = (u^+^u',v^+^v')
  negateV (u,v)     = (negateV u,negateV v)

Pairs of vector spaces, over the same scalar field, form vector spaces:

instance (VectorSpace u,VectorSpace v, Scalar u ~ Scalar v)
      => VectorSpace (u,v) where
  type Scalar (u,v) = Scalar u
  s *^ (u,v) = (s*^u,s*^v)

Given vector spaces u and v, a basis representation for (u,v) will be one basis representation or the other, tagged with Left or Right:

instance (HasBasis u, HasBasis v, Scalar u ~ Scalar v)
      => HasBasis (u,v) where
  type Basis (u,v) = Basis u `Either` Basis v

The basis vectors themselves will be (ub,0) or (0,vb), where ub is a basis vector for u, and vb is a basis vector for v. As expected then, the dimensionality of the cross product is the sum of the dimensions.

  basisValue (Left  a) = (basisValue a, zeroV)
  basisValue (Right b) = (zeroV, basisValue b)

The decomposition of a vector (u,v) contains left-tagged decompositions of u and right-tagged decompositions of v.

  decompose (u,v) = decomp2 Left u ++ decomp2 Right v

where

decomp2 :: HasBasis w => (Basis w -> b) -> w -> [(Scalar w, b)]
decomp2 inject = fmap (first inject) . decompose

Triples etc, can be handled similarly. Instead, the library implementation reduces them to the pair case:

instance ( HasBasis u, s ~ Scalar u
         , HasBasis v, s ~ Scalar v
         , HasBasis w, s ~ Scalar w )
      => HasBasis (u,v,w) where
  type Basis (u,v,w) = Basis (u,(v,w))
  ...

Exercise: complete this instance definition (without peeking).

Bases in spaces

What about other vector spaces, particularly infinite dimensional ones? The result type for decompose is not convenient:

decompose :: v -> [(Basis v, Scalar v)]

Moreover, its definition for pair types would have to be changed, e.g., to use interleaving instead of append. On the other hand, this type can be thought of as an association list, representing Basis v -> Scalar v. Instead, we might use the function representation directly:

decompose :: v -> (Basis v -> Scalar v)

In that case, the definition of decompose on pairs is

decompose (u,v) = decompose u `either` decompose v

which beautifully mirrors the basis type definition:

type Basis (u,v) = Basis u `Either` Basis v

I guess we’d have to somehow extend the definition of recompose as well, or avoid it.

One example of an infinite dimensional vector space is a function over an infinite domain. The additive group and vector space instances follow a standard form for applicative functors applied to an additive group or vector space. In this case, the applicative functor is ((->) a).

instance AdditiveGroup v => AdditiveGroup (a -> v) where
  zeroV   = pure   zeroV
  (^+^)   = liftA2 (^+^)
  negateV = fmap   negateV
 
instance VectorSpace v => VectorSpace (a -> v) where
  type Scalar (a -> v) = Scalar v
  (*^) s = fmap ((*^) s)

As a basis for a function space a->u, let’s use the subset of functions that map one domain value to some basis vector for u and map all other domain values to zero. By linearly combining these functions, we can construct any function in a->u that is nonzero for finitely many domain values. If we stretch the notion of linear combinations beyond finite combinations, perhaps we can go further and cover at least the countably infinite domain types.

To represent these functions, it suffices to record the choice of domain element and the representation of the corresponding basis vector for u:

instance (Eq a, HasBasis u) => HasBasis (a -> u) where
  type Basis (a -> u) = (a, Basis u)
 
  basisValue (a,b) a' | a == a'   = basisValue b
                      | otherwise = zeroV

The actual implementation is a bit more efficient, avoiding recomputation of basisValue b for each a'.

Decomposing a function into its coordinates is even simpler:

  decompose g (a,b) = decompose (g a) b

Some isomorphisms

This instance rule for functions will be applied repeatedly for curried functions. For instance, Basis (a -> b -> u) == (a, (b, Basis u)). The isomorphic uncurried form has an isomorphic basis: Basis ((a,b) -> u) == ((a,b), Basis u).

Pairing in the range instead of domain gives rise to another pair of isomorphisms:

Basis (a -> (b,c)) == (a, Basis b `Either` Basis c)
 
Basis ((a -> b, a -> c)) == (a, Basis b) `Either` (a, Basis c)

The rules for basis types look like logarithms, especially if we add a basis for ():

  type Basis () = Void
 
  type Basis (u,v) = Basis u `Either` Basis v
 
  type Basis (a -> u) = (a, Basis u)

Compare with:

The rules are also essentially the same as the ones used for memo tries, but phrased in terms of logarithms instead of (explicit) exponents.

Two earlier posts described a simple and general notion of derivative that unifies the many concrete notions taught in traditional calculus courses. All of those variations turn out to be concrete representations of the single abstract notion of a linear map. Correspondingly, the various forms of mulitplication in chain rules all turn out to be implementations of composition of linear maps. For simplicity, I suggested a direct implementation of linear maps as functions. Unfortunately, that direct representation thwarts efficiency, since functions, unlike data structures, do not cache by default.

This post presents a data representation of linear maps that makes crucial use of (a) linearity and (b) the recently added language feature indexed type families (“associated types”).

For a while now, I’ve wondered if a library for linear maps could replace and generalize matrix libraries. After all, matrices represent of a restricted class of linear maps. Unlike conventional matrix libraries, however, the linear map library described in this post captures matrix/linear-map dimensions via static typing. The composition function defined below statically enforces the conformability property required of matrix multiplication (which implements linear map composition). Likewise, conformance for addition of linear maps is also enforced simply and statically. Moreover, with sufficiently sophisticated coaxing of the Haskell compiler, of the sort Don Stewart does, perhaps a library like this one could also have terrific performance. (It doesn’t yet.)

You can read and try out the code for this post in the module Data.LinearMap in version 0.2.0 or later of the vector-space package. That module also contains an implementation of linear map composition, as well as Functor-like and Applicative-like operations. Andy Gill has been helping me get to the bottom of some some severe performance problems, apparently involving huge amounts of redundant dictionary creation.

Edits:

• 2008-06-04: Brief explanation of the associated data type declaration.

Linear maps

Semantically, a linear map is a function f :: a -> b such that, for all scalar values c and “vectors” u, v :: a, the following properties hold:

f (c *^ u)  == c *^ f u
f (u ^+^ v) == f u ^+^ f v

where (*^) and (^+^) are scalar multiplication and vector addition. (See VectorSpace details in a previous post.)

Although the semantics of a linear map will be a function, the representation will be a data structure.

data a :-* b = ...

The semantic function is

lapply :: (LMapDom a s, VectorSpace b s) =>
          (a :-* b) -> (a -> b)  -- result will be linear

The first constraint says that we know how to represent linear maps whose domain is the vector space a, which has the associated scalar field s. The second constraint say that b must be a vector space over that same scalar field.

Conversely, there is also a function to turn any linear function into a linear map:

linear :: LMapDom a s => (a -> b) -> (a :-* b)  -- argument must be linear

These two functions and the linear map data type are packaged up as the LMapDom type class:

-- | Domain of a linear map.
class VectorSpace a s => LMapDom a s | a -> s where
  -- | Linear map type
  data (:-*) a :: * -> *
  -- | Linear map as function
  lapply :: VectorSpace b s => (a :-* b) -> (a -> b)
  -- | Function (assumed linear) as linear map.
  linear :: (a -> b) -> (a :-* b)

The data definition means that the data type (a :-* b) (of linear maps from a to b) is has a variety of representations, each one associated with a type a.

These two conversion functions are required to be inverses:

{-# RULES
 
"linear.lapply"   forall m. linear (lapply m) = m
 
"lapply.linear"   forall f. lapply (linear f) = f
 
 #-}

Scalar domains

Consider a linear function f over a scalar domain. Then

f s == f (s *^ 1)
    == s *^ f 1  -- by linearity

Therefore, f is fully determined by its value at 1, and so an adequate representation of f is then simply the value f 1.

This observation leads to LMapDom instances like the following:

instance LMapDom Double Double where
  data Double :-* o  = DoubleL o
  lapply (DoubleL o) = (*^ o)
  linear f           = DoubleL (f 1)

Non-scalar domains

Maps over non-scalar domains are a little trickier. Consider a linear function f over a domain of pairs of scalar values. Then

f (a,b) == f (a *^ (1,0) ^+^ b *^ (0,1))
        == f (a *^ (1,0)) ^+^ f (b *^ (0,1))  -- linearity
        == a *^ f (1,0) ^+^ b *^ f (0,1)      -- linearity twice more

So f is determined by f (1,0) and f (0,1) and thus can be represented by those two values.

instance LMapDom (Double,Double) Double where
  data (Double,Double) :-* o = PairD o o
  PairD ao bo `lapply` (a,b) = a *^ ao ^+^ b *^ bo
  linear f = PairD (f (0,1)) (f (1,0))

and similarly for triples, etc.

This definition works fine, but I want something compositional. I’d like linear maps over pairs of pairs and so on.

Composing domains

We can still use part of our linearity property. Using zeroV as the zero vector for arbitrary vector spaces,

f (a,b) == f ((a,zeroV) ^+^ (zeroV,b))
        == f (a,zeroV) ^+^ f (zeroV,b)

We see that f is determined by its behavior when either argument is zero.

In other words, f can be reconstructed from two other functions over simpler domains:

fa a = f (a,zeroV)
fb b = f (zeroV,b)

If f :: (a,b) -> o, then fa :: a -> o and fb :: b -> o.

Exercise: show that fa and fb are linear if f is. We can thus reduce the problem of representing the linear function f to the problems of representing fa and fb. This insight is captured in the following LMapDom instance:

instance (LMapDom a s, LMapDom b s) => LMapDom (a,b) s where
  data (a,b) :-* o = PairL (a :-* o) (b :-* o)
  PairL ao bo `lapply` (a,b) = ao `lapply` a ^+^ bo `lapply` b
  linear f = PairL (linear (\ a -> f (a,zeroV)))
                   (linear (\ b -> f (zeroV,b)))

Of course, there are similar instances for triples, etc, as well as for tuple variants with strict fields, such as OpenGL’s Vector2 and Vector3 types.

What have we done?

If you’ve studied linear algebra, you may be thinking now about the idea of a basis of a vector space. A basis is a minimal set of vectors that can be combined linearly to cover the entire vector space. Any linear map is determined by its behavior on any basis. For scalars, the set {1} is a basis, while for pairs of scalars, {(1,0), (0,1)} is a basis. It is not just coincidental that exactly these basis vectors showed up in the definitions of linear for Double and (Double,Double).

In the general pairing instance of LMapDom above, bases are built up recursively. Each recursive call to linear results in a data structure that holds the values of fa over a basis for a and the values of fb over a basis for b. Each of those basis vectors corresponds to a basis vector for (a,b), by zeroV-padding.

The dimension of a vector space is the number of elements in a basis (which is independent of the particular choice of basis). For vector space types a and b,

dimension (a,b) == dimension a + dimension b

which corresponds to the fact that our linear map representation (as built by linear) contains samples for each basis element of a, plus samples for each basis element of b (all zeroV-padded).

Working with linear maps

Besides the instances above for creating and applying linear maps, what else can we do? For starters, let’s define the identity linear map. Since the identity function is already linear, simply convert it to a linear map:

idL :: LMapDom a s => a :-* a
idL = linear id

Another very useful tool is transforming a linear map by transforming a linear function.

inL :: (LMapDom c s, VectorSpace b s', LMapDom a s') =>
        ((a -> b) -> (c -> d)) -> ((a :-* b) -> (c :-* d))
inL h = linear . h . lapply

where the higher-order function h is assumed to map linear functions to linear functions.

We don’t have to stop at unary transformations of linear functions.

-- | Transform a linear maps by transforming linear functions.
inL2 :: ( LMapDom c s, VectorSpace b s', LMapDom a s'
        , LMapDom e s, VectorSpace d s ) =>
        ((a  -> b) -> (c  -> d) -> (e  -> f))
     -> ((a :-* b) -> (c :-* d) -> (e :-* f))
inL2 h = inL . h . lapply

The type constraints are starting to get hairy. Fortunately, they’re entirely inferred by the compiler.

Let’s do some inlining and simplification to see what goes on inside inL2:

inL2 h m n
  == (inL . h . lapply) m n                -- inline inL2
  == inL (h (lapply m)) n                  -- inline (.)
  == (linear . (h (lapply m)) . lapply) n  -- inline inL
  == linear (h (lapply m) (lapply n))      -- inline (.)

Ternary transformations are defined similarly. I’ll spare you the type constraints this time.

inL3 :: ( ... ) =>
        ((a  -> b) -> (c  -> d) -> (e  -> f) -> (p  -> q))
     -> ((a :-* b) -> (c :-* d) -> (e :-* f) -> (p :-* q))
inL3 h = inL2 . h . lapply

Look what happens when these operations are composed. As an example,

inL h . inL g
  == (linear . h . lapply) . (linear . g . lapply)
  == linear . h . lapply . linear . g . lapply    -- associativity of (.)
  == linear . h . g . lapply                      -- rule "lapply.linear"
  == inL (h . g)

This transformation is not actually happening in the compiler yet. The “lapply.linear” rule is not firing, and I don’t know why. I’d appreciate suggestions.

There are a few more operations defined in Data.LinearMap. I’ll end with this simple, general definition of composition of linear maps:

-- | Compose linear maps
(.*) :: (VectorSpace c s, LMapDom b s, LMapDom a s) =>
        (b :-* c) -> (a :-* b) -> (a :-* c)
(.*) = inL2 (.)

Derivative towers again

A similar, but recursive, definition is used in the new definition of the the general chain rule for infinite derivative towers, updated since the post Higher-dimensional, higher-order derivatives, functionally.

(@.) :: (LMapDom b s, LMapDom a s, VectorSpace c s) =>
        (b :~> c) -> (a :~> b) -> (a :~> c)
(h @. g) a0 = D c0 (inL2 (@.) c' b')
  where
    D b0 b' = g a0
    D c0 c' = h b0