I’ve heard from a few people that my blog’s RSS feeds were broken. I just upgraded my blog to Wordpress 2.9.1, which appears to have fixed the problem with RSS generation.

If you notice any problems with RSS or anything else, please let me know.

There is a lot of confusion about the meaning of “functional” and “declarative” as descriptions of programming languages and paradigms. For instance, Haskell is sometimes advertised as a “purely functional programming language” and other times as “the world’s finest imperative programming language”. I added some playful confusion (and clarity, I hope) with my post The C language is purely functional.

I still regularly hear people ask and disagree about whether Haskell’s monadic I/O is “functional”. I would say yes in one (technical) sense of “functional”. But not the same sense in which we say “functional programming has simple, precise semantics” or “functional programming has good compositional properties” or “functional programming is good for reasoning about”. Monadic I/O is a clever trick for encapsulating sequential, imperative computation, so that it can “do no evil” to the part that really does have precise semantics and good compositional properties.

It’s because of this confusion that I’ve started using the more specific term “denotational programming” in my blog subtitle and elsewhere, as an alternative to what I used to call “functional programming”. While there are other notions of “functional”, applicable even to monadic IO, I think “denotational” captures the fundamental and far-reaching benefits that we called “good for reasoning about” and “powerfully compositional”.

When I bash monadic I/O, my real complaint isn’t with the technical invention–which I like. Rather, I’m cranky about confusion and misleading communication, and about distraction from what we originally meant by “functional programming”–from what I call “denotational programming”.

I don’t mind that we haven’t yet been liberated from the von Neumann model. As long as remember we haven’t.

As long as we keep on searching.

This post describes an idea I’ve had in mind for years and have chatted about with friends but haven’t written down before.

Functional programmers used to worry about how to solve “the I/O problem”. Functional programming (FP) eschews any notion of side-effect or order of execution, and instead has an order-independent notion of evaluation. In contrast, input & output (I/O) seems to be an inherently effectful, ordered notion.

For a long time, I’ve been dissatisfied with the current Haskell solution to “the I/O problem”. Of course it’s very clever and effective. With “monadic IO” we can program both functionally and imperatively, and the imperative semantics doesn’t compromise the functional semantics.

I look at monadic IO not as any solution to functional I/O, but rather as a (clever) way not to bring I/O into the functional world. There’s a ton of confusion buzzing around monadic IO, so let me try to restate more carefully: “monadic IO” is a way to generate imperative computations (not just I/O) without having to say anything at all about the semantic content of those computations. Rather than retooling our mental habits completely away from the imperative model of computation, monadic IO is a way to hang onto these old mental habits, and use them in a hybrid functional/imperative programming style, in which the imperative doesn’t contaminate the functional, semantically.

John Backus offered us a gift and a blessing in his 1977 Turing Award lecture, Can Programming Be Liberated from the von Neumann Style? A functional style and its algebra of programs. He told us how the sequential/imperative model enslaves us in weak thoughts, inhibiting manageable scaling (sections 1 through 5.1). In the abstract, Backus says

Conventional programming languages are growing ever more enormous, but not stronger. Inherent defects at the most basic level cause them to be both fat and weak: their primitive word-at-a-time style of programming inherited from their common ancestor–the von Neumann computer, their close coupling of semantics to state transitions, their division of programming into a world of expressions and a world of statements, their inability to effectively use powerful combining forms for building new programs from existing ones, and their lack of useful mathematical properties for reasoning about programs.

As well, Backus showed one way to throw off the shackles of this mental and expressive slavery — how to liberate our notions of programming by leaving the von Newmann mental model behind.

Haskell is often described as a “purely functional programming language”, and in a technical sense it is. (In good company, too; see The C language is purely functional.) However, as commonly practiced, imperative computation (IO) still plays a significant role in most Haskell programs. Although monadic IO is wily enough to keep a good chunk of purity in our model, Haskell programmers still use the imperative model as well and therefore have to cope with the same fundamental difficulties that Backus told us about. In a sense, we’re essentially still programming in Fortran, with its mixture of statements and expressions (semantically, of commands and values). Don’t get me wrong; Haskell is a much better Fortran than the original. (If you have an urge to inform me how monadic IO works, re-read what I’m really saying. Honestly, I know.)

(Sadly, the very definition of “program” in Haskell requires that there be a main of type IO ()! Haskell programs thus cannot be purely functional (except technically), and so cannot be composed except very weakly, as Backus describes. For an alternative model, see Tangible Functional Programming: a modern marriage of usability and composability.)

In a sense, I see us as in a worse position than before. Since the adoption of monadic IO, it’s been less immediately apparent that we’re still enslaved, so there’s been much less activity in searching for the promised land that the prophet JB foretold. Our current home is not painful enough to goad us onward, as were continuation-based I/O and stream-based I/O. (See A History of Haskell, section 7.1.) Nor does it fully liberate us.

Since I/O is an intrisically sequential, effectful notion, and we need some way of doing input & output, I guess the best we can do, is what we’ve done with monadic I/O: support both imperative and functional programming and proctect the functional from the imperative.

Or can we?

In my recent post, Why program with continuous time?, I had suggested that the abstraction of continuous time is useful even if “in the end all streams are discrete”. I advised designing APIs based on the principle that “programming is more about the middle than the end, i.e., more about composition than about output.” As an example, I pointed out that we don’t represent numbers as strings even if “in the end”, numbers are output as strings, suggesting that the key reason to choose an abstraction other than strings is composability (here, arithmetic). The post gives a few other examples of this principle of designing an API for composition, not for input or output.

Roly Perera noted that

… you never really need to reach “the end”. (It really is all about composition.)

To extend an example in the previous post, after numbers, then strings, and then pixels, are phosphors the end? (Sorry for the obsolete technology.) After phosphors come photons. After photons comes retina & optic nerve. Then cognitive processing (and emotional and who-knows-what-else). Then, via motor control, to mouse, keyboard, joystick etc. Then machine operating system, and software again. Round & round. More importantly, our interactions with other wetware organisms and with our planet and cosmos, and so on.

Roly added:

What looks like imperative output can be just what you observe at the boundary between two subsystems.

Which is exactly how I look at imperative input/output.

Representation shifts often happen at programming interfaces. Major interfaces in a whole system sometimes require a major representation shift, namely marshalling to and unmarshalling from serialized, memory-external representations. For instance, conversion from strings to numbers on input to a computer or brain and conversion of numbers to strings on the way out. Or replace strings with other rendered representations. Another example: conversion from continuous time & space to discrete time & space (“sampling”) on input to a digital machine, and conversion from discrete time & space to continuous time & space (“reconstruction”) on the way from digital machine (computer, CD/DVD player) to a brain.

This perspective is what’s behind my confidence that we will someday really learn to look at whole systems in a consistently functional/denotational style (simple & precise semantics). The imperative interfaces in today OSs and data-bases are troubling at first, and indeed I often hear people (even on #haskell) using these examples as demonstration that the imperative programming model is inescapable. These examples don’t trouble me, however, because I see that we’ve already dealt with others of their kind. Underneath the implementation of our current functional abstractions (numbers, strings, trees, functions, etc), there are imperative mechanisms, such as memory allocation & deallocation, stack frame modification, and thunk overwriting (to implement laziness). Every one of these mechanisms can be used as evidence against the functional paradigm by someone who hasn’t yet realized how the mechanism can be shifted out of the role of a programming model notion and into the role of implementation of a programming model notion. The new notion is more composable by virtue of being semantically simpler and mathematically more well-behaved.

Stack and register munging and jump/GOTO are implementations of the semantically simpler notion of function application. (I mean “function” in the sense of math and of pure functional programming.) Moreover, stack & register munging is the input/output (IO) part of the implementation of function application. Information goes out of one context and into another. My vision of “solving the I/O problem” for functional programming is nothing like Haskell’s semantically imperative (non-)solution (caled “IO” in Haskell), but rather to continue shifting semantically imperative mechanisms out of the programming model and into the implementation of (semantically simple) function application.

How do we get from here (still stuck in the quagmire of imperative programming models) to there (simple semantics and consistently excellent composability)? First, stop rehearsing the thinking that keeps us stuck. When we keep telling ourselves and each other that imperative notions are inevitable, accompanying such statements with hand-waving arguments, we are weaving self-fulfilling prophecies. A person convinced of (and perhaps ego-invested in) the nonexistence of new possibilities is a person actively resisting their opportunity to discover of new possibilities — too stuck in the obvious to discover new truths.

The significant problems of our time cannot be solved by the same level of thinking that created them. – Albert Einstein

They are ill discoverers that think there is no land, when they can see nothing but sea. – Francis Bacon

Hand-waving is an important factor in staying stuck in impossibility thinking, since rigorous argument uncovers unconscious limiting assumptions. I’m just as happy if you are working on showing impossibility as discovering the possible, as long as you are rigorous and get rigorous peer review. Otherwise, we all fall into the trap of self-deceit:

Nothing is easier than self-deceit. For what each man wishes, that he also believes to be true. – Demosthenes

If you wish to see the truth, then hold no opinion for or against – Osho (Rajneesh Chandra Mohan Jain)

The discovery of this reality is hindered rather than helped by belief, whether one believes in God or believes in atheism. We must make here a clear distinction between belief and faith, because, in general practice, belief has come to mean a state of mind which is almost the opposite of faith. Belief, as I use the word here, is the insistence that the truth is what one would “lief” or wish it to be. The believer will open his mind to the truth on the condition that it fits in with his preconceived ideas and wishes. Faith, on the other hand, is an unreserved opening of the mind to the truth, whatever it may turn out to be. Faith has no preconceptions; it is a plunge into the unknown. Belief clings, but faith lets go. In this sense of the word, faith is the essential virtue of science, and likewise of any religion that is not self-deception. – Alan Watts

In summary, I’m suggesting we take a fresh look at I/O and functional programming. Instead of putting I/O into functional programming model (as in continuation- and stream-based I/O), or adopting a hybrid functional/imperative model (as in monadic I/O), let’s explore how to move I/O entirely out of our programming model into the implementation of a denotationally simple model for whole systems.

Ever since ActiveVRML, the model we’ve been using in functional reactive programming (FRP) for interactive behaviors is (T->a) -> (T->b), for dynamic (time-varying) input of type a and dynamic output of type b (where T is time). In “Classic FRP” formulations (including ActiveVRML, Fran & Reactive), there is a “behavior” abstraction whose denotation is a function of time. Interactive behaviors are then modeled as host language (e.g., Haskell) functions between behaviors. Problems with this formulation are described in Why classic FRP does not fit interactive behavior. These same problems motivated “Arrowized FRP”. In Arrowized FRP, behaviors (renamed “signals”) are purely conceptual. They are part of the semantic model but do not have any realization in the programming interface. Instead, the abstraction is a signal transformer, SF a b, whose semantics is (T->a) -> (T->b). See Genuinely Functional User Interfaces and Functional Reactive Programming, Continued.

Whether in its classic or arrowized embodiment, I’ve been growing uncomfortable with this semantic model of functions between time functions. A few weeks ago, I realized that one source of discomfort is that this model is mostly junk.

This post contains some partially formed thoughts about how to eliminate the junk (“garbage collect the semantics”), and what might remain.

There are two generally desirable properties for a denotational semantics: full abstraction and junk-freeness. Roughly, “full abstraction” means we must not distinguish between what is (operationally) indistinguishable, while “junk-freeness” means that every semantic value must be denotable.

FRP’s semantic model, (T->a) -> (T->b), allows not only arbitrary (computable) transformation of input values, but also of time. The output at some time can depend on the input at any time at all, or even on the input at arbitrarily many different times. Consequently, this model allows respoding to future input, violating a principle sometimes called “causality”, which is that outputs may depend on the past or present but not the future.

In a causal system, the present can reach backward to the past but not forward the future. I’m uneasy about this ability as well. Arbitrary access to the past may be much more powerful than necessary. As evidence, consult the system we call (physical) Reality. As far as I can tell, Reality operates without arbitrary access to the past or to the future, and it does a pretty good job at expressiveness.

Moreover, arbitrary past access is also problematic to implement in its semantically simple generality.

There is a thing we call informally “memory”, which at first blush may look like access to the past, it isn’t really. Rather, memory is access to a present input, which has come into being through a process of filtering, gradual accumulation, and discarding (forgetting). I’m talking about “memory” here in the sense of what our brains do, but also what all the rest of physical reality does. For instance, weather marks on a rock are part of the rock’s (present) memory of the past weather.

A very simple memory-less semantic model of interactive behavior is just a -> b. This model is too restrictive, however, as it cannot support any influence of the past on the present.

Which leaves a question: what is a simple and adequate formal model of interactive behavior that reaches neither into the past nor into the future, and yet still allows the past to influence the present? Inspired in part by a design principle I call “what would reality do?” (WWRD), I’m happy to have some kind of infinitesimal access to the past, but nothing further.

My current intuition is that differentiation/integration plays a crucial role. That information is carried forward moment by moment in time as “momentum” in some sense.

I call intuition cosmic fishing. You feel a nibble, then you’ve got to hook the fish. – Buckminster Fuller

Where to go with these intuitions?

Perhaps interactive behaviors are some sort of function with all of its derivatives. See Beautiful differentiation for an specification and derived implementation of numeric operations, and more generally of Functor and Applicative, on which much of FRP is based.

I suspect the whole event model can be replaced by integration. Integration is the main remaining piece.

How weak a semantic model can let us define integration?

Thanks

My thanks to Luke Palmer and to Noam Lewis for some clarifying chats about these half-baked ideas. And to the folks on #haskell IRC for brainstorming titles for this post. My favorite suggestions were

• luqui: instance HasJunk FRP where
• luqui: Functional reactive programming’s semantic baggage
• sinelaw: FRP, please take out the trash!
• cale: Garbage collecting the semantics of FRP
• BMeph: Take out the FRP-ing Trash

all of which I preferred over my original “FRP is mostly junk”.

When people talk, especially via the Internet, I expect misunderstandings, since human language is so fraught with ambiguity. And though I work very hard at clarity, I know my skill is limited and I sometimes fail.

So I see communication as an iterative process: I say something that you don’t quite get, and you say “Huh?” about the what or why or how of some particular part, and I refine my message. Likewise, I listen to you, and check out the places were I’m puzzled.

I’m very sensitive where I think I’m seeing criticisms/dismissals without inquiry/curiosity, and hence some peevishness in some remarks (now edited) in my previous post, Why program with continuous time?. See Fostering creativity by relinquishing the obvious for related remarks.

In some venues, like reddit, aggressive non-inquiry can become the dominant mode of discussion. Because I prefer high-quality, open-minded inquiry, I mostly choose moderated blogging instead. If I don’t pass a comment through, I’ll usually offer the writer some suggestions, perhaps suggesting a possible misreading of whatever was being responded to, and invite re-submission. If I’m particularly annoyed with the writer, I’ll usually take time to get over my annoyance before responding, so there can be a delay.

In this way, I try to keep a high signal-to-noise ratio, where noise includes assumptions, reactions to misreadings, and often compounded public attempts by people to get each other to listen more carefully.

I’m starting to discover that people I don’t get along with sometimes have a very different style from mine. I like to invite many possibilities into a space and explore them. My mother shared with me a quote from Henry Nelson Wieman, which she now uses as her email signature:

To get the viewpoint of the other person appreciatively and profoundly and reconcile it with his own so far as possible is the supreme achievement of man and his highest vocation.

While I’m agnostic about the “supreme/highest” part (and open to it), I very much like Wieman’s description of individual and collective learning as progressing most powerfully by integrating different viewpoints, as founded on working to understanding each other “appreciatively and profoundly”.

I’m learning that some other folks have an oppositional style of learning and discovering.

When Thomas (Bob) Davie and I worked together in Antwerp, he told me that his style is to fiercely resist (battle) any new idea proposed to him. Whatever breaks through his defenses is worth his learning. I was flabbergasted and the distance between his style and mine. And greatly relieved also, because I had to work with him, and I had previously interpreted his behavior as non-curious and even anti-creative. Although I wasn’t willing to collaborate in the battle mode at the time, fortunately he was willing to try shifting his style. And the recognition of our differing style toward similar ends helped greatly in relieving the building tension between us. Now I enjoy him very much.

Since this surprising discovery, I’ve wondered how often friction I have with other people coincides with this particular difference in personal styles and whether there are additional style that I hadn’t been aware of. So when friction arises, I now try to find out, via a private chat or email.

Edits:

• 2010-01-04: Filled in Bob’s name (with permission).

Is program proving a viable and useful endeavor? Does it create more problems than it solves?

Today I participated in a reddit discussion thread. The discussion brought up some issues I care about, and I think others will care also.

Although I’m quoting from another person whose viewpoint may contrast with mine, I mean no insult or disrespect to him. I think his views are fairly common, and I appreciate his putting them into writing and stimulating me to put some of mine into writing as well.

[...] you could specify the behavior it should have precisely. However in this case (and in most practical cases that aren’t tricky algorithms actually) the mathematical spec is not much shorter than the implementation. So meeting the spec does not help, the spec is just as likely to be wrong/incomplete as the implementation. With wrong/incomplete I mean that even if you meet the spec you don’t attain your goal.

I like the observation that precise specifications can be as complex and difficult to get correct as precise programs. And indeed specification complexity is problematic. Rather than give up on precision, my own interest is in finding simpler and clearer ways to think about things so that specifications can be quite simple. Thus my focus on simple and precise semantic models. One example is as given in Push-pull functional reactive programming. I’m proud of the simplicity of this specification and of how clearly it shows what parts of the model could use improving. (The Event type doesn’t follow the type class morphism principle, and indeed has difficulty in practice.) And since such semantics-based software design methods are still in their infancy, I agree that it’s usually more practical to abandon preciseness in software development.

For more examples of formally simple specifications, see the papers Denotational design with type class morphisms and Beautiful differentiation.

I suspect the crux of the issue here is that combining precision and simplicity is very difficult — whether for specifications or programs (or for many other other human endeavors).

Any intelligent fool can make things bigger and more complex … it takes a touch of genius — and a lot of courage — to move in the opposite direction. – Albert Einstein

We had been writing informal/imprecise “programs” for a long time before computers were invented, as various kinds of instructions/recipies. With mechanization of program handling in the 1940s came the requirement of precision. Since simple precision is difficult (according to my hypothesis), and precision is required, simplicity usually loses out. So our history of executably-precise programs is populated mostly by complexity.

I conclude that there are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is make it so complicated that there are no obvious deficiencies. The first method is far more difficult. – C.A.R. Hoare, The Emperor’s Old Clothes, Turing Award lecture, 1980

As much as one assumes that the future is bound to perpetuate the past, one might believe that we will never escape complexity. In my own creative process, I like to distinguish between the notion of necessarily true and merely historically true. I understand progress itself to be about this distinction — the overturning of the historically true that turns out (often surprisingly) not to be necessarily true.

What about specification? Are precise specifications necessarily complex, or merely historically complex? (I.e., must we sit by while the future repeats the complexity of the past?) This question I see as even more unexplored than the same question for programs, because we have not been required to work with formal specifications nearly as much as with formal programs. Even so, some people have chosen to work on precise specification and so some progress has been made in improving their simplicity, and I’m optimistic about the future. Optimistic enough to devote some of my life energy to the search.

Simplicity is the most difficult thing to secure in this world; it is the last limit of experience and the last effort of genius. – George Sand

Everything is vague to a degree you do not realize till you have tried to make it precise. – Bertrand Russell

I expect that future software developers will look back on our era as barbaric, the way we look at some of the medical practices of the past. Of course, we’re doing the best we know how, just as past medical practitioners were.

I know people are apt to hear criticism/insult whether present or not, and I hope that these words of mine are not heard as such. I appreciate both the current practitioners, doing their best with tools at hand, and the pioneer, groping toward the methods of the future. Personally, I play both roles, and I’m guessing you do as well, perhaps differing only in balance. I urge us all to appreciate and respect each other more as we muddle on.

For me functional reactive programming (FRP) has been mainly about two principles.

One is denotational design (i.e., based on simple, precise denotational semantics), which has been integral to FRP since the first incarnation as ActiveVRML. I’ve written several things recently about denotational design.

My second core principle is continuous time, which has been present since Fran & ActiveVRML’s predecessor TBAG.

Today I read a confusion that I’ve heard many times before about continuous time, which I’d like to bring up, in part because I love continuous time, and in part because there’s a much broader underlying principle of software design.

[...] I don’t see why the lack of continuous streams leaves a “gap”. In the end all streams are discrete.

“In the end”, yes. Just as in the end, numbers are displayed as ascii numerals. However, programming is more about the middle than the end, i.e., more about composition than about output. For that reason, we don’t generally use strings in place of numbers. If we did, composition operations (arithmetic) would be very awkward. Similarly, continuity in space and in time is better for composition/modularity, leaving discreteness to the output step.

Another name for “continuous” is “resolution-independent”, and thus able to be transformed in time and space with ease and without propagating and amplifying sampling artifacts.

As another example, consider the data types in a 3D graphics API. In the end, all graphics is pixels, isn’t it? So what gap is left in a pixel-oriented API that doesn’t address higher-level continuous notions like triangles or curved surfaces? (Hint: it’s not just speed.)

One could go further than strings and pixels, and say that “in the end” my data types will end up as phosphors or electrical impulses, so programming at those levels is perfectly adequate. Again, composability would suffer.

Another example is functional vs imperative programming. It’s all side-effects in the end. Functional programming excels in composability, as explained and illustrated by John Hughes in Why Functional Programming Matters. And I’m not just talking about pure functional programming, but the availability of compound expressions, as introduced in Fortran (controversially), despite that machines just execute sequences of atomic, side-effecting statements in the end.

Another example, and really the heart of John’s paper, is finite vs infinite data structures. We only access a finite amount of data in the end. However, allowing infinite data structures in the middle makes for a much more composable programming style.

Some unsolicited advice to us all: Next time you see someone doing something and you don’t understand their underlying motivation, ask them. Many issues are not immediately obvious, so don’t be shy! Reading papers can help as well.

For more on continuous time:

• TBAG: A High Level Framework for Interactive, Animated 3D Graphics Applications
• Functional Reactive Animation
• An Embedded Modeling Language Approach to Interactive 3D and Multimedia Animation
• Functional Implementations of Continuous Modeled Animation

Edits:

• 2010-01-03: Trimmed & tweaked the unsolicited advice. Hopefully less peevish/patronizing now. Thanks for the feedback.
• 2010-01-07: Trimmed quote.

This post describes a simple way to integrate a function over an interval and get an exact answer. The question came out of another one, which is how to optimally render a continuous-space image onto a discrete array of pixels.

For anti-aliasing, I’ll make two simplying assumptions (to be revisited):

• Each pixel is a square area. (With apologies to Alvy Ray Smith.)
• Since I can choose only one color per pixel, I want exactly the average of the continuous image over pixel’s subregion of 2D space.

The average of a function over a region (here a continuous image over a 2D interval) is equal to the integral of the function across the region divided by the size (area for 2D) of the region. Since our regions are simple squares, the average and the integral can each be defined easily in terms of the other (dividing or multiplying by the area).

To simplify the problem further, I’ll consider one-dimensional integral, i.e., integrating a function of R over a 1D interval. My solution below involves the least upper bound operator I’ve written about (and its specialization unamb).

A first simple algorithm

Integration takes a real-valued function and an interval (low & high) and gives a real.

integral :: (R->R) -> R -> R -> R

Integration has a property of interval additivity, i.e., the integral of a function from a to c is the sum of the integral from a to b and the integral from b to c.

∀ a b c. integral f a c == integral f a b + integral f b c

which immediately leads to a simple recursive algorithm:

integral f low high = integral f low mid + integral f mid high
  where mid = (low + high) / 2

Extending to 2D is simple: we could divide a rectangular region into four quarter subregions, or into two subregions by splitting the longer dimension. The quartering variation is very like mipmapping, as used to anti-alias textures. Mipmapping takes a pixel array and builds a pyramid of successively lower resolution versions. Each level except the first is constructed out of the previous (next higher-resolution) level by averaging blocks of four pixels into one. The simple integral algorithm above (extended to 2D) is like mipmapping when we start with an infinite resolution (i.e., continuous) texture.

Umm …

Maybe you’re thinking what I’m thinking: Hey! We don’t have a base case, so we won’t even get off the ground.

Given that our domain is continuous, I don’t know what to use for a base case. So let’s consider what the purpose of a base case is, and see whether that purpose can be accomplished some other way.

What’s so important about base cases?

Does every recursion need a base case in order to avoid being completely undefined?

Here’s a counter-example: mapping a function over an infinite stream, taken from the source code of the Stream package.

data Stream a = Cons a (Stream a)
 
instance Functor Stream where
  fmap f ~(Cons x xs) = Cons (f x) (fmap f xs)

The key thing here is that Cons is non-strict in its second argument, which holds the recursive call. (Definition: a function f is “strict” if f ⊥ == ⊥.)

Non-strictness of if-then-else is exactly what allows more mundane recursions to produce defined (non-⊥) results as well, e.g.,

fac n = if n <= 0 then 1 else fac (n-1)

In this fac example, if-then-else must be (and is) non-strict in its third argument (the recursive call).

So “base case” is not the heart of the matter; non-strictness is.

Finding non-strictness

The trouble with our elegant recursive derivation of integral above is that addition is strict in its arguments (where the recursive integral calls appear). This strictness means that we cannot get any information at all out of the right-hand side until we get some information out of the recursive calls, which won’t happen until we get information out of their recursive calls, ad infinitum.

To escape this information black hole, can we find some scrap of information about the value of the integral that doesn’t (always) rely on the recursive calls?

Interval analysis (IA) provides an answer. The idea of IA is to extend functions to apply to intervals of values and produce intervals of results. The interval versions are sloppy in that a result interval may hold values not corresponding to anything in the source interval. (In math-speak, a result interval may be a proper superset of the image of the source interval under the function.)

Applying an interval version of the function to the source interval results in lower and upper bounds for f. The average must be between these bounds, so the integral is bounded by these bounds multiplied by the interval size. (Or use more sophisticated variations on IA, such as affine analysis, etc.)

Now we have some information. How can we mix it in with the sum of recursive calls to integral? We can use (⊔) (least upper bound or “lub”), which is perfect for the job because its meaning is exactly to combine two pieces of information. See Merging partial values and Lazier function definitions by merging partial values.

Instead of treating IA as operating on intervals, think of it as operating on “partial numbers”, i.e., inexact values. Suppose we define a type of numbers that consistently generalizes exact numbers but is additionally populated with inexact values.

type Partial R  -- abstract
 
between :: Ord a => a -> a -> Partial a
exact   :: a -> Partial a

Perhaps exact a = between a a.

Now we can escape the black hole:

integral f low high = ((high - low) * f (between low high)) ⊔
                      (integral f low mid + integral f mid high)
  where mid = (low + high) / 2

Representation

One representation of a partial value is an interval. In this representation, (⊔) is interval intersection.

type Partial a = (a,a)  -- first try
 
(l,h) ⊔ (l',h') = (l `max` l', h `min` h')

If the lower and bounds are plain old exact numbers, then this choice of representation and (⊔) has a fatal flaw. The max and min functions are strict, so (⊔) can easily produce less information than it is given, while its job is to combine all information present in both arguments. (For instance, let l == 3 and l' == ⊥. Then l `max` l' == ⊥, so we’ve lost the information from l.)

One possible solution is to change the kind of numbers used in the bounds in such a way that max and min are not strict. Let’s use two new types, for lower and upper bounds:

type Lower a  -- abstract
type Upper a  -- abstract

These two types also capture partial information about numbers, though they cannot express exact numbers (other than maxBound and minBound, where available).

Now we can represent partial numbers.

type Partial a = (Lower a, Upper a)
 
(l,h) ⊔ (l',h') = (l ⊔ l', h ⊔ h')

Notice that I’ve replaced both max and min each by (⊔). Now I realize that I only used max and min above as a way of combining the information that the lower and upper bounds (respectively) gave us. The (⊔) operator states this intention directly.

Pleasantly, we don’t even have to state this definition, as (⊔) is already defined this way for pairs. (Well, not quite; see Merging partial values.)

Improving values and intervals

This idea for representing partial values is very like what Warren Burton and Ken Jackson called “improving intervals”, which is a two-sided version of Warren’s “improving values” (corresponding to Lower above). (See Encapsulating nondeterminacy in an abstract data type with deterministic semantics (JFP, 1991) and Improving Intervals (JFP, 1993). As these papers are hard to find, you might start with Ken Jackson’s dissertation.)

Warren and Ken represented improving values and intervals as lazy, possibly-infinite lists of improving approximations. While an improving value is represented as a sequence (finite or infinite) of monotonically increasing values, an improving interval is represented as a sequence of monotonically shrinking intervals (each containing the next). The denotation of one of these improving representations is the limit of the sequence. Any query for information not specifically present in a partial value would yield ⊥, just as applying a partial function (or data structure) outside its domain yields ⊥. For instance, one could ask a partial number for successive bits. If a requested bit cannot be determined from the available bounds, then that bit and all later bits are ⊥.

Dropping lub

There’s a subtlety in the second definition of integral above. My goal is that integral yield a completely defined (exact) number. We can meet this goal for fairly well-behaved functions, since IA gives decreasing errors as input intervals shrink, and the result intervals are multiplied by shrinking interval sizes. Even discontinuities in the integrated function will be smoothed out, if there are only finitely many, thanks to the multiplication.

Completely defined values are at the tops of the information ordering. (I’m assuming we don’t have the “over-defined” (self-contradictory) value ⊤.) The sum of recursive calls is also fully defined, and starter partial value, f (between low high), has strictly less information, i.e.,

   (high - low) * f (between low high)
⊑  integral f low high
=  integral f low mid + integral f mid high

So we can get by with a less general form of (⊔). If we represent our numbers as lazy lists of improving intervals, then we can simply use (:) in place of (⊔).

Hm. My reasoning just above is muddled. I think the crucial property is that

     (high - low) * f (between low high)
⊑    (mid  - low) * f (between low mid )
   + (high - mid) * f (between mid high)

To see why this property holds, first subdivide:

     (high - low) * f (between low high)
==   ((mid - low) + (high - mid)) * f (between low high)
==   (mid  - low) * f (between low high)
   + (high - mid) * f (between low high)

Next apply information monotonicity, i.e., more information (smaller interval) in yields more information out:

f (between low high) ⊑ f (between low med)
 
f (between low high) ⊑ f (between med high)

from which the crucial property follows.

Note: the notation is potentially confusing, since smaller intervals means more information, and so (⊑) means (⊇), and (⊔) means (∩).

Efficiency and benign side-effects

There is a serious efficiency problem with this (lazy) list-based representation of improving values or intervals, as pointed out by Ken and Warren:

At any point in time, it is really only the tightest bounds currently in the list that are important. [...] Hence, the list representation consumes more space than is necessary, and time is wasted examining the out-of-date values in the list. A better representation, in a language that allows update-in-place, would be a pair consisting of the tightest lower bound and the tightest upper bound found so far. This pair would be updated-in-place when better bounds become known. (Improving Intervals, section 5.2)

The update-in-place suggested here is semantically benign, i.e., does not compromise pure functional semantics, because it doesn’t change the information content. Instead, it makes that content more efficient to access a second time. The same sort of semantically benign optimization underlies lazy evaluation, with the run-time system destructively replacing a thunk upon its first evaluation.

This benign-update idea is also explored in Another angle on functional future values.

Averaging vs integration

In practice, I’d probably rather use a recursive interval average function instead of a recursive integral. Recall that with the recursive integral, we had to multiply by the interval size at each step. The intervals get very small, and I worry about having to combine numbers of greatly differing magnitudes. With a recursive average, the numbers get averaged at each step, which I expect means we’ll be combining numbers of similar magnitudes.

average f low high = f (between low high) ⊔
                     average f low mid `avg` average f mid high
  where mid = low `avg` high
 
a `avg` b = (a + b) / 2
 
integral f low high = (high - low) * average f low high   -- non-recursive

Here’s a modified form that can apply to higher dimensions as well:

average f iv = f iv ⊔ mean [average f iv' | iv' <- subdivide iv]
 
mean l = sum l / length l
 
integral f iv = size iv * average f iv   -- non-recursive

Exact computation

The algorithms described above can easily run afoul of inexact numeric representations, such as Float or Double. With such representation types, as recursive subdivision progresses, at some point, an interval will be bounded by consecutive representable numbers. At that point, sub-dividing an interval will result in an empty interval plus the pre-divided interval, and we will stop making progress.

One solution to this problem is use exact number representations. Another is to use progressively precise inexact representations.

The central question for me in designing software is always

What does it mean?

With functional programming, this question is especially crisp. For each data type I define, I want to have a precise and simple mathematical model. (For instance, my model for behavior is function-of-time, and my model of images is function-of-2D-space.) Every operation on the type is also given a meaning in terms of that semantic model.

This specification process, which is denotational semantics applied to data types, provides a basis for

• correctness of the implementation,
• user documentation free of implementation detail,
• generating and proving properties, which can then be used in automated testing, and
• evaluating and comparing the elegance and expressive power of design decisions.

For an example (2D images), some motivation of this process, and discussion, see Luke Palmer’s post Semantic Design. See also my posts on the idea and use of type class morphisms, which provide additional structure to denotational design.

In spring of 2008, I started working on a functional 3D library, FieldTrip. I’ve designed functional 3D libraries before as part of TBAG, ActiveVRML, and Fran. This time I wanted a semantics-based design, for all of the reasons given above. As always, I want a model that is

• simple,
• elegant, and
• general.

For 3D, I also want the model to be GPU-friendly, i.e., to execute well on (modern) GPUs and to give access to their abilities.

I hadn’t thought of or heard a model that I was happy with, and so I didn’t have the sort of firm ground I like to stand on in working on FieldTrip. Last February, such a model occurred to me. I’ve had this blog post mostly written since then. Recently, I’ve been focused on functional 3D again for GPU-based rendering, and then Sean McDirmid posed a similar question, which got me thinking again.

Geometry

3D graphics involves a variety of concepts. Let’s start with 3D geometry, using a surface (rather than a solid) model.

Examples of 3D (surface) geometry include

• the boundary (surface) of a solid box, sphere, or torus,
• a filled triangle, rectangle, or circle,
• a collection of geometry , and
• a spatial transformation of geometry.

First model: set of geometric primitives

One model of geometry is a set of geometric primitives. In this model, union means set union, and spatial transformation means transforming all of the 3D points in all of the primitives in the set. Primitives contain infinitely (even uncountably) many points, so that’s a lot of transforming. Fortunately, we’re talking about what (semantics), and not how (implementation).

What is a geometric primitive?

We could say it’s a triangle, specified by three coordinates. After all, computer graphics reduces everything to sets of triangles. Oops — we’re confusing semantics and implementation. Tessellation approximates curved surfaces by sets of triangles but loses information in the process. I want a story that includes this approximation process but keeps it clearly distinct from semantically ideal curved surfaces. Then users can work with the ideal, simple semantics and rely on the implementation to perform intelligent, dynamic, view-dependent tessellation that adapts to available hardware resources.

Another model of geometric primitive is a function from 2D space to 3D space, i.e., the “parametric” representation of surfaces. Along with the function, we’ll probably want some means of describing the subset of 2D over which the surface is defined, so as to trim our surfaces. A simple formalization would be

type Surf = R2 -> Maybe R3

where

type R  -- real numbers
type R2 = (R,R)
type R3 = (R,R,R)

For shading, we’ll also need normals, and possibly tangents & bitangents, We can get these features and more by including derivatives, either just first derivatives or all of them. See my posts on derivatives and paper Beautiful differentiation.

In addition to position and derivatives, each point on a primitive also has material properties, which determines how light is reflected by and transmitted through the surface at the point.

type Surf = R2 -> Maybe (R2 :> R3, Material)

where a :> b contains all derivatives (including zeroth) at a point of a function of type a->b. See Higher-dimensional, higher-order derivatives, functionally. We could perhaps also include derivatives of material properties:

type Surf = R2 :~> Maybe (R3, Material)

where a :~> b is the type of infinitely differentiable functions.

Combining geometry values

The union function gives one way to combine two geometry values. Another is morphing (interpolation) of positions and of material properties. What can the semantics of morphing be?

Morphing betwen two surfaces is easier to define. A surface is a function, so we can interpolate point-wise: given surfaces r and s, for each point p in parameter space, interpolate between (a) r at p and (b) s at p, which is what liftA2 (on functions) would suggest.

This definition works if we have a way to interpolate between Maybe values. If we use liftA2 again, now on Maybe values, then the Just/Nothing (and Nothing/Just) cases will yield Nothing. Is this semantics desirable? As an example, consider a flat square surface with hole in the middle. One square has a small hole, and the other has a big hole. If the size of the hole corresponds to size of the portion of parameter space mapped to Nothing, then point-wise interpolation will always yield the larger hole, rather than interpolating between hole sizes. On the other hand, the two surfaces with holes might be Just over exactly the same set of parameters, with the function determining how much the Just space gets stretched.

One way to characterize this awkwardness of morphing is that the two functions (surfaces) might have different domains. This interpretation comes from seeing a -> Maybe b as encoding a function from a subset of a (i.e., a partial function on a).

Even if we had a satisfactory way to combine surfaces (point-wise), how could we extend it to combining full geometry values, which can contain any number of surfaces? One idea is to model geometry as an structured collection of surfaces, e.g., a list. Then we could combine the collections element-wise. Again, we’d have to deal with the possibility that the collections do not match up.

Surface tuples

Let’s briefly return to a simpler model of surfaces:

type Surf = R2 -> R3

We could represent a collection of such surfaces as a structured collection, e.g., a list:

type Geometry = [Surf]

But then the type doesn’t capture the number of surfaces, leading to mismatches when combining geometry values point-wise.

Alternatively, we could make the number of surfaces explicit in the type, via tuples, possibly nested. For instance, two surfaces would have type (Surf,Surf).

Interpolation in this model becomes very simple. A general interpolator works on vector spaces:

lerp :: VectorSpace v => v -> v -> Scalar v -> v
lerp a b t = a ^+^ t*^(b ^-^ a)

or on affine spaces:

alerp :: (AffineSpace p, VectorSpace (Diff p)) =>
         p -> p -> Scalar (Diff p) -> p
alerp p p' s = p .+^ s*^(p' .-. p)

Both definitions are in the vector-space package. That package also includes VectorSpace and AffineSpace instances for both functions and tuples. These instances, together with instances for real values suffice to make (possibly nested) tuples of surfaces be vector spaces and affine spaces.

From products to sums

Function pairing admits some useful isomorphisms. One replaces a product with a product:

(a → b) × (a → c) ≅ a → (b × c)

Using this product/product isomorphism, we could replace tuples of surfaces with a single function from R² to tuples of R³.

There is also a handy isomorphism that relates products to sums, in the context of functions:

(b → a) × (c → a) ≅ (b + c) → a

This second isomorphism lets us replace tuples of surfaces with a single “surface”, if we generalize the notion of surface to include domains more complex than R².

In fact, these two isomorphisms are uncurried forms of the general and useful Haskell functions (&&&) and (|||), defined on arrows:

(&&&) :: Arrow       (~>) => (a ~> b) -> (a ~> c) -> (a ~> (b,c))
(|||) :: ArrowChoice (~>) => (a ~> c) -> (b ~> c) -> (Either a b ~> c)

Restricted to the function arrow, (|||) == either.

The second isomorphism, uncurry (|||), has another benefit. Relaxing the domain type to allow sums opens the way to other domain variations as well. For instance, we can have types for triangular domains, shapes with holes, and other flavors of bounded and unbounded parameter spaces. All of these domains are two-dimensional, although they may result from several patches.

Our Geometry type now becomes parameterized:

type Geometry a = a -> (R3,Material)

The first isomorphism, uncurry (&&&), is also useful in a geometric setting. Think of each component of the range type (here R3 and Material) as a surface “attribute”. Then (&&&) merges two compatible geometries, including attributes from each. Attributes could include position (and derivatives) and shading-related material, as well as non-visual properties like temperature, elasticity, stickiness, etc.

With this flexibility in mind, Geometry gets a second type parameter, which is the range type. Now there’s nothing left of the Geometry type but general functions:

type Geometry = (->)

Recall that we’re looking for a semantics for 3D geometry. The type for Geometry might be abstract, with (->) being its semantic model. In that case, the model suggests that Geometry have all of the same type class instances that (->) (and its full or partial applications) has, including Monoid, Functor, Applicative, Monad, and Arrow. The semantics of these instances would be given by the corresponding instances for (->). (See posts on type class morphisms and the paper Denotational design with type class morphisms.)

Or drop the notion of Geometry altogether and use functions directly.

Domains

I’m happy with the simplicity of geometry as functions. Functions fit the flexibility of programmable GPUs, and they provide simple, powerful & familiar notions of attribute merging ((&&&)) and union ((|||)/either).

The main question I’m left with: what are the domains?

One simple domain is a one-dimensional interval, say [-1,1].

Two useful domain building blocks are sum and product. I mentioned sum above, in connection with geometric union ((|||)/either) Product combines domains into higher-dimensional domains. For instance, the product of two 1D intervals is a 2D interval (axis-aligned filled rectangle), which is handy for some parametric surfaces.

What about other domains, e.g., triangular, or having one more holes? Or multi-way branching surfaces? Or unbounded?

One idea is to stitch together simple domains using sum. We don’t have to build any particular spatial shapes or sizes, since the “geometry” functions themselves yield the shape and size. For instance, a square region can be mapped to a triangular or even circular region. An infinite domain can be stitched together from infinitely many finite domains. Or it can be mapped to from a single finite domain. For instance, the function \ x -> x / abs (1-x) maps [-1,1] to [-∞,∞].

Alternatively, we could represent domains as typed predicates (characteristic functions). For instance, the closed interval [-1,1] would be \ x -> abs x <= 1. Replacing abs with magnitude (for inner product spaces), generalizes this formulation to encompass [-1,1] (1D), a unit disk (2D), and a unit ball (3D).

I like the simple generality of the predicate approach, while I like how the pure type approach supports interpolation and other pointwise operations (via liftA2 etc).

Tessellation

I’ve intentionally formulated the graphics semantics over continuous space, which makes it resolution-independent and easy to compose. (This formulation is typical for 3D geometry and 2D vector graphics. The benefits of continuity apply to generally imagery and to animation/behavior.)

Graphics hardware specializes in finite collections of triangles. For rendering, curved surfaces have to be tessellated, i.e., approximated as collections of triangles. Desirable choice of tessellation depends on characteristics of the surface and of the view, as well as scene complexity and available CPU and GPU resources. Formulating geometry in its ideal curved form allows for automated analysis and choice of tessellation. For instance, since triangles are linear, the error of a triangle relative to the surface it approximates depends on how non-linear the surface is over the subset of its domain corresponding to the triangle. Using interval analysis and derivatives, non-linearity can be measured as a size bound on the second derivative or a range of first derivative. Error could also be analyzed in terms of the resulting image rather than the surface.

For a GPU-based implementation, one could tessellate dynamically, in a “geometry shader” or (I presume) in a more general framework like CUDA or OpenCL.

Abstractness

A denotational model is “fully abstract” when it equates observationally equivalent terms. The parametric model of surfaces is not fully abstract in that reparameterizing a surface yields a different function that looks the same as a surface. (Surface reparametrization alters the relationship between domain and range, while covering exactly the same surface, geometrically.) Properties that are independent of particular parametrization are called “geometric”, which I think corresponds to full abstraction (considering those properties as semantic functions).

What might a fully abstract (geometric) model for geometry be?

Part one of this series introduced the problem of memoizing functions involving polymorphic recursion. The caching data structures used in memoization typically handle only one type of argument at a time. For instance, one can have finite maps of differing types, but each concrete finite map holds just one type of key and one type of value.

I extended memoization to handle polymorphic recursion by using an existential type together with a reified type of types. This extension works (afaik), but it is restricted to a particular form for the type of the polymorphic function being memoized, namely

-- Polymorphic function
type k :--> v = forall a. HasType a => k a -> v a

My motivating example is a GADT-based representation of typed lambda calculus, and some of the functions I want to memoize do not fit the pattern. After writing part one, I fooled around and found that I could transform these awkwardly typed polymorphic functions into isomorphic form that does indeed fit the restricted pattern of polymorphic types I can handle.

Awkward types

The first awkwardly typed memoizee is the function application constructor:

type AT = forall a b . (HasType a, HasType b) => E (a -> b) -> E a -> E b
 
(:^) :: AT

Right away AT misses the required form. It has two HasType constraints, and the first argument is parameterized over two type variables instead of one. However, the second argument looks more promising, so let’s flip the arguments to get an isomorphic type:

forall a b . (HasType a, HasType b) => E a -> E (a -> b) -> E b

And then move the quantifier and constraint on b inside the outer (first) ->:

forall a . HasType a => E a -> (forall b. HasType b => E (a -> b) -> E b)

We’re getting closer. Next, define a newtype wrapper.

newtype A2 a = A2 (forall b. HasType b => E (a -> b) -> E b)

So that AT is isomorphic to

forall a . HasType a => E a -> A2 a

i.e.,

E :--> A2

The function inside of A2 doesn’t have the required form, but another newtype wrapper finishes the job.

newtype EF a b = EF {unEF :: E (a -> b) }
 
type H' a = EF a :--> E
 
newtype H a = H { unH :: H' a }

The AT type is isomorphic AP where

type AP = E :--> H

Curried memoization

A “curried memo function” is one that takes one argument and produces another memo function. For a simple memo function, not involving polymorphic recursion, there’s a simple recipe for curried memoization:

memo2 :: (a -> b -> c) -> (a -> b -> c)
memo2 f = memo (memo . f)

Our more polymorphic memo makes currying a little more awkward. First, here’s a helper function for working inside of the representation of an H:

inH :: (H' a -> H' a) -> (H a -> H a)
inH h z = H (h (unH z))

The following more elegant definition doesn’t type-check, due to the rank 2 polymorphism:

inH f = H . f . unH  -- type error

Now our AP memoizer is much like memo2:

memoAP :: AP -> AP
memoAP app' = memo (inH memo . app')

(A more general, consistent type for memoAP is (f :--> H) -> (f :--> H).)

Isomorphisms

Now, to define the isomorphisms. Define

toAP   :: AT -> AP
fromAP :: AP -> AT

The definitions:

toAP app ea = H $ \ (EF eab) -> app eab ea
fromAP app' eab ea = unH (app' ea) (EF eab)

If you erase the newtype wrappers & unwrappers, you’ll see that toAP and fromAP are both just flip.

I constructed fromAP from the following specification:

toAP (fromAP app') == app'

Transforming step-by-step into equivalent specifications:

\ ea -> H $ \ (EF eab) -> (fromAP app') eab ea == app'
 
H $ \ (EF eab) -> (fromAP app') eab ea == app' ea
 
\ (EF eab) -> (fromAP app') eab ea == unH (app' ea)
 
(fromAP app') eab ea == unH (app' ea) (EF eab)
 
fromAP app' eab ea == unH (app' ea) (EF eab)

Memoizing vis isomorphisms

Finally, I can memoize

memoAT :: AT -> AT
memoAT app = fromAP (memoAP (toAP app))

Again, a more elegant definition via (.) fails to type-check, due to rank 2 polymorphism.

The Lam (lambda abstraction) constructor can be handled similarly:

Lam  :: (HasType a, HasType b) => V a -> E b -> E (a -> b)

This time, no flip is needed.

I wonder

How far does this isomorphism trick go?

Is there an easier way to memoize polymorphic functions?

Memoization takes a function and gives back a semantically equivalent function that reuses rather than recomputes when applied to the same argument more than once. Variations include not-quite-equivalence due to added strictness, and replacing value equality with pointer equality.

Memoization is often packaged up polymorphically:

memo :: (???) => (k -> v) -> (k -> v)

For pointer-based (“lazy”) memoization, the type constraint (“???”) is empty. For equality-based memoization, we’d need at least Eq k, and probably Ord k or HasTrie k for efficient lookup (in a finite map or a possibly infinite memo trie).

Although memo is polymorphic, its argument is a monomorphic function. Implementations that use maps or tries exploit that monomorphism in that they use a type like Map k v or Trie k v. Each map or trie is built around a particular (monomorphic) type of keys. That is, a single map or trie does not mix keys of different types.

Now I find myself wanting to memoize polymorphic functions, and I don’t know how to do it.

Flavors of polymorphism

If a recursively defined function f is polymorphic, then the recursion may still be monomorphic. That is, recursive calls may be restricted to the same type instance as the parent call. Most recursive polymorphic functions fit this form, because most polymorphic recursive data types are “regular”, meaning that a polymorphic data type is included in itself only at the same type instance. For instance, the usual polymorphic lists and trees are regular.

Example: GADTs

Among other places, non-regular, or “nested data types” arise in statically typed encodings of typed languages. For instance, here’s a GADT (generalized algebraic data type) for typed lambda calculus expressions:

-- Variables
data V a = V String (Type a)
 
-- Expressions
data E :: * -> * where
  Lit  :: a -> E a                      -- literal
  Var  :: V  a -> E a                   -- variable
  (:^) :: E (a -> b) -> E a -> E b      -- application
  Lam  :: V a -> E b -> E (a -> b)      -- abstraction

The Type type is sort of like TypeRep, except that it is statically typed.

data Type :: * -> * where
  Bool   :: Type Bool
  Float  :: Type Float
  ...
  (:*:)  :: Type a -> Type b -> Type (a,b)
  (:->:) :: Type a -> Type b -> Type (a->b)

These GADTs (E and Type) are both non-regular, and so recursive functions over them will involve more than one argument type.

So how can we memoize?

A first try

Let’s consider a specific case of polymorphic functions:

type k :--> v = ∀ a. HasType a => k a -> v a

HasType is to Typeable as Type is to TypeRep. The memo implementation I’m playing with relies on HasType.

The memoizer can have type

memo :: (k :--> v) -> (k :--> v)

which uses rank 2 polymorphism (because of the argument type’s ∀, which cannot be moved to the outside).

My implementation of memo is similar to the discussion in Stretching the storage manager: weak pointers and stable names in Haskell, but modernized a bit and adapted for polymorphism. It uses an IntMap of lists of pairs of stable names (akin to pointers) for arguments and values for results. The idea is to first use hashStableName to get an Int to use as an IntMap key, and then linearly traverse the resulting list of binding pairs, comparing stable keys for equality. (StableName has Eq but not Ord.) Although hashStableName can map different stable names to the same hash value, collisions are rare, so the StableBind lists rarely have more than one element and the linear search is cheap.

type SM k v = I.IntMap [StableBind k v]  -- Stable map
 
data StableBind k v = ∀ a. HasType a => SB (StableName (k a)) (v a)

The reason for hiding the type parameter a in StableName is so that different bindings can be for different types.

The key tricky bit is managing static typing while searching for a particular StableName a in a [StableBind]. Here’s my implementation:

blookup :: ∀ k v a. HasType a =>
           StableName (k a) -> [StableBind k v] -> Maybe (v a)
blookup stk = look
 where
   look :: [StableBind k v] -> Maybe (v a)
   look [] = Nothing
   look (SB stk' v : binds') 
     | Just Refl <- tya `tyEq` typeOf2 stk', stk == stk' = Just v
     | otherwise                                         = look binds'
   tya :: Type a
   tya = typeT

The crucial magic bit is

tyEq :: Type a -> Type b -> Maybe (a :=: b)

where a :=: b represents a proof that the types a and b are the same type. The proof type has a simple GADT representation:

data (:=:) :: * -> * -> * where Refl :: a :=: a

This simple type definition ensures that only valid type-equality proofs can exist. Well, except for ⊥. The guard’s pattern match with Just Refl will force evaluation, so that ⊥ can’t sneak by us. That match also informs the type-checker that stk and stk' are stable names of the same type in that clause, which then makes stk == stk' be well-typed, and makes Just v have the required type, i.e., Maybe (v a).

Finally, the typeOf2 function is a simple helper that peels off two type constructors and extracts a Type:

typeOf2 :: HasType a => g (f a) -> Type a

Wishing for more

The type of memo above is too restrictive for my uses. It only handles polymorphic functions of type k a -> v a, and only with the single constraint HasType a.

The reason I want lazy memoization now is that I’m compressing expressions to maximize representation sharing, as John Hughes described in Lazy Memo-functions. Once sharing is maximized, pointer-based memoization works better, because equal values are pointer-equal. To compress an expression, simply use a memoized copy function, as John suggested.

compress :: HasType a => E a -> E a
compress e = mcopy e
 where
   mcopy, copy :: HasType b => E b -> E b
   -- Memo version
   mcopy = memo copy
   -- Copier, with memo-copied components
   copy (u :^ v)  = appM (mcopy u) (mcopy v)
   copy (Lam v b) = lamM v (mcopy b)
   copy e         = e
   -- memoized constructors
   appM :: (HasType a, HasType b) => E (a -> b) -> E a -> E b
   appM = memo2 (:^)
   lamM :: (HasType a, HasType b) => V a -> E b -> E (a -> b)
   lamM = memo2 Lam

The memo2 function is defined in terms of memo, using some some newtype trickery. Its type:

memo2 :: HasType a =>
         (k a -> l a -> v a) -> (k a -> l a -> v a)

But, sigh, the (:^) and Lam constructors I’m trying to memoize do not have the required types. My higher-order-polymorphic memo functions do not have flexible enough types.

And this is where I’m stuck.

I’d appreciate your ideas and suggestions.

There has been a scurry of reaction on twitter and reddit to Robert Fischer’s post saying that Scala is Not a Functional Programming Language. James Iry responded by saying that analogous reasoning leads to the conclusion that Erlang is Not Functional

My first inclination was to suggest that Haskell, as commonly practiced (with monadic IO), is not a functional language either. Instead, I’m going to explain how it is that the C language is purely functional.

The story of C

First, let me make that claim more precise. What I really mean is that almost everyone who “writes C programs” is really programming in a purely functional language. Modulo some minor historical accidents.

“C programmers” really program not in C, but in the purely functional language cpp. As with any purely functional language, cpp consists only of (nestable) expressions, not statements. Now here’s the clever part: when evaluated (not executed), a cpp expression yields a (pure) value of type C, which is an ADT (abstract type) that represents imperative programs. That value of type C is then executed (not evaluated) by the cpp language’s RTS (run-time system). As a clever optimization, cpp’s RTS actually includes a code generator, considerably improving performance over more naïve implementations of the C abstract data type.

(So now you can see that, “C programmer” is a almost always a misnomer when applied to a person. Instead people are cpp programmers, and cpp programs are the real C programmers (producers of C). I don’t think I can buck this misnomer, so I’ll stay with the herd and use “C programming” to mean cpp programming.)

In this way, the geniuses Brian Kernighan and Dennis Ritchie applied monads (from category theory) to programming language design considerably before the work of Moggi and Wadler. (Even earlier examples of this idea can be found, so I’m taking liberty with history, choosing to give K&R credit for the sake of this post.)

Now along the way, some mistakes were made that tarnished the theoretical beauty of cpp+C:

• Although cpp is in spirit purely functional, pragmatists insisted on the unnecessary and theoretically awkward construct #undef, which allows a single name to be re-defined.

• Scoping rules are not as clean as, say, the lambda calculus or Scheme, (But hey — even John McCarthy got scoping wrong.)

• The C ADT is implemented simply as String (or char *, for you type theorists, using a notation from Kleene), and the representation is exposed rather than hidden, so that cpp programs operate directly on strings. The wisdom of data abstraction caught on later.

Fortunately for purists, the relatively obscure programming language Haskell restored the original untarnished beauty of K&R’s vision by fixing these defects. The type system was modernized, scoping rules improved, and the C type (renamed to “IO“, to avoid expensive legal battles) made abstract.

(Admittedly, Haskell also had some shameful pre-teen influences from hooligans like Church, Curry, Landin, and Milner, before it settled its true cpp+C nature, under the guiding influence of Wadler (under the influence of Moggi).)

What about Haskell?

Which leads to the question: is Haskell+IO a purely functional programming language?

Sure it is, even as much as its predecessor C (more precisely, cpp+C).

Some who ought to know better would claim that Haskell+IO is only technically pure. (See claims and follow-up discussion, in response to a remark from James Iry.)

I must then respond that, in that case, even C programming is only technically pure.

Which is absurd.

Lately I’ve been learning that some programming principles I treasure are not widely shared among my Haskell comrades. Or at least not widely among those I’ve been hearing from. I was feeling bummed, so I decided to write this post, in order to help me process the news and to see who resonates with what I’m looking for.

One of the principles I’m talking about is that the value of a closed expression (one not containing free variables) depends solely on the expression itself — not influenced by the dynamic conditions under which it is executed. I relate to this principle as the soul of functional programming and of referential transparency in particular.

Recently I encountered two facts about standard Haskell libraries that I have trouble reconciling with this principle.

• The meaning of Int operations in overflow situations is machine-dependent. Typically they use 32 bits when running on 32-bit machine and 64 bits when running on 64-bit machines. Implementations are free to use as few as 29 bits. Thus the value of the expression “2^32 == (0 ::Int)” may be either False or True, depending on the dynamic conditions which it is evaluated.
• The expression “System.Info.os” has type String, although its value as a sequence of characters depends on the circumstances of its execution. (Similarly for the other exports from System.Info. Hm. I just noticed that the module is labeled as “portable”. Typo? Joke?)

Although I’ve been programming primarily in Haskell since around 1995, I didn’t realize that these implementation-dependent meanings were there. As in many romantic relationships, I suppose I’ve been seeing Haskell not as she is, but as I idealized her to be.

There’s another principle that is closely related to the one above and even more fundamental to me: every type has a precise, specific, and preferably simple denotation. If an expression e has type T, then the meaning (value) of e is a member of the collection denoted by T. For instance, I think of the meaning of the type String, i.e., of [Char], as being sequences of characters. Well, not quite that simple, because it also contains some partially defined sequences and has a partial information ordering (non-flat in this case). Given this second principle, if os :: String, then the meaning of os is some sequence of characters. Assuming the sequence is finite and non-partial, it can be written down as a literal string, and that literal can be substituted for every occurrence of “os” in a program, without changing the program’s meaning. However, os evaluates to “linux” on my machine and evaluates to “darwin” on my friend Bob’s machine, so substituting any literal string for “os” would change the meaning, as observable on at least one of these machines.

Now I realize I’m really talking about standard Haskell libraries, not Haskell itself. When I discussed my confusion & dismay in the #haskell chat room, someone suggested explaining these semantic differences in terms of different libraries and hence different programs (if one takes programs to include the libraries they use). One would not expect different programs (due to different libraries) to have the same meaning.

I understand this different-library perspective — in a literal way. And yet I’m not really satisfied. What I get is that standard libraries are “standard” in signature (form), not in meaning (substance). With no promises about semantic commonality, I don’t know how standard libraries can be useful.

Another perspective that came up on #haskell was that the kind of semantic consistency I’m looking for is impossible, because of possibilities of failure. For instance, evaluating an expression might one time fail due to memory exhaustion, while succeeding (perhaps just barely) on another attempt. After mulling over that point, I’d like to weaken my principle a little. Instead of asking that all evaluations of an expression yield same value, I ask that all evaluations of an expression yield consistent answers. By “consistent” I mean in the sense of information content. Answers don’t have to agree, but they must not disagree. Failures like exhausted memory are modeled as ⊥, which is called “bottom” because it is the bottom of the information partial ordering. It contains no information and so is consistent with every value, disagreeing with no value. More precisely, values are consistent when they have a shared upper (information) bound, and inconsistent when they don’t. The value ⊥ means i-don’t-know, and the value (1,⊥,3) means (1, i-don’t-know, 3). The consistent-value principle accepts possible failures due to finite resources and hardware failure, while rejecting “linux” vs “darwin” for System.Info.os or False vs True for “2^32 == (0 ::Int). It also accepts System.Info.os :: IO String, which is the type I would have expected, because the semantics of IO String is big enough to accommodate dependence on dynamic conditions.

If you also cherish the principles I mention above, I’d love to hear from you.

I have another paper draft for submission to ICFP 2009. This one is called Beautiful differentiation, The paper is a culmination of the several posts I’ve written on derivatives and automatic differentiation (AD). I’m happy with how the derivation keeps getting simpler. Now I’ve boiled extremely general higher-order AD down to a Functor and Applicative morphism.

I’d love to get some readings and feedback. I’m a bit over the page the limit, so I’ll have to do some trimming before submitting.

The abstract:

Automatic differentiation (AD) is a precise, efficient, and convenient method for computing derivatives of functions. Its implementation can be quite simple even when extended to compute all of the higher-order derivatives as well. The higher-dimensional case has also been tackled, though with extra complexity. This paper develops an implementation of higher-dimensional, higher-order differentiation in the extremely general and elegant setting of calculus on manifolds and derives that implementation from a simple and precise specification.

In order to motivate and discover the implementation, the paper poses the question “What does AD mean, independently of implementation?” An answer arises in the form of naturality of sampling a function and its derivative. Automatic differentiation flows out of this naturality condition, together with the chain rule. Graduating from first-order to higher-order AD corresponds to sampling all derivatives instead of just one. Next, the notion of a derivative is generalized via the notions of vector space and linear maps. The specification of AD adapts to this elegant and very general setting, which even simplifies the development.

You can get the paper and see current errata here.

The submission deadline is March 2, so comments before then are most helpful to me.

Enjoy, and thanks!

I’ve just finished a draft of a paper called Denotational design with type class morphisms, for submission to ICFP 2009. The paper is on a theme I’ve explored in several posts, which is semantics-based design, guided by type class morphisms.

I’d love to get some readings and feedback. Pointers to related work would be particularly appreciated, as well as what’s unclear and what could be cut. It’s an entire page over the limit, so I’ll have to do some trimming before submitting.

The abstract:

Type classes provide a mechanism for varied implementations of standard interfaces. Many of these interfaces are founded in mathematical tradition and so have regularity not only of types but also of properties (laws) that must hold. Types and properties give strong guidance to the library implementor, while leaving freedom as well. Some of the remaining freedom is in how the implementation works, and some is in what it accomplishes.

To give additional guidance to the what, without impinging on the how, this paper proposes a principle of type class morphisms (TCMs), which further refines the compositional style of denotational semantics. The TCM idea is simply that the instance’s meaning is the meaning’s instance. This principle determines the meaning of each type class instance, and hence defines correctness of implementation. In some cases, it also provides a systematic guide to implementation, and in some cases, valuable design feedback.

The paper is illustrated with several examples of type, meanings, and morphisms.

You can get the paper and see current errata here.

The submission deadline is March 2, so comments before then are most helpful to me.

Enjoy, and thanks!