Clojure: Interview with Rich Hickey

Clojure’s creator, Rick Hickey, took some time to tell Computerworld about his choice to create another Lisp dialect, the challenges of getting Clojure to better compete with Java and C#, and his desire to see Clojure become a ‘go-to’ language.

What prompted the creation of Clojure? 

After almost 20 years of programming in C++/Java/C#, I was tired of it. I had seen how powerful, dynamic and expressive Common Lisp was and wanted to have that same power in my commercial development work, which targeted the JVM/CLR. I had made a few attempts at bridging Lisp and Java, but none were satisfying. I needed something that could deploy in a standard way, on the standard platforms, with very tight integration with existing investments. At the same time, throughout my career I have been doing multithreaded programming, things like broadcast automation systems, in these OO languages, and seen nothing but pain. As a self-defense and sanity-preserving measure, I had moved my Java and C# code to a nonOO, functional style, emphasising immutability. I found this worked quite well, if awkward and non-idiomatic. So, I wanted a dynamic, expressive, functional language, native on the JVM/CLR, and found none.

 Where does the name Clojure come from? 

It’s a pun on the closure programming construct (and is pronounced identically). I wanted a name that involved C (CLR), L (Lisp) and J (JVM). There were no search hits and the domain was available – what’s not to like?

 Was there a particular problem the language aimed to solve? C

lojure is designed to support writing robust programs that are simple and fast. We suffer from so much incidental complexity in traditional OO languages, both syntactic and semantic, that I don’t think we even realise it anymore. I wanted to make ‘doing the right thing’ not a matter of convention and discipline, but the default. I wanted a solid concurrency story and great interoperability with existing Java libraries.

Why did you choose to create another Lisp dialect instead of extending an existing one? 

While Lisps are traditionally extremely extensible, I had made some design decisions, like immutability for the core data structures, that would have broken backward compatibility with existing Scheme and Common Lisp programs. Starting with a clean slate let me do many other things differently, which is important, since I didn’t want Clojure to appeal only to existing Lispers. In the end Clojure is very different and more approachable to those having no Lisp background.

 Why did you pick the JVM? 

I originally targeted both the JVM and CLR, but eventually decided I wanted to do twice as much, rather than everything twice. I chose the JVM because of the much larger open source ecosystem surrounding it and it has proved to be a good choice. That said, the CLR port has been revitalised by David Miller, is an official part of the Clojure project and is approaching feature-parity with the JVM version.

Clojure-in-Clojure: self-hosting is usually a big milestone for programming languages – how is that going? 

It is going well. We are approaching the end of the foundation-laying phase. There were a few base capabilities of Java which I leveraged in the implementation of Clojure for which there was no analogy in Clojure itself. Now the last of these is coming into place. Then there will be nothing precluding the implementation of the Clojure compiler and the Clojure data structures in Clojure itself, with efficiency equivalent to the original Java implementation.

Did you run into any big problems while developing the language?  

One of the biggest challenges was getting the persistent data structures right, with sufficient performance such that Clojure could be a viable alternative to Java and C#. Without that, I wouldn’t have gone forward.

 We’ve all read The rise of ‘Worse is Better’ by Richard Gabriel. Do you feel that a project like Clojure can help reverse that attitude? 

The arguments made in Worse is Better are very nuanced and I’m not sure I understand them all, so Clojure tries to take both sides! It values simplicity of interface and of implementation. When there is a conflict, Clojure errs on the side of pragmatism. It is a tool, after all. 

With multi-core CPUs becoming more common and a resurgence of hyperthreading, dealing with concurrent tasks is now more important. How does Clojure deal with this?

 Good support for concurrency is a central feature of Clojure. It starts with an emphasis on functional programming. All of the core data structures in Clojure are immutable, so right off the bat you are always working with data that can be freely shared between threads with no locking or other complexity whatsoever, and the core library functions are free of side-effects. But Clojure also recognises the need to manage values that differ over time. It supports that by placing values in references, which both call out their stateful nature and provide explicit concurrency semantics that are managed by the language. For example, one set of references in Clojure are transactional, which lets you conduct database-like transactions with your in-memory data and, like a database, automatically ensures atomic/consistent/isolated integrity when multiple threads contend for the same data. In all cases, Clojure’s reference types avoid the complications and deadlocks of manual locking. 

What can you tell us about the support for parallelism and the upcoming Java ForkJoin framework?

 While concurrency focuses on coordinating multiple tasks, parallelism focuses on dividing up a single task to leverage these multi-cores to get the result faster. I didn’t build any low-level infrastructure for parallelism into Clojure since the Java concurrency experts were already doing that in the form of the ForkJoin framework, a sophisticated thread pool and work-stealing system for parallel computation. As that framework is stabilising and moving towards inclusion in Java 7 (and usable with Java 6), I’ve started implementing parallel algorithms, like mapping a function across a vector by breaking it into subtasks, using ForkJoin. Clojure’s data structures are well suited for this decomposition, so I expect to see a rich set of parallel functions on the existing data structures – i. e., you won’t have to use special ‘parallel’ data structures. 

What about running on distributed systems? MapReduce did come from Lisp . . .

 I don’t think distribution should be hardwired into a general purpose programming language. Clojure can tap into the many options for distribution on the JVM – JMS, Hadoop, Terracotta, AMQP, XMPP, JXTA, JINI, JGroups etc, and people are already leveraging many of those.

 How did you choose the Eclipse License for Clojure? 

The EPL has the advantage of being reciprocal without impinging on non-derivative work with which it is combined. Thus, it is widely considered to be commercial-friendly and acceptable for businesses. 

Web frameworks? I notice there’s one called ‘Compojure.’ Do you see this as a direction in which Clojure could grow? 

Definitely, there are already interesting frameworks for Clojure in many areas. One of the nice things about libraries for Clojure is that they can leverage tried-and-true Java libraries for the low-level plumbing and focus on higher-level use and flexibility. 

What books would you recommend for those wanting to learn Clojure? 

Programming Clojure, by Stuart Halloway, published by Pragmatic Programmers is the book right now and it’s quite good – concise and inspiring, I highly recommend it. I know of a couple of other books in the works.