Metaphysical Developer

Idiomatic Clojure with LightTable

Posted in Clojure, Software Development by Daniel Ribeiro on November 18, 2013

LightTable, the famous innovative IDE (slash reactive work surface, as described by the author) recently got some pretty amazing additions: user defined plugins and user defined custom expressions.

Custom expressions allow users to define functions that get any block of text and replace with the result, stream information back to LightTable, or display the result inline:

image

The clojure project Kibit immediately came to mind when I saw the new addition. Kibit is a static code analyzer for Clojure that allows you to find a more idiomatic way of writing a given block of text.

So I wrote this short LightTable integration for it. With it, LightTable can replace any s-expression with its more idiomatic version. For instance, this code:

Screen Shot 2013-11-17 at 5.53.01 PM

becomes:

Screen Shot 2013-11-17 at 5.53.21 PM

To see how Kibit did, here is the before and after shots of the expression broken down in its components:

s0

After:

s1

In order to use Kibit with LightTable, you just need to:

1. add lein-kibit it to your project.clj:

Screen Shot 2013-11-17 at 5.56.49 PM

2. add the integration function to your usermap:

Screen Shot 2013-11-18 at 10.54.40 PM

The sample project can be found on GitHub.

Tagged with: ,

Conjcraft: A Minecraft Mod implemented in Clojure

Posted in hacking, Languages, Minecraft by Daniel Ribeiro on April 20, 2012

When you don’t create things, you become defined by your tastes rather than ability. Your tastes only narrow & exclude people. So create.

– why the lucky stiff

TL;DR:  Source here, and here a video of the mod in action:

Conjcraft

Conjcraft is a simple and extensible Mod for Minecraft written in Clojure (and some Java). Besides introducing two new blocks (Clojure and Github, which is hosting the source here), it brings an extremely simple and small DSL for writing Minecraft recipes.

The recipe DSL cleans up on on Minecrafts original one (which is alredy terse for a Java DSL). Compare these simple ones:

addRecipe(new ItemStack(Block.rail, 16), new Object[]
                {
                    "X X", "X#X", "X X", 'X', Item.ingotIron, '#', Item.stick
                });
(recipe-dsl {\X :ingotIron \# :stick}
  "X X
   X#X
   X X" 'rail 16)

Small explanation: the Clojure version is essentially the ascii art of this recipe:

Disclaimer: I’ll not try to teach Clojure here (besides saying it is a Lisp). If you need more info, there are great resources on the web.

This gain in expressiveness (which is come from the fact that Clojure is extremely more expressive than Java) is compounded in multiple recipes, specially after defining a consistent character to block/item mapping:

(def char-block (create-input-char-binding
                  '(
                     d dirt
                     o cobblestone
                     g github
                     c clojure
                     r redstone
                     )))

Many recipes can use them:

(defn recipes []
  (recipe-dsl char-block
     "d
      d" 'github

     "o
      o" 'clojure

     "c
      c
      c" 'swordGold

     "c c
      c c" 'bootsGold

     "cgc
      cgc" 'bootsDiamond

     "ccc
      c c
      c c" 'legsGold

     "ccc
      cgc
      cgc" 'legsDiamond
    ))

And finally, all of this is encoded in plain text Clojure files, stored in the conjcraft directory inside  user.home (which on Linux and Mac OS it is usually the user’s home directory, aka ~).

This way Conjcraft is very extensible, as it allows the users to add blocks and recipes, without requiring Eclipse or MCP, or to recompile and obfuscate the de-obfuscated Java code.

Such simplicity, though, did not come easily…

Origins

One of the things that has always amazed me about Minecraft is how simple its concept is. I believe this simplicity is actually paramount to its success: by giving you very solid and small building blocks (no pun intended), the game steps away and let the user create its own goals and be shine on its own.

This simplicity also lets other developers step in and create a huge variety of amazing mods (out of which, one my personal favorites is the Aether mod, for being a very ambitious project, and showing how much great content you can create on top of such a simple and powerful platform).

“Simplicity Ain’t Easy”: Stuart Halloway masterfully made this argument, exploring what simple is (one of the key points being that simple is “not compound”), its importance, and how Clojure is a simple language, which actually makes it very powerful. Inspired on the simplicity and power of both Clojure and Minecraft (and continuing my healthy(?) obsession with Minecraft and Clojure) it seemed only natural for me to set to create a simple mod on top of both platforms (natural because both of them run on top of JVM).

Modding Minecraft with Java is quite straightforward with the help of Minecraft Coder Pack (aka MCP) and ModLoader. Calling clojure from Java is also very straightforward, to the point that you basically need a Java class like this:

public class mod_Conjcraft extends BaseMod {
    public void load() {
        try {
            File file = new File(new File(System.getProperty("user.home"), "conjcraft"), "conjcraft_main.clj");
            System.out.println("Loading clojure mod files from " + file.getAbsolutePath());
            clojure.lang.Compiler.loadFile(file.getAbsolutePath());
            clojure.lang.RT.var("conjcraft", "call").invoke();
        } catch (IOException e) {
            throw new IllegalStateException(e);
        }
    }
}

And then I was able to create a very small function, in 5 lines of Clojure, to add a recipe that would take one block off dirt and output 7 blocks of dirt:

(ns conjcraft)
(import '(net.minecraft.src Block ModLoader ItemStack))
(defn call []
  (let [dirt Block/dirt]
    (ModLoader/addRecipe (ItemStack. dirt 7) (to-array ["#" \# dirt]))))

This actually works pretty well when using Eclipse, or the recompile.sh script that comes with MCP. The fun really began when I started preparing to release it…

The 1st rule of the Obfuscator Club is:

You can’t defeat the obfuscator. This is actually really important. Minecraft is obfuscated in its original distribution, which makes a lot of sense for a proprietary and commercial game. MCP tools de-obfuscate the original java code from its original form, giving methods and classes names very straightforward and sensible names.

The problem is that, in general, gamers will need your mod in the obfuscate code, as they game expects classes to use the obfuscated names. Therefore you absolutely must obfuscate your mod.

The 2nd rule of the Obfuscator Club is:

You can’t defeat the obfuscator.

Clojure does have the capability of generating .class files with its Ahead of Time (AOT) compiler.  Since the obfuscator does not operate on java source code, but on .class files, this could have helped. But it doesn’t. Other languages that run on JVM like Scala (which compiles to pretty Java-like bytecode) and Mirah (which can even compile to Java source code) can actually get around the obfuscator this way, as long as you don’t use features that require reflection.

To understand why it doesn’t work with Clojure, let me show you what this simple AOT example:

(ns core
  (:gen-class :main true))

(defn -main []
  (println "Hello World!"))

With some help of JD-GUI we can see the equivalent Java code of the generated class files, in particular:

public class core
{
  private static final Var main__var = Var.internPrivate("core", "-main");
  private static final Var equals__var = Var.internPrivate("core", "-equals");
  private static final Var toString__var = Var.internPrivate("core", "-toString");
  private static final Var hashCode__var = Var.internPrivate("core", "-hashCode");
  private static final Var clone__var = Var.internPrivate("core", "-clone");

These seemly innocuous lines actually break in runtime. This happens because the obfuscator has another very important property: it puts everything on top level namespace (no packages). Note that the package “core” is written as a literal string, which the obfuscator will not touch. And currently there is no way to use AOT with empty namespaces

You could change the Clojure compiler, or use tools to manipulate the byte code on the class files, but there is actually a much simpler solution:

Breaking the rules: Defeating the Obfuscator

Clojure is famous for supporting one of the most powerful types of metaprogramming: template macros. I have not exploited it on the project because macros can be very hard to understand (think of them as functions that take code in its raw Abstract Syntax Tree form, and output another raw Abstract Syntax Tree), and I wanted to keep the project very accessible.

The point is that I used Clojure to generate Java source code, on compile time (the type of metaprogramming you always have the option to use, no matter the platform or base language you are based on).

This is done by the create_constants.clj script, which actually imports the de-obfuscated code and generates a Java file mapping all block, item and material names to their actual objects (the result cannot be published without breaking both Minecraft and MCP licenses, but reading the code you can get an idea of what the result looks like).

Using the property highlighted before, that the literal strings will not be obfuscated, and knowing that the obfuscator will not obfuscate the attribute names of classes you create (only make stripe their package), this static maps are available to be used directly by interpreted Clojure code.

The final element of defeating the obfuscator is the ExtendableBlock class. It essentially takes Clojure functions (clojure.lang.IFn interface), and delegate methods to them (some methods have to be re-exposed even when public, as the original public method names will be obfuscated).

Conclusions

Modding Minecraft is extremely fun, and it gets a lot more enjoying when doing it in languages that that are fun to use. I’ve used Clojure here, but there are many other languages that could have been used. So have fun, and create.

Thanks

Thanks Notch for making Minecraft and supporting the modding community. Thanks for all the presenters at ClojureWest for inspiring me to bring Clojure to new places. Thanks Robert for making one of the best Minecraft modding tutorials out there. And finally thanks to all the creators of MCP and ModLoader for making modding a simpler and pleasant experience.

Tagged with: , , ,

ClojureScript vs Coffeescript

Posted in hacking, Languages by Daniel Ribeiro on August 28, 2011

A language that doesn’t affect the way you think about programming, is not worth knowing

Alan J. Perlis

Edit Feb/2014: Please note that this post is from 2011, a few weeks after Clojurescript was released. Things changed a lot in the mean time…

In the past few years Javascript has gained a lot of attention and ubiquity: HTML5 technologies leverage a lot of Javascript, which enables people to create amazing dream worlds (like the in the ROME project) with WebGL, V8 brings a lot of JIT techniques to a JavaScript Virtual Machine which helps Google Chrome be a very fast browser, and powers NodeJS (allowing people to create a web page in a single programming language).

Javascript also powers queries in NoSQL databases like Mongo and CouchDB, and it can be used when making QT applications, 3d Games in Unity and even mobile apps with frameworks like WebMynd and PhoneGap. It has been a long way from the old days when it was confined to the browser, and mostly used for form validation.

In spite of all of this attention, Javascript has been so misunderstood that attention to its Good Parts had to drawn. It doesn’t help that its prototype based OO was first introduced by the rather unknown language Self, despite the several advantages it has when compared to traditional class based OO (the paper Organizing Programs Without Classes, written by Google’s Senior VP of Operations Urs Hölzle, who, among other things, also contributed to key JIT techniques like polymorphic inline caching).

Therefore it is not surprising that there are many projects that compile existing languages to Javascript. Many of these were too focused on the web platform (like Google Web Toolkit and the amazing Cappuccino‘s Objective-j). On more recent years languages are targeting the whole JS ecosystem (which makes a very poignant argument that JavaScript is Assembly Language for the Web). Coffeescript is one of such languages, which very fond (I’ve written about it recently).

About a month ago ClojureScript was released, porting the Clojure language from Java ecosystem to the JS. Clojure is quite an amazing effort of engineering, not only for being a very successful Lisp on the JVM, but also for its novel approach of handling time and state (which its creator, Rich Hickey, explains really well).

I was really excited to see the examples, but I was a bit bummed out that the most interesting example was a Twitter visualization tool (which feels a bit too much like a 2010 app). Since both CoffeeScript and Clojure are fun languages, and I wanted to see how ClojureScript would compare to CoffeeScript, I took the challenge and crafted a simple game HTML5 physics based game on both languages, using Box2dWeb, a js port of Box2D (the physics engine, created by Erin Catto, that is behind Angry birds).

The game consists of clicking on the objects to destroy them, so that they don’t reach the top of the canvas (and, in another very Tetris like fashion, the elements pop out faster the more you play). It really sticks to the bare minimum of Terrano’s Hierarchy of Gamer Needs. All the code is open source and can be found on Github. The CoffeeScript‘s version source can be found here, and the ClojureScript‘s here.

Lessons Learned

Disclaimer: ClojureScript is pretty much in alpha status, so many things are likely to improve in the future.

Compiling: The first thing that really pops up is how fast Coffeescript compiles down to JS. The watch behavior allows you to fire the compilation process and forget it. ClojureScript takes me about 5 seconds to compile a single file. Granted it gives warning about unused/undefined variables, but I’d really prefer it to compile instantly and let the browser tell me this on runtime.

Namespaces: Clojure’s namespace are implemented as global variables, which are shadowed by local variables with the same name. For instance, if you are in a namespace called game, don’t use local variables and arguments named game. This is really important, as ClojureScript will use the global namespace for every single function defined in that namespace, so shadowing it is likely to give all sorts of errors.

ClojureScript is not Clojure: Fogus wrote an interesting piece on the lack of eval on ClojureScript. Even though I find it might make sense for some web pages, when making WebGL games, or even Canvas 2d games, the assets size can easily overshadow then entire library’s size. Which is not a big deal if you use HTML5′s Cache manifest. In the end, it felt very much like the opposite of the Lisp spirit (epitomized by Paul Graham on his Five Questions about Language Design: “Give the Programmer as Much Control as Possible“).

The documentation is quite clear that eval is not supported. What it is not clear, is that this argument against eval permeates many other functions: resolve is not implemented (neither ns-resolve, or the *ns* definition). Without both of them, there is no way to transform a string into a function. For people more used to OO languages, like Javascript, Ruby and Python, this essentially means that ClojureScript doesn’t have any reflection APIs. On the game:

createElement: ->
  randomY = (0.2 + 0.4 * Math.random())*  H / @scale
  randomX = (Math.random() * (W - 50) + 25) / @scale
  type = @objectList[randomInt(@objectList.length)]
  @["create#{type}"] randomX, randomY, Math.random() + 1

The last line of the Coffeescript version uses reflection to get the correct method name (to decide to invoke createTriangle, createCircle or createSquare ). The ClojureScript version had to be translated into:

(defn- create-element [game]
  (let [randomY (/ (* H (+ 0.2 (* 0.4 (rand)))) scale)
        randomX (/ (+ 25 (* (rand) (- W 50))) scale)
        type (rand-nth [:circle :square :triangle])
        method (keyword (str "create-" (name type)))]
    ((@game method) game randomX randomY (inc (rand)))
    )
  )

Which is possible because the game has the respective functions as keyword attributes, which can be easily converted from string interpolation.

User Macros are not supported: At the moment at least (support for it is likely to come on following updates). This makes the situation above much harder to take. But this is mostly due to ClojureScript’s alpha status. This does make the code a bit longer (the CoffeeScript version has 236 lines, while the ClojureScript has 301 lines). In order to circumvent what I consider that would be one of the ugliest bits caused by lack of macros (several (set! (. obj attr) value) calls), I defined a js-set function:

(defn- js-set
  "Sets an attribute name to a value on a javascript object
Returns the original object"
  ([jsobject attr value]
    (do (native-set-wrapper jsobject attr value)
      jsobject))
  ([jsobject & values]
    (do (doseq [[attr value] (apply hash-map values)]
          (native-set-wrapper jsobject attr value))
      jsobject)))

This is actually really against Clojure’s spirit, as Clojure really promotes immutable code. However Javascript libraries, in particular Box2dWeb, really expect mutable state. Therfore handling native js objects require such functions (note that converting them to clojure and keeping it on clojure land can be easily done with the nice undocumented function js->clj function, which is actually used on TwitterBuz).

Therefore we can write functions this way

(defn- create-fixture
  ([shape] (js-set (b2FixtureDef.)
  :density 3
  :friction 0.3
  :restitution 0.9
  :shape shape
))
([] (create-fixture nil))
)

Instead of:

(defn- create-fixture
  ([shape] (let [f (b2FixtureDef.)]
  (set! (. f density) 3)
  (set! (. f friction) 0.3)
  (set! (. f restitution) 0.9)
  (set! (. f shape) shape)
))
([] (create-fixture nil)))

Which makes it look a lot like assoc function for creating maps with updated values. This version inspired me to refactor the Coffeescript version using a similar assoc function:

createFixture = (shape) ->
f = new b2FixtureDef
f.density = 3.0
f.friction = .3
f.restitution = .9
f.shape = shape if shape?
return f

became:

assoc = (o, i) -> o[k] = v for k, v of i; o

createFixture = (shape) ->
assoc new b2FixtureDef,
density: 3
friction: .3
restitution: .9
shape: shape

Which is quite similar to ClojureScript’s version (the colons are on the right instead of the left, and it requires a comma). Which reduces the amount of accidental complexity to a minimum.

Edit: Thanks everybody for pointing out that you can use macros with Clojurescript. However, at the moment, the are clojure macros (so no js), and they require you hacking your clojurescript to add the macro files in the classpath. Hiccups and cljs-3d are two projects that do this, so you can see on their build files how they do this. Even then, you still need to use require-macros. All of this makes macros less of a native feature on Clojurescript, and it makes a lot harder to share code seamlessly.

IDE support: Clojure’s IDE support is really nice. Intellij’s La Clojure (avaiable on its free Community Version) does a lot more than mere syntax highlight: minimal refactoring support, rainbow parenthesis, smart parenthesis, syntax highlighted repl, great autocomplete support, awesome code navigation, autocomplete for java classes and live templates. And it works pretty well for ClojureScript as well. Other IDEs are also great, even though Emacs support can be a bit more intense on its setup (which is not something emacs users are unfamiliar with).

Even though I am really happy with Github’s founder Chris Wanstrath work on the Emacs mode for Coffeescript, it doesn’t have the same support that Clojure does. It is getting more and more support, but nowadays Clojure has the upper hand.

Debugging support: Browser support for debugging languages that compile down to javascript is coming, but at the moment Cofffeescript compiles down to such a readable JS that it not a big problem. This is a known issue with ClojureScript at the moment. Even on pretty print compile mode.

Conclusions

Since Javascript on the web has a much more simple execution model than Java, Clojure’s amazing concurrency control mechanisms are not as shining. Nevertheless ClojureScript is a delight to work with. As it moves out of its Alpha status, many of the issues are likely to be gone. I also expect it to support the full Clojure language (including things like resolve, letfn, macros and eval), as writing web apps in a single language on client and server is a really nice feature. This would also make ClojureScript even more interesting, as it would allow developers do leverage all the power of existing Clojure libraries into their Javascript work (and possibly use it on a more polyglot environment).

Coffeescript is more suitable for production apps right now, but it is nice to see all these developer efforts to allow people to be more productive and happy with their work on Javascript platforms (this way we don’t have to wait for Google’s NaCl and PNaCL, which promise to bring even more languages to the environment).

Tagged with: , ,
Follow

Get every new post delivered to your Inbox.