Over the past year, I’ve been completely changed by the knowledge of an entire untold history of things we take for granted—personal computing, object-oriented programming, the GUI.
For the vast majority of my coding lifetime, I had considered the act of constructing software as being identical to describing it, using some appropriate or inappropriate language. Anything beyond that, such as ‘visual’ tools, were mere crutches for those who do not know how to Program. Even though they’re nice and pretty, they hide what’s really going on—with no prospect of opening up the friendly abstractions they offer.
This reached its culmination last summer, when I got interested in User Interface Management Systems, and the idea of GUI interaction as a formal language (the Seeheim model). Previously I had used cumbersome general-purpose languages to code the event-handling logic of GUIs; now, I searched for and tried to construct custom languages; it was only a matter of finding the correct one. Language comes first; everything is describable, and everything in a computer, including GUIs, boils down to machine language in the end, so that is that.
Despite this, cracks did show, and I had allowed myself to dwell on one of them. Even if interaction is “essentially” linguistic (as I saw it), it is quite paradoxical to insist on describing, using words, the visual qualities of shapes, colours, positions, and so on. Especially since a picture paints a thousand words!
In fact, typing out pixel coordinates of vertices and adjusting them prior to recompilation is not a mere paradox—it is perverse. This forced me to consider visual editing appropriate for visual data, even if everything else is code.
I started applying this to my central motivation, games. Normally, to specify the content of a level, I’d write numbers in a text file and parse this in the game code to build Java objects or whatever. The “level editor” was just my text editor. But it also meant I could not see what it would look like, or even whether it would parse properly, so I had to use my imagination to predict the results. This did not strike me as particularly odd; I’d seen this done a lot, and besides, it’s good to support text editing; it’s how everything else is done, after all.
I now conceded that the idea of a level-editing tool really was a better way to do this, no matter how good one is at coding. But then, a crucial thought: why should such a tool be separate from the game itself? In fact, why not have an “edit mode”, that you can dip in and out of at will—play, pause, edit, resume? This gave way to some headaches: how to treat editing and ordinary interaction in a semantically appropriate way, how to reify object properties to make them editable; how far to bother with all this before I am no longer really using Java. There were hints of treating everything uniformly as stuff that can be edited and interacted with.
I was very muddled and unclear and dreaded implementing it. I was also clinging to the interaction-as-language thread, which was not as tangled. And I think this was the point where I finally spent time discovering the work of Alan Kay.
I’d heard of the guy before, but didn’t know anything. He designed a system called FLEX for his PhD, which came up as I was researching interaction grammars. I think it used them. Anyway, his story was fascinating and somewhat sad. He was originally inspired by the scaling potential of biological cells: small, relatively unintelligent components that can comprise fully functioning organisms a trillion times their size. This was his main motivation for OOP, a very different conception to how it turned out in the mainstream.
He makes reference to many research directions and people that I’d never heard of. It has been both enlightening and disheartening to discover the incredible things that researchers did decades ago, that have been ignored, forgotten or poorly re-invented by programmers and computer scientists since.
I stumbled across Alan Kay and his colleagues’ work at Viewpoints as if by destiny. I easily recognised their concepts of introspective, dynamic, easily self-modifiable systems as the logical conclusion—many steps ahead but in the same direction—of what I had been trying to do with my games and level editors. And that was a really good feeling, because it told me that I wasn’t crazy to have insisted on the experience of software development being different; I no longer had to fear it being a dead end that I’d have to abandon sooner or later. I could leap ahead to build on what better minds had already done.
But even with the fresh new attitude of VPRI, and some emphasis on direct-manipulation interfaces, I can’t shake the fact that much of what they do is very language-focused. Don’t get me wrong; while I can’t blame somebody who brushes off Open Reusable Object Models, or Combined Object Lambda Architectures, because of those ill-fated terms “Object model” and ‘Architecture’, I would try to get across that there is something genuinely novel about the ideas presented inside these papers, miles ahead of whatever the terminology might evoke. And there really isn’t anything wrong with them at their core.
It’s just that, earlier this year, I also came across Bret Victor, who showed me how static-text interfaces limit our ability to create, and did some pretty incredible demos to emphasise how much direct manipulation could help us in constructing software.
The COLA platform is supposed to be bootstrapped by way of source files, translators, and languages. Piumarta and Warth did some seriously cool recursive knot-tying in the Object Models paper, in developing a minimal possible pervasively late-bound system which can be used to develop itself. But it’s all done using language, and I wonder if there is a more interesting question of how to bootstrap the same thing visually rather than linguistically. This forms the basis of a new project I’ll be writing about.
I used to consider language as fundamental, and everything else reducible to it. Despite this being technically true in this case of computing, it also completely misses the point. This is so hard to see, because language is ubiquitous in software construction. It is all around us. I would dare to say that it is the only way the effective totality of software is developed, and has been since computing began.
But nowhere else in life is this true. When human beings construct anything else, it is often a combination of direct manipulation, drawing diagrams, and teaching by demonstration, with language as one of several tools. Even in mathematics, its formulation in language and symbols is often the final stage of a thought process involving imagery and demonstration.
So no; I defy my original viewpoint that got me started on this circular path. GameMaker got it right. Scratch got it right. Even rubbish ‘kiddie’ tools that don’t offer any language-based coding whatsoever fail to get it any more wrong than the flipside of insisting exclusively on descriptions.
But I should not be too hard on myself. It was not my fault that so much of programming consists of specifying dynamic, emergent behaviour ahead of time in a static, eternal form such as text. How could I be expected to think any differently? In fact—how did I get to where I currently stand??
It’s always easy to rationalise the status quo, it’s “just the way things are”. Everything currently works a similar way, why not be compatible with that? Everyone is guaranteed to have primitive text-based tools; if I make my game levels text files, then I can edit them using SSH on the other side of the world! I can use Git to version them without feeling guilty! They’ll be compatible with Unix tools! That’s cool, huh?
I refused to rationalise the complexity of my game programming experiments. Instead of reluctantly hacking it together in C++ or Java or C or Lua or Java again, I said: No! There has to be an easier way! I traded several tangled, inelegant, unmaintainable messes of “project” for a journey that’s shown me why they were doomed in the first place. Not necessarily for others who could try the same thing, but why they were doomed for me (if that makes sense). If you recognise any of this in yourself, you’re not alone.
Programming is complicated. We do scatter
printf()s across our code. Software doesn’t scale easily. The main code, the graphics libraries, and the system itself are guaranteed to each define their own custom yet functionally identical versions of Vector, Color, Shape, etc. We do have a gazillion hot new Web frameworks every goddamn month to, I don’t know, circumvent the DOM to make GUIs, or make CSS actually usable.
But it doesn’t have to be this way. There is no reason why any of the problems facing computing, from software scaling, to Windows 10 freezing if I don’t shut it down every night (indeed, it did so just as I finished writing that down, no kidding), are simple inevitabilities of the nature of computer systems, or software engineering, despite what we may be led to believe. Anyone who suggests they are is forgetting that computing is still less than a century old, and that alone is ample reason to suspect that only a small fraction of what is possible has been covered so far.
But we won’t make any progress if we remain stuck in the past. And the past, not the present, is what an embarrassing majority of our fundamental tools and ideas preserve as state-of-the-art. And over-emphasis on language is but one facet.