Polling the state of my head on this date

The small fraction of my waking brainpower that I can allocate to developing my computing ideas is, at this moment, divided between pondering these questions:

(and, once I’ve got them down, I will probably change track to some others)

• When in doubt — use bullet points?

Position and binary on/off input

• Input from myself to the computer is (99% case) keyboard and mouse.
• But for my phone it is touch — there, the keyboard(s) are simulated.
• Virtual buttons are much more flexible than physical buttons. The former can behave in any way we wish, as usual, while the latter has the sole benefit(*) of giving tactile feedback.
• Typing on a physical keyboard consists of jabbing one key at a time (no matter how fast).
• Alternatively — using a single button (e.g. left mouse button, or just “the screen“) to represent all “jabbing” — to act as a single, privileged finger with access to the machine — we demultiplex this through spatial position (set via mouse motion, or touchscreen sensors, or even gaze tracking).
• (Of course, this is exactly what happens in the real world, but we do not have access to the source code of the real world, so we are stuck with it.)
• Sometimes we hold down modifier keys, and because they naturally return to ‘up’, they give us ‘sticky’ or inherently temporary modes, because nobody is going to keep a key held. Caps Lock is an exception to this.
• Should I bootstrap a system using the physical keyboard, or should I try and use the mouse to create simulated keyboard and bind the real one to it afterwards?
• The advantage is exemplified by vim. If I want a circle, I don’t have to remember it’s on the ‘g’ key; I can just click the ‘circle’ button in the UI. Same with OK (Enter), Yes (y), Move Unit, Summon Dragon, etc…
• But then typing is channelled through a single button and I get Repetitive Strain Injury.
• But the touchscreen solves this, save for the tactile feedback…?
• If Bret Victor had his dream for widespread haptics realised, maybe we could have the best of all worlds?
• The weakness of the real world is also its strength. It is fixed, but that means it is predictable. (*)
• Paper has, like, four? failure modes: water, fire, tearing, age.
• Note that “suddenly displaying my handwriting in the Wingdings font for no apparent reason” is not on that list.
• Virtual paper has infinitely many failure modes. Because when we could in principle program any behaviour we wanted … and what tiny fraction of that space consists of behaviours we can actually work with?
• As $PERSON said: a “failure” is just a behaviour that we don’t like.
• Perhaps, then, it is OK to begin the bootstrapping process using my keyboard.
• I am sure somebody is reading this and giggling at my philosophical paralysis.
• In my defence, I used to be a lot worse, and I have been actually doing some real experiments this year 🙂

Specialisation

I believe that one of the key advantages of the ‘object’ idea is the emphasis on supporting specialisation. Objects might share certain behaviours at certain times, but ultimately an object can always choose, in the end, to act completely as a class unto itself. It allows us to represent the knowledge of the exceptions to the rules, as well as the rules themselves.

(TODO: even if all programming is “just” early-bound / ad-hoc / manual / labour-intensive knowledge representation and inference — does this mean we should all pack up and spend time researching AI? But then isn’t the whole point that this is what we’re already doing all the time without realising?? And anyway, in order to program AI, you need to program i.e. without the aid of the AI you’re developing. Bootstrapping applied to AI?)

One concrete instance of this idea is that every object ought to have its own user interface. To reiterate, this doesn’t mean that no two objects can share the same UI or do things the same way, but instead that specialisation is accounted for and that it is encouraged that each object present itself with the most appropriate interface that it judges in its particular context at that moment.

(Listing the variables a, b, c, d, e allows for the possibility that they’re all the same, even though they are all listed under different names. But the fact that they are listed as different names also allows them to all be uneqivalent. This is not permitted by listing a single variable: a. A single thing is indistinguishable from many things that just happen, at the moment, to be equal to it. If two things can ever be different then they have to occupy separate boxes. We don’t need to say A is A, but even saying that “A is the same as B” can only relate them because one is called A and the other is not.)

After reading some of Bret Victor’s Magic Ink, I cannot stop thinking about his separation of tool and platform and his JPEG example.

• A JPEG “file” should actually be an application. Everything should be an application. Meaning: everything should be able to be a “dynamic picture”; nothing should be limited from the outset to any less.
• A JPEG should be its own viewer and be its own editor. And the same with everything else. (self-extracting archives are ahead of the game ;))
• True, many things will share components (e.g. visual editing GUIs, compression algorithms, etc.). But we already have a solution (or should have one) for this: shared libraries for application programs — application programs just like the ones we already require to work with any data (explicit image editors or video players or document viewers or IDEs)
• (Tools (viewers/editors) and files are tightly coupled, but this is only managed by manual association. If you have an unknown file type then it can’t just “play itself” – you need to do lots of work yourself to try and figure out whether it matches some tool already out there.)
• (Even finding out what “type” a file is, is a completely heuristic exercise not guaranteed to return a true answer — first we check the filename extension, then we fall back to looking at the first few bytes of the file, then we do statistical analysis on the file contents, … is this a legitimate criticism?)
• (If I want the file to be a JPEG but a “little bit different”, say insert some bitpatterns or records or re-order them, then I have to submit this to a standards board and get it accepted after several years — even if I could easily write code to present an ordinary JPEG interface from my odd custom format, and distribute that as part of the file that would be activated.)
• There are two objections:
• (1) inter-program (object) communication and co-operation
• (2) SECURITY!!
• (1) is nothing new to us and we already do it with file formats and protocols (is this fair?)
• But (2) is facetious. The desire for security is, of course, perfectly valid; but blaming “dynamic picture”-ness for this is just plain wrong.
• The problem is that our current implementations of dynamic simulations (applications) allow them to make permanent changes to their environment by default.
• If, however, we begin with strict isolation and think of outside access (e.g. file deletion) as a set of actions to be explicitly approved (or, at a coarser level, a set of lifelong “privileges” or “permissions”) then we don’t need to cripple the creative capacities of today’s technology with the static-ness of yesterday’s.
• An encapsulated simulation of an emergent time-evolving system should be no more “dangerous” than simulation of a fixed time-evolving system. Or: a Turing machine is no more inherently “insecure” than an animated GIF.
• Thankfully, this is the way everything is going — look at web apps despite the history of HTML; look at mobile apps and Docker despite the archaic set of metaphors they remain entangled with.
• (Not perfect — but a step in the right direction.)
• I must look into the Principle of Least Authority and the Web’s Rule Of Least Power.
• Was Tim Berners-Lee genuinely on to something, or was he truly one of the “amateurs” resented by Mr Kay?
• (Surely, a Turing machine can be “semantic” or “machine-readable” simply by  having a protocol.)
• (“Damage” risks are: denial-of-service to transient resources (e.g. starvation of CPU, power usage, filling up RAM, taking up display space, blasting white noise out of the speakers, hogging the network) and unwanted changes to persistent resources (e.g. your important files). We often talk about preventing the latter, but true encapsulation that gives the user control would also encompass the former.)
• (“Access” risks are to do with the keeping of secrets and are the domain of cryptography etc.)

The GUI mystery

• What is it about text that appears to lend itself so well to the automation of repetitive tasks (programming)? Why must we go back to the old teletype even if I want to automate dragging a set of files into the recycle bin?
• Is it: the symbolic representation of things that are much bigger / longer than a word?
• Or is the situation just that we are very used to one?
• Basically, I’m wondering why I have never seen any of the GUI automation described in Watch What I Do used in real life.
• Obviously it’s not the concrete syntax that’s the key, but — the abstract syntax is just an interface to sequence, selection, repetition right?
• I suspect the real reason is that, because computing grew out of a desire to automate clerical work — number-crunching, i.e. the shuffling of textual symbols — nobody knows what “computation” based on “any old shapes” even looks like. Right at the very beginning, computation was defined as symbol shuffling.
• The question doesn’t need to be: how can we “write programs” with shapes
• Instead the question is: how can we automate arbitrary graphical tasks in the same way we automated arbitrary textual tasks in the past half-century.
• (It’s tempting to presume otherwise, but Computer Vision and Neural Networks are not needed here any more than they were needed to get text-processing off the ground)
• (Perhaps I should look at Turing’s original 1936 paper, On Computable Numbers, With An Application To The Entscheidungsproblem, for inspiration. Preliminary glance points to sections 1 and 9)
• There is a split: text can be automatic, but other graphics is manual only.
• The UNIX “fear of pixels“; the sheer convenience of knowing that your program will always at least have a terminal window for output; that you can type printf and rely on it working. It’s just that you have to do a ton of work to draw a line.
• Then, in User-Land: the dual “fear of automation”; the sheer convenience of knowing that you will at least be able to see things and point to them; that you can probably find a graphics program that will let you draw lines. It’s just that you have to do a ton of work positioning each point over and over again.
• Remember — text is not special; it requires someone to have written code to translate keystrokes into bitpattern changes, and those bitpattern changes into glyph shapes, which are also still shapes by the way.
• Source code is nothing but a long-feedback user interface for writing simulations … of what? Predominantly, of changes to a hierarchical filesystem and output to a teletype.
• That is precisely what printf() makes convenient. And conversely, working with the semantic properties of text in graphical environments is harder than writing code to process ASCII values. Meanwhile, manipulating text glyphs and layout is precisely what is so infuriating about graphics APIs and game programming.
• The UI reproduces itself in its output, or something. The UI of automation is currently text; therefore, text is what it works best with, because it can be directly represented (even manipulated…?) in the program description itself.
• More historically accurate, though: the output / environment shaped the UI in the first place. Poor Kerninghan and Ritchie didn’t have pixels; they had a typewriter.
• It is clearly a matter of “feedback cycle”.
• I think I have my answer: because we can program any shape-manipulations by writing enough text, people don’t see the value in exploring ways to automate without text. (**) They are just that: unknown. In exactly the same way that automation of symbol-shuffling was unknown before we went ahead and did it.
• (**) But consider: there must be some threshold past which writing text doesn’t save us that much labour. In fact, this happens all the time: we perform the direct-manipulation GUI steps manually because we know it will take less time than figuring out how to write the shell script to do it.
• If the time saved by the program working is outweighed by the time spent on setup, maintenance and Stack Overflow — then ask yourself, is this really “automation”??
• In this sense, we do not have “arbitrary” computation. Viewed from the abstract mathematical perspective, anything is possible and we can stop worrying. But the practical reality screams otherwise: for some types of problems, if text is our interface, then we really might as well not be using computers at all.

Just a better common tongue?

• UNIX won; the low-level metaphors embodied by it form the common language of computer software — they are the way all of us develop, use and think about “software” or “that which is achievable using computers” on a daily basis. Network effects suck us in. It is difficult to see that there could be alternatives, and even harder to try them out.
• My collaborators seems to agree that this common language is incomplete or severely sub-optimal.
• But even if it is crap, it is still a set of concepts that can be reliably assumed to be supported on various systems (e.g. environment variables, hierarchical file systems, print statement / terminals) — the fact that things developed by different people can work together (despite whatever amount of effort spent on setup, maintenance and StackOverflow) means that it works better than the absence of a common set of concepts. (I am stating the obvious here but making it explicit.)
• Many advances can be made without necessarily requiring some new standards or protocols or conventions to be adopted by the software world at large.
• But certainly, for some strategies to succeed, they will need to achieve a critical mass of practical use or agreement.
• Question: is it the case that we are largely advocating for “just” a new common representation of high-level concepts? To replace JSON / XML / YAML / etc with (a) some serialised format that humans needn’t touch, and (b) widely-adopted manual- and automatable-interfaces for it?
• Even if not, how much of an impact would this still have?
• And if we accept this as a goal, it is doomed to the usual problems of trying to get widespread adoption of some standard(s)?
• Ideally, what would happen is a usable artefact that reliably makes life so much easier in software that you and I and everyone else would just want to adopt it.
• Question: are all efforts to qualitatively improve computing doomed without the aid of network effects? Are there some that could succeed without needing to be or have a “common language”?