Ending the Absurdity

"If at first, the idea is not absurd, then there is no hope for it."
- Albert Einstein

Peter and I were talking the other day about how absurd the ROC (resource oriented computing) model can appear. This came about because I'd become re-acquainted with the inner workings of the NetKernel kernel whilst implementing a number of low-level changes for the upcoming version 5 release.

How can something so convoluted that every atom of computation resolves to an endpoint through a set of arbitrary and dynamically interrelated address-spaces by an abstract identifier possibly work? These computations are then cached using a deep dependency based validity model and their computation costs determined and accumulated to minimize future computation. Surely all this complexity can't work, or at least can't work well?

Now, if you're thinking what I thought on first seeing Einstein's quote then you're going to be saying to yourself "yeah sure, but I bet that there is no hope for most of the absurd ideas either!" And yes I can't argue with that, but it's the converse that's interesting; why is it that most enterprise software lacks any absurdity? Why is it in general that new hot trends and technologies are so conservative in ambition and only willing to make single safe steps forward? Why do the same old ideas get rehashed in new clothing again and again destined to fall short? (*1)

I find the conservative nature of the software industry very difficult to understand. We are working in an industry with the most fluid of materials: concepts and information. And of course, with NetKernel, we have taken a very different tack. So trying to think of some reasons, I came up with the following:

1. Big business gets stuck with the innovators dilemma - big new ideas that disrupt existing markets are not usually fostered.
2. Good ideas are in short supply - if there is one takeaway from entropy it's that there are many many more disorganized, broken or useless permutations of state than good and useful ones.
3. Abstract is abstract - there is no getting away from the fact that NetKernel's power and ultimate flexibility comes at a price. That price is the need for a conceptual leap. The dictionary definition doesn't just describe abstract as "expressing a quality or characteristic apart from any specific object or instance"(*2) but also "difficult to understand; abstruse". Yet abstraction is absolutely needed for us to make sense of the world. From the day we are born we are finding and refining our abstractions.
4. Right place, right time...

Both Peter and I had been working independently in different organizations (I in a large utility company, and Peter in Hewlett Packard) on developing approaches to reducing the cost of software. XML was showing itself to be a powerful generic data language and was gaining traction to the extent that a rich set of tools around it were developing. When we came together at HP our joint backgrounds in physics, innate curiosity, irreverence to convention and desire for unification led us down a new path. We saw many distinct strands that could be tied together. We had/have a creative tension. Peter always has an eye on the big picture, the ability to pick apart an obscure technology or standard and see what makes it tick, and most importantly the drive to ensure what needs to be done gets done. I, on the other hand, have a desire to create deep symmetry, an obsession for optimization (from my misspent youth developing 8bit games) and an eye for aesthetics and design. Together we created NetKernel.

With the imminent release of NetKernel 5 we have a solid, rounded platform, battle hardened and embodying the state-of-the-art ROC. There is still work to do however. We must now turn our attention to the learning curve and make the abstract clear, uncomplicated and obvious. If you are using NetKernel now you are one of the self selecting few. For every one of you here there are many more who have bounced off the absurdity or the subsequent learning curve. This is our next challenge...

~~~~~~~~~~

*1 I understand some of what I've said with regard to the general lack of ambition in software may be controversial. (Please comment.) I am of course generalizing; there is lots of great software out there often hidden away quietly doing it's job well. But it does frustrate me that we as a society could be benefiting hugely if we could more economically create and maintain large software systems. We need to remove the inherent brittleness that goes right to the foundations. That is what ROC does.

*2 dictionary.reference.com