Durable Scope

Ending the Absurdity

2011-10-06T20:10:00.000+01:00

"If at first, the idea is not absurd, then there is no hope for it."
- Albert Einstein

Peter and I were talking the other day about how absurd the ROC (resource oriented computing) model can appear. This came about because I'd become re-acquainted with the inner workings of the NetKernel kernel whilst implementing a number of low-level changes for the upcoming version 5 release.

How can something so convoluted that every atom of computation resolves to an endpoint through a set of arbitrary and dynamically interrelated address-spaces by an abstract identifier possibly work? These computations are then cached using a deep dependency based validity model and their computation costs determined and accumulated to minimize future computation. Surely all this complexity can't work, or at least can't work well?

Now, if you're thinking what I thought on first seeing Einstein's quote then you're going to be saying to yourself "yeah sure, but I bet that there is no hope for most of the absurd ideas either!" And yes I can't argue with that, but it's the converse that's interesting; why is it that most enterprise software lacks any absurdity? Why is it in general that new hot trends and technologies are so conservative in ambition and only willing to make single safe steps forward? Why do the same old ideas get rehashed in new clothing again and again destined to fall short? (*1)

I find the conservative nature of the software industry very difficult to understand. We are working in an industry with the most fluid of materials: concepts and information. And of course, with NetKernel, we have taken a very different tack. So trying to think of some reasons, I came up with the following:

Big business gets stuck with the innovators dilemma - big new ideas that disrupt existing markets are not usually fostered.
Good ideas are in short supply - if there is one takeaway from entropy it's that there are many many more disorganized, broken or useless permutations of state than good and useful ones.
Abstract is abstract - there is no getting away from the fact that NetKernel's power and ultimate flexibility comes at a price. That price is the need for a conceptual leap. The dictionary definition doesn't just describe abstract as "expressing a quality or characteristic apart from any specific object or instance"(*2) but also "difficult to understand; abstruse". Yet abstraction is absolutely needed for us to make sense of the world. From the day we are born we are finding and refining our abstractions.
Right place, right time...

Both Peter and I had been working independently in different organizations (I in a large utility company, and Peter in Hewlett Packard) on developing approaches to reducing the cost of software. XML was showing itself to be a powerful generic data language and was gaining traction to the extent that a rich set of tools around it were developing. When we came together at HP our joint backgrounds in physics, innate curiosity, irreverence to convention and desire for unification led us down a new path. We saw many distinct strands that could be tied together. We had/have a creative tension. Peter always has an eye on the big picture, the ability to pick apart an obscure technology or standard and see what makes it tick, and most importantly the drive to ensure what needs to be done gets done. I, on the other hand, have a desire to create deep symmetry, an obsession for optimization (from my misspent youth developing 8bit games) and an eye for aesthetics and design. Together we created NetKernel.

With the imminent release of NetKernel 5 we have a solid, rounded platform, battle hardened and embodying the state-of-the-art ROC. There is still work to do however. We must now turn our attention to the learning curve and make the abstract clear, uncomplicated and obvious. If you are using NetKernel now you are one of the self selecting few. For every one of you here there are many more who have bounced off the absurdity or the subsequent learning curve. This is our next challenge...

~~~~~~~~~~

*1 I understand some of what I've said with regard to the general lack of ambition in software may be controversial. (Please comment.) I am of course generalizing; there is lots of great software out there often hidden away quietly doing it's job well. But it does frustrate me that we as a society could be benefiting hugely if we could more economically create and maintain large software systems. We need to remove the inherent brittleness that goes right to the foundations. That is what ROC does.

*2 dictionary.reference.com

Enterprise Visualizer Tools

2011-09-27T21:55:00.000+01:00

As promised in the previous post here is a sneak peak of the new NetKernel Enterprise Edition Visualizer tools. These tools do not make any new information available as the core visualizer shows every gory detail. However these newviews perform in-depth analysis of the low-level data to determine information that can be quitelabour intensive to infer when working with large request trees. These new views are:

an expiry determinants tool to allow rapid determination of why any particular response is expired.
a scope determinants tool to find which sub-requests are de-scoping into the request scope during resolution.
a compare caching tool which finds differences in two requests that are stopping them being cached.

These tools appear as new items in the context menu of every captured request in the visualizer.

Expiry Determinants Tool
When the response from an endpoint is marked as expired it is often difficult to determine why if that expiration has occurred if you have deep sub-request tree. Expiration propagates up to all dependent resources by default and one expiring request deep down can cause a whole sub-tree to fail to cache. WIthout this tool you must manually search down through the tree of requests until you find the cause.

Using the tool you simply select a captured request in a trace and select "View Expiry Determinants" and you will get a filtered tree view of the sub-requests with requests that are cause of expiration highlighted in red. All irrelevant branches of the tree (that are not expired) are trimmed.

Scope Determinants Tool
A feature of the refreshed visualizer is a cachability status indicator (traffic light). Red means no caching is possible, amber mean cache retrieval is constrained by request scope and green means caching is possible. When a response has amber cachability status this is an indication that the response depends upon the context that the request was issued. This causes a non-zero response scope depth. What this means is that the request itself or one of it's subrequests has de-scoped to resolve a resource. Using this tool you can rapidly see which request(s) this is. Select a captured request in a trace and select "View Scope Determinants". You will get a filtered tree view of sub-requests with requests whose resolution is descoping above the selected requests scope are highlighted in yellow. Also depth indicator on the right showing how deep they resolve. All irrelevant branches of the tree (that don't de-scope or contain de-scoping requests) are trimmed.

In this screenshot you can see that resolution of pdsConfig.xml resource required three spaces from the request scope but the major culprit is the XMLToHDS transreptor which de-scoped all the way back to the "Control Panel (private)" space. Armed with this information it might make sense to import the xml core module which supplies this transreptor into "NKEE / Tools".

Compare Caching Tool
If you have a resource that is being used within a request that isn't being retrieved from cache but you expect it to be this tool will help. You need to capture two separate requests for the resource and then sequentially select them in the visualizer and select "Compare Caching". This tool will step through all the possible causes for non-retrieval.

Visualizer - Time Machine Debugging

2011-08-05T14:30:00.005+01:00

Over the summer I've been working upgrading NetKernel's Visualizer ready for a major new release planned for the autumn/fall. All this work combined with comments from various folks set me thinking that not much has been said about this critical tool which must appear so alien on first approach.

The Visualizer technology was developed along side the NetKernel 4 kernel in the summer of 2007. After earlier experiments with more traditional breakpoint based approaches to debugging we realized that the ROC abstraction was at just the right level to allow a much more powerful debugger to be implemented. In a nutshell the difference between the visualizer and a regular debugger is that the visualizer captures everything that happens over a period of time rather than simply showing the instantaneous state at the time the breakpoint is activated.

Capturing everything probably sounds crazy/impossible right? But when you consider the pervasive immutability of state in Resource Oriented Computing (ROC) it starts to sound feasible and that combined with the naturally more granular approach that ROC encourages over object oriented and functional approaches I hope you see that it is indeed possible! (If your still not convinced then download NetKernel now and give it a try - it'll take you less than 5 minutes to get it up and running.) What's more, capturing visualizer traces have close to zero performance overhead to an executing system. Again this is due to the immutability of state - capturing simply involves hanging on to state for longer than it otherwise would be - so, yes, this does take extra heap space.

The real win with the visualizer is that the trepidation of overstepping the unknown critical line of code is gone. No more spending lots of time elaborately setting up some situation and carefully stepping through it in the debugger only to find that you were a bit zealous with that step-over button or you realize you realize that what you've just spend and hour tracking down was a symptom and not the cause. So maybe the physicists can't pull off time travel, but you can with the help of the visualizer. Time machine debugging really means stepping back in time as well as forward. I really find it hard going back to a regular debugger.

Other benefits of the visualizer include:

Realtime profiling - with NetKernel Enterprise Edition all timings are captured so local and elapsed times of all requests are captured and can be explored to look for hotspots.
Testing - visualizer traces can be programmatically captured and introspected enabling rich testing of not just what but how and why.
Modeling - capturing the full details of how requests are processed can lead to some interesting visualisations!
Support - Visualizer traces can be saved and re-loaded aiding rapid remote support and all the necessary information for bug fixing.
Evaluation happens in real-time - so their are no "heisenberg effects" and debugging happens after the fact at your leisure.

Visualizer Walkthrough
Let's have a look at some of the key aspects of the implementation. In a later post I'll maybe go into more detail if there is interest. I'm going to show the new version - the core concepts and layout remain unchanged. The main differences are in refinement of the UI and new extended views and tools.

First step is to enable the visualizer and capture some traces. The main page lets you do this as well as manage the traces you have captured:

Figure 1: Table of captured request traces

Clicking the start button will start the capturing of traces, clicking it again will stop. Each entry in the table is one captured trace which corresponds to the full execution state of one root request (an external event/request injected into NetKernel through a transport) from start to end. You have the option to capture all requests or specify a regular expression filter on the root request identifier, so for example only capture requests from a particular transport or to a particular application.

It is worth stating here, for those less familiar with ROC, that the root request usually initiates sub-requests and those sub-requests initiate further sub-requests down to arbitrary depths into your application. The resource oriented approach isn't just surface deep. It is this tree of sub-requests that form the captured trace.

Clicking on one of the captured traces we get a context menu with a choice of actions to perform. If we select "View Request Trace" we can drill down into the details of what happened during the execution. (One of the changes that has recently been implemented is full integration with NetKernel's Space Explorer. Not only does this give full cross-linking of entities such as spaces and endpoints but also allows the extensible action framework that provides these context menus.)

Figure 2: View of request trace showing recursive tree of sub-requests

Once we get into the details of the request trace you can begin to see how the visualizer approach is different. The table here shows the full recursive request execution tree laid out not just the current call stack. A call stack view (from the perspective of a particular sub-request) is available as are various other views. Each row represents one sub-request, the endpoint that handles it and the response returned. The columns in the request trace view are:

The call tree with the endpoints that are invoked for each request.
The address space which hosts the endpoint
The request verb
The request identifier
The elapsed duration of processing the request
The local time spent inside the endpoint not including sub-requests
Expiration status of response
Representation of response

Clicking one of the request rows presents a context menu. For the moment we are going to click "View Request Details" and see all the gory details of a single sub-request. The request details view is broken down into several sections which show the issued request, how it was resolved, how the endpoint processed the request (if any, often processing as cached), and finally the response.

Figure 3: Request Details

I don't want to go into any discussion here about each of the displayed fields. The point I want to make is that all the details are there captured.

Figure 4: Resolution Details

The resolution process is mechanism that the NetKernel kernel uses to locate an endpoint to evaluate a request. The detail here shows how the request scope is systematically interrogated until a resolution is found. If a match is found the endpoint is then used for evaluation. For more details on the scope concept see my earlier post on Dynamic Resolution.

Figure 5: Endpoint evaluation details

The endpoint details section is only shown if the endpoint is evaluated. Often times
the response can be pulled from cache. Details of any direct sub-requests issued are
shown along with their expiration and duration, both useful for determining correct
and efficient operation.

Figure 6: Response details

Again all the details of response are here. Nothing is hidden.

I hope this post gives a good overview of the visualizer. I plan to talk in more detail at a later stage about some of the new views that we'll be releasing soon and what has motivated them. In writing this post I notice there are a lot of details of the NetKernel implementation of ROC that are not really talked about elsewhere, for example, what does cost on the response mean? what does the expiry field really show? Let me know if there are other dark corners you would like me to shed light on.

Dropbox with nCoDE - Part 3

2011-06-15T12:15:00.000+01:00

In this third and final part in this series of three post we'll fill in the final piece of the puzzle and look at how we achieve the Dropbox authentication using OAuth. The previous two parts are found here and here.

First a disclaimer, I don't claim to be knowledgeable about OAuth! I made the point in the last post that you don't really need to be because the compositional approach hides the unnecessary details. However I've got to agree with the recommendation that you must understand the security implications before putting OAuth into a production environment.

The authentication process in our demo client application involves two services. The first is called Prepare. Remember from the previous post the user will be redirected to this service if they get an authentication failure whilst trying to browse a Dropbox directory.

Figure 1: OAuth Prepare Service (click for full-size view)

Prepare initiates communication with Dropbox service by sending the our demo client's
consumer key and secret. The consumer key and secret are obtained from Dropbox when you register a new application on their developer pages. These are stored and returned by the OAuth Settings service. (We factor this out here rather than having a literal inside prepare because they are also needed by the second service, Prime.)

Figure 2: OAuth Settings Service (click for full-size view)

As we can see the settings contain a number of URLs in addition to the key and secret which are required to drive this negotiation. These URLs are:

requestTokenURL - service which the client application should call to obtain an access token using it's consumer key and secret.
authorizeWebSiteURL - the web site the user should be redirected to, to login if necessary and approve access by the client application.
callbackURL - the URL that the authorizeWebsite will redirect back to after successful login.(in our case this is the prime service)
accessTokenURL the service used by the prime service to obtain an access token once the user has approved access and Dropbox has returned a uid.

The OAuth Prepare endpoint wraps the first part of the negotiation with Dropbox which involves sending the demo application specific consumer key and secret to obtain a request token. The request token and callbackURL are then appended as parameters onto the authorizeWebSiteURL which is put into the response. Looking at the nCoDE program in figure 1 we can see that the response from OAuth prepare is used twice. Firstly we sink it into the users session for use later by the Prime service. Secondly we extract the authorizeURL using XPath, that we sink to the httpResponse:/redirect. This has the effect of redirecting the users browser to the Dropbox web site where they will be presented with a page something like this:

You'll notice the Sequence accessor is connected to the response. The sequence accessor allows a sequence of requests to be evaluated in order regardless of if they needed. In fact only the response from one of the arguments (the first one named response) is used and returned as it's response. This is something that is not normally possible with the lazy functional evaluation of nCODE. In functional evaluation requests are only made to satisfy requirements made by the arguments to other endpoints.

When the Prepare service is requested it should redirect the users browser to Dropbox, they will login if necessary and then Dropbox will redirect back to the callback URL we specified in the OAuth settings with a UID. This will cause the invocation of our second service Prime. Prime performs the necessary final parts of the OAuth negotiation to tie the request token with the UID to obtain the access token it needs to invoke the dropbox API on behalf of the user.

Figure 3: OAuth Prime Service (click for full-size view)

We can see this final part of the negotiation is handled by the OAuth Prime accessor. It has three arguments, our OAuth settings, the response from OAuth Prepare that we stored in the users session and the UID parameter. The response from OAuth prime is the access token that we'll to pass into subsequent HTTP requests to attach the appropriate OAuth headers to a request. In Prime we use the sequence accessor again to firstly store the access token in the session before using it to retrieve the users account information. First we request /account/info from Dropbox, this returns back a JSON representation of the users account details and usage information. We convert this to HDS using the JSONToHDS accessor and the style it into a HTML page using XSLT. This page then contains a "continue to files link..." which returns the user back to listing their root directory.

Wrap up
I hope this proved useful and instructive. I also hope it shows the compositional approach and how a small set of general purpose components working with a small set of generic representations can lead to rich solutions. If you want a copy of the module please drop me an email at tab at 1060.org. I'm reluctant to put up a put general download because of the uncertain status of the consumer key and secret provided by Dropbox. Really they should be obscured inside an application and will only be transmitted over the wire encrypted by HTTPS, of course this isn't really possible with a code demo.

Dropbox with nCoDE - Part 2

2011-06-10T09:00:00.012+01:00

This posting is walk-though of developing an OAuth client service on NetKernel with the nCoDE visual composition environment. If you haven't read the background material in the first post I recommend you do that first.

OAuth involves a three way orchestration between a user on a web browser, a client web application and a service web application. The basic idea is that a client application with limited trust can access a service application without ever gaining access to the users credentials and in a way that can be be managed in terms of scope and duration by the service application. This is great because it leaves a user in control to share data on an ad-hoc basis without any undue privacy concerns.

However the specification can be a little intimidating and implementing client and service applications can be more than a little involved. Indeed work on OAuth 2.0 is already underway with one of it's design intents to simplify client implementations. Luckily for us there is already OAuth support built into NetKernel's HTTP Client module and that combined with compositional development is going to make our life easy. We don't need to worry about the finer details of the negotiation.

Our Dropbox client isn't going to do anything fancy. It actually does nothing that the web application running on Dropbox's own site does. We'll simply allow browsing of the folder structure and downloading of files. Let's get started by taking a look at the main entrypoint service to the dropbox application. Our application is hosted in a NetKernel module which exposes itself on HTTP port 8080. We don't worry about the detail of how this is configured but I'll make the module available soon for those who want to play. NetKernel uses grammars to define the resource identifiers that services will handle. So for this main entrypoint service we are using this:

(The res: scheme is simply a generic internal scheme that the external http: scheme is rewritten to when it enters NetKernel.) This grammar says that any external path that starts with /dropbox/view is passed to this service and anything that follows is treated as the path argument that is passed to the service.

Figure1: Dropbox view service (click for 100% view)

nCoDE services are executed as a functional program closely aligned to the resource-oriented foundations of NetKernel. Execution is driven by creating the response which will usually appear on the right-hand side of the diagram and is a dark grey box. Each pale yellow box is a NetKernel endpoint which produces a response and has a number of input arguments. Blue boxes are literal values and pale grey boxes are other nCoDE services. The interconnecting arrows show the transfer of state from the response of one endpoint to an argument of another. For the sake of brevity I'll just call all the boxes on the diagram endpoints unless the distinction is necessary as they are all treated consistently.

The view service simply evaluates a two line groovy script to determine if the path argument ends with a slash, it returns a boolean. (The context object used within the groovy script comes from the NKF API available to all NetKernel language runtimes - it gives them access to the ROC address-space) Based on the response from groovy the if accessor chooses either to run the view folder service or the view file service passing in the path argument and returns the sub-service as the response.

Ok, so let's look at the view file service. This service will download the file at the given path from the users Dropbox:

Figure2: Dropbox view file service (click for 100% view)

The view file service generates a Dropbox API URL by filling the path argument into a Freemarker template. The URL is then requested using the HTTP Get accessor which also receives some OAuth credentials pulled from the users session. (We'll see how this gets setup later). The response from the HTTP Get is returned directly as the response. At this level of detail we don't need to know how the credentials are set up or what they are. And the beauty of the HTTP Get accessor is that it hides all the details of how these credentials are applied to the GET request.

Figure3: Dropbox view folder service (click for 100% view)

The view folder service is a step up in complexity from the last two services we've seen. This service in it's essence is the same as the view file service but rather than simply return the response from the HTTP GET we convert the JSON response to HDS using the aptly named JSONtoHDS accessor. (HDS is an internal NetKernel representational form that is like an optimized XML for data-structures but not documents.) The HDS is then styled using XSLT into a human readable directory listing with hyperlinks.

The additional complexity comes about because we interrogate the HTTP response code looking for an authentication failure. This will happen if the user has not yet gone through the authentication process and no valid OAuth credentials are available. In this case we redirect the user to the start of the OAuth authentication process at /dropbox/prepare. It's worth noting the use of the generic Math Engine which is put to the simple task of comparing two integers but it is capable of a lot more! Also notice the way the HTTP redirect is performed by SINKing state into the httpResponse:/redirect resource using the Sink accessor. The functional program nCoDE runtime normally issues all requests as SOURCE because this is directly analogous to function invocation but helper accessors such as the Sink allow mutation of state.

The discussion of the OAuth authentication process and the two nCoDE services that comprise it will be the topic of the third and final post in this series.

Dropbox with nCoDE - Part 1

2011-06-03T15:39:00.011+01:00

With any worry about having too many buzzwords and technologies in one place cast aside, this post is the first in a series which will show you how I developed a Dropbox web application using the Dropbox restful OAuth API with a compositional approach using new and little discussed NetKernel compositional development environment nCoDE.

First a little background on compositional development and nCoDE. Compositional development is an approach that focuses on the assembly and configuration of pre-existing components to build systems. It's an approach that has always been at the heart of NetKernel from the early days of XML processing pipelines with a rich set of accessors and DPML to bind them. With the release of NetKernel 4 our emphasis and efforts shifted ever so slightly. The resource oriented abstraction enabled lots of exciting new patterns and the freedom of representational expression (read "you can use any old immutable object for state transfer"). That was all well and good, very good in fact, but it left the compositional approach in a little bit of a dilemma. The problem is that composition works best in very constrained environments and now we had a general purpose architecture building abstraction. It's not surprising that since then most development both internally at 1060 and with users of NetKernel has been done with procedural code, either Java or one of the scripting runtimes such as Groovy.

You've probably guessed where this is leading. Yep, I wasn't happy about this state of affairs. Mainly because one on the tangible benefits of NetKernel has been complexity management. If you read this blog you know my Death Star end-game but on a day to day basis baby sitting technology isn't my idea of fun. I'd rather it just worked. To truly achieve this you need the ability to step above the minutia of code and build with well designed and implemented blocks. We had achieved this with spacial structure using the standard module infrastructure and the space explorer but not with information processes.

DPML was ported to NetKernel 4 early on in the development cycle but didn't get huge amounts of love. For those who don't know it, DPML stands for Declarative Process Markup Language. Think of it as an assembly language for resource oriented processes. It's kind of unique as a language in that it really cannot do much by itself. What it does do though is orchestrate resource requests and their responses and arguments. Even basic constructs such as conditional processing and exception handling are external to the language. This is perfect for composition.

But DPML is not the whole story. Freedom in the abstraction has lead to many different resource representations and argument passing conventions. Let me share the example of forEach with you. There are currently two forEach accessors in the code NetKernel libraries. The first one is in Layer1 module. It makes extensive use of declarative requests (an XML syntax for defining a resource request) for defining the operations of how and what their arguments are. Whilst declarative requests are very flexible, the problem is (when it comes to composition) that it makes these services need to be external which leads to a lot of fine grained services. Another forEach acessor was developed for use in nCoDE (yes we're getting to it soon!). This one takes quite a different approach by injecting a space containing all the resources sub-requests might need. This fits more closely with lazy evaluation approach of DPML. The point of this example is not to say one approach is right and one wrong, but rather to show one of the ways in which flexibility of the abstraction has not helped the compositional approach.

Now given how quiet I've been of the past months you might think I've had my feet up in the hammock. But you have to remember that you only get a few good hammock days in the UK, the rest of the time it's rainy and windy. So I've been keeping busy working towards making compositional development work. This work has included:

determining conventions and patterns that give flexibility whilst still being composable. This has led to lots of work that has been described in the weekly newsletters as suitably vague "metadata updates" but in reality has been a lot of teasing apart of assumptions to allow a compositional development environment to ...just... work.
rewriting DPML to interact cleanly with the ROC abstraction as we see it now rather than how it might of been before we knew what it was! The benefits of this is better debug-ability, better caching, higher performance and no obscure corner cases.
fleshing out low-level common endpoints libraries needed to avoid needing to drop down to procedural code.
developing a workable vision and first implementation of a visual composition environment nCoDE.

So where are we at? Well we are doing well. nCoDE was released to coincide with the NKWest conference in April but has received considerable refinement since and now runs on the new DPML. All of the metadata updates have bedded in and formed the foundation that drives nCoDE. Coverage of compositional libraries is kind of patchy but that is why I'm at the stage I am of pushing nCoDE to do real things. And that leads us nicely to the topic of the mini-series of posts - Dropbox with nCoDE. TADA

Watch this space for part two and we'll actually do something rather than just talk about it...

Ok to keep you happy hear is a teaser nCoDE process. See if you can work out what it does:

(note ironic use of groovy)

BTW if you want to start playing with nCoDE download NetKernel then go to the apposite package manager and install the nCoDE module. After it's installed the new module wizard within the developer tools on the web based admin panel will let you create a new module and get playing straight away.

Home Monitor Reloaded

2011-02-10T13:50:00.025Z

I can't believe that the last time I mentioned my home monitor project was exactly six years ago! For those who haven't heard of it before the home monitor project is a home automation system running in my house which attempts to capture as much useful information and control as much as is possible with the absolute minimum of expensive equipment. It is also acting as a bit of a playground for doing fun things with NetKernel.

Here are few snapshots of some of the data visualizations it generates (using Protovis, this needs javascript and iframe support to display):

_
This chart shows a summary of the weather over the last 2 days. Legend: yellow: daylight, gray: wind speed/direction, green: barometric pressure, red: temperature, pink: dewpoint, blue: rain
_
This chart shows a summary of house data over the last day. Legend: black: humidity%/2, red/brown lines: upstairs/downstairs temp, red/blue bars: indoor/outdoor motion detection, black: at home indicator. (pale green and blue lines show derivative calculations relating to humidity)
_
This chart shows electicity usage over last day. Legend: red line: power usage (watts), black line: total energy usage (KWh*100)

Hardware

It's actually been up and running now for since 2004, though over that time it's migrated from it's original Pentium 166MHz MMX laptop running Windows 98(!) through to a Pentium III 866Mhz Dell laptop running Windows XP and only a week ago an almost modern Pentium M 1.6GHz Laptop running Ubuntu. Over that time too the interfaced hardware has changed too. In 2008 I got a real weather station with serial port connectivity from Santa. This mean that my uncalibrated thermistor temperature sensors could be retired along with the dodgy rain sensor. I've still got the original light dependent resistor monitoring daylight levels that has now seen over 2300 sunrises. Also the magnetic reed switches and PIR motion detectors are still working enabling me to determine when the house is occupied. These sensors were until recently connected up to a legacy joystick port but recently I moved them over to a future proofed USB experiment interface board the Velleman K8055. This board also allowed me to eliminate the need for a legacy parallel port which is interfacing via opto-isolators to the 4 channel wireless power switches. These wireless power switches are used to control central heating, dehumidifier and some lights. One other additional device I recently got connected was a Current Cost electricity monitor. This continually measures the total power usage as well as providing an additional handy temperature sensor. In it's current form the server also has a USB DVB-T TV tuner and external hard drive with myth-tv running. Total steady state power consumption is 27W.

Software

The data architecture of the software is pretty much unchanged from my original description. It is still a three stage data pipeline. The first and second stages are issued by CRON jobs running at 1 second and 5 minute intervals respectively. The first stage monitors all the motion and door sensors detecting and counting transitions and firing off asynchronous requests for immediate action for events such as arriving home to an empty house. The state of this first stage is embodied in a resource call active:quickPoll which along with various other data resources from the weather station and power monitor are used by the second stage. The second stage, active:slowPoll aggregates all the information for the last 5 minutes and writes a record to the database, the data looks this:


<slow>
<house>
<moisture>0.0</moisture>
<upstairs_deg_C>17.4</upstairs_deg_C>
<downstairs_deg_C>16.3</downstairs_deg_C>
<humidity>57</humidity>
<atHome>false</atHome>
<lastAtHome>5189024</lastAtHome>
<inner>30</inner>
<outer>0</outer>
<innerDiff>0</innerDiff>
<outerDiff>0</outerDiff>
<power>195</power>
<cpu_deg_C>39</cpu_deg_C>
</house>
<weather>
<temperature_deg_C>10.3</temperature_deg_C>
<humidity>92</humidity>
<dewpoint>9.0</dewpoint>
<light>0.43</light>
<rain>246.0</rain>
<pressure>1001.0</pressure>
<wind_mph>0.0</wind_mph>
<wind_angle>77.1</wind_angle>
</weather>
</slow>

The third stage is a suite of groovy scripts which generate visualizations of the data and determine the actions that occur when different events take place. I have a web interface in place that allows these to be remotely edited. These are run on demand but are cached where appropriate.

This is the Static Structure Diagram of the module. The architecture can be summarized as being three layer. The first layer (home monitor) controls access to the second layer (home monitor private /branch-merge) and splits out requests into one of two channels the first provides an authenticated session and the second allows unauthenticated access to certain resources. The third layer (data layer) hosts the hardware and database accessors. I've used a few interesting external projects to help with connectivity, these are libk8055, openJaWS, and RXTXcomm.

I won't go into any more detail about the code. If anyone is interested I can send you the module.

Photos

Here are a few photos of the installation:

Installed and tidy in a cupboard!

The interface board and wireless transmitter in a box

Spaghetti wires under the floor boards.

Closeup of the opto-isolators

Modelling injected spaces

2010-11-30T14:21:00.014Z

One of the enhancements to the ROC abstraction that was introduced with NetKernel 4 was the ability to dynamically inject address spaces into the scope of resource requests. In NetKernel 3 the scope was determined purely from the requestors scope and the resolution path through any space imports to the resolved endpoint. In NetKernel 4 this same basic approach is used but any endpoint can now inject new spaces into the scope before issuing a request. In essence what this new capability does is allow local (private) address spaces to be shared by any sub-requests. The practical uses of this include such things as:

unifying "pass-by-value" and "pass-by-reference" to always be pass-by-reference but with a reference to a resource in an injected space. For more details see the NKSE documentation.
providing a mechanism for late/lazy access to transport state. This is used by the HTTP transport bridge to provide access to all areas HTTP envelope if needed but only if needed. This not only reduces the overhead of parsing the various parts into representations but more importantly automatically maximizes the cachability of the response. (Those who used NetKernel 3 will remember needed to configure the HTTP bridge to determine which parts of HTTP envelope should be passed as explicit arguments to sub-requests.)

Anyway the reason for this post is to show how these injected spaces are now modeled in the space explorer. This is a new feature of the Static Structure Diagram and will be available in the repositories soon.

(click image for a bigger version)

This static structure diagram shows a Fulcrum space with the HTTPTransport and with HTTPBridge wrapping Space 2. The new notation shows the HTTP Request Response Space injected into all requests that get delegated from the HTTPBridge into Space 2. We can see that the HTTP Request Response Space has two endpoints the HTTPRequestEndpoint and HTTPResponseEndpoint which provide access to the request and response through the httpRequest: and httpResponse: schemes respectively for all sub-requests.

Space 2 contains the HTTPSession overlay which delegates all requests through to Space 3. We can see the HTTPSession overlay similarly injects a HTTP Session Space into the scope of all the requests it delegates.

Here is the module definition xml (module.xml) for this functionality:

I hope this new notation is going to be useful to throw light onto what previous was one of the darker corners of the ROC abstraction and will help with visualizing your systems. As ever all feedback welcome!

Tony

Resource Oriented Modelling Language

2010-10-12T14:17:00.010+01:00

We are getting close the first anniversary of the NetKernel 4. With the release of Netkernel 4 finally we had the solid core ROC abstraction that we had always talked about. Along with the core abstraction came a metadata model which we haven't really talked a lot about. Metadata means a lot of different things to different people so just to be clear, what I'm talking about here is the system state that is generated and captured by the core NetKernel infrastructure of the kernel in combination with standard modules. This includes the data on the deployed functionality and it's structure as well as runtime data about requests, who issues and receives them. It has had some use in tools like the visualizer and space explorer but really these tools are complex and overwhelming to most people because not a lot of work has gone into really making them usable- they are power tools that show err on the side of showing all the information available.

Why did we bother?

Over the last year I've spent quite a bit of time thinking and working to make NetKernel easier - trying to tame the abstraction. One thing I've come to realize is that it's all well and good making great things possible but it is much harder to make them easy. Unifying the concepts of ROC into a cohesive whole was a big investment in intellectual energy in years past and that provides us with a sound basis but leaves much work to do in the areas of modeling, patterns, conventions and tooling.

We've known for a long time about the componentized basis of ROC enables the Construct Compose Constrain development methodology that leads to the big reductions in system complexity and code. I've always had the vision that this was amenable to a visual development approach in the same way that dataflow languages are in tools like Labview. This is where we are heading - a full visual development environment backed by the ROC abstraction.

Anyway to cut a long story short we released a ROC Architecture Explorer Preview which sparked of some interest a couple of weeks ago. The roadmap for this tool is that it will replace the space explorer and provide a pluggable tool for exploring the interrelationships of the metadata model. One of the initial views included is what we are calling a Static Structure Diagram this has a lot in common with the static diagrams of UML.

The Static Structure Diagram shows, primarily, Spaces with their set of Endpoints and the request delegation relationships between those spaces. It can also show the physical packaging of those spaces within their hosting modules. This diagram provides a great way to understand the architecture of the system.

Here is a diagram of the NetKernel CRON module (I think clicking on it will produce a larger version):

Fig 1: Static Structure Diagram of Cron module

I'll describe this diagram now and doing so show how a picture paints a thousand words! From this diagram we can see that the CRON module consists of three spaces. The leftmost space is the only public space (meaning it can be imported by other modules), this is signified by the + character at the beginning of it's title (just like UML). The second space is delegated to by the first through the PrivateFilterEndpoint, this provides the encapsulation by hiding all the resources marked as private in the second space. This second space contains six endpoints, the first is a transport which receives external events. We can see this by the line started with a circle and leading into the transport. In this case the CRON transport is the endpoint which receives the triggered actions from the underlying CRON engine and issues the requests. The space also consists of a number of StaticResourceEndpoints which map files in the module structure into the address space, then a mapper which delegates requests into the third space and finally an import of the Layer1 module. The third space will only receive mapped requests delegated from the second and it fulfills those requests through a another StaticResourceEndpoint and some imports of other modules.

Ok now you know the notation see if these make sense. Here are two variants of the the Static Structure Diagram for the Frontend fulcrum on my development machine show both Wink and PhotoNK applications dynamically imported as well as a few other odds and ends. The first diagram omits the module boundaries and the second shows them, both diagrams hide the detail of imported libraries.

Fig 2: Frontend Fulcrum hiding module boundaries

Fig 3: Frontend Fulcrum showing module boundaries

One objective of the Static Structure Diagram is that is can be both human drawn and machine generated. Internally the tool generates SVG's so there should be good for printing big! I'll love to hear any feedback about this diagram and how it can be improved.

Cheers, Tony

Separating Architecture from Code

2010-03-05T17:17:00.007Z

The MP3 format is a great technology for compressing audio because it can be progressively configured to reduce the complexity in an audio stream in the parts that humans are less able to perceive. It understands the psycho-acoustics of human hearing and the typical waveforms that constitute music. By working in a constrained domain it can represent the data in a more concentrated form. Similarly JPEG is a great technology for compressing photographs because it can reduce the complexity in parts of an image that humans are less able to perceive. It knows that typical photographs contain large areas of flat colour or slowly changing gradients; it knows that our eyes see less colour information that they see luminance. The algorithms behind these technologies are very smart, they understand the nature of the data and take advantage of that fact to represent the information in a more concentrated form. There are many more examples of these domain specific compressions in video encoding.

At first sight ZIP is great technology for compressing general purpose data streams. It can produce a representation of the original data which is smaller and without loss. The ZIP algorithm works by finding common patterns throughout the data and uses a language to express those patterns. Whilst this works for "typical" data it doesn't always work. By "typical" data I mean real world data than contains patterns and is not completely random. Hold that thought...

Now lets switch gears to think about software. Imagine a language that could express typical software patterns and architectures in a more compact way - it would translate to less code. If we understood what constituted the essential essence of typical software systems with a language to express it then we could "decompress" this to create an instantiation of that system in all it's gory detail needed by the complex hardware of today's computers. By this I do not mean just putting some source code through a compression algorithm. That would be no good - because the compressed data could not be authored. What I mean is a language for expressing the system which enables you to achieve a lot with a little.

One example of what I mean is a technology like Ruby of Rails, a fairly small set of files can create a rich application. A small set of input data can create large software system, in fact orders of magnitude smaller that it would be if that same application was create using a procedural programming language. It achieves this by starting with the concept of what it is trying to create, in this case a database backed web application, and just requires the minimal "configuration" to make that work. Then additional data can be added and that deviates the application away from it's initial form in well defined and useful ways. So you can end up with a rich and powerful application with a very small amount of code.

Let me put this example into context. I long time ago I used to work on developing games on 8bit home computers. In that day hardware was very simple, particularly audio hardware. It was common to use what was called an Envelope to define sound effects and notes for music. The envelope consisted of a list of dozen or so integers expressed as bytes that defined various characteristics of the sound to be played such as how fast it's amplitude rose from silence called attack and the pattern of how it's pitch varied to create tremolo effects of musical arpeggios. Those dozen bytes defined all the sounds that you could create. You could actually get a surprising richness of sounds! If those bytes were put in an raw WAV file you would get much much less. But it is clear that you are never going to get the realistic sound of a piano never mind the expressiveness of the human voice from such a technology. Certain sounds, in fact most sounds, were just outside it's capability.

Looking again at the example of Ruby on Rails, we can see that although you can create some very rich and useful applications there are many things it cannot be used for. For example it couldn't be used for creating an Internet DNS server and it couldn't be used for creating a 3D first person shooting game. There are lots of things it couldn't be used for. That is not to be negative about it. It makes a very deliberate trade-off - it enables a small representation for systems which fit within it's problem domain.

There seems to be a trend in the IT industry towards solution/frameworks like these. They appear on face value to be a good solution to creating the necessary rich and complex applications that are required whilst managing the development costs and the specialist knowledge required. What can be wrong with this? With enough frameworks we can completely cover the space and then it's just a matter of picking the right one - right? However I believe they are not ideal. To understand why lets look at these three questions:

1) What to do when you hit a limit to their degrees of freedom?
When you hit a constraint in a given technology then if you are lucky it's a soft limit and you can drop out of the abstraction it offers to a lower level and solve your problem. Depending upon what you want to do that might completely sidestep any benefits derived from using it in the first place. It is essential that you have this capability though.

2) Can you understand enough about their scope up front to make a good choice for your future requirements?
Committing to technology is hard. You know that it's making you job easy up front or you'd already have passed it by -right? But what happens when you've got deep into you solution and you hit a limitation? What happens if you hit a performance bottleneck in production and it is an inherent characteristic of the approach? The best you can do is to do your homework before committing and hope that some subtle issue doesn't bite hard.

3) What happens when different parts of system needs different technologies to be integrated?
Merging one or more technologies into a cohesive application can be hard. Different technologies come from different development teams and are based on different paradigms and approaches. The glue code to connect them is often nasty, hard to maintain and is hard to separate from your applications custom code.

So just like the ZIP compression is to MP3 let us look at software representation technology that isn't so narrow in scope. Let us look at one which aims to be able to represent pretty much any enterprise software whilst still concentrating the representation; reducing the amount of code. Unless you thought I'd had a last minute change of mind then you know what's coming next. Yep NetKernel. A common theme across pretty much everyone who has put a NetKernel system into production is the surprise at how little code there is. In fact one of our customers was a little embarrassed to share too much of the details of their solution to the client for worry that they'd be expecting much more for their money.

So how is this possible? That's a good question and one that isn't that easy to answer. We started out trying to change the economics of software in the narrow domain of XML message exchange systems back in last century and because of our backgrounds and the isolation from commercial pressures in the labs of HP through ruthless pursuit of the ideal we serendipitously discovered a new way. Over time we managed to distil this way into what we now call Resource Oriented Computing (ROC). An abstraction which when combined with a number of core technologies such as the standard module definition create basis for richly representing enterprise software very concisely, liberating the essence of software architecture from the nitty gritty details of code.

Asynchronicity

2009-12-18T17:35:00.004Z

It's funny how many coincidences of parallel thoughts about the same thing seem to be happening recently. Maybe I'm going mad or getting telepathic? Anyway the latest of these coincidences regards the topic of this post. I've just noticed that Darren Cruse has been thinking about asynchronous processing in NetKernel and wondering where it was. Yesterday I was thinking the same thing too after reviewing the new NetKernel Quick Reference sheet I've been working on. (Feedback most welcome on this - print it out and keep it you desk!)

The first thing to understand is how the physical threads are decoupled from the logical requests with NetKernel. Root requests are initiated by transports when they receive external events and as these requests are handled by endpoints then may issue sub-requests. If all the sub-requests are issued synchronously then they will all remain on the one physical thread. This physical thread will be the thread that was created by the transport. Looking at the Java call stack it will look a lot like a regular Java call stack getting as deep as the depth of nested sub-requests. So far so good, nothing new.

Synchronous requests are typically more common in applications. This is simply because they are the easiest to think about and write processing around. However NetKernel has an intrinsically asynchronous messaging middleware. Within the NetKernel kernel they are actually all treated asynchronous calls but wrapped with logic to ensure they wait for responses (using the java.util.concurrent functionality) and are optimized to continue execution on the same thread.

So now let's look at how an asynchronous request is issued. A good example is the new Asynchronous HTTP transport that we have in Enterprise Edition. It is based on NIO and it can handle more concurrent requests than it has threads - this makes it highly scalable to large numbers of concurrent requests. When it receives a HTTP request it uses NKF (NetKernel Foundation API) to issue an asynchronous sub-request and attaches itself as a listener. The issuing of an asynchronous request completes immediately and the transport thread can continue processing more incoming requests. When the response is ready the HTTP transport receives a callback on it's listener interface and the transport can complete it's work by returning the HTTP response back to it's client.

Inside NetKernel this asynchronous request gets pushed into the request queue where a pool of worker threads wait to pull them off to process. This thread then takes on the responsibility for processing the request. So for example if the request resolves to an endpoint that issues further synchronous sub-requests they will all happen on this thread. However if further asynchronous sub-requests are issued this will again go into the queue and this thread will return to the pool if it has no more work to do. There is subtle complexity here though, because if synchronous requests are sat there waiting in the physical thread's call stack that thread cannot return to the pool. In this situation is must wait for the synchronous request's response and then return it back to the caller.

So far we have looked at how asynchronous requests are issued. NetKernel can also handle requests asynchronously. What I mean by this is that when an endpoint receives a request it doesn't need to send a response right away. It can hold on to it, do some work, maybe issue some asynchronous requests and then complete without issuing a response. Of course it must issue a response at some stage otherwise it's caller will be left dangling. NKF has a special method setNoResponse() for this purpose, because by default if no response is set an endpoint will return back a null representation.

The implications of all this asynchronicity is that you can easily implement some quite sophisticated software patterns without needing to worry about the threading and concurrency issues. In fact constructs such as request throttles and many of the Enterprise Integration Patterns can be implemented as black box endpoints in libraries without needed heavy weight JMS implementations. This is an interesting area of exploration for the new year!

Where's the API? Where's the code?

2009-11-20T13:17:00.003Z

These are common questions leveled at NetKernel and the systems developing using NetKernel. They reveal a mismatch of expectations which is caused by a fundamental difference in the structure of software using the Resource Oriented Computing (ROC) abstraction upon which NetKernel is based.

In the closing remarks of a previous post I mentioned that ROC liberates systems from being implemented in a single language by virtue of the fact that the ROC middleware acts as an integration platform for different programming languages and technologies. In this post I want to delve into this a bit deeper to explore some of the implications as well as answering the questions posed in the title.

Let us first look at software platforms such as the Java Virtual Machine (JVM) and the .Net Common Language Runtime (CLR). They also provide a mechanism to execute code from multiple programming languages. They compile them down into a common bytecode. They provide services such as memory management, security and an underlying operating system abstraction. They abstract away pointers and only allow access to the system through trusted API's. At first glance there is a similarity to ROC middleware but fundamentally at the low-level they are still executing arbitrary imperative code as an amorphous whole communicating without any intervention by a kernel or other intermediary.

Let's just be clear that at the physical level NetKernel is an application running upon a Java Virtual Machine. But don't let that detract from it's form, what it does primarily is to instantiate the alternate reality of the ROC abstraction. The ROC abstraction plays host to resource address spaces embodied by software components (endpoints) which communicate via asynchronous requests-response pairs. Each endpoint acts as a black box, it is free to execute whatever code it desires (within the limits it's sandbox) though they are constrained to using the Context API to communicate outside their box. This API is common to across all endpoints though each endpoint has it's own instance for each handled request. It is this Context API that gives the endpoint its place in the world. It is this API which enables the handling of incoming requests and the issuing of responses for those requests. It also enables the creation and issuing of sub-requests and receipt of their responses. This is the only fundamental API that NetKernel has - for those familiar with the details of NetKernel this API is called NKF - the NetKernel Foundation API.

It is quite possible to develop and deploy endpoints which execute code against the Context API. Practically speaking any programming language which can be compiled down to JVM bytecode can be used to implement endpoints, this includes Java, Groovy, Ruby and Clojure. Whilst this is very useful it is not the be all and end all of endpoints. Because endpoints are treated as black-boxes there is a pattern called Language Runtime which allows the creation of endpoints to embody any implementable programming language or domain specific language (DSL).

The Language Runtime is a pattern which consists of an language runtime endpoint which embodies a particular programming language, a program representation, and a program compiler endpoint. If necessary for a particular implementation, the program compiler is responsible for converting the human editable representation of the program into a machine optimized representation. The language runtime will then execute the program. Typically language runtimes are stateless, when they receive a request they execute the program in isolation working with any supplied input arguments and possibly issuing sub-requests for resources they need to work with. Upon completion they return a response. NetKernel has language runtimes for many common programming languages such as Javascript, XSLT, Beanshell (interpreted Java).

There is broad spectrum of capability in the types of programming language that can be modeled in this way. In some examples of language runtimes the program can degenerate into what might look more like a configuration document. However in this case the endpoint might be better considered as a configurable transform or data source depending upon it's intent. But what this does show is that code and configuration are one and the same thing. I think that is pretty interesting. It shows that with sufficiently specialized endpoints a system can be composed by mere configuration. It is this which provides the answer to the "Where's the code?" question that I have seen asked many times when systems developed leverage the rich library of endpoints supplied with NetKernel.

Caching: How and why it works

2009-11-11T16:04:00.007Z

NetKernel's embodiment of Resource Oriented Computing (ROC) has the unique property of being able to transparently, efficiently and safely (i.e. not return stale values) cache computation. This is achieved through a process similar to Memoisation in functional programming languages but is generalized to be applied beyond functions without side-effects and even to functions that can change at non-deterministic times.

Wikipedia describes caching as:

...a collection of data duplicating original values stored elsewhere or computed earlier, where the original data is expensive to fetch (owing to longer access time) or to compute, compared to the cost of reading the cache... Once the data is stored in the cache, it can be used in the future by accessing the cached copy rather than re-fetching or recomputing the original data.

NetKernel has two distinct caches. Firstly a Representation Cache keeps response representations keyed against the requests that where issued for them. This is only useful for the usually idempotent request verbs of SOURCE, EXISTS and META. Don't worry right now about stale cache entries or non idempotent requests, NetKernel has a rich and powerful mechanism for expiry of representations that we will talk about in a minute. Before any request is processed by an endpoint, NetKernel will first check the Representation Cache to see if it can just pull on a pre-computed representation.

Secondly a Resolution Cache keeps request resolutions keyed against requests that are issued. Remember from our previous discussion of Dynamic Resolution how the resolution process can potentially become quite involved when large modular address-space is instantiated within a complex system. This cache acts to shortcut the resolution traversal. The Resolution Cache acts as backup behind the representation cache so that even when the response from a request cannot be cached NetKernel can start executing the request processing endpoint immediately.

Representation Validity
HTTP uses smattering of headers to define response expiration heuristics [Expires, Cache-Control, Last-Modified, If-Modified-Since, ETags]. Part of the proliferation is legacy caused and partly because of the number of approaches to best optimising sharing best knowledge of potential expiry for different server implementations over the latency and cost of the network.

Luckily things are much simpler in ROC. A response has a single IExpiry interface:

boolean isExpired()

The expiry function is should be a monotonic function which should never return false after it has returned true. I.e. once expired, always expired.

Standard expiry implementations are provided:

ALWAYS_EXPIRED - response should never be reused. The request is not idempotent or lacks referential transparency.
NEVER_EXPIRED - response will always be valid
TIMED_EXPIRY - response is valid until a given absolute time in the future.
DEPENDENT_EXPIRY - response is valid until any sub-requests response is invalid. This is usually the default expiration function and ensures that expiration propagates to all dependent resources.

In addition custom expiry functions can be programmatically attached to a response allow rich and dynamic expiration mechanism such as watching for modified filesystem files or the golden thread pattern used to layer expiration semantics over external data sources such as relational databases.

Scope
Earlier in this discussion we said that the Representation and Resolution caches are keyed on the request. The actually fields used from the request are the resource identifier, the request scope, and the request verb. However to improve hit rates we need to be a bit more subtle. NetKernel 3 employed a user setable flag "isContextSensitive" on each response, if false the response did not depend upon the request scope, i.e. the context of the request would not effects it's response. If true the response depends upon the request scope and a request issued with a slightly different scope would be considered and computed independently. NetKernel 4 automatically determines how much of the requests scope is needed to bound the response. Not only is transparent but it is optimal.

If you found that last paragraph a little dry your certainly not going to be the only one. A possibly easier way understand the problem we solve here is to consider a real world request. If I should shout out "Hey Peter I want a cup of tea!" what will happen? If I'm sat in the office and Peter is in a good mood then this request will resolve to Peter Rodgers, CEO of 1060 Research (for want of a better identity) will jump up and make me a cup of my favorite tea. In this example Peter (tea making endpoint) was resolved in the scope of the office. He used his current mood (scope) used my tea preferences (scope) to determine his response. (Of course real world cups of tea are not cachable but lets pretend they are.) Now if I change my scope by going out into the street and shout "Hey Peter I want a cup of tea!" what will happen? If I'm lucky enough to get a resolution of this request from some passing Peter then they are not going to have my tea making preferences in scope and this request will need to be re-evaluated. Let's say I ring in to the office to speak with Peter and ask for a cup of tea, now my scope is different however after a puzzled pause Peter says "The cup I made you earlier is still on your desk". Pulled back from the cache I get my cup of tea with out any effort. Peter didn't care about my location (scope) to know what tea I wanted. Ok I admit it, that was slightly contrived!

Temporal locality
One of the things we came to realize was how effective caching was. We thought that an inevitable consequence of high level of abstraction that ROC brings would be a performance overhead. To quote the SOA Manifesto "Flexibility over optimization", we expected a compromise to gain the malleability and scalability of ROC. But in actuality we got increased performance too. Even small caches can have big effects on performance. It turns out that this is due to Temporal locality. The real world has tendency for requests to form normal distributions with many resources being used again and again over a period of time and only a few that are one offs. Even with unique requests on the edge of a system the internals often gain a lot from caching. We find real world systems typically reduce computation by 50%.

I hope this posting has helped explain some of the magic behind caching in NetKernel.

ROC: Dynamic Resolution

2009-10-26T12:17:00.018Z

Resource Oriented Computing (ROC) uses a variation on dynamic scope to resolve requests. Dynamic resolution substitutes resolution of variables and functions with request resolution, and replaces statement blocks with spaces.

Each issued request has stack of spaces which we call the request scope. The resolution process involves attempting resolution in each of the spaces in turn until an endpoint is found. If no endpoint is found the request is deemed unresolvable.

One important point here is that resolution is always relative to the request scope. The implication of this is that resources are not absolute but relative to the observer, or should we say requestor.

Let's look at how a request gets a request scope:
To understand this we need a bit of background. In ROC any computation is always initiated by a root request. A root request is issued by what is called a transport. A transport acts as a client, constructing and issuing a request based upon some external stimulus. Examples include incoming network packets or events on a graphical user interface. Transports are initialized into a space. It is this single space which constitutes the request scope of the root request.

In resource oriented systems, endpoints issue subrequests to other endpoints just like functions call functions in functional languages or methods invoke methods in object oriented. Any non-trivial ROC system will involve deep subrequest trees. This happens when the response from one endpoint is computed using the responses from subrequests. (Just to be clear here, endpoints don't directly issue subrequests to endpoints, I use that language here to be concise. Any sub-request is always resolved to an endpoint using the method described in this post as a whole).

Let's look at how subrequests get a request scope:

My previous discussion on modular addressing discusses how a tree of spaces is built with delegation. When the root request is resolved into a modular address space it may resolve down through imports and overlays. As it does it accumulates the spaces it resolves through, such that when resolution is complete the request scope has additional stack frames for each spaces the resolution has passed through. Subrequests issued by the resolved endpoint gain this extended scope.

A subrequest is issued with the scope gained from the resolution of the parent request. Each space in the request scope, which may contain delegating endpoints, is resolved in, in turn, until a resolution is found. If a space fails to resolve it is popped and the request scope shrinks.

As a closing note it is worth considering why ROC doesn't use the simpler static scoping that is more common in modern programming languages: Because ROC isn't constrained within the syntax of one programming language but, rather, is distributed across heterogeneous space and endpoint implementations there is no sense of absolute lexical scope. It is the middleware of ROC that supports resolution rather than any particular language. It turns out that this rather unique characteristic is one that provides a unique benefit. Systems can be developed that span multiple programming languages enabling them to take advantage of the best tool for the job with a fine degree of control.

Rich Representations

2009-10-22T23:14:00.008+01:00

Resource representations: they capture a snapshot of the state or intended state of a resource.

When you think of REST it's natural to think of resource representations as a stream of binary data sent over a network ready to be consumed by a client. However, except in the degenerate case of serving static data and downloading it to your hard disk, that representation must always be "serialized" by the server and "parsed" by the client. That's obvious you say! Well it is if your client and server are network separated but in a Resource Oriented Computing (ROC) platform most requests are local, running on the same host.

Marshalling resource representations through a binary stream has two negative implications. The most obvious is the overhead of serialization and parsing processes. The second relates to the development and on going maintenance costs of the system. It you are going to have a serialized form that may well incur additional work, agreeing the wire-format and developing the serializing and parsing.

In ROC a resource representation can be any old object model with one proviso; it must be immutable. Without immutability multiple clients could communicate through side-effects either between themselves or with the service.

The first implication of this freedom is that representations gain an additional second dimension of typing. The first, original, dimension is the logical type, i.e. what is it? In an Internet systems this is the MIME type. The second dimension is the physical object model of the representation. In networked resource systems this second dimension was fixed, always being a binary data stream.

Here are some examples:

An open office text document with a MIME type application/vnd.oasis.opendocument.text embodied as a binary data stream.

The RSS feed to the 1060 Forum [http://www.netkernel.org/forum/rss/recent/1] embodied as a DOM object model

The Powered by NetKernel [http://www.1060research.com/netkernel/poweredbynetkernel.png] logo embodied in a java.util.Image object model

The complete works of William Shakespeare as an audio-book embodied as a binary data stream in MP3 format.

Employee "Tony Butterfield" embodied as a application specific POJO representation with the 1060 Research HR application.

The second, more interesting, implication is that these rich representations lead the way for unification of many processes which currently have separate names and are considered separately. Processes such as compilation, parsing, serialization, encoding, indexing etc. can all be considered as changing the representation of resource from one physical form to another; trans-representation, or the contraction transreption. Within ROC transreption is performed by "transreptor" endpoints resolved within address spaces. Transreptors may be invoked implicitly to adapt physical representations between service and client or explicitly requested. Implicit transreption leads to loose coupling by allowing services to be built without functionality to understanding anything other that the object model they work with. It allows two services to communicate even if they work with different object models as long as a transreptor can be resolved to bridge the gap.

Modularity of Addressing

2009-10-21T16:41:00.006+01:00

This post looks at the thought process behind modular address spaces within NetKernel and Resource Oriented Computing (ROC).

The Internet is a global information space. This is the design intent and it is reflected in it's data structures and registries including URI syntax, DNS, MIME types, and port number assignments. Intranets and such, the lower-case internet, allow for local address-space which provide a neighborhood around a node possibly backed by the Internet. However this is a pretty static and limited structure limited to local overrides with fallback and complex network configurations.

By contrast, software languages have demanded much richer namespace abstractions. Most modern languages support scoping which provides a mechanism for information hiding and encapsulation by restricting access to variables and expressions. Static scoping based upon the syntax of the programming language being the most common form. In addition many languages support the concepts of classes, packages or namespaces orthogonal from scope. They often employ the ability to create a flat list or hierarchical set of named spaces. These namespaces can then be imported or referenced from other spaces providing mechanism for managing the inherent complexity in having a large number of parts.

It is clear that programming languages have developed much richer addressing mechanisms and the likely reason is the increased complexity at this scale. The inherent complexity of systems is on an inevitable upward trend that is unlikely to end any time soon. This is due to a number of factors including the increasing interconnectedness of systems, the relentless march of new functionality and the difficulty and time involved in unifying the old with the new.

So it was clear to us that ROC needed to support rich modularity and we needed to find a way to marry the addressing mechanism of the internet with that. There is obviously a concern here that in doing this we damage the desirable properties of the web. Interestingly none of constraints and principles of the web as described in Fielding's thesis say anything about the necessity of a global information space. My guess is that it was done this way because Berners-Lee had a egalitarian vision of the web; that everybody should see the same thing. This was then re-enforced by the complexity of engineering more sophisticated solutions on a global scale. Of course today we see sophisticated content filtering firewalls but these don't attempt to create a coherent subset of the webs address space, they simply put crude barriers on certain areas.

I am very happy with the solution we came to to support modularity. We introduced the concepts of space and endpoint. A space is a resolution mechanism for requests. Exactly how a space resolves requests is not constrained but typically it will have a list of endpoints that it will test in order to determine if they will handle the request. Again exactly how and endpoint specifies what requests it can handle is implementation specific. Example endpoints would include Servlet like services or bindings of a portion of a filesystem into the address space.

Now consider an implementation of an endpoint which acts as a proxy to another space. It achieves this by resolving all requests that would resolve within the other space and then forwards them. We now have modularity. We can construct patterns of spaces like libraries of resources which are aggregated by a parent space. We call this endpoint an import.

As a specialisation of an import we can have an endpoint that intercepts requests entering a space and the response returning from the endpoint in that space. This is analogous to Aspect Oriented techniques. It enables
many things such as exception handling, security gatekeepers and quality of service auditors. We call this endpoint a transparent overlay. It is transparent because it doesn't effect the address space, the same address space is exposed but the behavior of the resources might be different.

A further specialization of a transparent overlay is a non-transparent overlay. Here the endpoint will resolve certain configured requests that differ from the space it delegates too. It then maps the requests it receives to requests into the delegate space. Non-transparent overlays include filters and general purpose address space transformation technologies.

These examples give a good idea of the modularity constructs that ROC enables. One thing that is missing from this discussion is how scoping is achieved. Scoping introduces relativity to addressing and opens the way some powerful patterns. I'll discuss how ROC uses a variant of dynamic scoping in a later post.

ROC: The Web inside

2009-10-20T13:52:00.007+01:00

We like to say NetKernel does Resource Oriented Computing (ROC). But what is ROC? One way to help understand is to ask "What drove us to create NetKernel?" The answer is simple; to take the properties of the World Wide Web and apply at the micro-scale to the inner workings of software systems. The motivation being to better manage the ever increasing scale and complexity with the vision of enabling the next generation of more ambitious systems.

Consider something as huge and complex as the Death Star. You don't even want to imagine the number of interconnected systems which must inter-operate on it, everything from the nuclear reactor to the tactical command systems and from life support systems to storm trouper shift rostering. They all need to work pretty well together when you are relying on it to keep quarter of a million people alive in the depths of space whilst a battling against the Rebel Alliance. Think of use-cases like diverting power from the canteen to boost the shields or the forecasting from the battle intelligence systems feeding into the staffing allocations. It is clear that there is big difference between what is required here and the most complex systems of today. Complexity grows almost to the power of 2 with the number of interconnecting parts. There are a number of dimensions to the problem including the time to design, implement and test as well as managing and maintaining the systems in production but the key to many of them is finding ways to manage the complexity.

It is widely acknowledged that the web works well. It has scaled way beyond expectation in terms of number of nodes and in generations of technology it supports. However it's remit ends once it dips just below the surface of the clients and servers which constitute much of it's mass. This is intentional and is not a criticism. What lives below the surface and also in the multitude of other systems is software and lots of it. This is where we looked to apply the properties of the web. I'm not going to criticize what happens down here - a lot of smart people have worked hard for a long time. However if you feel paralyzed by layer after layer of complex and unfathomable APIs and you wonder why you now need a gigabyte of RAM to run anything non-trivial then you'll understand our motivations.

The principles and constraints of the web are well documented in Fielding's Representational State Transfer thesis. It turns out, maybe surprisingly, that all these constraints and principles apply equally well when applied at the micro scale within software. In our work we also discovered further refinements to the approach due the differences of scale. In future posts I'm going have a go elucidating them.

Context

2009-10-20T09:36:00.002+01:00

As this is the first post I thought I'd say why I started this blog. However I'm not sure how long it will last because usually I find more pressing demands on my time than blabbing. So what I'd like to do, when time permits, is explore some of the concepts and thoughts behind the development of NetKernel and the evolution of what we now see as Resource Oriented Computing (ROC). It might also include some snippets of interesting technology discovered along way.