Event Models for RESTful APIs

Event Models for RESTful APIs

Adding event-driven processing to REST APIs is an important concept for the emerging Internet of Things. REST APIs provide for high level system interoperability and software usability, while event driven processing is needed for autonomic capability and efficiency.

An event is a significant change in the state of some element of a system. Events are asynchronous, that is they occur at arbitrary times relative to other operations in the system.


REST APIs enable web scale resource interaction and simplify application programming. The state of the system and values of observable properties can be easily discovered and operated on using GET and SET operations in URI addressable end points that have memory-like semantics. Metadata can easily be added to functional data to enable linking resources into composite horizontal applications.

Event driven systems process real time data efficiently. These systems are often based on asynchronous messages sent between end points that invoke software at the end points. Application programming is generally more difficult because of the need to interact using a stateful communication protocol. One example is the difference between programming for sockets and messages (asynchronous) vs. memory-like read/write semantics (synchronous).

The Internet of Things needs to be an event driven system for efficient real time response and processing, and needs REST APIs for resource discovery and programming efficiency, and to support rich metadata for interoperability in horizontally integrated applications.


In the context of a REST API, an event occurs when a resource in the API is updated and/or it's value is changed, or changed significantly. Events can be any updates, or filtered updates that meet some criteria, for example temperature too high, temperature too low, or temperature changed by more than 2 degrees since the last update. Typically, Client-Server REST systems don't provide for asynchronous events, instead software would need to perform periodic GET operations to sample the state of a resource.

We propose to add event driven processing to REST by introducing a common set of patterns and abstractions that provide a well defined event model. The event model will enable asynchronous event processing and message oriented communications to be configured and controlled through the REST API. 

Many of these patterns and abstractions are familiar and in use today in various forms. For example, CoAP has support for both synchronous and asynchronous REST operations, there are message brokers with REST endpoints (Qest, Xively), and there are hooks for instrumenting RESTful endpoints to add callbacks on updates (Real Time REST hooks). This will summarize the fundamental patterns and build a complete event model based on these patterns.


REST endpoints as event sources

An event occurs when some resource is updated or changes. To support events in a REST architecture, there needs to be some means to detect and generate events when resources are updated, a means to propagate the events and direct them to the proper destination, for example to update another REST endpoint, and a means of acting on the event, by storing a record of the event or invoking event handler software to respond to the event.

The diagram above shows an example event processing graph:

(1) sensor data are updated when the sensor endpoint makes an observation and sends an update to the Smart Object Observable Property, which is a REST resource. 

(2) an Observer resource associated with the Observable Property informs the system to notice updates and forward them according to routing instructions contained in the Observer resource.

(3) The event is routed to the destination resource

(4) The Observable Property at the destination Smart Object is updated

(5) A handler is invoked based on an observer associated with the local resource. This updates a physical actuator endpoint.

HTTP provides for asynchronous operations from Client to Server using POST, PUT, or CREATE. By arranging to have a server endpoint at each resource location, as is the case in with the Smart Object API shown above, asynchronous updates can be propagated using standard HTTP protocol operations POST, PUT, and CREATE. 

Other protocols like MQTT can also be used to send events between resources. The Observer resource can program the topic, broker, service, QOS, etc. in it's stored representation. 

Using MQTT, the topic can be a representation of the URL and the data can be the representation of the data, e.g. JSON. This way, the URLs can be send between endpoints whilst tunneling through MQTT, preserving the semantics of REST updates and, more importantly, preserving the granular resource access control system based on capabilities that REST enables.

We will use MQTT as an example asynchronous protocol for this document, but we could also use HTTP POST, websockets, CoAP, XMPP, AMQP, Dbus, MODBUS, RS-232, WSNs, or any other asynchronous communication mechanism.

Observers and Subscribers in a general REST event model

The Observer is a graph resource that is associated with a data resource. The Observer informs and programs the system to generate events on updates of the associated resource. When the observed resource is updated, the system performs the actions specified by the Observer's graph end points, either propagating a representation of the resource update to another endpoint or invoking a handler. Graph endpoints can be added to describe the action, for example to specify the QOS of an MQTT publish operation.

This example above shows an Observer associated with a REST resource (Observable Property) that programs the endpoint to publish updates to a specified MQTT broker endpoint whenever the Observable Property is updated. The protocol, IP address, QOS, and topic are all specified as local graph relationships that the endpoint uses to program it's behavior when the Observer is created. 

Observers can use HTTP POST, websockets, XMPP, or any asynchronous event notification method available in the underlying platform.

The Subscriber is a graph resource associated with a destination resource endpoint which is to be updated. The Subscriber programs the destination endpoint to subscribe to a data source, such as a MQTT broker or a Smart Object Observable Property that uses HTTP POST. This creates a graph link from destination to source, and could be used to create an Observer resource at the source endpoint.

This example shows an MQTT subscriber that creates a subscription with a broker to publish updates to the associated REST resource (Observable Property). This creates a graph link back to the data source. This pattern can be used to control HTTP POST, CoAP extended GET, and other asynchronous notification mechanisms from the destination endpoint.

Note that Observers and Subscribers are ways of controlling asynchronous updates from either the source or the destination endpoints, or both. If both are used between 2 endpoints, there is a 2 way resource graph connection established.

Message broker with REST endpoints

Another set of abstractions is to enable the configuration and of existing message systems such as MQTT using REST endpoints. There are two interfaces, a message interface and a REST interface, to shared stored resources. 

There are two message operations supported, to enable  clients or brokers to publish updates to REST endpoints through a message interface, and to publish updates of REST endpoints to brokers or clients subscribing through a message interface. There is also a subscribe operation using the message interface. Incoming messages update resources, and resource updates send outgoing messages.

The convention is to use URIs as topics that allow the system to construct URLs for the REST resource endpoints. The payload consists of the resource representation, e.g. JSON. This handles incoming and outgoing publish operations and acts as a message broker with REST access to topics and, if there is storage, topic history storage. 

REST Message Broker allows message clients to publish to REST endpoints and subscribe to REST endpoints. An application can monitor an MQTT endpoint by observing the REST resource it publishes to, and an application can publish to an MQTT client by updating to the REST endpoint the MQTT endpoint is subscribed to. For example, sensors could connect with MQTT and publish updates for application software to read, and application software could update a REST endpoint, resulting in the system publishing the update to a sensor that is subscribed to the URL as topic.

The above example shows how an MQTT broker endpoint can be added to a REST server to enable REST clients and MQTT clients to share data by mapping URLs to topics. When either a REST client updates a resource an MQTT client publishes to a topic, the corresponding resource update is published to any subscribers. 

A General Event Model for REST APIs and architectures

In summary, a complete event model is presented for adding event driven real time capability to RESTful architectures. The event model enables RESTful abstractions for event sources and destinations, message broker endpoints, client and server endpoints, and asynchronous software event handlers. 

Through the Observer, Subscriber, and REST Message Broker abstractions, and associated graph resource definitions, a general facility is created that can abstract the functions of common event and message passing systems to provide high level REST API control of the low level protocols, for example QOS control for MQTT endpoints. These high level models provide a consistent API for software interoperability, while allowing diverse protocols to be used in the underlying platform.

What is the Internet of Things?

What is the Internet of Things?

Michael J Koster 4/29/2013

So far in these articles, I’ve discussed in broad terms, and in relative depth, some ideas around architecture for the Internet of Things. This time I want to take up a specific question that helps to frame the differences between various standards and platforms. That question is about the fundamental distinction between Machine to Machine communication, known as M2M, and where I think we are going with the Internet of Things

What is M2M and what is IoT?

Of course I don’t have a hard definition, but I’m going to offer a broad contrast, based on what I see to be different, complementary optimization points. Let’s start with what we expect to get out of IoT as our context for the discussion. What is it we expect IoT to do for us?

IoT Value Proposition and Value Model

IoT will be a lot of things to a lot of people. Rather than try to list them all, I think there are some common patterns that define the basic value model delivered by the Internet of Things.

First, connecting things to software results in a big gain in the manageability of things and starts to enable the life cycle management of everyday objects. For business this means that capital assets are converted to managed services. The value begins to shift from a scarcity model to a usage model, with huge incremental increase in the return on investment (ROI).

Next, and even more exciting, is the network effect. As more and more things are connected and made available as managed services, whole collections of resources can be managed as systems. It’s clear that this enables the smart home and smart city convenience scenarios that many use to portray IoT. Beyond that, there is another, more exponential, increase in created value when resources are managed together as a system. This is the network effect applied to the physical world, and what I think generates the ultimate value of the Internet of Things.

To realize the full value of the IoT, then, things need to be connected to software, and they also need to be made available to be used together as a system. This is an important distinction that helps frame the discusion of the original question around M2M vs. IoT.

M2M enables connection of objects to software

Machine to Machine communication has been used for decades, in experimental form since the days of Tesla, and in practice since teletype. Nowadays, M2M connects sensors or connected objects to software services and to each other. Industrial automation has been handled by computer software since the 1970s.

M2M standardization has been a goal of industry for most of this time. M2M standards are focused on protocols and message formats, and recently, to some extent, software APIs. They provide for vertical integration, end-to-end, of systems. M2M standards allow for message level connectivity between endpoints in systems.

What’s missing from M2M is the high level mechanisms to build out horizontally and connect diverse resources together as a system to create and exploit the network effect.

The result is silos, which are separately controlled platforms and standards that are incompatible with one another, making it difficult to integrate devices and services from multiple providers.

The Internet of Things connects M2M systems together

M2M communication provides the essential underlying mechanism for IoT. But simply creating one standard for M2M is not sufficient. We think of IoT as adding a layer of discovery, interoperability, and autonomy on top of M2M communication services, using open internet-style standards. The focus is on data models and APIs that enable horizontal integration across different M2M systems. IoT standards enable collections of sensors and other connected objects using different M2M protocols to be managed as larger systems, creating and exploiting the network effect. IoT is built on M2M. An IoT platform connects across silos and discourages separate data models that lead to the creation of silos.

What about MQTT and other M2M Standards? How do they relate to IoT?

MQTT is at it's heart a message transport protocol. It moves messages from one endpoint to another, through a message queue. A publish/subscribe interface is provided to associate endpoints. An endpoint having data to send publishes the availability of the data stream, and any interested endpoint registers a subscription. After that any messages or, in the case of sensors, updates are pushed from the publisher to all the subscribers.

The value for IoT is efficient data push to potentially a large number of subscriber endpoints. Also, the MQTT service can itself be responsible for guaranteed delivery in the case of lossy networks. There are some different QOS models you can specify.

MQTT is a standard protocol that can be used to send data messages or updates from one endpoint to another. I have seen XMPP and PubSub also proposed to do this. There are some advantages to MQTT that make it a very good M2M standard.

HTTP and CoAP can also be used for M2M communication, based on RESTful interfaces at the endpoints. Efficient data push is native to CoAP using an extended GET operation, and for small systems and low data bandwidth, can easily be done in HTTP based systems with websockets and webhooks. RESTful interfaces have their own advantages.

RPC based systems based on .NET and SOAP are also being promoted as M2M standards, and getting some adoption. So as I see it, there are many ways to accomplish M2M communication with different strengths and weaknesses.

CoAP is a good example of both M2M and IoT, as it can be run on different sensor net radio stacks, while still providing interoperable APIs and data models across them all.

The open source platform we're building for IoT will be able to use MQTT "under the covers" to update exposed RESTful endpoints. The idea is to provide standard abstractions for the endpoints that can use different underlying M2M systems as needed to adapt to existing platforms or exploit unique advantages.

So, you can build systems with MQTT as the M2M layer. But MQTT is one component of many that will make up the Internet of Things.

Broad Interoperability has other benefits

Lack of interoperability tends to lead to distorted business models that attempt to monetize the basic connectivity and communication channels, creating a "fat middle" ecosystem full of multiple incompatible competing information pipes, and distracting developers from the human touch points where real value is created.

The Interoperability problem also makes it difficult to reuse and repurpose existing sensing capabilities in new applications. Much of the value will be created from repurposing existing data, as in the well-known example of using GE turbine and commercial aircraft sensor data for real-time weather forecasting. Existing sensors need to be made available for use in new systems.

IoT is an Internet

Just as the Internet is made of many platforms, standards, and networks, so is IoT made of many Sensor Nets, M2Ms, Application platforms, user platforms, and so on, creating a community ecosystem.

Meaningful IoT standards will be build in the manner of Internet standards, based on community process and open contribution, drawn from diverse use cases brought by participants from industry, research, the DIY and Maker community, and others.

We already have common internet languages and protocols, including HTTP, CoAP, MQTT, XMPP, and more. There are plenty of M2M options, but there is not yet a well known architecture for interoperability, discovery, and autonomy. This is, by definition, something we will all use and will need to build together.

Building a shared common platform will create a richer ecosystem for everyone, by enabling development of new systems to focus on the value-added human physical touch points of connected objects and the services that manage them. New services based on high value horizontal integration of resources will be enabled.

It will accelerate the deployment of the IoT at scale, leading to near term economic and human social benefits for all.

An Open Source Platform for the Internet of Things

Michael J Koster 2/11/2013

The Internet of Things connect people and the physical world together in a network of sensing, reasoning, and action. It is real-time, context-aware, hyper-local, and operates at a granular scale.

It is the social graph and the physical graph together: people connecting to things, things connecting to things, people connecting to people.

The Internet of Things: connecting people and things together with software. 

The Internet of Things is often described as "machine-to-machine communication" (M2M) or simply that everything will be connected. Connected to what? How does your refrigerator know what to say to your bathroom mirror? 

It doesn't, really. The information in your refrigerator is only relevant in the context of you, the user, where and when you interact with your "things", and your intentions at the time. Those connections can only be made by application software.

So the Internet of Things is really about connecting people and things together with software.

For people, it may mean greater DIY customization of their experience, through connected awareness, automation, telepresence, and in general a more "hands free" interaction with the physical systems of everyday life.

For business and enterprise, it means exploiting the network effect of connecting physical objects and features together with software. This will produce results like the conversion of capital assets to managed resources, with an expected multiplicative effect on efficiency and resource utilization.

IoT Applications are user-directed graphs

For the purpose of this discussion, we can to think about IoT applications as consisting of sensor and actuator end points, user device end points, and application software that connects the endpoints in the form of a directed graph. 

The application software may exist on more than one server, as internet applications do today, but the important distinction is that these connections are made based on user context and intentions rather than javascript programmers, and that the graph may be persistent and dynamically updated per user needs.

Once we understand that the goal is to connect people and things together with software, we can look at ways of using network and data technology we have today, and that which we might invent, to solve the problem.

IoT is Deployed in Silos today

The Internet of Things is being built today, but in a highly vertical fashion. Many companies are emerging with a top-to-bottom technology stack including sensors, actuators, gateways, cloud services, and phone apps. These are typically not interoperable with any other vendor's system at any level (sensor, gateway, service, or app). They all use the internet in similar but incompatible ways.

There are many problems with this situation, which I dive into in my other articles: users vulnerable to single vendor, lack of robustness in solutions, duplication of core platform services, developer code locked to a vendor, and slow industry progress. 

This is to be expected early on in an emerging area like the IoT, as many different solutions are being tried. This has been going on for several years now in the IoT, and a loose infrastructure has emerged based on a common set of patterns and data models. While the various systems are not seamlessly interoperable at any level, the common patterns that are emerging suggest that interoperability would be a practical goal.

WSN Interoperability 

Wireless Sensor Networks are key technology for the IoT, so much that WSN was probably the number one issue in the early deployment. There are many competing standards: Zigbee, Z-Wave, EnOcean, KNX, XRF, IEEE 802.15.4, Bluetooth, WiFi, RFID, RFM12B...

WSN diversity is a good thing: the differences between the systems are important, such as power, range, data rate, packet size, and there is no one-size-fits-all nor is there likely to be. Some systems support ultra low power energy harvesting like a temperature sensor that runs from a fingernail-sized PV cell and works 100 hours in the dark, vs. richness of protocol such as a complex appliance that needs real time monitoring.

As a guess, there may be 3-4 most common WSN technologies e.g. BT4 moving to 802.15.4 (body area, fitness), Enocean (energy harvesting home and bnuilding automation), WiFi Direct (powered appliances), RFID (sensor tags and keys), 802.15.22 (TVWS) for metro/wide area sensor networking, and these will change and evolve over time.

The new IPV6 protocol for the internet has a set of low power constrained network protocols that allow WSN connection to the internet through a simple router and proxy. These protocols are forward-thinking toward the Internet of Things, and will likely take over  more of the WSN domains mentioned above, as Moore's law pushes greater logic complexity into ultra-low power circuits.

Service interoperability

Connected things connect through WSN gateways and routers to Internet services that fulfill the application logic for the user. Today, for the most part, each vendor provides a cloud service for the devices they sell, e.g. Twine. There are also cloud services that allow any connection, providing an API for anyone to connect, for the purpose of integrating multiple devices. 

There is the beginning of an ecosystem here, where some devices are being built to use existing services, e.g. Good Night Lamp uses Cosm as their cloud service. Other services that allow open API connectivity include Thingworx and Digi Device Cloud. These services all use very similar RESTful APIs to JSON and XML objects, but have different underlying data models. As a result, sensors and gateways must be programmed for each service they need to interact with. 

Proprietary Cloud Service + Open Client

Many IoT services today are based providing easy access to the devices and gateway, with open source client code and reference hardware designs, selling hardware on thin margins, and kickstarter campaigns. There is typically a proprietary cloud service with a proprietary or ad-hoc API from the device or gateway to the service, and a structured API to the service offering "cooked" data. 

These systems contain a highly visible open source component, but much of the functionality comes from the cloud service. If a user wishes to use the open source part of the system with another service, the APIs will need to be adapted on either the device/gateway end or service end, or both. It's not exactly a lock-in, but there is a fairly steep barrier to user choice.

This also leaves users vulnerable to outages of a single provider. Much better and more robust would be an ability to configure more than one service provider in parallel in an application graph, for a measure of robustness in the face of service outages. Even more, it should be possible to run user application code in IoT gateways, local user-owned servers, or user-managed personal cloud services. Today's infrastructure and business models are at odds with this level of robustness for users.

User Agency

Another issue that goes even beyond the IoT is that of user identity and user control of resources. While many are imagining the next Facebook of the IoT or the next Twitter of the IoT, these systems are examples of users giving up control of their data, their experience, and often their very identity. 

The very nature of the Internet of Things requires a new paradigm based on user agency. Users need to be enabled to customize and define their own IoT applications. We want to give users full control of their identity, their resources, and their experience on the Internet of Things. 

Toward An Internet of Things Platform

A platform for the Internet of Things need to broadly address the needs of users, developers, and vendors. The features described here are a subset of the entire platform, but those proposed to address the above needs and requirements.

Here are what we propose as base features for an open source Internet of Things platform that promotes interoperability between sensor networks, gateways, services, and user devices.The individual elements have been developed, and are discussed in more detail, in earlier articles of this series. 

The Smart Object API defines a data model for broad interoperability

The Smart Object API is a RESTful API to an encapsulation of resources at a node in an application graph. A Smart Object contains IoT data, metadata, and software agent code (application software event handler) resources. 

The Smart Object API is also a Semantic Web Linked-Data application that provides integrated, semantic web based, resource discovery and linkage. Semantic Web standards are also used to structure the resources inside Smart Objects. This allows all of the software to be driven by graphs rather than pre-programmed state machines. New APIs can easily be added without breaking any existing code.

The resources presented by the API include RDF descriptions in various web formats, Agent resources to manage application software handlers, and Observable properties, which carry data and data streams.

Observable properties are also structured resources, with their own RDF descriptions and Subscription (also called Observer) resources to enable changes in the observable property to be pushed to URI endpoints in the application graph. Observers may use a number of protocols, for example, Websockets, HTTP POST, PubSub, MQTT, XMPP depending on the application.

This enables a distributed data flow model using software agents + event handlers to implement application logic state machines, filters, triggers, etc. based on changes in data or context.

Use of an abstract observable property enables flexible data typing based on semantic constructors and JSON, XML data objects.

Programming languages

The semantic + event model can be integrated into general purpose languages through libraries. High level methods for semantic linkage and handler connection will allow developers to build graphs and define event processing rules, filters, and triggers. 

The RESTful internal resource model enables APIs to easily and automatically be built for any language. Special purpose languages may also be developed, sporting embedded Semantic Event processing models and control structures. 

Universal Gateways integrate local sensor net resources

Smart Object gateways will host all smart object resources, performing Semantic integration and tagging of WSN resources, WSN-to-internet proxy, adding encryption and security to constrained-protocol WSNs, caching and buffering of accesses, and function as an always-on resource for data collection from energy harvesting sensors and other chirpy data sources. Software agents may be locally executed in the gateway.

The Smart Object API resources provide a semantic backplane for multiple WSNs and multiple services, enabling integration of IoT resources from top to bottom of the stack, and even accommodating IoT vendor "silos" that only provide high level cooked APIs from their cloud services.

Smart Object gateways and services talk to each other using the same API, which is strongly consistent with the low level object programming model provided to the library. This enables application code to run anywhere, in any cloud service or gateway, providing robust service availability in the face of network and service outages.

Native applications using the Smart Object API will have access to load balancing, failover, redundant persistence, and strong object storage models.

Personal Object affords user control of identity, resources, and experience

Personal identity and agency are of special importance for the Internet of Things. We have decided to build in a strong model for user agency into the system from the beginning. This allows users to control their identity, access to their resources based on their strong identity, and also to control their experience on the Internet of Things.

The Personal Object is a special case of a Smart Object that represents a user's identity and preferences on the IoT. A Personal Object can be constructed using the Friend Of A Friend ontology, which provides for creation of a social graph that connects people to other people that they know and who can verify their identity. A subgraph of this socal graph can then be used as a strong identifier for a person.

To provide for IoT interaction we propose to connect the social graph to the physical graph using concepts from a new Friend Of A Thing (FOAT) ontology. These concepts and relationships will define people's relationships to the physical resources they control and that also help establish their graph-based identity.

This method allows for the definition of very granular resource access control based on strong identity. Resources are shared with people in one's social graph by making connections to other nodes which can be used to verify both the identity and the capability of resource sharers. These links allow very granular resource control of each resource or property according to specific capabilities. At a minimum there will be API level capability keys, extensible to more granular endpoints e.g. graph objects but not subjects.

Personal Cloud service gives users a vendor-neutral identity and point of control

People today are basing more and more of their digital existence on cloud services such as Facebook, Google+, and Twitter. These vendors often make us the product and hold our data hostage. Even with the best of the platforms, we are ever more vulnerable to channels and storage and software that others control. 

Going forward, it's difficult to imagine an IoT experience where service vendors like Facebook set the constraints and define the parameters of how you turn your lights on and off. 

It's also difficult to see how multiple competing vendors could provide truly compatible services that allow users to freely move their devices from one to the other.

We will therefore provide in the platform the means for users to create personal cloud services. This will enable vendor-independent storage and service locations to host user's Personal Object and Smart Objects. 

Personal cloud services can provide redundancy and persistence, and can be a user's IoT home location on the internet.

Personal cloud services can also function as Gateway-as-a-Service to integrate other cloud APIs into the common data model.

Perhaps most importantly, the creation of a personal cloud system allows users to root all of their social and physical graphs at their own service, analogous to a personal computer, providing a user locus of control. One's facebook graph can be rooted in and synced to one's Personal Object running in their personal cloud service. 

User Identity and Agency on the Internet of Things

The Personal Object: User Control of Identity, Resources, and Experience on the Internet of Things

1/20/2013 Michael J. Koster

User Agency on the Internet of Things

The Internet of Things connects people and things together with software. 

The most recent article on Open Infrastructure describes the Internet of Things application as a graph which contains sensors, actuators, and user interface devices at the edges and rules, triggers, and other application logic at the nodes, executing in the sensors themselves, in gateways, and in cloud services. These application graphs are built and customized according to user intentions and user context.

This article discusses the idea of user agency on the Internet of Things; users will need ways to express their intentions and constraints to the system in the management of IoT resources and devices owned and controlled by the user.

Scalable, granular, and simple user authentication and control

Users and creators are roles that exist at all levels of the system. Users create application graphs based on intentions and context. Creators are users of a platform, and give users tools and systems with which to customize their own experience.

Users share data with other users, with service providers, with everyone, or with no one, as needed, in a granular way, to compose these application graphs. Users also import data shared out by others to compose into their own application graphs.

Clearly, the ability of users to control and manage their resources will become a limiting factor, if current authentication and control mechanisms are used. Passwords and user accounts using existing permission schemes will place severe limits on the granular scalability and robustness achievable for the future Internet of Things. 

It has become clear that a system of user control for the Internet of Things will need to be built into the basic access mechanism of the IoT platform and APIs from the beginning, rather than something added on. It's also clear that the design should be based on user intentions and user control as underlying principles. 

User control of identity, resources, and experience on the Internet of Things

From this a basic requirement emerges. For an effective, scalable deployment of the Internet of Things in a way that respects users intentions, we need to build in a system that places the center of control with the user. As a start, we propose to build affordances that give users control of their identity, their resources, and their experience.

The stability and robustness of IoT applications depends on a stable base of identity from which users can be confident that only their intentions are being implemented in their application graphs. User identity must be consistently maintained across multiple service providers, in a way that is also portable and replicable across identity platform providers. This is necessary in order to achieve an ecosystem that allows users to choose services and providers that best support their needs.

It is important that users have control of both the identity itself and of the platform used to provide it. This provides a base of assured user agency and retention of user resources and context in the face of changing business conditions. It also allows presentation of a free market choice to users.

User resources (data, sensors, "things", virtual objects) must be accessed under user control or by entities delegated by the user to access the resources. Control needs to be delegated on a fine grain basis so that the user can maintain a "need to access" policy for individual data elements, e.g. their height and weight but not their cholesterol count or blood pressure, and perhaps for a limited time period or for a one-time access. 

The control of resources thus needs to be linked to user identity, and given out only under user authority. There needs to be ways in which users specify their intentions for resource control and sharing easily and unambiguously, while taking into changing account user context and user situations.

Users must be in control of their experience: to create IoT application graphs based on their own intentions and context, and to choose or compose whatever user interface is appropriate. Users intentions and context will direct the composition of user application graphs. Semantic description frameworks with discovery and linkage mechanisms allow user intentions to be determined through context and unambiguously applied. 

For example, a user may have a graph describing their house such that a generic energy management ontology can be applied to control lights and heaters, further customized by adding knowledge of user preferences to the graph. It should be easy to "tinker" with a generic template to customize the user experience.

System based on knowledge and transparency

The Internet of Things protocols, APIs, and platforms need built-in methods to apply user intentions to resource access control. The idea is to build in mechanisms that allow users to define their intentions and apply them to the information graph of the system, and for the API to only be capable of resource interactions that are consistent with user/owner intentions. 


A system of shared secrets and passwords for accounts and login mechanisms won't work for the scale and granularity of the IoT. This system barely works for the web today, and is being replaced by lower friction systems like Oauth. 

We propose to base user authentication and resource control on strong knowledge of graph relationships. This allows the user's intentions to be plainly known throughout the application graph, and can plainly expose attempts to falsify credentials or manipulate the system.

We can use the social graph to authenticate users, by using the unique set of graph connections that a user has to other users and to the objects they own. These graph connections can be validated by checking links back from multiple graph objects at different locations. The concepts that define relationships between people are defined in the Friend Of A Friend (FOAF) Ontology

We can also build a resource graph consisting of Smart Objects connected to services and other Smart Objects, based on the Smart Object, Sensor net, and related IoT ontologies, and the Smart Object API. 

What's needed now is the mechanism to connect the two graphs, such that users are authenticated by not only their social relationships, but also their verifiable connections to "things", for example an RSA key or RFID badge, or both. Connecting the social graph to the physical graph will also allow direct expression of the authenticated users intentions within the application graph.

Resource control will be defined by the graph linking users to their resources and defining predicate concepts that, for example, define that a particular entity in the user's FOAF graph "may observe" a particular resource under the user's control. These concepts can be defined in a "Friend Of A Thing" (FOAT) ontology. The idea is to allow resource owners to precisely define the resources to be shared and the constraints, e.g. time limit, number of observations, on the sharing.

Personal Object

We believe that the only way to accomplish this responsibly is to give users full control of their social and physical graphs. For this we need a provider-neutral identity API and platform based on open internet standards. We propose an Internet Personal Object, or Personal Object, to be constructed using a Linked Data API (Smart Object API or generic Linked Data) to hold user's social graph connections, preferences, and other identity information. 

The Personal Object is essentially a Smart Object that represents a person's identity and some of their preferences and intentions, from which users can add graph relationships to define access control for Smart Objects and other internet resources under control of the user.

We propose to create a Personal Graph API, using the Smart Object API resources and service layer, to create a Personal Object. This API, along with the extended FOAF and FOAT ontologies, would be used to create strong identity-control graphs between owners, their delegates, and the things they control. 

Personal Cloud Instance 

There is another area of user agency to be considered in the deployment of the Internet of Things. Users get benefits in mobile accessibility, and may gain something in redundancy and professional data management, by keeping our information resources in proprietary cloud application services, but what is lost is control. We don't have any guarantee of continued access to our data, and we depend on a proprietary platform that we can't replicate in order to interact with our data anyway. If we can create or purchase software, it will be locked into a proprietary platform.

With cloud computing we have the opportunity to provide a new level of robustness and user control that couldn't easily be achieved through hard drives on PCs. We need to create a new class of platform-as-a-service that gives the user control over their environment and applications, much the same as the PC environment provided. These services can be linked in redundant fashion with multiple points of service, including local hardware gateways.

For ultimate user control, we propose a personal cloud service, for example an Amazon micro instance, that can host the personal object in a vendor-independent way. This can be updated through various social graph APIs in order to keep up to date with changes in one's social network. 

We propose a basic user-customizable Open Source tool for users to manage their social and physical graphs; sort of an Internet of Things "address book". The user's social graph connections (of those who mutually choose to participate) are maintained in FOAF, and the physical graph connections are maintained using FOAT. The tool would allow broad preferences to be established, including cultural context, with personal customization.

PCaaS

A platform based on Open Source software can be built and deployed as a standard OS image that has all the necessary services pre-configured. There could be multiple instances in different cloud providers managed as redundant copies of a user's Personal Object. The cloud platform would also include instances of Smart Object services for user-created IoT application graphs and graph elements, and be able to run IoT applications on behalf of the user in the personal cloud service. 

This framework would extend to physical IoT gateways, providing user control all the way to the IoT endpoints. IoT applications may then be distributed easily across cloud services and into local gateways.

This Personal Cloud Service can interact and share with other proprietary services and walled gardens, etc., but users must have a stable information base that is under their control from which other relationships can be managed. This will form a stable sub-structure for user agency on the internet.

How to enable users to control their identity, resources, and experience on the Internet of Things:

1. Create a personal social and local resource graph that users have control over, using an Open Source freely available platform and linked data API

2. Create a Personal Object that stores the resource graph, and additionally manages the users preferences and intentions as policies. Create and manage links from the social graph to the physical graph using a freely available Open Source tool such as an "IoT Address Book".

3. Host the Personal Object in a Personal Cloud service that is under control of the user. Use Open source stacks and tools for all essential infrastructure.

The Internet of Things needs an Open Infrastructure

The Internet of Things needs an Open Infrastructure

12/30/2012
Michael J Koster

An Internet of Things application may be best represented by a graph, with sensors and data sources connected to filters, triggers, rule evaluators, and other processes, with these outputs connected to actuators, storage, visualizations, etc. The connections are made according to user intentions and are customized to the user's context.

It's often observed that one thing holding the Internet of Things back is the difficulty of connecting things together. Some are trying to solve this at the platform level, by providing connectivity within the platform e.g. SmartThings, and some are working to connect data between vendor's offerings, e.g. IFTTT. 


A general solution to the problem would be a common connection and communication mechanism between the various sensors, actuators, gateways, services, and apps. It's generally realized that some degree of "open-ness" is needed for the systems to be connected.  Various claims of open-ness are being made for different IoT systems and in different contexts. In this post I would like to discuss the meaning and impact of open participation in various aspects of Internet of Things technology and processes.

First, let me describe the meaning of some common concepts as I understand them for the purpose of this discussion:


What is a Platform?

A platform is a common set of resources used by developers to build and deploy applications. There are other types of platforms, but for this discussion there is a common set of resources used by IoT developers on which to build IoT applications. 

For the common IoT deployment model of sensors, data sources, and actuators connected to services, the platform could consist of sensor nets, gateways, service enablement, APIs, servers, clients, application libraries, smartphone UX enablement, and other common resources. The platform would allow application developers to build or add sensors as needed and develop applications that can be hosted on the sensors, gateways, services, and smartphones. A minimum of customization of the platform should be necessary to support new applications. 

Platforms allow developers to share resources and patterns, thus minimizing the new code needed to create new applications. Platforms are often specific to application domains; there may be many different platforms across the IoT just as there are many platforms on the WWW. A platform is often defined by a programming language or an operating system, e.g. "the Windows platform" or "the Java Platform". Applications built on a particular platform are often interoperable with other applications on the same platform, and portable from one instance or implementation of the platform to another.

Examples of platforms that are built on the WWW are WS, .NET, AJAX

We could say that the WWW is itself an application built on IP, where IP is a platform. Other IP applications include ftp, smtp, POP, IMAP. 


What is Infrastructure?

Infrastructure consists of common resources used for deployment of platforms. 


Physical infrastructure includes the copper, fiber, silicon, and wireless switching fabric of the internet.

DNS is an infrastructure service for the Internet. 

Anytime some layer in the stack becomes ubiquitous and is used to build platforms on, it could be considered infrastructure. IP could be considered infrastructure on which the WWW runs.

The IoT infrastructure consists of wireless networks, the Internet, routers and switches, and in many ways the basic protocols used. For example, HTTP and CoAP enable IoT platforms to be built using these web protocols as APIs to the common internet infrastructure.

What is an Ecosystem?

An ecosystem is a system of systems that all share resources and protocols, and interact and influence the outcome of each other. The WWW can be considered an ecosystem. The Internet and supporting protocols HTTP, etc. are it's infrastructure. There are a number of platforms that all interoperate, more or less, within the WWW as an ecosystem. 

In a healthy ecosystem, it's natural for platforms and protocols to come and go. There is a natural interdependence that is balanced against competition. The concept of "all boats must rise" is important to an ecosystem. 

In this context, it's difficult for an ecosystem which has multiple competing platforms to be contained within a single company or institution. The sustainable ecosystem seems to be based on open participation at all levels.

What is Open?

There may be many definitions of "open", but for this purpose it should refer to an open avenue of participation. By this I mean that any capable participant is allowed to participate on an equitable basis. Participation can occur at many levels in an Internet based system; here are a few example categories:

Open participation for Users

Participants at this level trade for goods and services that benefit them in their life or business. Open participation for users has to do with access to goods and services at fair prices with fair policies, through accessible channels and outlets, and enabling user participation across languages, etc. without any obligation or limitation on the part of the user as to their fair use of the goods or services, including sharing what they own and control with others.

Internet of Things users will participate in creating their own applications and managing their own information in ways not previously possible. The role of user as composer of IoT application graphs gives the user agency in customizing their own experience. At the same time, it requires a new level of compose-ability in services and data sources. It will also drive new solutions to today's problems of password based security and user control of information. 

Open platform for Developers

Participants at this level add value to goods and services by creating new offerings based customizing, programming, combining, or otherwise building on top of the offering as a platform. Open participation here requires equitable access to specifications and design information without requiring intellectual property licenses, regardless of institutional affiliation or lack thereof. It should be possible for a developer to create a new application on an open platform without encountering barriers to entry. 

Internet of Things application developers will need to provide high level compose-able services and data sources, in a way that users can easily manage. There will need to be standard, frictionless, transparent mechanisms to pass user authentication and provenance along with commands and data, built into platforms. This will require a level of cooperation between developers and standards bodies throughout the process.

Providers (Service Provider, Hardware Manufacturer)

Open access for service providers enables multiple sources. Open participation at this level requires open access to specifications and access to the standards process, without any licensing restrictions, and regardless of institutional affiliation or lack thereof. It should be possible for a start-up company to enter the business as an open platform service provider or manufacturer without barriers.

Standard infrastructure services, APIs, and platforms will enable a service provider business model, where someone with domain expertise and an idea to solve a problem can easily build a new service business focused on solving the domain problem. Open source reference implementations of infrastructure components will enable IaaS and PaaS providers to support rapid prototype deployment of new platforms and services.

Standards Body

Standards bodies are responsible for the common specifications of a platform or system of platforms. They are usually composed of representatives from corporations and public institutions, and experts in the field. An open standards process is one which allows equitable participation of any qualified person, without any licensing restrictions, regardless of institutional affiliation or lack thereof.

There are currently a number of different standards processes being used to develop IoT standards, each resulting in subtly different qualities of open participation at the different levels. Here are a few examples of broad categories.

Types of Standards

Examples of standards process include de facto, Industry SIG, and the Internet standards process (IETF). 

An example of an industry de facto standard is Microsoft Windows. In this case, avenues for participation are limited to a single vendor provider and single corporation develops the specification, i.e. only one vendor provides the Windows platform. Participation by software application programmers is most open, with practically no barriers. User participation is limited by a restrictive end user license. 

Many current standards are based on Industry Special Interest Groups (SIG). A common SIG purpose is to share and cross-license Intellectual Property between corporate and institutional members. The specification is often not publicly available, but can often be obtained by joining the SIG and executing the license and NDA. An example of this type of SIG is the Weightless TV whitespace M2M standard. Participation as a developer or vendor is limited to SIG members. Participation as a user is usually based on a hardware purchase or in the case of software a binary license.

The Internet standards, for example the Internet Engineering Task Force (IETF) are based on open participation and industry consensus. Avenues for participation are mostly open to all, at all levels. Tim Berners-Lee didn't have to ask permission from a TCP/IP SIG to develop the WWW application and HTTP.

In Europe, there is a broad consortium of Industry, government, and educational institutions working with standards bodies to develop a comprehensive framework for the IoT. This is an open process that allows many avenues of participation. There isn't enough space here to break it down, but there are some excellent participatory models being used. 


What's the license?

When it comes to answering the question "How open is it?", the final word is the license. Any formal work these days, whether software, hardware, or process, e.g. a standard or benchmark, comes with a license. It's the details and requirements of the license that ultimately determine how open your avenues of participation are. It's worth noting that the term "open" is used by different providers to describe a wide range of license restrictions, and it's very important to understand exactly what the license requires and how it limits or enhances your value in participating.

Examples of licenses include the Microsoft Windows EULA, the GNU General Public License (GPL), THe Apache 2.0 License, The Facebook Terms of Service agreement, and basically every long document you need to scroll through and check "agree" on before participating in a service or downloading software or activating a hardware.

Open Source software

Open Source software is generally recognized as software which is available in human-readable and maintainable source code form, to any developer that follow the provisions of the license. Open Source software has both copyright and license. The concept of "copyleft" is sometimes used to describe open source license terms. Some open source licences require derivative work to be published with the license intact, e.g. the GPL. Other licenses do not restrict the use of the code, and allow a developer to produce closed source derivative works. 

Sometimes source code is published for informational purposes, for example to write interface code to an API, but is not available under an open source license, that is the work may not be reproduced or used for derivatives. This is not open source.

It's generally agreed that infrastructure services and platforms benefit from being Open Source, where multiple contributors are involved in revisions and bug fixes. This provides a level of transparency that has been proven to result in very robust code.

The infrastructure of the Internet, that is the DNS servers, web servers, programming languages, and client components, is built from Open Source software. SInce the Internet is an ecosystem, or system of systems, there are compatible closed license source alternatives for many of the services. 

Open Source Hardware

Lately the notion of Open Source Hardware has become popular. The basic idea is the same as Open Source software, in that the design specifications, schematics, board layout, and firmware are available for anyone "skilled in the art" to reproduce the hardware for any purpose as long as the terms of the license are followed. A good example of this is the Arduino. 

Again, it's the license that matters, even if the design specifications are published for reference. Arduino is available under an open hardware license that allows anyone to manufacture and sell the designs. Note that this is not the case with Raspberry Pi. The Pi hardware license is controlled by the Pi foundation, and is only made available on a case-by-case basis to their manufacturing partners.

Open API

An Application Programming Interface allows developers to build applications on a platform or to communicate with other applications and services. The idea of an open API is to open the system at a particular point to allow developers to build onto the system without signing a license or NDA. This gives the platform provider control over how developers can use the platform, while allowing anyone to develop on, and add value to, the platform. 

The API a platform provides becomes essentially an extension to the application programming language, baking the API into application programs developed on the platform. The unfortunate consequence of this is that applications developed on one platform and API can not be used, or sometimes even ported easily, to other platforms that use different a different API.

So very often, an open API is a tool for a platform provider to attract developers and lock them into the platform. Sometimes the lock-in is not intentionally done to exploit the developers, but rather an unintended consequence of simply developing a unique API for a new platform. That is, there is no well known API that exists, no a new one is created.

Open Source vs. Single Source

Another impact of closed development is the creation of a single source infrastructure. For example, a number of the current IoT system deployments include sensors, a gateway, and a cloud service. For the system to perform it's intended function, all components must be functional. If any component fails or if the provider of a component goes out of business, the entire system fails. Many of the vendors have addressed the sensor/actuator part of this equation by providing an open API or open source client code, etc. but still present a single point of failure in the cloud service. 

In this sense, then, the opposite of Open Source is Single Source, and the attendant single point of failure. Of course, we need to provide value in unique services that, if open sourced, would allow competitors to offer equivalent services... Where is the business model? 

One solution is an open source service enablement layer that allows the creation of a service ecosystem. Vendors can offer different, competing services that are targeted to user groups and use cases. Users can have a choice of differentiated services to perform a particular function in the application graph. The Internet of Things is going to be big enough to not only enable this, but soon to require it.

Impact on innovation cycle

When everything was developed under the closed IP based system of the industrial age, innovation proceeded at a pace regulated by the patent system and the pace of product development cycles. We had to wait to see what the competitive field produced in secret and then released in public to drive each innovation cycle. These cycles often got out of phase, resulting in some spectacular self-indulgent technology spirals like the iAPX432 or, more recently, the CPU pipeline MIPS blowout ca. 2000 where, out of competitive pressure, all the vendors produced designs that were way too expensive and couldn't be cooled. Upon further thought, it may be engineer turnover between companies that becomes the rate limiting factor in the closed IP development world...

On the other hand, the opening up of software engineering that came with Open Source and Open coding platforms like Python, PHP, etc, has resulted in sweeping innovation of the software process, with agile development cycles measured in days and continuous release processes. Use of common patterns like RESTful APIs and platform support for common patterns have resulted in most software effort focused on building an implementation of the idea, instead of building scaffolding and plumbing.


Access to one's own inventions

I hold a number of patents in computer architecture. Actually, I don't hold them, I assigned them to my employers who sold them to their buyers etc. and as a result are now owned by completely different people than when I made the inventions. I could expect to be sued by these people if I try to advance my work in these areas for another company, or at the very least my prior patents could be used against me. 

What I'm getting at is the current system of IP assignment and value based on secrecy is a limiting factor in my development of any technology I worked on while under the IP agreements imposed by the corporations that employed me. In reality there is a concept of fair use but no one knows exactly what it is without a legal test. All this creates a bias in me to want to work on open source systems where I can maintain access to my own work.


Impact on scalability and single point of failure

Open systems have a distinct advantage in the ability to rapidly scale up and to tolerate single points of failure. IN an open Source ecosystem, if one company fails or is unable or unwilling to continue maintaining a component, there will be another to take over or sometimes provide resources to keep the project going. In a closed system, the single points of supply become the weak links. 

Consider the Raspberry Pi. The single supplier situation combined with unexpected demand has resulted in continuous shortages, to the point of being a business risk for anyone considering using the Pi as a component. If the RPi design were licensed as Open Hardware, there may have been an opportunity for an second source supplier to come up to speed and meet the demand. At least the business risk would be mitigated by the possibility of an independent second source.

Business models

One item on every "must have" list for outside capital is something no one else has, preferably with a lock on the Intellectual Property. The requirement for exclusivity should be balanced against the advantages to everyone of a healthy sharing of infrastructure. 

It's often assumed that a business must possess some secret in order to be competitive. It's also assumed that Open Source programmers don't get paid. Both these notions should be challenged. Plenty of businesses thrive without needing to keep secrets. Businesses based on Intellectual Property are relatively new. It's worth noting that most of the modern infrastructure around software patents was created during the ascent of the Microsoft corporation, with much help from a certain Mr. Gates Sr. in the legal department.

The Open Internet of Things provides clear opportunities in two areas. The first is in providing new hardware for sensors, actuators, gateways, and data centers plus the required hardware connectivity in sensor nets and public infrastructure. The second is in providing services that add value based on the rich connectivity afforded by an open infrastructure. 

These devices and services can and should certainly be unique and don't need to be Open Source. The important thing is to not create lock-in or lock-out situations which result in unfair access or single points of failure. If there is good use of a common set of protocols and infrastructure, the ecosystem should support plenty of redundancy in heterogenous solutions and open source alternatives. 

I believe the best business models will be the ones that take advantage of connectivity and standards, rather than the ones that create vertical silos, walled gardens, and lock-in, regardless of the exclusivity of the Intellectual Property. Besides, recent legal battles illustrate the futility of pretending to own something everyone uses or does.

The Emerging Internet of Things needs an Open Infrastructure

To realize the Internet of Things application graph shown in the introduction, a level of open-ness is needed at all levels of participation. Open standards are needed to enable broad industry participation without any legal obligations. Open platforms are needed to allow users to compose, share, and migrate their application graphs across multiple providers. 

Open standards and platforms enable developers to focus on the business model, domain oriented, value added, areas of devices and services, and allow the cost of developing and maintaining shared infrastructure to be shared across the industry. This is facilitated by the adoption of Open Source platforms, both hardware and software, for the emerging Internet of Things infrastructure.