23rd Annual IEEE International Symposium on Personal, Indoor and Mobile Radio Communications

Introducing the QEST broker: Scaling the IoT by bridging MQTT and REST

Matteo Collina∗ , Giovanni Emanuele Corazza† , Alessandro Vanelli-Coralli†

∗ ARCES (Advanced Research Centre for Electronic Systems)
University of Bologna, Viale Risorgimento 2, 40136 Bologna, Italy
Email: mcollina@arces.unibo.it
† DEIS (Department of Electronics, Computer Science, and Systems)
University of Bologna, Viale Risorgimento 2, 40136 Bologna, Italy
Email: {giovanni.corazza, alessandro.vanelli}@unibo.it

Abstract—In the ”Internet of Things” (IoT) vision the 1) Firstly, in a several-billions network, a standard,
physical world blends with virtual one, while machine-to- machine-to-machine protocol is needed, so smart ob-
machine interaction improve our daily life. Clearly, how these ject developers could focus on functionalities, and this
virtual objects are exposed to us is critical, so that their user
interface must be designed to support the easiness of usage that protocol must be tailored to the machines needs.
is driven by the users’ needs, which is different from what 2) Secondly, it should be simple for people to interact
machines requires. These two requirements must be solved, with machines, usually through their handheld device;
and an integrated solution should emerge, if we want to bring the latest years in the mobile industry have shown how
the IoT to the 50 billions network that is predicted to became the communication should be engineered to support
in the next years.
We believe that these requirements cannot be met by the the user experience, and not vice versa.
same communication protocol, and so we propose a new kind In general the requirements of machines and people are
of broker, named QEST that can bridge the two worlds, distinctly different, and it may be hard to define a protocol
represented by their state-of-the-art protocols: MQTT and and the associated best practices to statisfy both of them.
REST. In this paper, we demonstrate that our approach allows
rapid development of user-facing IoT systems, while grating We propose a solution that, through the use of widespread
machines all the performance they need. patterns in two different areas, can solve the dilemma.
Keywords-Internet of Things, Web of Things, REST, MQTT, B. A case for the Web of Things
smart objects, publish subscribe
The ”Web of Things” (WoT) initiative envisions a world
I. I NTRODUCTION where devices are exposed using the lingua franca of
The last 20 years of research in the pervasive computing the Internet technologies: the World Wide Web [3]. The
area have seen very important steps towards the realization WoT architecture builds upon the Representational State
of Mark Weiser’s vision of ubiquitous computing [1]: a Transfer (REST) principles [4]: this paradigm envisions
world were technology vanishes in the background. a world where digitally enhanced objects (smart things)
The advent of the World Wide Web revolutionized the are accessible using Uniform Resource Identifiers (URIs)
way we work, communicate, socialize, and how business that can be exchanged, referenced and bookmarked. Even
is done: the Internet has become one the most pervasive further, Guilard et al. [3] propose a way to enable tech-
technology. Recently, smartphones and mobile broadband savvy end-users to create physical mashups. This approach
enabled us to carry the Internet in our pocket, seamlessy enables application developers to leverage their existings
integrating it in our lives. However, everyday objects remain skills (HTML, JavaScript, PHP, Ruby, Python) to build new
disconnected from the virtual world, while the ”Internet of ways to interact with objects , somewhat serendipitously.
Things” (IoT) movement is exploring how to interconnect Semantic discovery services, such as DiscoWoT [5], for the
them. This technology shift is supposed to be greater than Web of Things have also been proposed: thus enabling both
the advent of mobile phones, and in [2] a 2020 scenario humans and machines to semantically discover, select and
where non-phone interconnected devices will be 10 times use smart things.
the phone devices (50 vs 5 billions) is foreseen. The WoT as presently conceived, is structured as a huge
Service Oriented Architecture (SOA) [6], where each device
A. Scalability issues offers some services to the others. While this approach
Billions of interconnected devices is a challenge for the fits perfectly a business driven use case, it lacks the sup-
whole Internet and for the objects themselves, and in order port needed by the autonomous interaction between smart
to operate at that scale several major issues should be devices. In [3], the authors noted that in most pervasive
resolved. We identified two main scalability issues regarding scenarios, humans and machines are interested in real-time
communication: data, and not in syndication. While the HTTP protocol,

978-1-4673-2569-1/12/$31.00 ©2012 IEEE 36

 HTTP Clients MQTT Clients
which is the underlining medium of the WoT, has not
been designed to support persistent communication, the
recent draft of HTML 5 specification proposes to intro-
duce Web Sockets: a bidirectional communication channel
between client and server. However, Web Sockets totally REST Server MQTT Server
hides the naming scheme that makes REST so powerful:
every resource has a standard unique identifier, the URI. The QEST
Web Sockets approach results in non-standard solutions for
manipulating resources, whereas, in order to clearly support
collaboration between devices, there is the need to unify the
Data Layer
naming scheme of smart objects and the URIs, combined
with the REST pattern, provide the state-of-art solution.

C. A case for MQTT

Today’s real world embedded devices usually lacks the Redis
ability to handle high-level protocols like HTTP and they
may be served better by lightweight binary protocols. In this Figure 1. QEST broker architecture
regard, Andy Stanford-Clark, and Arlen Nipper proposed in
1999 what now has became the MQTT protocol [7].
The MQTT protocol is fast, lightweight, power efficient how to achieve this two-way bridging between MQTT and
and implements various levels of Quality of Service. MQTT REST.
implements a classic publish/subscribe (pub/sub) pattern
with a central broker. The protocol revolves around the A. Exposing MQTT topics as REST resources
concept of topic, where clients might publish updates or MQTT topics do not retain state: however, in order to
subscribe for getting the updates from other clients. The expose them as a REST resource, we have to syndicate the
MQTT community claims that a pub/sub protocol is what is last payload seen on the topic as the resource value: thus,
needed to build a true IoT. our broker has state. After this consideration, exposing the
MQTT libraries have been provided for all major IoT de- topics is possibile by manipulating a single value. Following
velopment platforms, like Arduino, for several programming a pure REST style, we can create a topic just by issuing
languages (C, Java, PHP, Python, Ruby, Javascript) and for an HTTP POST request at /topics containing both the
the two major mobile platforms (iOS and Android). topic’s name and its original payload. For example, if the
The aim of our work is to bring toghether the REST- topic’s name is light_bulb we can:
oriented web architecture with the real-time properties of • read the last published value by doing an HTTP GET
the MQTT protocol, in order to bridge the gap between request at /topics/light_bulb;
machines and developers in the billionaire IoT. We demon- • publish a value in the topic by doing an HTTP PUT
strate that MQTT and REST might and should work together request at /topics/light_bulb.
by defining and implementing a new interaction protocol The HTTP protocol allows proper querying of the state
between devices and the Web. The implementation of this of a MQTT topic, because through the use of the HTTP
protocol is a broker that support the publish/subscribe pattern caching protocol, such as the headers Last-Modified,
through multiple communication protocols. We identify our Last-Modified-Since and ETag, clients can safely
broker as QEST, aiming at exposing a MQTT topic as a poll the broker about the state of the topic.
REST resource. This approach can guarantee only a best-effort level of
Quality of Service, because that is the true nature of the
II. I NTRODUCING THE QEST BROKER
HTTP protocol.
The QEST broker requires bridging two diverse ecosys-
tems, with a different semantic model of communcation. B. Exposing REST resources as MQTT topics
MQTT implements pub/sub, while the Hyper Text Transfer The MQTT protocol is stateless in regard of payloads, but
Protocol (HTTP) - which is the basis of the REST pattern - as stated in the previous paragraphs, we need to store the
is just a request/response protocol. QEST, by bridging these latest published value of a topic for syndication. Thus we
two approaches, realizes a new hybrid paradigm: firstly it need to improve the MQTT semantic in order to achieve a
modifies the broker semantic to retain and syndicate the last better correspondence with that of HTTP.
payload seen on a topic, secondly it exposes that payload as In MQTT, when a machine subscribes to a topic, it does
a REST resource. In the following paragraphs, we explain not know the value of the topic until a new payload is

37
published. Obviously this is very far from HTTP syndication HTTP/MQTT Clients
model, so the broker sends the last published value to the
newly connect client. This is an important change because Load Balancer
it deviates from the pub/sub nature of the MQTT protocol.

C. Real-time updates for the Browser

Web browsers might want to subscribe for topic updates
directly on the broker, thus getting near real-time updates. REST Server MQTT Server REST Server MQTT Server

In order to implement this feature using a REST pattern,

we propose to use the Long Polling approach [8], which
QEST
Data Layer
... QEST
Data Layer

allows clients to connect to a resource and wait until it

changes or a timeout is passed. As we are adhering to
the REST principles, we do not want to change the URI,
Redis
so the long polling can be triggered by adding a custom
”Long-Polling: enabled” HTTP header to the GET
request. Figure 2. QEST multiple node architecture
Instead of using Long Polling we could have used Web
Sockets [9] as the proposals of the WoT suggests [3].
WebSockets however do not implement the concept of URI As shown in figure 1, a single QEST node consists of two
after opening the communication channel, i.e messages in main components, the REST (web) server, and the MQTT
the stream are free form. The URI is the pillar of the REST server: both provide a representation of the data stored in
architecture, and we deem necessary to adhere to a pure Redis, which is accessed through the data abstraction layer.
REST approach in regard of the representation of MQTT QEST is a multiprocol broker with a common semantics,
topics. and its main responsibility is to notify subscribers on the
update of a particular topic, which is achieved as follows:
D. Real-time updates for system integration 1) the client publishes the update either through the
The QEST broker is not limited at the communication REST or the MQTT front-end;
with Web Browser and smart things, but it also supports 2) the front-end writes the new value on the data layer;
other back-end systems, such as logistics applications. These 3) the data layer stores the value and publishes an update
are built with procedural or object-oriented paradigms, and on a specific Redis key.
they usually require mantaining a connection using threads, In order to receive updates on a topic, the client acts as
or forking processes. It is often quite hard to integrate real- follows:
time communication in these legacy applications, as tradi-
1) the client subscribes for updates to either the REST
tional back-end technologies are ill suited for this purpose.
or the MQTT front-end, according to its nature;
In the previous section we have shown how Web browsers
2) the front-end registers for changes to the data layer;
can subscribe to changes using a standard HTTP technique.
3) the data layer subscribes to changes to a particular
However, it is hard to implement Long Polling in legacy
Redis key;
systems, as maintaining an open connection, as Long Polling
4) whenever is published a value on the Redis key, it is
dictates, may be impossible if the technology does not
forwarded to the data layer;
support it. In order to scale the IoT we need simplify the
5) the data layer notifies the subscribed clients through
integration with these systems, enabling a communication
the front-ends.
between them and the QEST broker.
In this regard we propose to use webhooks [10], which are
A. Scalability
a pub/sub implementation using the HTTP medium. Using
simple POST requests, the client can subscribe to updates on In order to support the interaction between billions of
the broker by specifying the URI that will be used to send devices, the broker must be able to scale horizontally,
the updates. The broker assures to call all the subscribed i.e. the single broker must be replicated mantaining the same
URIs when there is an update to the topic. functionalities. The QEST architecture allows to extend the
single node architecture to multiple nodes, as the single node
III. QEST S YSTEM ARCHITECTURE is totally stateless and it can be replicated at will (fig. 2).
QEST revolves around the fast key-value store Redis [11], The multiple node architecture introduces a load balancer,
which acts as the internal broker of the application. Redis which is necessary to divide the load between all nodes and
has various features, but the most important for our case is it can be implemented using several software components,
an implementation of the publish/subscribe pattern. such as HAProxy [12].

38
 state-of-art QEST-based
approach to IoT apps solution to IoT apps In the first scenario, we crafted a set of Arduino [15]
devices that switches on and off a led based on the state of
a common topic, which is modified by pressing a button on
Web App Web App
one of the devices. This allows us to trigger on and off all
the leds in our ”forest” by pushing the button of one of the
devices, or just from our web interface.
In the second scenario, we built a do-it-yourself temper-
Bridge

QEST ature monitoring for the house, and exposed the value on
a web page. Using this information, we were able to track
IoT and graph the temperature of the house.
Device
3 2 1 0 9 8 7 6 5 4 3 2 1 0

GND
SCL

AREF
SDA
1 1 1 1 DIGITAL

RX
TX
PWM
PWM
PWM

PWM
PWM

PWM
L

TX
Arduino UNO ON

Broker
RX

1
ICSP

RESET
IOREF

3V3
www.arduino.cc

POWER
5V Gnd Vin
ANALOG IN
0 1 2 3 4 5

V. E XPERIMENTAL RESULTS
The main point of evaluation of our solution is the easi-
ness of usage for software developers. Also, in order to show
Device that QEST does not penalize the performance1 we compared
3 2 1 0 9 8 7 6 5 4 3 2 1 0
GND
SCL

AREF
SDA

1 1 1 1 DIGITAL
RX
TX
PWM
PWM
PWM

PWM
PWM

PWM

TX
RX
Arduino UNO ON
1

ICSP

www.arduino.cc
RESET
IOREF

POWER ANALOG IN
3V3

5V Gnd Vin 0 1 2 3 4 5

our solution to two implementations of MQTT: the Really

Figure 3. Example architectures for IoT applications, implemented with Small Message Broker (RSMB) [16], which is a proprietary
the current state-of-art (on the left), or with QEST (on the right) implementation made by IBM, and Mosquitto [17], which
is open source.
We compared the performance of raw publishing and
B. Enabling technologies receiving a notification on multiple clients, as we believe
The principal technology requirement for the QEST bro- it is the most challenging for a MQTT broker and hence
ker is the ability to handle thousands of concurrent connec- for QEST. As shown in figure 4, QEST performance is
tions with a low overhead. Traditional server technologies comparable with that of Mosquitto in the considered interval
use a fork or threading approach, where a process or a (1-1000 subscribers). The RMSB implementation performs
thread serve only one request at a time. Clearly this approach better up to 100 subscribers, while it does not support a
cannot support thousands of concurrent connections larger number of users. Finally, the two nodes QEST setup
A solution for handling this case is to use evented in- is faster than the single one, as expected.
put/output capabilities, where there is a single process/thread We also benchmarked the QEST broker HTTP perfor-
that handle each connection when it needs to receive or send mance on publishing to a topic with a growing number of
data on it. This approach allows a lower memory footprint subscribers (fig. 5).
and low CPU usage; thus is scalable. Results demonstrate that publishing a value on the QEST
In particular, the QEST broker is built on the Node.js [13] broker through HTTP is equally performant to MQTT.
platform, which is based upon the Chrome JavaScript virtual
VI. S ECURITY ISSUES
machine. Node.js implements the reactor pattern: a particular
evented I/O system where every computation is executed In the future, the need to secure communication between
in response to an event, and this approach allows to build things and humans, with a billion devices IOT, is one
highly concurrent network applications. In order to support of the biggest challenges we will have to cope with, to
the MQTT protocol, we used the MQTT.js implementa- avoid a massive privacy issue. In this regard, the European
tion [14]. Union is investigating how to regulate the future Internet
of Things [18]. The state-of-art in M2M protocols does not
IV. U SE C ASES address the security and privacy needs of our society [19],
and MQTT makes no exception.
The primary objective of QEST is simplifying the de-
In order to protect a resource, a system should adhere to
velopment of IoT applications, by allowing the developers
two different requirements:
to integrate real world devices easily in web and mobile
• authentication, i.e. allowing only known user to connect
applications. In figure 3, we compare the state-of-art ap-
proach and our QEST proposal. In the first one, we have to the system;
• authorization, i.e. denying the users the right to access
to implement a custom bridge between the web application
and the broker, which limits the diffusion of the interaction the resource they are forbidden to access.
between the real and virtual worlds. On the contrary, QEST The MQTT protocol was not designed with security in
simplify the solution by concentrating two functions on the mind, and we highlight problems in both areas. Regarding
same component. In order to demonstrate this approach, we 1 All benchmarks where done with a PC running Ubuntu Linux and
developed two main use cases: ”forest of leds” and ”temp equipped with an Intel i5-2400 processor (6MB Cache, 3.10 GHz) and
monitoring”. 4GB of RAM.

39
 applications, must be designed and developed. Developing
such a solution is out of the scope of our current work, but
we believe a mixed MQTT/REST solution might allow the
user to pair their devices with their data.

VII. C ONCLUSIONS AND F UTURE W ORK

In a billion-devices Internet of Things, there is the need
of a common medium to support the interaction between
machines, and to allow new form of applications between
smart objects and end users. We discussed the requirements
for the two types of interactions, and we concluded that the
state-of-art of communication protocols does not support all
of them.
In order to solve the dilemma, we introduced a new
broker, QEST, that bridge the gap between the things and
the web, allowing existing developers to use their skills
to interact with smart objects, thus fostering innovation.
Figure 4. Comparative benchmark on MQTT performance between
We demonstrated that QEST performance are comparable
Mosquitto, RSMB and QEST to other MQTT implementation, while providing a much
more efficient programming interface between things and
developers.
As for future developments, we plan to expand the QEST
broker to fully support the MQTT specification, e.g. it
only implements best-effort quality of service and, it lacks
some peer-to-peer capabilities which are present in other
implementations.
Finally, it is possible to extend the QEST broker to support
sensor networks, i.e. devices that are too much energy
efficient to support the permanent TCP connection needed
by TCP, which is the basis of the MQTT protocol. Thanks
to the multi-protocol architecture of our broker we plan to
integrate the MQTT-S protocol, which has been designed to
support such devices.

R EFERENCES

[1] M. Weiser, “The computer for the 21st century,” Scientific

Figure 5. QEST benchmark on publishing a value on a topic from HTTP, American, Feb. 1991. [Online]. Available: http://www.ubiq.
and listening to it from a growing number of MQTT subscribers. com/hypertext/weiser/SciAmDraft3.html

[2] “More than 50 billion connected devices,” Ericsson, February

the first one, even though it support username and pass- 2011. [Online]. Available: http://www.ericsson.com/res/docs/
word based authentication, it transmits credentials without whitepapers/wp-50-billions.pdf
encryption.
[3] D. Guinard, V. Trifa, and E. Wilde, “A resource oriented
In [19], the authors cite Virtual Private Networks (VPN) architecture for the web of things,” in Proc. Internet of Things
as a mean to secure IoT deployments, and we propose it (IOT), Tokyo, Japan, December 2010, pp. 1–8.
too.
As for authorization, all the main MQTT implementations [4] R. T. Fielding and R. N. Taylor, “Principled design of the
support Access Control Lists (ACL) for limiting users to modern web architecture,” ACM Trans. Internet Technol., pp.
access resources, but they cannot be updated in real-time, 115–150, May 2002.
and from the protocol itself. If the IoT system is going
[5] S. Mayer and D. Guinard, “An extensible discovery service
to became ubiquitous and consumer-friendly, then an au- for smart things,” in Proceedings of the 2nd International
thorization protocol similar to OAuth [20], which is used Workshop on the Web of Things (WoT 2011). San Francisco,
on the web to grant access to personal data by third-party USA: ACM, Jun. 2011.

40
 [6] D. Guinard, V. Trifa, S. Karnouskos, P. Spiess, and D. Savio, [13] “Node.js,” Joyent, Inc, April 2012. [Online]. Available:
“Interacting with the soa-based internet of things: Discov- http://nodejs.org
ery, query, selection, and on-demand provisioning of web
services,” IEEE Transactions on Services Computing, vol. 3, [14] A. Rudd, “MQTT.js,” April 2012. [Online]. Available:
no. 3, pp. 223–235, 2010. http://bit.ly/hO9cZP

[7] “MQ Telemetry Transport,” April 2012. [Online]. Available: [15] M. Banzi, D. Cuartielles, T. Igoe, G. Martino, and D. Mellis,
http://mqtt.org “Arduino,” April 2012. [Online]. Available: http://arduino.cc/

[8] S. Loreto, P. Saint-Andre, S. Salsano, and G. Wilkins, [16] “Really Small Message Broker,” April 2012. [Online].
“Known issues and best practices for the use of long polling Available: http://ibm.co/GQ7vwr
and streaming in bidirectional http. internet engineering task
forceprincipled design of the modern web architecture,” RFC [17] “Mosquitto,” April 2012. [Online]. Available: http:
6202, April 2011. //mosquitto.org

[9] “The WebSocket API,” World Wide Web Consortium [18] “EU investigating IoT regulations,” April 2012. [Online].
(W3C), April 2012. [Online]. Available: http://dev.w3.org/ Available: http://bit.ly/HyYSb2
html5/websockets/
[19] D. Giusto, A. Iera, G. Morabito, L. Atzori, C. M. Medaglia,
[10] “Webhooks,” April 2012. [Online]. Available: http://www. and A. Serbanati, “An overview of privacy and security issues
webhooks.org in the internet of things,” in The Internet of Things. Springer
New York, 2010, pp. 389–395.
[11] S. Sanfilippo, “Redis,” April 2012. [Online]. Available:
http://redis.io [20] “The OAuth 2.0 Authorization Protocol,” April
2012. [Online]. Available: http://datatracker.ietf.org/doc/
[12] “HAProxy,” April 2012. [Online]. Available: http://haproxy. draft-ietf-oauth-v2
1wt.eu

41