An Elasticity Framework for Distributed Message Queuing Telemetry Transport Brokers

Nội dung text: An Elasticity Framework for Distributed Message Queuing Telemetry Transport Brokers

1. VNU Journal of Science: Comp. Science & Com. Eng, Vol. 37, No. 1 (2021) 1-14 Original Article An Elasticity Framework for Distributed Message Queuing Telemetry Transport Brokers Linh Manh Pham*, Xuan-Tung Hoang VNU University of Engineering and Technology, 144 Xuan Thuy, Cau Giay, Hanoi, Vietnam Received 23 October 2020 Revised 25 February 2021; Accepted 27 February 2021 Abstract: Internet of Things (IoT) applications are increasingly making impact in all areas of human life. Day by day, its chatty embedded devices have been generating tons of data requiring effective network infrastructure. To deliver millions of IoT messages back and forth with as few faults as possible, participation of communication protocols like Message Queuing Telemetry Transport (i.e., MQTT) is a must. Lightweight blueprint and battery friendly design are just two of many advantages of this protocol making it become a dominant in IoT world. In real application scenarios, distributed MQTT solutions are usually required since centralized MQTT approach is incapable of dealing with huge amount of data. Although distributed MQTT solutions are scalable, they do not adapt to fluctuations of traffic workload. This might cost IoT service providers because of redundant computation resources. This leads to the need of a novel approach that can adapt its volume changes in workload. This article proposes such an elastic solution by proposing a flexible MQTT framework. Our MQTT framework uses off-the-shelf components to obtain server’s elasticity while keeping IoT applications intact. Experiments are conducted to validate elasticity function provided by an implementation of our framework. Keywords: MQTT broker, Elasticity, Internet of Things, Cloud computing. 1. Introduction * throughout the Internet either with or without human interventions. At present, 31 billion It is a fact that Internet of Things (IoT) has devices are connected as IoT devices and it is been widely spreading in many domains with predicted that by 2050 this number will surge different scales from homes, enterprises, pass 170 billion limit [1]. Also, an IoT network institutions to industries. Behind IoT can hold up 50 to 100 trillion connected objects, application scenarios are millions of connected and this network can track the movement of devices trying to communicate and deliver data every single of objects. Each people living in ___ urban areas can be surrounded by 1000 to 5000 * Corresponding author. tracking IoT things. In the same context, Email address: linhmp@vnu.edu.vn currently, there are about 4 billion people connected, more than 25 million applications, 1
2. 2 L.M. Pham, X.T. Hoang / VNU Journal of Science: Comp. Science & Com. Eng., Vol. 37, No. 1 (2021) 1-14 more than 25 billion embedded and intelligent Elasticity is a native characteristic of Cloud systems, which generate 50 trillion gigabytes of computing according to NIST [4]. Thanks to data. The IoT market can bring up to 4 trillion elasticity, cloud resources are not overused or USD in revenue for its service providers [2]. underused, which not only saves cloud To support the communication of billions of providers’ money but also improves customer IoT devices and delivery of its huge generated experience. Many IoT applications have been data, the IoT service providers need to implement being implemented on Cloud or ready to be and maintain robust and scalable network moved on it. It is worth mentioning that IoT infrastructures. Especially, when IoT applications resources like MQTT brokers implemented on cross the boundary of home-wide to reach the Cloud can also benefit from cloud elasticity. skyline of city- or country-wide systems, the In this article, we propose a framework number of IoT devices can increase extremely at making MQTT brokers elastic. To obtain this an unpredictable rate. That is a point for goal, the following contributions are made: development of not only brawny but also scalable i) We propose a novel framework that can IoT infrastructures. Such modern IoT infrastructures flexibly support elasticity while retain all nowadays contain an essential component called features of MQTT protocol; Message Queuing Telemetry Transport (i.e., MQTT) ii) We make a concrete implementation of servers (a.k.a. brokers). These brokers are the proposed framework using an open-source implemented on MQTT protocol devised since MQTT broker software (EMQX) and a private 1999, which is an open Machine-to-Machine cloud platform (OpenStack); protocol (M2M) originally. It is also an openly iii) We conduct experiments to validate the industrial standard released by OASIS and ISO soundness of the proposed approach using the (ISO/IEC 20922) [3]. With its advantages such as open-source implementation aforementioned. lightweight blueprint, bandwidth-efficient design, The rest of the article is organized as low power consumption, or spatial/temporal follows. After highlighting various related work decoupling, MQTT brokers are dominating the in Section 2, we describe in detail the overall architecture of our proposed MQTT framework IoT world indisputably. Although recent MQTT brokers implement in Section 3. The validating experiments and both centralized and distributed approaches to results are reported in Section 4. Finally, we be able to handle millions of connected clients conclude in Section 5. in a short period of time, only a few one proposed solutions to keep up with fluctuation 2. Related Work of workload generated by IoT clients. This 2.1. Message Queuing Telemetry Transport Brokers might happen when number of IoT devices increase or decrease unpredictably during MQTT is a Message Oriented Middleware certain times. In reality, the city-wide IoT (MOM), which follows publish/subscribe applications often deal with disperse and pattern (pub/sub for short) [5]. With lightweight intermittent devices causing a change in the design in mind, MQTT is suitable for many IoT number of involving clients. For instance, smart applications where various restricting cars often join given connected vehicle networks requirements must be satisfied such as low in the rush hours rather than regular hours, bandwidth, low energy, or intermittent sensor generating more IoT data within specific periods. nodes. The pub/sub model brings to the This leads to the need for development of new decoupling in both space and time, where MQTT systems that know not only how to scale MQTT clients (i.e. publishers and subscribers) but also keep pace with changes of the workload do not have knowledge about locations of each generated by the clients. In other words, an other and do not need to be present at the same approach makes MQTT brokers elastic. time while sending or receiving the messages,
3. L.M. Pham, X.T. Hoang / VNU Journal of Science: Comp. Science & Com. Eng., Vol. 37, No. 1 (2021) 1-14 3 respectively. This is possible because the In recent decade, many IoT applications protocol uses an intermediate service called have implemented MQTT brokers such as MQTT brokers to mediate messages among [6-11]. A typical structure of an IoT application participants (publishers and subscribers). While using MQTT brokers deployed with centralized space decoupling characteristic helps IoT datacenter remotely (like in the Cloud application separate its high volume of environment) is depicted in Figure 1. The goal available data from the origin of data, time of the application is to collect data from many decoupling one is a necessity for IoT IoT devices and sensors, then process and store applications because of its distributed nature. these data, and at last send notifications and The communication mechanism of MQTT is reports to the final users (using laptop, mobile, relatively simple. First, the MQTT clients need to tablet, etc.). In some cases, the gathered data establish a connection to MQTT broker by can be published directly to the topics sending a CONNECT message. A CONNACK subscribed by final users without any data message from the broker sent back to the client is analysis. Control commands can be published to confirm for a successful connection. After that, to the command topic in the broker like any the operations of publishing/subscribing messages other type of MQTT messages by the final to/from the broker can be done. The publisher users. These messages will be archived in the needs to send a PUBLISH message containing a cloud storage and transmitted to the IoT devices topic name. A topic (i.e. subject of interest) is a or sensors by some scheduling mechanism. In string used by broker, where subscribers register the case of time-sensitive application, the to it for getting copies of needed messages. To do command messages can also be sent directly to that, the subscriber must send a SUBSCRIBE the IoT devices without travelling to the Cloud. message containing its interesting topic to the We see that IoT devices, end-user interfaces, and data analyzing system are all MQTT clients broker. Topics can be organized in a hierarchical producing and consuming telemetry data. way (i.e. topic trie) to take advantage of wildcard Many IoT applications often implement a filters such as “#” or “?”. In general, the clients centralized MQTT broker keeping all can publish/subscribe to more than one topic not subscribed topics. However, the broker in this using or using these wildcards for convenient. topology is easy to become a bottle neck of the In MQTT, the communication reliability entire system. To avoid this, some solutions can be obtained by specifying the levels of have been proposed, which can be categorized Quality of Service (QoS). There are three levels into two types of distributed system: bridged of QoS including “At most one” (0), “At least brokers and clustered brokers. In the first one” (1), and “Exactly one” (2). At level 0, the model, two brokers can be bridged to be able to delivery is not acknowledged and the message serve more messages from clients while is sent only once in any case. At level 1, if no keeping separation of both broker’s locations. acknowledgement is received by the publisher, Published messages are forwarded from a it will try to resend the message multiple times. broker to its bridged one according to specific At level 2, exactly one copy of the message is access policy. A full-mesh network needs to be received by the subscriber by a two-way formed among brokers (i.e. each one pairs with handshake agreement between the publisher all others) in order that any MQTT client can and subscriber. To guarantee no data is lost, IoT connect to any broker it wishes to. Therefore, using bridged model to obtain elasticity is too applications obviously need a lightweight and complex. It is only suitable for networks having vigorous solution like the MQTT broker model. a few MQTT brokers. MQTT brokers support Some of the most widely used MQTT brokers the bridging method including HiveMQ, so far are Mosquitto, HiveMQ, moquette, EMQX, JoramMQ, moquette, mosquitto, VerneMQ, EMQX, etc. VerneMQ, etc. [12]. Some implementations of
4. 4 L.M. Pham, X.T. Hoang / VNU Journal of Science: Comp. Science & Com. Eng., Vol. 37, No. 1 (2021) 1-14 this model are reported in research of Collina et reduced significantly in comparison to the al. [13], Schmitt et al. [14], and Zambrano et al. bridged model. Moreover, the knowledge of [15]. MQTT brokers following the clustered topic trie and route table are transferred among model take advantage of subtopics in the brokers, thus any MQTT client can hierarchical trie. One of the brokers (B0) keeps connect/reconnect to any broker it wants to root topic and subtopics which its subscriptions establish/resume its sessions. Not many MQTT involve. Other ones (B1, B2, etc.) only keep brokers support full features of the clustered their involved subtopics originating from the topology including EMQX, HiveMQ, RabbitMQ, root topic located at B0. The topic branches are VerneMQ [16]. Some research projects in this dynamically created in a broker corresponding trend that can be mentioned are the work of to MQTT subscriptions to this broker. Jutadhamakorn et al. [17], Thean et al. [18] and Therefore, the overhead among the brokers is Detti et al. [19]. Figure 1. A typical structure of an IoT application using MQTT brokers. 2.2. Elastic Message Queuing Telemetry Very few elastic solutions mentioning Transport Broker MQTT broker among various types of pub/sub servers have been proposed such as Brokel [24] Elasticity defined as one of essential and E-SilboPS [25]. Brokel defines a characteristic of Cloud Computing. Thanks to multi-level elasticity model for Pub/Sub brokers this special feature, cloud resources can be (including MQTT) in general, thus many provisioned or released corresponding to MQTT-specific tweaks have been simplified or demand. Nowadays, IoT applications are often omitted. E-SilboP is an elastic content-based installed in Cloud to take advantage of this publish/subscribe system specifically designed environment like on-demand measured service, to support context-aware sensing and broad network access as well as rapid elasticity. communication in IoT-based services. Therefore, it also omits many adjustable QoS Multiple solutions try to provide elasticity for parameters of MQTT protocol and only other components of IoT applications rather provides content-based elasticity. It is worth than MQTT broker, some of which are Proliot mentioning that these both research works are [20], DOCKERANALYZER [21], ACD [22], solutions for pub/sub servers in general, which BDAaaS [23]. do not focus only on MQTT broker. One of the
5. L.M. Pham, X.T. Hoang / VNU Journal of Science: Comp. Science & Com. Eng., Vol. 37, No. 1 (2021) 1-14 5 prerequisite to holistically obtain elastic MQTT [27], we prefer combining the most pertinent is that the solution must implement one of the open-source solutions into our framework. The distributed topologies mentioned in the overall novel architecture of the framework is subsection 2.1. depicted in Figure 2 composing of the following modules: 3. Elastic Message Queuing Telemetry i) MQTT broker cluster: A cluster of Transport Framework MQTT brokers implementing distributed This section introduces our new elastic pub/sub model with customizable QoS MQTT framework. The framework is designed parameters. The cluster consists of a number of to have flexible architecture containing a set of runtime systems called node. Nodes connect to representative modules. When an each other using TCP/IP sockets and implementation of the framework is deployed communicate by message passing. Each node on the Cloud, each of these representative ones keeps its parts of topic tries and current will be specialized into a concrete component- subscriptions. This mechanism helps published off-the-shelf (COTS) one. Therefore, the messages be routed across the cluster from the modules of framework can be substituted first node receiving the messages to the last one flexibly to obtain new features, to earn delivering the messages to the subscribers. The enhanced performance, or to lower software nodes can join cluster manually or licensing fees. We also present a concrete automatically. With automatic way, node implementation of each of the modules discovery and autocluster mechanisms such as constituting the framework. The IP multicast, dynamic DNS, or ETCD [26] need implementation mainly targets for cloud-based to be supported. The nodes can be deployed on IoT applications which require elasticity as an both public or private cloud networks. Public essential feature. These applications include, Cloud providers, such as AWS, Azure, or but not limited to, big data analytics, latency- private Cloud platforms, such as OpenStack, sensitive ones. With the principle of software CloudStack could be the good candidates. development serving the e-science community P Figure 2. Architecture of Elastic MQTT Framework. We choose EMQX [28] for our MQTT only one implementing all three levels of brokers. EMQX provides concurrent, fault- MQTT QoS, MQTT protocol for regular tolerant, and distributed broker nodes. It is one networks, and MQTT-SN protocol for sensor of few open-source MQTT solutions which ones. EMQX supports node discovery and offer clustered brokers. Moreover, EMQX is the autocluster with various strategies as in the case
6. 6 L.M. Pham, X.T. Hoang / VNU Journal of Science: Comp. Science & Com. Eng., Vol. 37, No. 1 (2021) 1-14 of IP multicast, dynamic DNS, etcd, and entire life cycle of all involving components. Kubernetes [29]. By that when a broker node Those components include resources such as arrives or leaves according to elastic actions, virtual machines, containers, images, security the cluster automatically recognizes the changes groups, alarms, scaling policies, etc. The and updates its configuration to reflect new grammar of the DSL can be derived from XML, number of nodes; JSON, or YAML. The main motivation is to ii) Load balancer: A Load Balancer (LB) is keep the thing simple and user-friendly. In the often deployed in front of a MQTT cluster to framework, the Orchestrator deploys and distribute MQTT connections and traffic from manages MQTT brokers as well as resources of devices across the MQTT clusters. LB also the elastic decision-making block such as enhances the high availability of the clusters, Metering, Metric Storage, and Alarming. balances the loads among the cluster nodes, and The main orchestrator supported by makes the dynamic expansion possible. The OpenStack is Heat service [32]. The links between the LB and cluster nodes are infrastructure for a cloud application is plain TCP connections. By this setup, a single described in a Heat template file. Infrastructure MQTT cluster could serve millions of clients. resources that can be described including Thanks to LB, MQTT clients only need to servers, volumes, users, security groups, know one point of connection instead of floating IPs, etc. Heat also provides an maintaining a list of MQTT brokers; autoscaling service integrating with Some commercial LB solutions are sub-modules of Telemetry, so a scaling group supported by EMQX such as AWS, Aliyun, or can be included as a resource in the template. QingCloud. In the terms of open-source This is a perfect fit for our elasticity goal. software, HAProxy [30] can serve as a LB for Templates can also delineate the dependencies EMQX cluster and establish/terminate the TCP between resources (e.g., this floating IP is connections. Many dynamic scheduling assigned to this VM). This helps Heat to create algorithms can be assigned by HAProxy such as all of managed components in the correct order round robin, least connection, or randomness; for completely launching application. Heat iii) Cloud infrastructure: It manages, manages the entire life cycle of the application provides, and releases dynamically virtual and it knows how to make the necessarily resources for gaining elasticity. To obtain dynamic changes. Finally, it also takes care of “unlimited” resources, an implementation of deletion of all the deployed resources when the private, public, or hybrid cloud may need to be application accomplishes; carried out. To serve e-science community, v) Telemetry: This module consists of three OpenStack [31], an open-source private Cloud sub-modules is chosen to provision and release virtual Metering: This module’s goal is to resources. With world-wide supported user efficiently collect, normalize, and transform community and large well-maintained services, data produced by orchestrated components. OpenStack is a fit for our goal. Some specific These data are intended to be used to create OpenStack services deployed for our different views and help solve various telemetry implementation are Nova, Keystone, Glance, use cases. Among them, data of specific metrics Horizon, Swift, and Neutron. Since we chose (i.e., measures) is collected and analyzed for OpenStack cloud, the following modules should elasticity triggering goal. Alarming and Metric deploy services supported officially by storage are two modules which directly exploit OpenStack; these measures. iv) Orchestrator: This module parses a Alarming: Its goal is to enable the ability of system-component description in its own triggering responsive actions based on defined high-level domain specific language (i.e. DSL) rules against sample or event data collected by and then deploys, manages, and monitors the Metering module. It consists of two main
7. L.M. Pham, X.T. Hoang / VNU Journal of Science: Comp. Science & Com. Eng., Vol. 37, No. 1 (2021) 1-14 7 sub-modules: “Alarm Evaluator” and “Alarm resolve dependencies between components and Notifier”. The former evaluates measures of a from there come up with a deployment plan given metric stored in Metric storage module containing the appropriate order of installment. whether they are over or under a threshold. The From the system description to the deployment latter then triggers a notification and sends to plan, the Orchestrator needs to use a chain of the Orchestrator who will perform solvers such as Learning Automata based corresponding elastic actions such as scaling allocator, Constraint Programming based out or in. solvers, Heuristics based solvers, and Metric storage: This database mainly stores Meta-Solvers. When an event or combination of aggregated measures of cluster nodes such as events and conditions occur at runtime, the system performance metrics. The metric is a list Orchestrator generates the corresponding of (timestamp, value) for a given managed elasticity plan and conducts the necessary resource. The resource can be anything from the modifications to convert the current topology to temperature of the nodes to the CPU usage of a the expected one described in the elasticity VM. Besides, the database also stores events, plan. The modifications include actions that is a list of things that happens in Cloud following ECA (event-condition-action) rule infrastructure: an API request has been such as resources’ scaling in/out or up/down received, a VM has been started, an image has when measures of a resource trespass the given been uploaded, whatever. Stored measures are thresholds. Figure 3 depicts one possible retrieved for monitoring, billing, or alarming, implementation of our framework using the where events are useful to do audit, aforementioned open-source solutions. performance analysis, debugging, etc. Correspondingly, OpenStack supports some official services for Telemetry module, 4. Validating Experiments including Ceilometer [33] for Metering, Aodh In order to validate functionalities of our [34] for Alarm, and Gnocchi [35] for Metric proposed framework, we conducted the Storage; implementation mentioned in Section 3 in our vi) Messaging server: It is needed for homegrown infrastructure at VNU University communication between framework’s modules of Engineering and Technology (VNU-UET). by exchanging messages. It creates connected We also make some discussions after the results channels using favoured communicating protocols of the experiments. such as AMQP, CoAP, or even MQTT. In OpenStack cloud, internal communication among 4.1. Experiment Testbed OpenStack services may be conducted by The testbed consists of two main parts: one RabbitMQ [36]. RabbitMQ is an open-source implementation of our proposed framework, message-oriented middleware supporting and one load injector for simulating multiple popularly communicating protocols such as MQTT clients and their workloads. The tester AMQP, STOMP, and MQTT. creates test plans with different scenarios using All modules of the framework are built-in functions of the load injector. The decoupling. It means the startup order is not simulated publishers generate MQTT messages quite important. In spite of that, it makes no and send them to the Load Balancer sense for some modules to work independently, (HAProxy). In turn, the LB distributes these thus requires the power-up of other ones as messages to one EMQX broker of the cluster prerequisites. Similarly, components and according to scheduling algorithm. The resources described and managed by the simulated subscribers make MQTT Orchestrator should be initiated at any given subscriptions to specific topics in the cluster by moment. The Orchestrator must have ability to connecting to the LB, too. The LB also
8. 8 L.M. Pham, X.T. Hoang / VNU Journal of Science: Comp. Science & Com. Eng., Vol. 37, No. 1 (2021) 1-14 distributes connecting requests of the 8 vCPU and 8 GB memory. We use OpenStack subscribers to one of the members of the Train, which was released on 2019. Our cluster. The routing messages from the source OpenStack cloud is built on 3 physical servers to the right destinations is conducted internally using Intel processors. Each physical server has by the cluster as mentioned in Section 3. 80 CPUs at 2.4 GHz, 256 GB memory and a We used Apache JMeter [37] as the load storage pool of 1.5 TB. CentOS 7 are installed injector in our experiments. It is an open-source on all physical machines as host operating tools for load test and performance evaluation. system. On top of that, KVM is used as a base It supports testing of many different protocol virtualization solution. For better resource types such as HTTP, HTTPS, SOAP, REST, isolation, instead of running OpenStack FTP, JMS, etc. Other protocols are included controller services (e.g. NovaAPI, into JMeter using plugins. To support the NeutronServers, Keystone, etc.) directly on experimentation, we have developed a MQTT physical servers, we install those components plugin for JMeter implementing some features on dedicated virtual server instances. Only of MQTT version 5.0 [38]. To do stress test, we hypervisor service (a.k.a. OpenStack used distributed testing paradigm with one NovaCompute) that takes care of running JMeter master and a couple of slaves to ensure virtual instances, will be installed directly on that there is no side-effect to the performance of physical servers. By this way, system services simulated MQTT clients. of OpenStack cloud itself will be completely Our private OpenStack cloud is installed in separated from virtual server instances created the data center for research at VNU-UET. by users of the cloud. In short, stress tests EMQX brokers and JMeter load injectors are conducted by our experiments, which are virtual machines provisioned by the cloud. Each running on OpenStack’s virtual servers will not EMQX broker instance has 2 vCPU and 2 GB affect performance of the cloud and vice versa. memory, and each JMeter virtual machines has I Figure 3. An implementation of Elastic MQTT Framework. 4.2. Experiment Scenarios This scenario simulates a huge number of IoT devices, for example smart plugs, publishing We conducted the experiments with two telemetry data to a central smart-home system. common scenarios usually found in IoT The devices are the publishers and the central applications using MQTT: Multi-publisher and smart-home system is the subscriber. Devices Multi-subscriber. Each scenario evaluates the are structured as a three-level tree. The top level effectiveness of elasticity with two models: represents the smart houses in a district. The centralized broker and clustered brokers. middle level is called households in a smart 4.2.1. Multi-publisher scenario house. The access point in each household
9. L.M. Pham, X.T. Hoang / VNU Journal of Science: Comp. Science & Com. Eng., Vol. 37, No. 1 (2021) 1-14 9 enables the smart plugs to reach Internet and multi-publisher scenario. We also define topic publish data to the smart-home center. The partitions representing 3600 smart plugs bottom level is the smart plugs who send to the providing a control interface, which each access point measures of devices plugged into partition is composed of: them. The three-level tree is mapped to MQTT i) 1 root topic “SmartHouse”; topics. A topic level is added below the device ii) 3 topics “Household” each smart house; to represent the telemetry parameters, for iii) 30 topics “SmartPlug” each household; instance the power consumption (kWh). The iv) 1 topic “Command” each smart plug. test scenario defines 40 topic partitions The testbed for this scenario is depicted in made of: Figure 5. i) 1 root topic “SmartHouse”; ii) 3 topics “Household” each smart house; iii) 30 topics “SmartPlug” each household 10 topics “Parameter” each smart plug. Therefore, a topic partition represents 90 smart plugs and each one publishes 10 telemetry parameters. We have 3600 smart plugs totally in the scenario. The testbed for this scenario is depicted in Figure 4. Figure 5. Multi-Subscriber Clustered Brokers Scenario. 4.3. Results The MQTT workloads are prepared using JMeter test plan. The workload starts with a short warm-up period and then dramatically increases when MQTT clients joins steadily to the simulation. EMQX servers are preconfigured following suggestions from EMQX documentation1. We chose IP multicast Figure 4. Multi-Publisher Clustered Brokers Scenario. method for the node-discovery and autocluster mechanisms. The scheduling strategy for 4.2.2. Multi-subscriber scenario This scenario also simulates a huge number HAProxy was set to roundrobin. of smart plugs, controlled by a central smart- The Ceilometer, Aodh, and Gnocchi were home system. The smart plugs are the configured to measure and store measurements subscribers and the central system is the of average %CPU utilization and number of publisher. Smart plugs provide two-way virtual CPUs (vCPU) metrics. Upper and lower communications. The final users and smart- thresholds for average %CPU utilization are set home center can send commands to the plugs. to 80% and 25% respectively. It means that if Besides, the smart-plugs can also respond to average %CPU utilization breaks these these commands, for example an indicator of an thresholds and the events caught by Ceilometer ON/OFF update. Topics are partitioned in a ___ couple of levels in the same way as in the 1
10. 10 L.M. Pham, X.T. Hoang / VNU Journal of Science: Comp. Science & Com. Eng., Vol. 37, No. 1 (2021) 1-14 and Aodh, a notification is sent to Heat for Adding one more broker (4vCPU totally) to conducting a corresponding elasticity action form the cluster can help to resolve the such as scaling in or out. Actually, Heat has to problem. In the centralized case, we see ask other OpenStack services such as Nova, obviously in Figure 7a that average %CPU Keystone, or Glance to get the elasticity actions utilization of the broker gets saturation after a done synchronously. Elasticity plan configured couple of minutes (at the 1st minute). At this in Heat ensures the number of VMs always in point, dropped message rate starts to increase. range of 1 to 3. With elasticity, operating cost reduces since We used two JMeter client machines for we do not have to always maintain multiple distributed tests. In each client machine, brokers (clustered brokers). In Figure 8a, we maximum of 5 JVM processes are allowed to see an elasticity effort to mitigate the pressure initiate. According to the test scenarios, each performed by our system. One virtual machine process is responsible for running 3600 MQTT of MQTT broker B1 is created to share the clients. Therefore, maximum 36000 MQTT workload. This broker automatically joins the clients can be started and run in two client cluster created beforehand by B0 using machines. To increase saturated probability of multicast method. The change in the topology is the brokers, QoS level of publishing and announced to HAProxy for reloading its subscribing MQTT messages is fixed to 2 and configuration. The reloading process needs to “clean session” flag set to FALSE in all be used instead of restarting one in order to experiments. lower the server downtime as much as possible. After reloading, HAProxy recognizes the new 4.3.1. Multi-publisher scenario server and distributes messages to all load-balancing members. At the end, average In the case of using a centralized MQTT %CPU utilization of the broker reduces under broker, there is only one subscriber per topic the lower threshold after a period of time. Thus, partition. This subscriber listens to all the topics we see another elasticity action (scaling in) at of the partition by subscribing to this time of the simulation when MQTT clients “SmartHouse/#” with wildcard mask “#” are finished or terminated. At this point when denoting all subtopics of the root topic the workload goes under 25%, number of “SmartHouse”. One publisher is created for VMs is decreased to one for minimizing every topic “SmartPlug” sending messages to operating cost. the topics “Parameter” below the topic 4.3.2. Multi-subscriber scenario “SmartPlug”. Totally, 3600 publishers send In centralized case, there is only one messages to 36000 topics “Parameter” at a publisher each topic partition. This publisher steady rate which is one message/second. sends messages to all the topics of the partition In the clustered case, the multi-publisher at a steady rate. One subscriber is created for scenario is tested with a cluster of two brokers every topic “SmartPlug”. Each subscriber B0 and B1 (B1 will be added dynamically when receives messages from the topic “Command” needed). Publishers (90 each partition) and under the topic “SmartPlug”. In all partitions, subscribers (one each partition) are equally 3600 subscribers receive messages from 3600 load-balanced across the two brokers. topics “Command” at a steady rate. Figure 6a shows average %CPU utilization In the clustered case, the multi-subscriber in both centralized and clustered cases without scenario is tested with a cluster of two MQTT elasticity. We see that the MQTT system with brokers B0 and B1 (B1 is added dynamically one broker (2vCPU) is easy to be saturated. when needed). Subscribers (90 each partition)
11. L.M. Pham, X.T. Hoang / VNU Journal of Science: Comp. Science & Com. Eng., Vol. 37, No. 1 (2021) 1-14 11 and publishers (one each partition) are equally With elasticity, we also see the same load-balanced across the two brokers B0 behaviours shown in Figure 8b like in the case and B1. of multiple publishers. The scaling out action Figure 6b shows average % CPU utilization with two more brokers is triggered later than in both centralized and clustered cases without the multi-publisher scenario. These two brokers elasticity. We see the same behaviors like the are added sequentially by Heat. One gap of one case of multiple publishers. Adding one more minute is set between broker additions to avoid broker to the cluster does not really help, but two elastic oscillation. The average %CPU more brokers (6 vCPU) can resolve the problem. In utilization stays above the upper threshold the centralized case, we also see obviously in Figure during the time longer than in the multi- 7b that average %CPU utilization of the broker publisher scenario. The reason is that the gets saturation after a couple of minutes (at the combination of QoS level set to 2 and “clean 2nd minute). At this point, dropped message session” flag set to FALSE keep retained messages at the brokers longer, thus the more rate also starts to increase. the subscribers are, the busier the brokers are. K (a) Multi-Publisher Scenario. (b) Multi-Subscriber Scenario. Figure 6. Without Elasticity: Average %CPU Usage of the Centralized and Clustered Brokers. (a) Multi-Publisher Scenario. (b) Multi-Subscriber Scenario. Figure 7. Average %CPU Usage of the Centralized Broker during Experimental Time.
12. 12 L.M. Pham, X.T. Hoang / VNU Journal of Science: Comp. Science & Com. Eng., Vol. 37, No. 1 (2021) 1-14 (a) Multi-Publisher Scenario. (b) Multi-Subscriber Scenario. Figure 8. Average %CPU Usage of the Clustered Brokers with Elasticity. 5. Conclusion References We have presented a flexible framework [1] N. Sharma, D. Panwar, Green IoT: Advancements that can support elasticity for MQTT broker and Sustainability with Environment by 2050. In: service in IoT applications. Our framework 8th International Conference on Reliability, Infocom Technologies and Optimization (Trends brings elasticity to the service by leveraging and Future Directions) (ICRITO), Noida, India, existing off-the-shelf components that are 2020, pp. 1127-1132. currently used in cloud environments. Our [2] V. Turner, D. Reinsel, J.F. Gantz, S. Minton, The elastic MQTT broker service has been Digital Universe of Opportunities: Rich Data and successfully implemented using EMQX as the Increasing Value of the Internet of Things, MQTT broker solution and OpenStack as cloud IDC Report Apr, 2014. environment. Experiments are conducted by [3] MQ Telemetry Transport. 2020 generating traffics to the service at varying load (30 October 2020). level to observe changes in number of broker [4] P. Mell, T. Grance, The NIST definition of cloud instances. Our experiment results show that our computing (draft), NIST special publication 800- 145 (2011) 1-3. MQTT broker service adapts relative well to [5] P.T. Eugster, P.A. Felber, R. Guerraoui, user load changes making the service fully A. Kermarrec, The many faces of accommodate incoming traffics as well as keep publish/subscribe, ACM Comput, Surv. 35(2) operating cost low. (2003) 114-131. [6] R. Kawaguchi, M. Bandai, Edge Based MQTT Broker Architecture for Geographical IoT Acknowledgements Applications, 2020 International Conference on Information Networking (ICOIN), Barcelona, This work has been supported by VNU Spain, 2020, pp. 232-235. University of Engineering and Technology [7] V. Gupta, S. Khera, N. Turk, MQTT protocol under project number CN19.09. We also send employing IOT based home safety system with sincere thanks to the staff of Center for ABE encryption, Multimed Tools Appl, 2020. Computer Network and eLearning, VNU-UET [8] A. Mukambikeshwari, Poojary, Smart Watering for supporting the implementation of project’s System Using MQTT Protocol in IoT, Advances in Artificial Intelligence and Data Engineering. infrastructure. Advances in Intelligent Systems and Computing,
13. L.M. Pham, X.T. Hoang / VNU Journal of Science: Comp. Science & Com. Eng., Vol. 37, No. 1 (2021) 1-14 13 Springer, Singapore 1133 (2020) số trang đầu- Based Publish-Subscribe Applications, in IEEE cuối. Transactions on Network and Service [9] Y.C. See, E.X. Ho, IoT-Based Fire Safety System Management 17(3) (2020) 1954-1968. Using MQTT Communication Protocol, [20] R.R. Righi, E, Correa, M.M. Gomes, C.A. Costa, International Journal of Integrated Engineering. Enhancing performance of IoT applications with 12(6) (2020) 207-215. load prediction and cloud elasticity, Future [10] S. Nazir, M. Kaleem, Reliable Image Generation Computer Systems 109 (2020) Notifications for Smart Home Security with 689-701. MQTT, International Conference on Information [21] M.H. Fourati, S. Marzouk, K. Drira, M. Jmaiel, Science and Communication Technology DOCKERANALYZER: Towards Fine Grained (ICISCT), Karachi, Pakistan, 2019, pp. 1-5. Resource Elasticity for Microservices-Based [11] P. Alqinsi, I.J.M. Edward, N. Ismail, Applications Deployed with Docker, 20th W. Darmalaksana, IoT-Based UPS Monitoring International Conference on Parallel and System Using MQTT Protocols, 4th International Distributed Computing, Applications and Conference on Wireless and Telematics (ICWT), Technologies (PDCAT), Gold Coast, Australia, Nusa Dua, 2018, pp. 1-5. 2019, pp. 220-225. [12] Comparison of MQTT Brokers, [22] M. Nardelli, V. Cardellini, E. Casalicchio, Multi- Level Elastic Deployment of Containerized of-mqtt-brokers.html”/, 2020 (30 October 2020). Applications in Geo-Distributed Environments, [13] M. Collina, G.E. Corazza, A. Vanelli-Coralli, 2018 IEEE 6th International Conference on Future Introducing the QEST broker: Scaling the IoT by Internet of Things and Cloud (FiCloud), bridging MQTT and REST, 2012 IEEE 23rd Barcelona, 2018, pp. 1-8, International Symposium on Personal, Indoor and [23] L.M. Pham, A Big Data Analytics Framework for Mobile Radio Communications - (PIMRC), IoT Applications in the Cloud, VNU Journal of Sydney, NSW, 2012, pp. 36-41. Science: Computer Science and Communication [14] A. Schmitt, F. Carlier, V. Renault, Data Exchange Engineering 31(2) (2015) 44-55. with the MQTT Protocol: Dynamic Bridge [24] V.F. Rodrigues, I.G. Wendt, R.R. Righi, C.A. Approach, 2019 IEEE 89th Vehicular Technology Costa, J.L.V. Barbosa, A.M. Alberti, Brokel: Conference (VTC2019-Spring), Kuala Lumpur, Towards enabling multi-level cloud elasticity on Malaysia, 2019, pp. 1-5. publish/subscribe brokers, International Journal of [15] A.M.V. Zambrano, M.V. Zambrano, E.L.O. Distributed Sensor Networks 13(8) (2017) 1-20. Mej´ıa, X.H. Calderon´, SIGPRO: A Real-Time [25] S. Vavassori, J. Soriano, R. Fernandez, Enabling Progressive Notification System Using MQTT Large-Scale IoT-Based Services through Elastic Bridges and Topic Hierarchy for Rapid Location Publish/Subscribe, Sensors, 2017. of Missing Persons, in IEEE Access. 8 (2020) [26] A distributed, reliable key-value store. 149190-149198. 2020 (30 October 2020). [16] The features that various MQTT servers (brokers) [27] D. Roure, C. Goble, Software Design for support. Empowering Scientists, IEEE Software 26(1) (2009) 88-95. -support”/, 2020 (30 October 2020). [28] EMQX Broker. [17] P. Jutadhamakorn, T. Pillavas, V. Visoottiviseth, R. 2020 (30 Takano, J. Haga, D. Kobayashi, A scalable and low- October 2020). cost MQTT broker clustering system, 2017 2nd [29] Kubernetes. 2020 (30 International Conference on Information Technology October 2020). (INCIT), Nakhonpathom, 2017, pp. 1-5. [30] HAProxy. [18] Z.Y. Thean, V. Voon Yap, P.C. Teh, Container- based MQTT Broker Cluster for Edge Computing, balancing/, 2020 (30 October 2020). 2019 4th International Conference and Workshops [31] OpenStack: Open Source Cloud Computing on Recent Advances and Innovations in Engineering (ICRAIE), Kedah, Malaysia, 2019, Infrastructure. 2020 pp. 1-6. (30 October 2020). [19] A. Detti, L. Funari, N. Blefari-Melazzi, Sub- [32] OpenStack Heat. Linear Scalability of MQTT Clusters in Topic-
14. 14 L.M. Pham, X.T. Hoang / VNU Journal of Science: Comp. Science & Com. Eng., Vol. 37, No. 1 (2021) 1-14 2020 (30 [36] RabbitMQ. 2020 (30 October 2020). October 2020). [33] OpenStack Ceilometer. [37] Apache Jmeter. 2020 2020 (30 (30 October 2020). October 2020). [38] L.M. Pham, T.T. Nguyen, M.D. Tran, A [34] OpenStack Aodh. Benchmarking Tool for Elastic MQTT Brokers in 2020 (30 IoT Applications, International Journal of October 2020). Information and Communication Sciences 4(4) (2019) 59-67. [35] Gnocchi - Metric as a Service. 2020 (30 October 2020). U k