The amount of data we generate today across all domains is exploding with millions of devices getting connected to the internet on a daily basis. Each IoT Application needs to handle Terabytes of data daily. Handling such a large volume of data is not feasible with a single-node server. Even though MQTT brokers in the market are well-scaled for throughput, large-scale production environments require MQTT high-availability clusters to meet the demands effectively.
The SaaS era of the Cloud has redefined the way the servers get deployed for high availability of the system. The applications built before the Internet of Things were all HTTP-based solutions where 99% of the requests were served in a single server (or a path of servers). MQTT high availability poses a different set of challenges for the data consistency. The protocol makes it mandatory that the messages that land on a particular server node in the MQTT Cluster should be made available to all the nodes in the cluster to make it available for the client connected to this system faster and more reliable way. Having said that the MQTT Cluster should also make the data available for the users via REST or the other modes who are also part of the overall ecosystem of IoT/IIoT Data.
MQTT Cluster Architecture
MQTT Protocol Specification does not support the storage of the data. However, the high availability cluster needs clear data storage for the transient data to make sure the data delivery is guaranteed as per the MQTT QoS Levels instructed in the MQTT Packets. The storage may not be needed when there is a single Node deployment as the data can be in transient memory. So MQTT Broker Clusters need more components beyond the MQTT Implementation. At a high level, the data storage for the transient data is needed in the following cases:
- When a new MQTT Broker joins the cluster
- When an MQTT Broker restarts due to the physical or functional reboot.
When all the brokers are running fine, there is a need for Inter Broker communication (IBC) :
- To send data to the other MQTT Server instances in the cluster
- To receive data from the other MQTT instances in the cluster
Load Balancer is something mandatory across all high availability deployments irrespective of the protocol used for the communication between the clients and the server. It plays a major role in setting up the MQTT Cluster as well as distributing data across the MQTT nodes in the cluster and also serving the REST APIs via them. There are two modes in which the MQTT cluster can be deployed. Load Balancer support plays a significant role in these configurations.
Active-Passive Mode
In this set up, there will be two MQTT Brokers deployed behind a load balancer. One of the MQTT nodes will be configured as Active and will serve the request. The secondary node will keep the status and data in sync with the primary and will be in stand-by mode. Whenever there is a functional or physical failure on the primary server, the secondary server will take over.
This setup is ideal for moderate data exchange scenarios and where a single MQTT node can handle the load. You can also conserve energy and cost by keeping the secondary node in a minimal configuration.
Multi Master Mode
When the requirement has more data exchange where a single node cannot handle the production data load, we need to deploy multiple masters behind the Load balancer and all MQTT Brokers will be serving the clients in parallel. Since it is a TCP based protocol, irrespective of the configuration, the clients will be sticky to the MQTT server to which it connects. The configuration can be made a weighted Round Robin to distribute the load in a better way.
How do the MQTT Clusters maintain Data consistency?
MQTT Broker has support for an in-built data exchange mechanism which we call as Inter Broker Communication Engine (IBC). IBC is enabled in each MQTT node in the cluster so that it sends and receives the data from the other MQTT cluster nodes. All nodes store data in parallel to the database, ensuring redundancy and enabling data recovery in case of failure.
MQTT Broker Redundancy
As outlined earlier, MQTT Protocol specification and the IoT Applications built over MQTT have a diversified set of requirements to make it more scalable and reliable in terms of data consistency in M2M and the integration with the data analysis platforms which is becoming an inherent part of the every deployment done today. We always ask our customers to store the data behind the broker and not as a subscriber. This will help them reduce bandwidth costs as well as the size of their deployments.
Physical Redundancy
Physical MQTT redundancy helps when one or two nodes fails and till the time you provision a new server or the same physical server comes back alive, the cluster should be fully functional.
Functional Redundancy
More than Physical monitoring of the nodes, the functional monitoring is crucial for the cluster support SLA. Each and every node should be monitored for all MQTT functions as well as the responsiveness of the nodes. These checks need to be done at the cluster level as well as a whole.
When we deploy large MQTT Clusters, we should have the overall configuration in such a way that each node server 60 to 70% of the overall load. This ensures the MQTT cluster health is optimal. Going beyond 80% will cause cluster failure when one or two nodes fail during production.
The HA setup is flexible and can be deployed on any cloud or on premise servers with a diversified set of Data Storage and load balancer options. The configuration in the load balancer depends on how the connections and data need to be balanced. However, making the best choice also depends on how & where the cluster set up is getting deployed.
Securing your MQTT Installation
Ensuring the security of your MQTT installation is crucial for protecting sensitive data and maintaining the integrity of your applications. It is only through effective MQTT Security measures, we can not only safeguard our data against unauthorized access but can also ensure the integrity of data throughout the entire communication process. This proactive approach is inevitable for mitigating potential threats and ensuring seamless operation in the IoT ecosystem.
Below are the key strategies to enhance the security of your MQTT installation :
- Use TLS/SSL Encryption - Implement TLS/SSL (Transport Layer Security/Secure Sockets Layer) to encrypt the data in transit between the clients and the brokers, preventing eavesdropping and tampering.
- Implement Authentication - Verify the identity of the clients that connect to the broker by implementing strong MQTT authentication mechanisms such as username/password, OAuth, token-based authentication, etc.
- Authorization Controls - Ensure that only authorized users can publish or subscribe to specific topics by setting up granular Access Control Lists (ACLs) to restrict client permissions based on roles.
- Secure the Load Balancer - Safeguard the load balancer against unauthorized access and attacks by incorporating security measures such as IP whitelisting and firewall rules.
- Limit Client Connections - Restrict the number of simultaneous connections and the rate of message publishing to mitigate the risk of denial-of-service (DOS) attacks.
- Monitor Network Traffic - Track and analyze the MQTT Traffic for unusual patterns or unauthorized access attempts by utilizing network monitoring tools.
- Safeguard Internal Communication - Establish a Virtual Private Network (VPN) to secure internal communications between MQTT brokers and clients, adding an additional layer of protection.
MQTT Clustering on Cloud
MQTT Cluster in AWS
Amazon Web Services provided a good set of options for the hardware (EC2) instances for running your MQTT nodes of the cluster. AWS EC2 comes with a set of internal IP Addresses that need to be used to communicate internally. The Elastic Load balancing has an option to redirect TCP load on port 1883 or 8883 (for TLS).
Elastic Load Balancer ( ELB) of the AWS should be used for the load balancing. Out of the 4 types of the options ELB offers, for MQTT the Network load balancer needs to be configured. The network load balancer supports TCP. Bevywise MQTT Broker ( CrystalMQ) supports the User interface as well which needs to be configured via the Application load balancer. However, you can use the same network load balancer for the IoT Dashboard as well.
MQTT High Availability in Azure
Azure Cloud provides a great set of tools that helps you set up the MQTT Broker on a highly redundant architecture. Like AWS, there are VMs, reliable database services and Azure Load Balancer that can be used to set up the most affordable MQTT Clusters on the Azure cloud in the region of your choice. Azure, like AWS, provides long-term commitment based discounted pricing to save costs on the servers.
Detailed step-by-step instructions for setting up the Azure IoT High availability application can be used to get your application production ready.
High Availability Clustering in DigitalOcean
We have been using Digital Ocean for a couple of years for all our hosting. Our Cloud Hosted MQTT Broker Deployments support Digital Ocean in addition to AWS. We have a great time working with them. Digital Ocean provides the most cost effective Droplets ( equivalent to EC2) for deploying MQTT Broker on your preferred OS. The set of the MQTT Server is same as all other servers. The Network load balancer of Digital Ocean supports the L3 / TCP Load sharing between the MQTT Servers behind them.
On the overall, if you are looking on the cloud based deployments, We would recommend a Ubuntu or Redhat based servers for running your MQTT instances and run the MySQL or PostgreSQL on a separate server or service like RDS where the redundancy of the data layer is taken care.
MQTT High Availability on-premise
Windows NLB Setup:
Most enterprises run all their workloads on a Windows based environment. If you are one of such enterprise, then the combination of the Windows Server and Network Load Balancer (NLB) should help in creating a robust MQTT Cluster for your production environment.
NLB runs as an integral part of all the Windows server instances. There is no need for a separate set of servers to run the load balancers. However some special Windows NLB configuration needs to be done on the Switch / Router you use to ensure smooth data sharing with all the Windows MQTT Broker running in the cluster. As advised earlier, MySQL or MSSQL needs to be run on a separate set of servers in either Master-Slave mode or Multi Master. If run on multi master, you may need to keep the ids unique with the odd-even configuration on the two DB instances.
Ubuntu Nginx Setup:
In the Linux environment, we need additional servers for running the Load balancer. The load balancer needs to be configured in fail over mode to ensure reliability. Nginx is the most reliable and TCP based load balancing application. Do refer to the detailed Nginx configuration manual to set up the MQTT Clusters.
Database Configuration:
Despite the number of MQTT Brokers we are running, we should have the ability to store the data to a singe database to which all the Broker will connect to. The Database will store all the incoming data traffic from all the nodes, which makes it more seamless to synchronize data within brokers. The key benefit of this centralized storage is that, irrespective of the client's connection with any MQTT brokers, it helps in bypassing data to all the available brokers with the help of IBC to smoothen the publish-subscribe flow.
For example, consider a clustered set up with 2 MQTT Brokers connected to the load balancer and database. 'A' client, a publisher is connected to the Broker 1 and 'B' client, a subscriber, is connected to the Broker 2 through load balancer. Now 'A' client publishes data in the topic mqtt1, and 'B' is subscribing to it. As per the set up, MQTT Brokers can talk to each other and share data from the database through IBC. Database will enable Broker 2 to get the data received in Broker 1 and send it to the subscribed client 'B'. Thus, enabling all the brokers to receive and send data irrespective of who is the client connected to?
By the way, it is not mandatory to have only centralized storage. Implementing a Master/Slave set up for your data storage will also add weightage based on the requirement.
While selecting your desired database you can consider choosing MySQL or PSQL as it can very well scale for such clustered high availability. Overall, the distributed architecture will provide a more robust and reliable network to handle traffic from a huge number of clients.
Maintaining data consistency and ensuring smooth communication across distributed nodes is crucial for large-scale IoT implementations. Only a robust MQTT cluster architecture can provide the confidence to handle high data traffic, maintain uptime, and deliver real-time data without interruptions. With this in mind, we’ve designed our MQTT Broker (CrystalMQ) to remain resilient, highly available, and reliable under all circumstances. Whether you’re scaling up or fine-tuning for mission-critical applications, we’ve got you covered!
Give our MQTT Broker a try today and experience the seamless connectivity for your IoT deployments!
