Rumqtt release R19: Performance testing, public infra, and persistence

Overview

Today we’re making release 19 for rumqtt 🎉. It builds on the last release that focused on adding rich monitoring capabilities to rumqtt.

Release R19 · bytebeamio/rumqtt

Significant Changes Stress test router to against 10k connections (#511)Add prometheus data collection support (#522)Remove Connect from ConnectionError::StateError (#541)Support retransmissio...

GitHubbytebeamio

We’ve benchmarked rumqttd to support 10,000+ clients, added a public MQTT broker, prometheus integration, and experimental support for storage backed persistence. For future we are prioritizing MQTT v5 support in rumqtt and it will be available in early 2023.

Read on to know more about what's included in this release 👇

⚡️ Load testing using mqttwrk

More and more IoT devices are being added to the internet every day. Being ready to handle such a scale is an important requirement for any MQTT broker – performance at scale is critical. Naturally, we wanted to figure out if rumqttd can meet the scalability requirements of the future.

To test an MQTT broker for the kind of loads we expect at scale, a benchmarking tool was required. Surprised by the lack of good ones, we decided to build one ourselves – mqttwrk! Mqttwrk makes it super simple to simulate different kinds of loads for testing an MQTT broker.

GitHub - bytebeamio/mqttwrk: Wrk inspired tool for scale and performance testing mqtt brokers

Wrk inspired tool for scale and performance testing mqtt brokers - GitHub - bytebeamio/mqttwrk: Wrk inspired tool for scale and performance testing mqtt brokers

GitHubbytebeamio

We simulated load from 10,000 publishers publishing 100 messages each using mqttwrk. Rumqttd could handle this load comfortably running on a single node. In a future blog post we will be writing about our testing strategy in depth.

We want to keep pushing rumqttd to stay atop the rapidly growing IoT demands. There is a need for achieving even greater scales for an MQTT broker. Our future releases will build on the current work to move in that direction.

🛣 Public MQTT broker

The friction of setting up the correct dev environment is a big hurdle for a new maker in the IoT space. We want to remove this hurdle and increase the development velocity for all IoT makers. To this end we’ve set up a test MQTT broker available for public usage!

Instead of running an MQTT broker on your local environment, you can connect to test.rumqttd.bytebeam.io:1883 and test your project quickly. This public instance of MQTT broker is backed by our very own rumqttd. This is suited to be used in production but you can use it to test out how MQTT brokers or clients work, or rumqttd specific features.

The public broker is live and available to use right now! 😄 You can try it out using rumqttc or mqttwrk.

🔥 Prometheus integration

Prometheus is a widely used to record real-time metrics. We're added Prometheus support to rumqttd with this release! With this rumqttd exposes metrics on an endpoint your Prometheus instance can scrape directly from.

This provides high visibility into the broker. We want the broker to be more than a black box for a user and plan on exposing more rich data points to the Prometheus collector.

💾 Storage backed persistence

Unreliable network environments and unexpected system failures are inevitable challenges in IoT. Remote devices may lose connectivity to the MQTT broker, or the broker or subscriber may restart due to planned or unplanned maintenance. This can lead to data loss.

We’ve added experimental support for storage backed persistence to our broker to mitigate these issues and ensure zero data loss of messages. You can try it out today here. We plan on writing in detail how it works in a future blog post.

Release R20-alpha · bytebeamio/rumqtt

Significant Changes Add support for storage backed disk persistent connections What’s Changed feat(rumqttd): Add support for disk persistent connections by @tekjar in #489feat(rumqttd): Add def...

GitHubbytebeamio

In summary, persistence mode solves the above mentioned issues with a two part solution – storing messages from publisher clients in disk, and maintaining persistent state for subscriber clients. Persistent mode is similar to how a computer game would save in-game progress and allow you to restart from the last saved point. It can be enabled with persistence = true on a desired tcp port in rumqttd configuration.

🧭 MQTT 5 Roadmap

We hear you! 👂 A lot of people have shown interest in MQTT v5 support for rumqtt. This is something on our roadmap and we want to make sure to bake MQTT v5 support to our client and broker the right way.

We plan to keep on releasing parts of v5 features while building towards full support for it.  👀 Expect all v5 features to land in rumqtt early 2023.

Learn more

For more details on this release you can have a look at the release notes and full changelog can be found at R18...R19.

We believe in the OSS philosophy and encourage contributions of all sizes! Please show us some love by ⭐️ing the project on GitHub. Better still, raise a pull request! You can also join our Discord server  and follow our development cycle closely.