Test Purpose
In EMQX 3.0, we have done the benchmark test of 10 million connections and 1 million message throughput. In that test, a cluster of 20 EMQX nodes was used, and the machine of each node had 16 cores & 32G.
In version 4.0 and subsequent 4.2 and 4.3, we redesigned and continuously optimized the architecture and internal modules, which greatly improved the performance of EMQX. In order to demonstrate the capabilities of EMQX 4.3 in large-scale MQTT concurrent connections, topic subscriptions, and high message throughput in this test, the tested cluster has only 5 nodes, and the machine resources of each node are 32 cores and 64G.
Test Scenarios
Huawei cloud host is used in this test, and 5 EMQX nodes and the test machines are in the same vpc. The main scenarios are 10 million MQTT connections, 10 million topic subscriptions，1 million QoS0 message throughput, and 500,000 QoS1 message throughput, as shown below.
Note: Unless otherwise specified, all connections have a heartbeat time of 300 seconds by default, and all message payloads are 50 bytes.
Scenario 1: 10 million MQTT connections and subscriptions + QoS0 broadcast scenario with 1 million message throughput
10 million MQTT clients connect to the EMQX cluster at a rate of 20,000 new connections per second. Each client subscribes to a topic after successful connection, and every 10 connections subscribe to the same topic. Therefore, the test reaches 1 million topics and 10 million subscriptions. After 10 million connections and subscriptions are completed, the message broadcast scenario will start (the number of receiving terminals is much larger than that of publishing terminals). 50 MQTT connections are used as pub clients to publish messages. Every 10 connections publish messages to the topic testn/${machineName()}/ ${threadNum} (n is 1～5) as a group. Each pub client publishes 100 QoS 0 messages per second. 1000 sub clients are also divided into 5 groups, each with 200 subscription topics of testn / # (n is 1 ~ 5).
Therefore, the total message publishing throughput rate is 5000 messages per second, and the total message receiving throughput rate reaches 1 million per second.
Scenario 2: 10 million MQTT connections and subscriptions + QoS0 one-to-one with 1 million message throughput
MQTT 10 million connections and subscriptions are the same as the above scenario. After 10 million connections and subscriptions are completed, one-to-one messaging will start (the number of receiving terminals is equal to the publishing terminals, and each receiving terminal subscribes to a corresponding pub topic). Both pub clients and sub clients are 25,000. Each pub client publishes 20 QoS 0 messages per second, and each sub client consumes 20 QoS 0 messages. Therefore, both publishing and receiving are 500,000 messages per second, and the total message throughput reaches 1 million.
Scenario 3: 10 million MQTT connections and subscriptions + QoS1 broadcast scenario with 500,000 message throughput
MQTT 10 million connections and subscriptions are the same as in scenario 1 above. After 10 million connections and subscriptions are completed, the message broadcast scenario with QoS of 1 starts (the number of receiving terminals is much larger than that of publishing terminals). 50 MQTT connections are used as pub clients to publish messages, and every 10 publish messages to the topic testn/${machineName ()}/${threadNum} (n is 1～5) as a group. Each pub client sends 50 QoS 1 messages per second. The 1000 sub clients are also divided into 5 groups, each with 200 subscription topics testn/# (N is 1 to 5).
Therefore, the total message publishing throughput rate is 2,500 messages per second, and the total message receiving throughput rate reaches 500,000 per second.
Scenario 4: 10 million MQTT connections and subscriptions + QoS1 one-to-one with 500,000 message throughput
MQTT 10 million connections and subscriptions are the same as in scenario 1 above. After 10 million connections and subscriptions are completed, one-to-one messaging with QoS of 1 will start (the number of receiving terminals is equal to the publishing terminals, and each receiving terminal subscribes to a corresponding pub topic). Both pub clients and sub clients are 25,000. Each pub client publishes 10 QoS 1 messages per second, and each sub client consumes 10 QoS 1 messages. Therefore, both publishing and receiving are 250,000 messages per second, and the total message throughput reaches 500,000.
Test Result Highlights
Scenario 1: 10 million MQTT connections and subscriptions + QoS0 broadcast scenario with 1 million message throughput
- The average connection speed reaches 20,000 per second, and the average response time for connection + subscription is 4.2ms
- Sub average response time is 0.33ms
- The average CPU usage of EMQX on each node is 34%. After 10 million clients are all connected, the CPU usage range of EMQX on each node during messaging period is 32%～60%.
- The average memory usage of EMQX on each node is 28.4GB. After 10 million clients are all connected, the memory usage range of EMQX on each node is 29GB～35.4 GB during messaging period.
| Min. response time (s) | Max. response time (s) | Average response time (s) | Average throughput | EMQX CPU usage | EMQX memory usage(G) | |
|---|---|---|---|---|---|---|
| MQTT Connection | 0.002 | 0.03 | 0.0042 | 19978 | 32%～60% | 29～35.4 |
| MQTT Pub | / | / | / | 5002.462 | ||
| MQTT Sub | 0 | 0.178 | 0.0003 | 1000490 |
Scenario 2: 10 million MQTT connections and subscriptions + QoS0 one-to-one with 1 million message throughput
- The average connection speed reaches 20,000 per second, and the average response time for connection + subscription is 5ms
- Sub average response time is 0.3ms
- The average CPU usage of EMQX on each node is 51%. After 10 million clients are all connected, the CPU usage range of EMQX on each node during messaging period is 56%～68%.
- The average memory usage of EMQX on each node is 28.7GB. After 10 million clients are all connected, the memory usage range of EMQX on each node is 28.6GB～35.2GB during messaging period.
| Min. response time (s) | Max. response time (s) | Average response time (s) | Average throughput | EMQX CPU usage | EMQX memory usage(G) | |
|---|---|---|---|---|---|---|
| MQTT Connection | 0.002 | 3.076 | 0.005 | 19923 | 56%～68% | 28.6～35.2 |
| MQTT Pub | / | / | / | 505045 | ||
| MQTT Sub | 0 | 0.04 | 0.0003 | 502322 |
Scenario 3: 10 million MQTT connections and subscriptions + QoS1 broadcast scenario with 500,000 message throughput
- The average connection speed reaches 20,000 per second, and the average response time for connection + subscription is 4.6ms
- Pub average response time is 1ms, and Sub average response time is 0.3ms
- The average CPU usage of EMQX on each node is 51%. After 10 million clients are all connected, the CPU usage range of EMQX on each node during messaging period is 35%～57%.
- The average memory usage of EMQX on each node is 28.5GB. After 10 million clients are all connected, the memory usage range of EMQX on each node is 29.6GB～33.8GB during messaging period.
| Min. response time (s) | Max. response time (s) | Average response time (s) | Average throughput | EMQX CPU usage | EMQX memory usage(G) | |
|---|---|---|---|---|---|---|
| MQTT Connection | 0.002 | 0.37 | 0.0046 | 19998 | 35%～57% | 29.6～33.8 |
| MQTT Pub | 0 | 0.123 | 0.001 | 2607 | ||
| MQTT Sub | 0 | 0.116 | 0.0003 | 521555 |
Scenario 4: 10 million MQTT connections and subscriptions + QoS1 one-to-one with 500,000 message throughput
- The average connection speed reaches 20,000 per second, and the average response time for connection + subscription is 4.5ms
- Pub average response time is 1.1ms, and Sub average response time is 0.6ms
- The average CPU usage of EMQX on each node is 47%. After 10 million clients are all connected, the CPU usage range of EMQX on each node during messaging period is 50%～65%.
- The average memory usage of EMQX on each node is 28.8GB. After 10 million clients are all connected, the memory usage range of EMQX on each node is 30.9GB～36.4GB during messaging period.
| Min. response time (s) | Max. response time (s) | Average response time (s) | Average throughput | EMQX CPU usage | EMQX memory usage(G) | |
|---|---|---|---|---|---|---|
| MQTT Connection | 0.002 | 3.022 | 0.0045 | 19803 | 50%～65% | 30.9～36.4 |
| MQTT Pub | 0 | 0.283 | 0.0011 | 249192 | ||
| MQTT Sub | 0 | 0.03 | 0.0006 | 521555 |
Test Tools
The enterprise version of the XMeter performance test platform is used in this test.
XMeter overview (https://www.xmeter.net/): XMeter is a performance test platform based on the extension of the open source test tool JMeter. In view of the large-scale access, flexible expansion requirements, multiple access protocols, and mixed scenarios of the Internet of Things, XMeter has modified JMeter to support large-scale and high-concurrency performance test, such as tens of millions of MQTT concurrent connection testing. In addition to testing the MQTT protocol, it can also support mainstream application test, such as HTTP / HTTPS.
JMeter-MQTT Plug-in: It is an open source MQTT performance test plug-in implemented by XMeter, which has been used in many projects and is currently the most popular MQTT plug-in in the JMeter community.
Tool version:
- XMeter：Enterprise 3.0
- JMeter-MQTT Plug-in：mqtt-xmeter-2.0.2
- JMeter：JMeter5.2.1