Webinar
Unifying Data from Vehicle to Cloud | Register Now →

Unified MQTT and Kafka with EMQX Streaming

Ingesting massive messages from millions of connected MQTT clients, storing them reliably, consuming them using Kafka clients directly, integrating with your backend services and systems seamlessly… Get all of these well done in one EMQX Platform.

Talk to an Expert →
Learn More →
Unified MQTT and Kafka with

What is EMQX Streaming?

EMQX Streaming is an innovative feature of EMQX that allows users to persistently store MQTT messages as data streams and consume them directly using Kafka clients.

MQTT is excellent for real-time message distribution and connecting many concurrent clients, but it doesn't directly support storing massive messages and on-demand access to historical data. As a result, MQTT brokers and Kafka have become the ideal partners and the combination of choice for most users.

However, maintaining separate Kafka clusters for MQTT workloads has become an additional burden, as it is very expensive, complex, and has high operational costs.

The EMQX Streaming feature is designed to solve this problem. Now, one EMQX Platform can meet these needs, and you no longer need to maintain separate Kafka clusters.

With EMQX Streaming:

  • Reliably store large amounts of MQTT message data
  • Directly use Kafka clients for historical data replay and real-time subscription
  • Seamlessly integrate with existing backend applications and big data systems
  • Directly interface with stream processors like Flink for data processing
Introduction

Benefits

Enhanced Data Persistence

MQTT Messages can be saved as data streams for a long time. Streams are also replicated across multiple nodes for fault tolerance and high availability.

Kafka Ecosystem Compatibility

EMQX Streaming implements the Kafka wire protocol, so it is compatible with the vast Kafka ecosystem, including Kafka clients, connectors, Flink, and more. It can also seamlessly work with your current Kafka applications.

Unified Experience

Operate and manage MQTT and Kafka workloads on one EMQX platform through a unified UI and APIs without any complicated integration process.

Simplified Operation and Maintenance

Managing Kafka clusters is tedious and complex. With EMQX Streaming, you no longer need to build and manage dedicated Kafka clusters for your MQTT data. This will significantly reduce your operational costs and mental burden.

How EMQX Streaming Works

How EMQX Streaming Works
Stream declaration
You must declare a stream with an MQTT topic filter before using it. All MQTT messages matched with the topic filter will be saved into the stream.
Stream data storage
The stream data persists in the storage system of EMQX, and to ensure fault tolerance and high availability, the data is replicated across multiple nodes.
Requests serving
Kafka clients send requests to EMQX to fetch stream data. Because EMQX can understand Kafka protocol, it can reply to Kafka clients correctly.
Group management
EMQX streaming also supports Kafka consumer group protocol. It can manage the cooperative consumption of multiple Kafka consumers and maintain their consumption progress.
Stream data retention
The data in streams will be saved for a specified period of time, and outdated data will be automatically deleted.

Use Cases

AutomotiveIndustrial IoT

Automotive

Nowadays, vehicles, particularly smart connected cars, generate a large amount of data during operation. These data play a crucial role in scenarios such as autonomous driving, smart cockpits, and diagnostics. By utilizing the MQTT protocol, vehicles transmit these data to the EMQX Platform in the cloud, where EMQX Streaming capabilities efficiently store the data in the corresponding data streams. This approach not only ensures data fault tolerance and high availability but also allows for direct integration with various backend applications and data systems through the standard Kafka protocol. This simplification of system architecture significantly enhances the overall stability of the data pipeline, as well as the efficiency of data integration and application development.

Automotive

Industrial IoT

In industrial and smart manufacturing, efficient data collection and persistent storage across multiple processes, production lines, and various equipment instruments are crucial for driving optimization, enhancing productivity, and ensuring quality. The efficient data persistence, fault tolerance, and high availability capabilities offered by EMQX Streaming play a significant role in scenarios such as production process monitoring and traceability, performance analysis and optimization, and predictive maintenance.

Industrial IoT

Related Products

EMQX Enterprise

Self-managed EMQX Platform for enterprise workloads.

EMQX Premium

Fully managed Enterprise-grade MQTT service with durable storage and advanced features.

Get Started with EMQX Streaming

Explore our detailed EMQX Streaming Documentation to learn how to configure and optimize this feature for your needs.
Documentation →
Have questions or need a tailored solution? Contact our sales team today to discuss how EMQX Streaming can enhance your IoT deployment.
Contact Us →
Contact Us