Overview
The Shenzhen Transportation Smart Integration IoT Platform Project, led by the Shenzhen Urban Transport Planning Center(referred to as SUTPC), is a universal core technical framework within Shenzhen's entire transportation network. The platform focuses on comprehensive management and control of perceptual devices in transportation scenarios, while also engaging in collaborative computation, processing, and data mining with other platforms and systems.
Utilizing the EMQX MQTT platform, SUTPC has achieved comprehensive access and perception of urban transportation elements. This solid digital foundation ensures the security of the construction of Shenzhen's Smart Transportation IoT Platform.
Challenges
Shenzhen's Smart Transportation IoT platform is deployed between the traffic big data platform and perceptual devices. It downwardly connects various sensors, terminals, and infrastructure monitoring nodes in smart transportation scenarios. Upwardly, it provides the necessary data and information to the traffic big data platform and intelligent applications.
The entire platform construction faces the following challenges:
- Multiple Access Methods: The platform not only needs to interface with existing monitoring platforms but also requires the addition of new devices through direct connections, monitoring node connections, and edge computing node connections.
- Diverse Protocol Types: The platform must support various protocols such as MQTT, Modbus, HTTP, etc., to enable data collection and distribution.
- High-Concurrent Device Access: In this phase, the platform needs to support the connection of 20,000 devices with a reserved capacity for 100,000 device connections in the future.
- Complex Data Flow: The platform interfaces with field devices, and the collected data needs to be parsed, processed, and transformed within the platform. Subsequently, the device data is sent to different business platforms such as the traffic big data platform and operation and maintenance platform.
- Numerous Non-Standardized Data: Due to the system's need to integrate a large number of existing devices, the platform must have the capability to parse various non-standardized protocols and messages. This includes messages related to device registration, device acquisition, and device status.
Solution
Since its release in the 1990s, the MQTT protocol has become the preferred protocol for diverse IoT applications due to its lightweight, simplicity, openness, ease of implementation, and support for Quality of Service (QoS). EMQX, an enterprise MQTT platform that provides integrated distributed MQTT messaging services and a powerful IoT rules engine, powers high-reliability, high-performance real-time data movement, processing, and integration for IoT.
Utilizing the EMQX MQTT platform, SUTPC has achieved the data collection and transmission of a large number of IoT devices in the urban transportation of Shenzhen through the MQTT protocol.
Secure Access with Multiple Methods
Unified communication protocols and data formats are standardized for MQTT-supported devices, including direct connection, monitoring node, and edge computing devices. EMQX on Intel X86 provides open-standard IoT MQTT protocol support, enabling easy access for new devices.
EMQX allows for easy integration of IoT device data and third-party systems into existing platforms. Secure communication with EMQX is guaranteed through TLS/SSL or QUIC, as well as various authentication mechanisms such as username/password, LDAP, JWT, PSK, and X.509 certificates. This ensures data security and privacy.
Easy Access with Various Protocols
EMQX offers open SDKs for multi-protocol access between platforms and devices. NeuronEX, the edge industrial protocol gateway, converts these protocols to MQTT, allowing for the integration of multiple protocols such as Modbus and OPC. NeuronEX also offers gateway configuration management, alarming, data processing, rule engine, and web-based UI. It simplifies the development process and improves efficiency.
Massive Device Concurrent Connections
EMQX's high-availability, distributed cluster architecture can support concurrent connections ranging from millions to billions, providing a stable and reliable platform access layer for large-scale device connections. To accommodate the connection of 20,000 devices, the IoT platform for the project adopts a microservices technology architecture, deploying multiple sets of data computation services and data storage services to expand the number of connected devices. In theory, it can support up to 100,000 device connections. Additionally, customized protocol adaptation and data parsing are provided for future device connections to meet potential access requirements.
Low-Code Data Processing and Integration
EMQX offers a powerful rule engine and data bridging based on SQL statements that allow end-to-end IoT data processing without coding. It enables flexible integration of IoT data into MySQL and ClickHouse for persistent storage, or Kafka for data sharing and management. The rule engine's message republishing feature enables the flow of data between IoT modules, allowing for rapid integration of data applications.
Efficient Data Parsing
The built-in Schema Registry enables high-performance encoding and decoding of MQTT messages in JSON, Avro, and Protobuf formats. This addresses the parsing of standardized and non-standardized protocols on IoT platforms, including parsing for device registration information, device collection messages, device status messages, and various structural data types. It allows the addition of custom parsing scripts on the IoT platform for data parsing and processing, significantly lowering the barrier for data parsing and reducing development workload.
Achievements
The platform of SUTPC’s project has established five core business subsystems: device access, device management, data management, data sharing, and platform security. This has significantly enhanced the level of governance for large-scale urban transportation.
Establishment of Standard Specifications
The project creates guidelines and standards for technology in transportation facilities, breaks down data barriers between monitoring platforms and standardizes device monitoring data format.
100,000+ Devices Connection Capability
In this phase of the project, a total of 113 facilities and over 20,000 devices have been connected, with a reserved capacity for 100,000 device connections. This ensures a data guarantee and the capability for future expansion.
Massive Data Storage and Distribution
The platform stored 10.1TB of 14.66B monitoring data within a three-month retention period. It can handle millions of transactions per second, facilitating real-time data movement and flexible distribution to other business systems, meeting diverse data demands in the IoT business.
Traffic Industry Device Model System
With the cloud-edge data access and integration solution provided by EMQ, the platform offers users flexible device modeling capabilities. It has established a complete set of traffic industry device models based on device attributes, services, and events, facilitating standardized interaction and management processes between edge devices and the cloud.
Traffic Industry Processing Algorithm Library
The data processing capabilities of EMQX, including encoding and decoding, along with the rule engine, preprocess various types of data from IoT monitoring devices. Based on this, a mature algorithm library for processing intelligent transportation data has been established, unlocking the real-time and historical value of streaming data.