Ingesting IoT Data From EMQX Enterprise to GCP Pub/Sub

Guowei Li

Apr 17, 2023

Ingesting IoT Data From EMQX Enterprise to GCP Pub/Sub

You may have already successfully connected your IoT devices to EMQX Enterprise using the instructions provided in our previous posts. This article will introduce how to ingest IoT data from EMQX Enterprise to GCP Pub/Sub service to help you complete the migration.

Create and Manage Pub/Sub Topics

If Pub/Sub has not been enabled yet, you can create a topic by following these steps.

Open the Pub/Sub console, click CREATE TOPIC, input a custom Topic ID, and click CREATE to create it.
Navigate to the topic details page by clicking on the corresponding Topic ID from the list. Once on the page, create a subscription to store messages. Choose the Pull type and specify a message retention duration of 7 days.

Please refer to GCP Pub/Sub subscription for comprehensive information regarding subscriptions.
Click Subscription ID → MESSAGES → PULL to view the messages sent to the topic.

After successfully creating a Pub/Sub topic and gaining familiarity with its basic management operations, we can now use EMQX Enterprise to forward data to the newly created topic.

Forward MQTT Messages to Pub/Sub

EMQX Enterprise offers GCP Pub/Sub integration for easy IoT Core migration. Even if you are not an IoT Core user, you can still forward client events and messages to the relevant Pub/Sub topics, integrating IoT data with applications and services on Google Cloud.

The following steps are to configure Pub/Sub integration on EMQX Enterprise.

Create a Service Account JSON

The Service Account JSON is a file containing authentication and authorization credentials, and EMQX Enterprise must access Pub/Sub resources.

Go to the GCP Console, input IAM in the search box, and go to the IAM & Admin page.
Navigate to the IAM & Admin page and click on Service Accounts → Email. Then select the appropriate email address and click on the KEYS tab. From there, click ADD KEY to create a new key in JSON format for authentication. Be sure to keep this file secure.

Create a Pub/Sub Resource

Open the EMQX Dashboard, go to Rule Engine → Resources, click Create, and select GCP PubSub on the pop-up box.
Provide a Resource ID to identify the resource and paste the previously generated Service Account JSON into it. Before creating the resource, it is recommended to click on Test to ensure connectivity. If no issues are found during the connectivity test, click Confirm to complete the creation process.
Once the resource is successfully created, it will appear as available. The next step is to configure the rule for forwarding data to this resource.

Create a Rule to Forward Data to Pub/Sub

Click Rule → Create to enter the rule creation page. In the SQL field, input the following content.
```
SELECT
  *
FROM
  "t/#"
```
The SQL statement will be executed when a message is received that matches the t/# topic. The SELECT clause will then retrieve the client context and message payload, which will be used in the action.

For more SQL functions and use cases, please refer to the EMQX Rule Engine.
Click Add action at the bottom of the page, select the action type Data forward, choose Data to GCP Pubsub, and select the resource just created to forward the SQL output to Pub/Sub.
Input the pre-created Topic ID into the GCP PubSub Topic field, and construct the message that needs to be forwarded in the Payload template. By default, the payload template is empty and will forward the entire SQL output result. Once you have made the necessary changes, click on the Confirm button to complete the creation of the action.
After confirming everything is correct, click Create to complete rule creation.

Testing and Verification

In the previous steps, a rule was created to forward all messages that match the t/# topic to the my-iot-core topic in Pub/Sub.

Next, we will use the WebSocket tool to perform a verification test.

Go to the Tool → WebSocket page, establish a client connection, and publish several messages to the t/1 topic.
On the Rule page, click the Monitor icon in the table to view the statistics of the specified rule. You can see that data_to_gcp_pubsub has 4 records, indicating that the data has been successfully written to Pub/Sub.
On the details page of the Pub/Sub subscription, you can view the corresponding message received by the topic.

Summary

We have covered the steps and procedures to migrate IoT Core services to an EMQX Enterprise cluster within the user's GCP environment.

To learn more about what you can do with EMQX Enterprise, please visit EMQX Enterprise Documentations or contact us to book a demo.

Ingesting IoT Data From EMQX Enterprise to GCP Pub/Sub

Create and Manage Pub/Sub Topics