How to send data to Azure Event Hubs with Apache Kafka

In this tutorial you will learn how to send data to Event Hubs with Apache Kafka

Benefits of the Apache Kafka output connector:

• Directly stream device data from akenza to Event Hubs with Apache Kafka and subsequently process it in Azure services

• Easily process low-power device data e.g. LoRaWAN device data with Apache Kafka

• Following benefits:

  • Real-Time Handling Apache Kafka can handle real-time data pipelines at scale with zero downtime and zero data loss.

  • High-throughput: Capable of handling high-velocity and high-volume data.

  • Low Latency: Handling messages with very low latency of the range of milliseconds

  • Variety of IoT Use Cases Manages a variety of IoT use cases commonly required for a Data Lake. (log aggregation, web activity tracking etc.)

In this tutorial, you will learn, how to send data to Event Hubs using the Apache Kafka output connector. The following steps are explained in more detail:

1. Setup Event Hubs namespace

2. Obtain an Azure Event Hub connection string

3. Setup a Data Flow in akenza

4. Use Azure Data Explorer storage to store and view messages

1. Setup Event Hubs namespace

First, you will need to set up an Azure IoT Hub in order to successfully process data from akenza in Azure services.

Create an Event Hub resource

• Sign in to the Azure portal

• Create a new Event Hubs resource, by searching for it in All Services and then creating a new Event Hubs namespace

  • HubsAssign a resource group by choosing an existing group or creating a new one and fill the IoT hub name and region fields; choose Public access in the networking tab to select a tier that fits your use case in the Management tab (use the free tier for testing)

• Click review and create to provision the IoT hub resource

Add a shared access policy

• Navigate to your Event Hub resource and select Shared access policies in the Settings section.

• Select + Add shared access policy, give a name, and check all boxes for the permissions (Manage, Send, Listen)

Adding a shared access policy

2. Obtain an Azure Event Hub connection string

Once the policy is created, obtain the Primary Connection String, by clicking the policy and copying the connection string. The connection string will be used as a JAAS password to create the connector in akenza.

Obtaining the connection string

3. Setup a Data Flow in akenza

In akenza, create a new Data Flow with Kafka as Output Connector. Define a Connector Name and insert the Kafka brokers, Kafka topic, JAAS username, JAAS password (which is previously obtained Connection String) obtained from the shared access policy. Save the Data Flow accordingly.

Apache Kafka template

Create a new device using the Data Flow and start sending data.

Important: Each Kafka topic will create a new instance in event hubs namespace resource.

The Event Hubs namespace resource can be found by navigating to Event Hub resource and then on the left tile under Entities choose Event Hubs. It will list all instances in that Event Hubs namespace.

4. Use Azure Data Explorer storage to store and view messages

One way to monitor incoming uplinks on Even Hubs is to create a database, more info about the setup can be found here.