How to send data to AWS Kinesis

In this tutorial you will learn how to send data to AWS Kinesis

Benefits of the AWS Kinesis output connector:

  • Directly stream device data from akenza to AWS Kinesis and subsequently process it in AWS Kinesis

  • Easily process low-power device data e.g. LoRaWAN device data in a AWS Kinesis

In this tutorial, you will learn, how to send data to AWS Kinesis. The following steps are explained in more detail:

1. Setup an AWS Kinesis stream

First, you will need to set up AWS kinesis in order to successfully process data from akenza in AWS services.

  • Sign in to the Amazon portal

  • Create a new AWS Kinesis stream resource, by searching for Kinesis and creating a data stream

    • copy the stream ARN for later use

  • Create a new API user by searching for IAM

    • create a new policy with access to the Kinesis service

    • Expand the tab Write and select PutRecords action and the newly created kinesis stream as resources

    • create a new user which uses access key as the credentials type and the previously created policy

2. Setup a Data Flow in akenza

In akenza, create a new Data Flow with AWS Kinesis as Output Connector. Define a Connector Name, enter the IAM user id and key as well as the Kinesis data stream ARN. Save the connector and Data Flow accordingly.

Create a new device using the Data Flow and start sending data.

Optionally: The custom payload can be selected for making use of templated payload in the desired structure. If the custom payload is not selected, the whole data sample will be forwarded.

3. Monitor Events

In order to monitor uplinks that are sent to AWS Kinesis, you can use AWS cli. To check if the uplinks arrived execute as a first step the following command and copy the return value. This value is the shard iterator used in the following commands.

aws kinesis get-shard-iterator --shard-id shardId-000000000000 --shard-iterator-type LATEST --stream-name {streamName} --query 'ShardIterator'

Send data to your device and execute the following command accordingly.

aws kinesis get-records --shard-iterator {shardIterator}

You should now see your base64 encoded records as well as the next shard iterator. If there is no data repeat the previous command with the next shard iterator.

4. Use of further AWS services

Once a device is connected to the kinesis, device data can be further used on various AWS services:

Possible services are:

  • AWS kinesis firehouse

    Real-time data stream processing

  • AWS S3

    Object storage is built to retrieve any amount of data from anywhere

  • AWS Lambda

    Run code without thinking about servers or clusters

Last updated