How to send data to AWS Kinesis

In this tutorial you will learn how to send data to AWS Kinesis

AWS Kinesis Output Connector

Benefits of the AWS Kinesis output connector:

• Directly stream device data from akenza to AWS Kinesis and subsequently process it in AWS Kinesis

• Easily process low-power device data e.g. LoRaWAN device data in a AWS Kinesis

In this tutorial, you will learn, how to send data to AWS Kinesis. The following steps are explained in more detail:

1. Setup an AWS Kinesis stream

2. Setup a Data Flow in akenza

3. Monitor events

4. Use of further AWS services

5. Useful Links

1. Setup an AWS Kinesis stream

First, you will need to set up AWS kinesis in order to successfully process data from akenza in AWS services.

• Sign in to the Amazon portal

• Create a new AWS Kinesis stream resource, by searching for Kinesis and creating a data stream

  • copy the stream ARN for later use

• Create a new API user by searching for IAM

  • create a new policy with access to the Kinesis service

  • Expand the tab Write and select

    • PutRecord

    • PutRecords

    • ListShards

  • Add the ARN of the newly created kinesis stream as resources

  • create a new user which uses access key as the credentials type and the previously created policy

Kinesis Policy

2. Setup a Data Flow in akenza

In akenza, create a new Data Flow with AWS Kinesis as Output Connector. Define a Connector Name, enter the IAM user id and key as well as the Kinesis data stream ARN. Save the connector and Data Flow accordingly.

New AWS Kinesis Data Flow

Create a new device using the Data Flow and start sending data.

Optionally: The custom payload can be selected for making use of templated payload in the desired structure. If the custom payload is not selected, the whole data sample will be forwarded.

3. Monitor Events

In order to monitor uplinks that are sent to AWS Kinesis, you can use AWS cli. To check if the uplinks arrived execute as a first step the following command and copy the return value. This value is the shard iterator used in the following commands.

aws kinesis get-shard-iterator --shard-id shardId-000000000000 --shard-iterator-type LATEST --stream-name {streamName} --query 'ShardIterator'

Send data to your device and execute the following command accordingly.

aws kinesis get-records --shard-iterator {shardIterator}

You should now see your base64 encoded records as well as the next shard iterator. If there is no data repeat the previous command with the next shard iterator.

4. Use of further AWS services

Once a device is connected to the kinesis, device data can be further used on various AWS services:

Possible services are:

• AWS kinesis firehouse

  Real-time data stream processing

• AWS S3

  Object storage is built to retrieve any amount of data from anywhere

• AWS Lambda

  Run code without thinking about servers or clusters

5. Useful Links

https://eu-central-1.console.aws.amazon.com/kinesis/home?region=eu-central-1#/homeeu-central-1.console.aws.amazon.com

Amazon S3 - Cloud Object Storage - AWSAmazon Web Services, Inc.

Serverless Function, FaaS Serverless - AWS Lambda - AWSAmazon Web Services, Inc.