Revision as of 15:30, 11 March 2018 (view source) Allen.chao (talk | contribs) (add concept of ingesting data into azure) ← Older edit		Revision as of 16:17, 11 March 2018 (view source) Allen.chao (talk | contribs) (add azure data storage) Newer edit →
Line 3:		Line 3:
	In this section, we will introduce steps to insert data from an Azure IoT hub to different Azure data stores. We assume that device messages are already received by an Azure IoT hub. If not, you can refer to Protocol converter with Node-Red section, which is designed to ingest IoT data from edge devices to Azure IoT hubs.		In this section, we will introduce steps to insert data from an Azure IoT hub to different Azure data stores. We assume that device messages are already received by an Azure IoT hub. If not, you can refer to Protocol converter with Node-Red section, which is designed to ingest IoT data from edge devices to Azure IoT hubs.
−	When an Azure IoT hub receives messages, we use Azure stream analytics job to dispatch the received data to other Azure services. In this case, we will ingest data into Azure CosmosDB and Azure blob storage.	+	When an Azure IoT hub receives messages, we use Azure stream analytics job to dispatch the received data to other Azure services. In this case, we will ingest data into Azure CosmosDB and Azure blob storage. All created Azure resources are stored in single Azure resource group named "azurestart-rg".
	[[File:IoTDataIngestion-az.png|center|750px|iot data ingest flow]]		[[File:IoTDataIngestion-az.png|center|750px|iot data ingest flow]]
		+
		+	== Create storage to save ingested IoT Data ==
		+
		+	Step 1: Login to Azure portal and create an Azure CosmosDB
		+
		+	* Search “azure cosmos db” and create new one
		+
		+	[[File:azure-cosmos-new.png|center|1000px|create cosmos DB]]
		+
		+	* ID: azurestart-cosmos
		+	* API: select “SQL”
		+	* Resource Group: click “Use existing” and select “azurestart-rg”
		+
		+	[[File:azure-cosmos-detail.png|center|450px|cosmos DB settings]]
		+
		+	Step 2: Create an Azure Blob Storage
		+
		+	* Search “storage account” and create new one
		+
		+	[[File:azure-blob-new.png|center|1000px|create blob storage]]
		+
		+	* Name: azurestartblob
		+	* Secure transfer required: Disabled
		+	* Resource Group: click “Use existing” and select “azurestart-rg”
		+
		+	[[File:azure-blob-detail.png|center|450px|blob storage settings]]
	= AWS =		= AWS =

---

Revision as of 16:17, 11 March 2018

Azure

In this section, we will introduce steps to insert data from an Azure IoT hub to different Azure data stores. We assume that device messages are already received by an Azure IoT hub. If not, you can refer to Protocol converter with Node-Red section, which is designed to ingest IoT data from edge devices to Azure IoT hubs.

When an Azure IoT hub receives messages, we use Azure stream analytics job to dispatch the received data to other Azure services. In this case, we will ingest data into Azure CosmosDB and Azure blob storage. All created Azure resources are stored in single Azure resource group named "azurestart-rg".

Create storage to save ingested IoT Data

Step 1: Login to Azure portal and create an Azure CosmosDB

• Search “azure cosmos db” and create new one

• ID: azurestart-cosmos
• API: select “SQL”
• Resource Group: click “Use existing” and select “azurestart-rg”

Step 2: Create an Azure Blob Storage

• Search “storage account” and create new one

• Name: azurestartblob
• Secure transfer required: Disabled
• Resource Group: click “Use existing” and select “azurestart-rg”

AWS

In this section, you can get experience about AWS IoT rule engine to insert data to different AWS storage. You can refer to Protocol converter with Node-Red which is designed to ingest IoT data to AWS IoT.

When you receive IoT data from AWS IoT. You can use rule engine to connect to another AWS service. In this case we will send IoT data to AWS S3 and DynamoDB

Step 1. Go to the AWS IoT console and click Act and click create rule.

Step 2. Enter {your rule name} and {description}

Step 3. Configure the rule as follows: Attribute : * Topic Filter: {your AWS IoT publish Topic}. The topic which used in protocol converter is “protocol-conn/{Device Name}/{Handler Name}”. In this case, we use wildcard # to get all message More Topic information can be found at

Step 4. Click Add action to store message in S3 bucket. Select “Store message in an Amazon S3 bucket”→click “configure action” more rule engine information can be found at

Step 5. In configure action. You need to choose a S3 bucket. If you don’t have any one, you can click “Create a new resource” to create one. In this case, we store data to json format and assort by data and hour. You can use SQL wildcard parse_time () and timestamp() to assort store folder and using newuuid() as filename.

${parse_time("yyyy-MM-dd", timestamp())}/${parse_time("HH", timestamp())}/${newuuid()}.json more AWS IoT SQL Reference information can be fount at

you need to choose a role which has permission can access AWS S3. After setting click update. Step 6. Click “add action” to connect DynamoDB Step 7. Select “Insert a message into a DynamoDB Table”→click “configure action” Step 8. Choose a DynamoDB table. If don't have DynamoDB table, you can click “Create a new resource” to create a table. In this case. We create a DynamoDB table which called ”IoT_Data_Ingestion” and it’s primary key is “uuid”. Step 9. Choose “IoT_Data_Ingestion” and enter Hash key value and Write message data to this column. Hash key value ${newuuid()} Write message data to this column payload you need to choose a role which has permission can access AWS DynamoDB. After setting click update. Step 10. After finishing setting click “create rule” Step 11. Now you can publish your message and check S3 and DynamoDB.