IoT Data Ingestion - Revision history

Angie.huang: Updates

2018-03-14T15:43:42Z

Updates

Angie.huang at 09:07, 14 March 2018

2018-03-14T09:07:43Z

Allen.chao: add external links

2018-03-14T06:07:13Z

add external links

Allen.chao: add azure storage verification

2018-03-11T19:52:33Z

add azure storage verification

Allen.chao: add azure stream analytics query

2018-03-11T19:07:39Z

add azure stream analytics query

Allen.chao: add azure stream analytics ouputs

2018-03-11T18:52:55Z

add azure stream analytics ouputs

Allen.chao: add azure stream analytics input

2018-03-11T15:59:47Z

add azure stream analytics input

Allen.chao: add azure stream analytics

2018-03-11T15:46:02Z

add azure stream analytics

Allen.chao at 14:56, 11 March 2018

2018-03-11T14:56:53Z

Allen.chao: add azure data storage

2018-03-11T14:17:11Z

add azure data storage

@@ Line 1: / Line 1: @@
 = Azure =
-In this section, we will introduce steps to insert data from an Azure IoT Hub to different Azure data stores. We assume that device messages are already received by an Azure IoT Hub. If not, you can refer to [http://ess-wiki.advantech.com.tw/view/Protocol_Connector#Azure Protocol converter with Node-Red section], which is designed to ingest IoT data from edge devices to the Azure IoT Hub.
+In this section, we will introduce steps to insert data from an Azure IoT Hub to different Azure data stores. We assume that device messages are already received by an Azure IoT Hub. If not, you can refer to [http://ess-wiki.advantech.com.tw/view/Protocol_Connector#Azure Protocol Converter with Node-RED&nbsp;section], which is designed to ingest IoT data from edge devices to the Azure IoT Hub.
 When an Azure IoT Hub receives messages, we use Azure Stream Analytics job to dispatch the received data to other Azure services. In this case, we will ingest data into Azure CosmosDB and Azure Blob Storage.
@@ Line 169: / Line 169: @@
 = AWS =
-In this section, you can get experience about AWS IoT rule engine to insert data to different AWS storage. You can refer to Protocol converter with Node-Red which is designed to ingest IoT data to AWS IoT.
+In this section, you can get experience using the AWS IoT Rule Engine to insert data to different AWS storage. You can refer to refer to [http://ess-wiki.advantech.com.tw/view/Protocol_Connector#AWS Protocol Converter with Node-RED&nbsp;section]&nbsp;&nbsp;which is designed to ingest IoT data to AWS IoT.
-When you receive IoT data from AWS IoT. You can use rule engine to connect to another AWS service. In this case we will send IoT data to AWS S3 and DynamoDB
   [[File:IoTDataIngestion.png|center|IoTDataIngestion.png]]
-Step 1. Go to the AWS IoT console and click Act and click create rule.
+Step 1. Go to the AWS IoT console and click '''"Act"''' and click '''"Create a rule"'''.
 [[File:2018-03-08 153047.png|center|1000px|2018-03-08_153047.png]]
@@ Line 183: / Line 185: @@
 [[File:2018-03-09 134506.png|center|1000px|2018-03-09_134506.png]]
-Step 3. Configure the rule as follows: Attribute&nbsp;: * Topic Filter: {your AWS IoT publish Topic}. The topic which used in protocol converter is “protocol-conn/{Device Name}/{Handler Name}”. In this case, we use wildcard # to get all message More Topic information can be found at
+Step 3. Configure the rule as follows: Attribute&nbsp;: * Topic Filter: {your AWS IoT publish Topic}. The topic which used in Protocol Converter is '''“protocol-conn/{Device Name}/{Handler Name}”'''. In this case, we use wildcard # to get all message More Topic information can be found at
 [[File:2018-03-09 150248.png|center|1000px|2018-03-09_150248.png]]
-Step 4. Click Add action to store message in S3 bucket. Select “Store message in an Amazon S3 bucket”→click “configure action” more rule engine information can be found at [[File:2018-03-09 150946.png|center|1000px|2018-03-09_150946.png]]
+Step 4. Click '''"Add action" '''to store message in S3 bucket. Select '''“Store messages in an Amazon S3 bucket”'''→click'''“Configure action”''' more rule engine information can be found at [[File:2018-03-09 150946.png|center|1000px|2018-03-09_150946.png]]
-Step 5. In configure action. You need to choose a S3 bucket. If you don’t have any one, you can click “Create a new resource” to create one. In this case, we store data to json format and assort by data and hour. You can use SQL wildcard parse_time () and timestamp() to assort store folder and using newuuid() as filename.
+${parse_time("yyyy-MM-dd", timestamp())}/${parse_time("HH", timestamp())}/${newuuid()}.json
-${parse_time("yyyy-MM-dd", timestamp())}/${parse_time("HH", timestamp())}/${newuuid()}.json more AWS IoT SQL Reference information can be fount at
+More AWS IoT SQL Reference information can be found&nbsp;at&nbsp; &nbsp;&nbsp;
-you need to choose a role which has permission can access AWS S3. After setting click update. [[File:2018-03-09 151301.png|center|1000px|2018-03-09_151301.png]] Step 6. Click “add action” to connect DynamoDB [[File:2018-03-09 155336.png|center|1002px|2018-03-09_155336.png]] Step 7. Select “Insert a message into a DynamoDB Table”→click “configure action” [[File:2018-03-09 160050.png|center|1003px|2018-03-09_160050.png]] Step 8. Choose a DynamoDB table. If don't have DynamoDB table, you can click “Create a new resource” to create a table. In this case. We create a DynamoDB table which called ”IoT_Data_Ingestion” and it’s primary key is “uuid”. Step 9. Choose “IoT_Data_Ingestion” and enter Hash key value and Write message data to this column. Hash key value ${newuuid()} Write message data to this column payload
+You need to choose a role which has permission can access AWS S3. After finishing the&nbsp;settings, click "Update". [[File:2018-03-09 151301.png|center|1000px|2018-03-09_151301.png]] Step 6. Click'''“add action”''' to connect DynamoDB [[File:2018-03-09 155336.png|center|1002px|2018-03-09_155336.png]] Step 7. Select'''“Insert a message into a DynamoDB Table”'''→click'''“configure action”''' [[File:2018-03-09 160050.png|center|1003px|2018-03-09_160050.png]] Step 8. Choose a DynamoDB table. If don't have DynamoDB table, you can click'''“Create a new resource” '''to create a table. In this case. We create a DynamoDB table&nbsp;called '''”IoT_Data_Ingestion”'''&nbsp;with a&nbsp;primary key '''&nbsp;“uuid”.''' Step 9. Choose '''“IoT_Data_Ingestion” '''and enter Hash key value and Write message data to this column. Hash key value '''${newuuid()} '''Write message data to this column payload
-you need to choose a role which has permission can access AWS DynamoDB. After setting click update. [[File:2018-03-09 160213.png|center|1004px|2018-03-09_160213.png]] Step 10. After finishing setting click “create rule” [[File:2018-03-09 160808.png|center|1005px|2018-03-09_160808.png]] Step 11. Now you can publish your message and check S3 and DynamoDB. [[File:2018-03-09 161209.png|center|1006px|2018-03-09_161209.png]] [[File:2018-03-09 161526.png|center|1007px|2018-03-09_161526.png]]
+you need to choose a role which has permission can access AWS DynamoDB. After setting click update. [[File:2018-03-09 160213.png|center|1004px|2018-03-09_160213.png]] Step 10. After finishing setting click '''“Create rule”''' [[File:2018-03-09 160808.png|center|1005px|2018-03-09_160808.png]] Step 11. Now you can publish your message and check S3 and DynamoDB. [[File:2018-03-09 161209.png|center|1006px|2018-03-09_161209.png]] [[File:2018-03-09 161526.png|center|1007px|2018-03-09_161526.png]]

@@ Line 1: / Line 1: @@
 = Azure =
-In this section, we will introduce steps to insert data from an Azure IoT hub to different Azure data stores. We assume that device messages are already received by an Azure IoT hub. If not, you can refer to [http://ess-wiki.advantech.com.tw/view/Protocol_Connector#Azure Protocol converter with Node-Red section], which is designed to ingest IoT data from edge devices to Azure IoT hubs.
+In this section, we will introduce steps to insert data from an Azure IoT Hub to different Azure data stores. We assume that device messages are already received by an Azure IoT Hub. If not, you can refer to [http://ess-wiki.advantech.com.tw/view/Protocol_Connector#Azure Protocol converter with Node-Red section], which is designed to ingest IoT data from edge devices to the Azure IoT Hub.
-When an Azure IoT hub receives messages, we use Azure stream analytics job to dispatch the received data to other Azure services. In this case, we will ingest data into Azure CosmosDB and Azure blob storage.
+When an Azure IoT Hub receives messages, we use Azure Stream Analytics job to dispatch the received data to other Azure services. In this case, we will ingest data into Azure CosmosDB and Azure Blob Storage.
-The IoT hub we're using to receive messages here is named '''"azurestart-hub"'''. All Azure resources used in this section are stored in single Azure resource group named '''"azurestart-rg"'''.
+The IoT Hub we're using to receive messages here is named '''"azurestart-hub"'''. All Azure resources used in this section are stored in single Azure resource group named '''"azurestart-rg"'''.
 [[File:IoTDataIngestion-az.png|center|750px|iot data ingest flow]]

@@ Line 1: / Line 1: @@
 = Azure =
-In this section, we will introduce steps to insert data from an Azure IoT hub to different Azure data stores. We assume that device messages are already received by an Azure IoT hub. If not, you can refer to Protocol converter with Node-Red section, which is designed to ingest IoT data from edge devices to Azure IoT hubs.
+In this section, we will introduce steps to insert data from an Azure IoT hub to different Azure data stores. We assume that device messages are already received by an Azure IoT hub. If not, you can refer to [http://ess-wiki.advantech.com.tw/view/Protocol_Connector#Azure Protocol converter with Node-Red section], which is designed to ingest IoT data from edge devices to Azure IoT hubs.
 When an Azure IoT hub receives messages, we use Azure stream analytics job to dispatch the received data to other Azure services. In this case, we will ingest data into Azure CosmosDB and Azure blob storage.
-The IoT hub we're using to receive messages is named '''"azurestart-hub"'''. All Azure resources used in this section are stored in single Azure resource group named '''"azurestart-rg"'''.
+The IoT hub we're using to receive messages here is named '''"azurestart-hub"'''. All Azure resources used in this section are stored in single Azure resource group named '''"azurestart-rg"'''.
 [[File:IoTDataIngestion-az.png|center|750px|iot data ingest flow]]
@@ Line 11: / Line 11: @@
 == Create storage to save ingested IoT data ==
-Step 1: Login to Azure portal and create an Azure CosmosDB
+Step 1: Login to [https://portal.azure.com Azure portal] and create an Azure CosmosDB
 *Search "azure cosmos db" and create new one
-[[File:azure-cosmos-new.png|center|1000px|create cosmos DB]]
+[[File:Azure-cosmos-new.png|center|1000px|create cosmos DB]]
 *ID: '''azurestart-cosmos'''
@@ Line 21: / Line 21: @@
 *Resource Group: click '''"Use existing"''' and select '''"azurestart-rg"'''
-[[File:azure-cosmos-detail.png|center|450px|cosmos DB settings]]
+[[File:Azure-cosmos-detail.png|center|450px|cosmos DB settings]]
 Step 2: Create an Azure Blob Storage
@@ Line 53: / Line 53: @@
 *Open "azurestart-sa" created at last step: '''Resource groups''' → '''azurestart-rg''' → '''azurestart-sa'''
-[[File:azure-sa.png|center|1000px|open sa]]
+[[File:Azure-sa.png|center|1000px|open sa]]
 Step 2.1: Add IoT hub input
@@ Line 59: / Line 59: @@
 *Click '''"Inputs"'''
-[[File:azure-sa-input.png|center|1000px|open sa input]]
+[[File:Azure-sa-input.png|center|1000px|open sa input]]
 *'''"Add stream input"''' → '''"IoT Hub"'''
-[[File:azure-sa-input-add-iothub.png|center|1000px|sa add input: iot hub]]
+[[File:Azure-sa-input-add-iothub.png|center|1000px|sa add input: iot hub]]
 *Input alias: '''iothub'''
@@ Line 121: / Line 121: @@
 *Click '''"Edit query"'''
-[[File:azure-sa-query.png|center|1000px|open sa query]]
+[[File:Azure-sa-query.png|center|1000px|open sa query]]
 *Copy following stream analytics query statements, paste to the text area and click '''"Save"'''
@@ Line 128: / Line 128: @@
 </pre>
-[[File:azure-sa-query-detail.png|center|1000px|sa query detail]]
+[[File:Azure-sa-query-detail.png|center|1000px|sa query detail]]
 *Back to '''"azurestart-sa"''' page, click '''"Start'''", then select '''"Now"''' for job output start time.
 *Click '''"start"''' to run stream analytics job, it may take a few seconds.
-[[File:azure-sa-start.png|center|1000px|sa start]]
+[[File:Azure-sa-start.png|center|1000px|sa start]]
 == Verify the saved data in Azure storage ==
@@ Line 141: / Line 141: @@
 Step 1: Confirm data in Azure blob storage
-* Go to "azurestart-blob": '''Resource groups''' → '''azurestart-rg''' → '''azurestart-blob'''
+*Go to "azurestart-blob": '''Resource groups''' → '''azurestart-rg''' → '''azurestart-blob'''
-[[File:azure-blob-1.png|center|1000px|blob storage list-1]]
+[[File:Azure-blob-1.png|center|1000px|blob storage list-1]]
-* '''"Containers"''' → '''"iot-data-ingestion"''', we can see a folder in container is named as pattern "YYYY-MM-DD". The ingested data will be saved in files under folder {YYYY-MM-DD}/{HH}
+*'''"Containers"''' → '''"iot-data-ingestion"''', we can see a folder in container is named as pattern "YYYY-MM-DD". The ingested data will be saved in files under folder {YYYY-MM-DD}/{HH}
-[[File:azure-blob-2.png|center|1000px|blob storage list-2]]
+[[File:Azure-blob-2.png|center|1000px|blob storage list-2]]
-* Click the context menu icon '''"…"''' → '''"Edit"'''
+*Click the context menu icon '''"…"''' → '''"Edit"'''
-[[File:azure-blob-display-1.png|center|1000px|blob storage display-1]]
+[[File:Azure-blob-display-1.png|center|1000px|blob storage display-1]]
-* We can see the received messages are saved line by line in the selected .json file
+*We can see the received messages are saved line by line in the selected .json file
-[[File:azure-blob-display-2.png|center|1000px|blob storage display-2]]
+[[File:Azure-blob-display-2.png|center|1000px|blob storage display-2]]
 Step 2: Confirm data in Azure cosmos DB
-* Go to "azurestart-cosmos": '''Resource groups''' → '''azurestart-rg''' → '''azurestart-cosmos'''
+*Go to "azurestart-cosmos": '''Resource groups''' → '''azurestart-rg''' → '''azurestart-cosmos'''
-[[File:azure-cosmos.png|center|1000px|azure cosmos]]
+[[File:Azure-cosmos.png|center|1000px|azure cosmos]]
-* '''Data Explorer''' → '''historic-data''' → '''Documents''': Each document in Cosmos DB represents one message received by Azure IoT hub. Click “id” to show the message content.
+*'''Data Explorer''' → '''historic-data''' → '''Documents''': Each document in Cosmos DB represents one message received by Azure IoT hub. Click “id” to show the message content.
-[[File:azure-cosmos-display.png|center|1000px|azure cosmos display]]
+[[File:Azure-cosmos-display.png|center|1000px|azure cosmos display]]
 = AWS =

@@ Line 15: / Line 15: @@
 *Search "azure cosmos db" and create new one
-[[File:Azure-cosmos-new.png|center|1000px|create cosmos DB]]
+[[File:azure-cosmos-new.png|center|1000px|create cosmos DB]]
 *ID: '''azurestart-cosmos'''
@@ Line 21: / Line 21: @@
 *Resource Group: click '''"Use existing"''' and select '''"azurestart-rg"'''
-[[File:Azure-cosmos-detail.png|center|450px|cosmos DB settings]]
+[[File:azure-cosmos-detail.png|center|450px|cosmos DB settings]]
 Step 2: Create an Azure Blob Storage
@@ Line 53: / Line 53: @@
 *Open "azurestart-sa" created at last step: '''Resource groups''' → '''azurestart-rg''' → '''azurestart-sa'''
-[[File:Azure-sa.png|center|1000px|open sa]]
+[[File:azure-sa.png|center|1000px|open sa]]
 Step 2.1: Add IoT hub input
@@ Line 59: / Line 59: @@
 *Click '''"Inputs"'''
-[[File:Azure-sa-input.png|center|1000px|open sa input]]
+[[File:azure-sa-input.png|center|1000px|open sa input]]
 *'''"Add stream input"''' → '''"IoT Hub"'''
-[[File:Azure-sa-input-add-iothub.png|center|1000px|sa add input: iot hub]]
+[[File:azure-sa-input-add-iothub.png|center|1000px|sa add input: iot hub]]
 *Input alias: '''iothub'''
@@ Line 121: / Line 121: @@
 *Click '''"Edit query"'''
-[[File:Azure-sa-query.png|center|1000px|open sa query]]
+[[File:azure-sa-query.png|center|1000px|open sa query]]
-* Copy following stream analytics query statements, paste to the text area and click '''"Save"'''
+*Copy following stream analytics query statements, paste to the text area and click '''"Save"'''
+<pre>        SELECT * INTO [blob] FROM [iothub]
          SELECT * INTO [cosmosdb] FROM [iothub]
 </pre>
-[[File:Azure-sa-query-detail.png|center|1000px|sa query detail]]
+[[File:azure-sa-query-detail.png|center|1000px|sa query detail]]
-* Back to '''"azurestart-sa"''' page, click '''"Start'''", then select '''"Now"''' for job output start time.
+* '''Data Explorer''' → '''historic-data''' → '''Documents''': Each document in Cosmos DB represents one message received by Azure IoT hub. Click “id” to show the message content.
-[[File:Azure-sa-start.png|center|1000px|sa start]]
+[[File:azure-cosmos-display.png|center|1000px|azure cosmos display]]
 = AWS =

@@ Line 86: / Line 86: @@
 [[File:Azure-sa-output.png|center|1000px|open sa ouput]]
-* '''"Add"''' → '''"Blob storage"'''
+*'''"Add"''' → '''"Blob storage"'''
 [[File:Azure-sa-output-add-blob.png|center|1000px|sa add output: blob]]
-* Output alias: '''blob'''
+*Output alias: '''blob'''
-* Choose '''"Select Blob storage from your subscriptions"'''
+*Choose '''"Select Blob storage from your subscriptions"'''
-* Storage account: '''azurestartblob'''
+*Storage account: '''azurestartblob'''
-* Container: choose '''"Create new"''' and input '''"iot-data-ingestion"'''
+*Container: choose '''"Create new"''' and input '''"iot-data-ingestion"'''
-* Path pattern: '''{date}/{time}'''
+*Path pattern: '''{date}/{time}'''
-* Date format: '''YYYY-MM-DD'''
+*Date format: '''YYYY-MM-DD'''
 [[File:Azure-sa-output-add-blob-detail.png|center|450px|sa output detail: blob]]
@@ Line 101: / Line 101: @@
 Step 3.2: Add cosmos DB output
-* '''"Add"''' → '''"Cosmos DB"'''
+*'''"Add"''' → '''"Cosmos DB"'''
 [[File:Azure-sa-output-add-cosmosdb.png|center|1000px|sa add output: cosmos DB]]
-* Output alias: '''cosmosdb'''
+*Output alias: '''cosmosdb'''
-* Choose '''"Select Cosmos DB from your subscriptions"'''
+*Choose '''"Select Cosmos DB from your subscriptions"'''
-* Account id: '''azurestart-cosmos'''
+*Account id: '''azurestart-cosmos'''
-* Database: choose '''"Create new"''' and input '''"iot-data-ingestion"'''
+*Database: choose '''"Create new"''' and input '''"iot-data-ingestion"'''
-* Collection name pattern: '''historic-data'''
+*Collection name pattern: '''historic-data'''
 [[File:Azure-sa-output-add-cosmosdb-detail.png|center|450px|sa output detail: cosmos DB]]
 = AWS =

@@ Line 54: / Line 54: @@
 [[File:Azure-sa.png|center|1000px|open sa]]
 *Click '''"Inputs"'''
@@ Line 63: / Line 65: @@
 [[File:Azure-sa-input-add-iothub.png|center|1000px|sa add input: iot hub]]
-* Input alias: '''iothub'''
+*Input alias: '''iothub'''
-* Choose '''"Select IoT Hub from your subscriptions"'''
+*Choose '''"Select IoT Hub from your subscriptions"'''
-* IoT hub: '''azurestart-hub'''
+*IoT hub: '''azurestart-hub'''
-* Endpoint: '''Messaging'''
+*Endpoint: '''Messaging'''
-* Shared access policy name: '''iothubowner'''
+*Shared access policy name: '''iothubowner'''
-* Encoding: '''UTF-8'''
+*Encoding: '''UTF-8'''
-[[File:azure-sa-input-add-iothub-detail.png|center|450px|sa input detail: iot hub]]
+[[File:Azure-sa-output-add-cosmosdb-detail.png|center|450px|sa output detail: cosmos DB]]
 = AWS =

@@ Line 5: / Line 5: @@
 When an Azure IoT hub receives messages, we use Azure stream analytics job to dispatch the received data to other Azure services. In this case, we will ingest data into Azure CosmosDB and Azure blob storage.
 The IoT hub we're using to receive messages is named '''"azurestart-hub"'''. All Azure resources used in this section are stored in single Azure resource group named '''"azurestart-rg"'''.
 [[File:IoTDataIngestion-az.png|center|750px|iot data ingest flow]]
@@ Line 15: / Line 15: @@
 *Search "azure cosmos db" and create new one
-[[File:azure-cosmos-new.png|center|1000px|create cosmos DB]]
+[[File:Azure-cosmos-new.png|center|1000px|create cosmos DB]]
 *ID: '''azurestart-cosmos'''
@@ Line 21: / Line 21: @@
 *Resource Group: click '''"Use existing"''' and select '''"azurestart-rg"'''
-[[File:azure-cosmos-detail.png|center|450px|cosmos DB settings]]
+[[File:Azure-cosmos-detail.png|center|450px|cosmos DB settings]]
 Step 2: Create an Azure Blob Storage
@@ Line 27: / Line 27: @@
 *Search “storage account” and create new one
-[[File:azure-blob-new.png|center|1000px|create blob storage]]
+[[File:Azure-blob-new.png|center|1000px|create blob storage]]
 *Name: '''azurestartblob'''
@@ Line 33: / Line 33: @@
 *Resource Group: click '''"Use existing"''' and select '''"azurestart-rg"'''
-[[File:azure-blob-detail.png|center|450px|blob storage settings]]
+[[File:Azure-blob-detail.png|center|450px|blob storage settings]]
 == Connect Azure IoT hub to Azure storage ==
@@ Line 39: / Line 39: @@
 Step 1: Login to Azure portal and create a Stream Analytics job
-* Search "stream analytics" and create new one
+*Search "stream analytics" and create new one
-[[File:azure-sa-new.png|center|1000px|create sa]]
+[[File:Azure-sa-new.png|center|1000px|create sa]]
-* Job name: '''azurestart-sa''' (for example)
+*Job name: '''azurestart-sa''' (for example)
-* Resource Group: click '''"Use existing"''' and select '''"azurestart-rg"'''
+*Resource Group: click '''"Use existing"''' and select '''"azurestart-rg"'''
-* Hosting environment: '''Cloud'''
+*Hosting environment: '''Cloud'''
-[[File:azure-sa-detail.png|center|450px|sa settings]]
+[[File:Azure-sa-detail.png|center|450px|sa settings]]
 Step 2: Set input of stream analytics job
-* Open "azurestart-sa" created at last step: '''Resource groups''' → '''azurestart-rg''' → '''azurestart-sa'''
+*Open "azurestart-sa" created at last step: '''Resource groups''' → '''azurestart-rg''' → '''azurestart-sa'''
-[[File:azure-sa.png|center|1000px|open sa]]
+[[File:Azure-sa-input.png|center|1000px|open sa input]]
-* Click '''"Inputs"'''
+*'''"Add stream input"''' → '''"IoT Hub"'''
-[[File:azure-sa-input.png|center|1000px|open sa input]]
+[[File:Azure-sa-input-add-iothub.png|center|1000px|sa add input: iot hub]]
-* '''"Add stream input"''' → '''"IoT Hub"'''
+* Input alias: '''iothub'''
-[[File:azure-sa-input-add-iothub.png|center|1000px|sa add input: iot hub]]
+[[File:azure-sa-input-add-iothub-detail.png|center|450px|sa input detail: iot hub]]
 = AWS =