Difference between revisions of "WISE-PaaS/SRP Services"

Revision as of 05:39, 26 January 2017

OVERVIEW

API Management

Adopt Azure API Management/Open-source (WSO2) to integrate various SRP services API.

Common Service APIs/Lib

Basic Account/Device/Group/Role operation APIs, including Add/Edit/Delete/Search/Advance Search WISE-PaaS administration information (Product Support, version, capability…etc.)

Custom/SRP Service

RMM/IMM/OTA/SCADA self-service APIs

Common Data Workers

Including device basic information and sensor data from broker. (Common MQTT Topic) For structure info (device Info) store into PostgreSQL For non-structure (sensor Info) store into MongoDB/DocumentDB/...etc.

Custom/SRP Data Workers

Process RMM/IMM/OTA/SCADA self-MQTT topic message and store into self-database (RMM/IMM/OTASCADA DB)

Message Broker (RabbitMQ)

Adopt RabbitMQ as default cluster broker, and plugin MQTT as communication protocol.

Device Agent

Define common MQTT topic handshake message and custom/SRP topic format.

Database (SQL/NoSQL)

PostgreSQL as default relation database and including Common/RMM /SCADA/IMM/etc. MongoDB/DoucmentDB for sensor information collect.

COMMON DATABASE SCHEME

Concept & Design Rules

Device belong to multi-groups or not to any group.
Each account have non or multi-groups.
Each group must belong to someone account.
Group of group (1-3 Levels)
Each account must belong to someone role.

Common Database Scheme

Common Database Table Details

Table Account

Table Role

Table Group

Table Device

Table MatchGroup

Non-Structure Data (MongoDB/DocumentDB)

Default Collections

Event
WISE-PaaS platform event message

Format
{ "product": "RMM",

"type": "DEVICE",

"subtype": "DEVICE_DISCONNECTED",

"severity": "ERROR",

"message": "DEVICE_DISCONNECTED",

"extMsg": <JSONObject>,

"device_id": 2,

"device_name": "AC09",

"account_id": 2,

"account_name": "admin",

"ts": ISODate('2016-11-29T05:13:46.759Z') }
Handler (Plugin)
Handler (Plugin) report data (source data)

Capability
Handler capability structure

Heartbeat
WISE-PaaS device’s keepalive mechanism, ensure device connection state.

Sensor Collections (Dynamic generate and transfer to key/value)

SUSIControl

HDDMonitor

Modbus

SenHub

IoTGW

…

API MANAGEMENT (RESTFUL APIS)

Common REST APIs

Device APIs

/wise-paas/common/device
<POST>    – Add Device

/wise-paas/common/device/<id>
<PUT>      – Edit Device

<DELETE>        –    Delete Device

<GET>      –    Get device, account, group information.

/wise-paas/common/device/advSearch
<POST>    – Advanced search by each field.

Group APIs

/wise-paas/common/group
<POST>    – Add Group

/wise-paas/common/group/<id>
<PUT>      – Edit Group

<DELETE>        –    Delete Group

<GET>      –    Get group, account information

/wise-paas/common/group/advSearch
<POST>    – Advanced search by each field.

Account APIs

/wise-paas/common/account
<POST>    – Add Account

/wise-paas/common/account/<id>
<PUT>      – Edit Account

<DELETE>        –    Delete Account

<GET>      –    Get account and group information

/wise-paas/common/account/advSearch
<POST>    – Advanced search by each field.

/wise-paas/common/account/login
<POST>    – Account login, verify the password, and return access token.

Role APIs

/wise-paas/common/role
<POST>    – Add Role

/wise-paas/common/role/<id>
<PUT>      – Edit Role

<DELETE>        –    Delete Role

<GET>      –    Get role information

/wise-paas/common/role/advSearch
<POST>    – Advanced search by each field.

Data Management APIs

/wise-paas/common/data/history/<id>/<beginTS>/<endTS>/<handler>/<sensorId>
<GET>      – Get row data, based on device Id, time range, handler and sensor Id.

/wise-paas/common/data/realtime/<id>/<beginTS>/<endTS>/<handler>/<sensorId>
<GET>      – Get real-time device data, based on device Id, time range, handler and sensor Id.

/wise-paas/common/data/set/<id>
<POST>    – Set device data to target device.

/wise-paas/common/data/sensorId/<id>/<handler>
<GET>      – Get sensor Id, based on device Id and handler.

/wise-paas/common/data/startReport
<POST>    – Set device start to report data.

/wise-paas/common/data/stopReport
<POST>    – Set device stop to report data.

NoSQL APIs

/wise-paas/common/noSQL/collection
<POST>    – Add new collection into noSQL DB, index options…etc.

<Delete> – Delete collection on noSQL DB.

/wise-paas/common/noSQL/collection/data
<POST>    – Add data into collection

<PUT>      – Replace data on collection, via conditions.

/wise-paas/common/noSQL/advSearch
<POST>    – Advance search on collection and conditions.

Custom REST API Categories

RMM

/wise-paas/rmm/apiInfo

/wise-paas/rmm/dashboard

/wise-paas/rmm/kvm

/wise-paas/rmm/msgNotify

…

IMM

/wise-paas/imm/apiInfo

…

SCADA

/wise-paas/scada/apiInfo

…

OTA

/wise-paas/ota/apiInfo

…

DEVICE CONNECTIVITY

Adopt open source RabbitMQ on server side to support both AMQP and MQTT protocol.
Transportation supports MQTT, AMQP and HTTP (Websocket)
- Define the common communication APIs for our WISE Agent, we called WISE Connector APIs
- Implement the WISE Connector APIs with different communication protocol.
- WISE Agent can link different communication protocol at compile time.

Supports secured connection (TLS/SSL) with anonymous and authenticated account

Message Topics

MQTT Topic Define
- From Server to Agent:
  /wisepaas/<product>/<agentId>/<function>ack
- From Agent to Server:
  /wisepaas/<product>/<agentId>/<function>req

Preserved Topic for Common use
- Agent Registration Topic: /wisepaas/device/<agentId>/agentinfoack
- Will Message Topic: /wisepaas/device/<agentId>/willmessage Or /wisepaas/device/<serverId>/willmessage
- Command Topic: /wisepaas/device/<agentId>/agentactionreq
- Response Topic: /wisepaas/device/<agentId>/agentactionack
- Report Data Topic: /wisepaas/device/<agentId>/devinfoack
Custom Topic Sample
- Custom Topic for power ON/OFF:

Server send command with topic: /wisepaas/custom/<agentId>/powerctrlreq

Agent send response with topic: /wisepaas/custom/<agentId>/powerctrlack

Handshake Sequence

Normal sign in and sign out sequence:

Agent lost connection with will message sequence:

Server lost connection with will message sequence:

Data Format

Agent ID Format

Format define
Agent ID Format is based on UUID format, which is represented as 32 characters, displayed in six groups separated by hyphens, in the form 8-4-4-4-12 for a total of 36 characters. In this format the UUID could map to Agent ID directly.

For example:

The UUID is:

123e4567-e89b-12d3-a456-426655440000

Mapping to the Agent ID is:

123e4567-e89b-12d3-a456-426655440000

Generate Agent ID

If the target device has it own unique ID, but less the 32 characters, then we could follow the steps to generate the Unique Agent ID:

Retrieve Unique ID as key, for example: MAC address or IMEI.

The first 8 digits is predefined to identify the Unique ID source.
00000001: MAC

00000002: IMEU

00000004: Hardware System config, ex: "/sys/fsl_otp/HW_OCOTP_CFG0"

00000005: Custom generates serial number. The serial number may not be unique, so we add the product or brand name before the serial number to make sure the ID is unique.

Add 'Zero'(0) to extended to 24 lowercase hexadecimal digits as UUID format.
For example:

Unique ID with MAC address:

MAC:
000babbb7ef3
Agent ID:
00000001-0000-0000-0000-000babbb7ef3

Custom Serial number:

Serial number for Advantech ARK 1120 :
KSA0451097
Agent ID:
00000005-0000-0000-000a-rkksa0451097

Basic JSON Format

JSON (JavaScript Object Notation) is a lightweight data-interchange format.

JSON is built on two structures:
A collection of name/value pairs. In various languages, this is realized as an object, record, struct, dictionary, hash table, keyed list, or associative array.

An ordered list of values. In most languages, this is realized as an array, vector, list, or sequence.

JSON Forms:

Object: { string : value}

Array: [ value ]

Value: string
      number
      object
      array
       true
       false
       null

WISE-PaaS predefined string:
agentID: target device ID, value type is string.

handlerName: target device supported function block (Plugin or Handler) name, value type is string.

commCmd: the sub command ID in function block (Plugin or Handler), value type is unsigned integer.

Device Registration Format:

{
"agentID":"0123456789012",
"handlerName":"general",         //defined for device registration
"commCmd":1,                          //defined for device registration
"hostname":"PC001104",            //target device name
"sn":"14DAE996BE04",                //target device serial number
"mac":"14DAE996BE04",            //target device mac address
"version":"1.0.0.0",                     //client application version
"type":"IPC",                                //target device type
"product":"",
"manufacture":"",
"account":"anonymous",             //bind target device to a specific account
"password":"",                             //encrypt with DES and Base64
"status":1                                     //1: registration, 0: un-registration
}

Report Data Format:

{
“agentID”:”0123456789012”,
“handlerName”:”custom”,
“commCmd”:123,
“custom_object”:{                      //User defined report data object.
    “custom_data1”:”string_value”,
    “custom_data2”:123,
    “custom_data3”:false
}
}

IPSO Format (Application Framework)

The IPSO Application Framework makes use of IETF standards as building blocks for a simple and efficient RESTful design model for IP smart objects. The framework may be used over either HTTP or CoAP web transfer protocols. HTTP, REST, XML, JSON, COAP and other key components of web technology are powerful mechanisms in an Internet of Things application.

Note: The objective of the Alliance is not to define technologies, but to document the use of IP-based technologies defined at the standard organizations such as IETF with focus on support by the Alliance of various use cases. Function Sets: The framework is organized into groups of resource types called Function Sets. A Function Set has a recommended root path, under which its sub-resources are organized. Each Function Set is assigned a Resource Type parameter, therefore making it possible to discover it.

Function Set

Root Path

Resource Type

General Purpose IO

/gpio

gpio

Power

/pwr

pwr

Load Control

/load

load

Sensors

/sen

sen

Light Control

/lt

lt

Message

/msg

msg

Location

/loc

loc

Configuration

/cfg

cfg

IPSO Smart Object

IPSO Smart Object [2] Guidelines provide a common design pattern, an object model, which can effectively use the IETF CoAP protocol to provide high level interoperability between Smart Object devices and connected software applications on other devices and services
The common object model is based on the Lightweight M2M (LWM2M 1.0) specification from the Open Mobile Alliance. OMA LWM2M is a device management and service architecture specification based on IETF CoAP, and provides a simple and flexible object template (object model) for constrained device management.
The object model from OMA LWM2M is reused to define application level IPSO Smart Objects. This enables the OMA Name Authority (OMNA) to be used to register new objects, and enables existing LWM2M compliant device libraries and server software to be used as an infrastructure for IPSO Smart Objects

Media Types for Sensor Markup Language (SENML)
Semantics

SenML

JSON

Type

Description

Base Name

bn

String

This is a string that is prepended to the names found in the entries

Base Time

bt

Integer

A base time that is added to the time found in an entry

Base Units

bu

String

A base unit that is assumed for all entries, unless otherwise indicated

Version

ver

Number

Version number of media type format

Measurement or Parameters

e

Array

Array of values for sensor measurements or other generic parameters

Name

n

String

Name of the sensor or parameter

Units

u

String

Units for a measurement value

Value

v

Float

Value of the entry

String Value

sv

String

Boolean Value

bv

Boolean

Value Sum

s

Float

Integrated sum of the values over time

Time

t

Integer

Time when value was recorded

Update Time

ut

Integer

Update time. A time in seconds that represents the maximum time before this sensor will provide an updated reading for a measurement.

The Data Type of Sensor Data Type

Data Type (type)

SenML Field

b (boolean)

bv

s (string)

s

e (enum)

e

i (integer)

v

d (decimal)

v

h(hexadecimal)

s

o(octet-stream)

s

Advantech Sensor Semantics

SenML

JSON

Type

Description

Min Range Value

min

Float

The minimum value that can be measured by the sensor

Max Range Value

max

Float

The maximum value that can be measured by the sensor

Access Mode

asm

String

The access mode of the resource. Ex: read (r), write (w), read/write (rw)

Standard Format

st

String

The sensor format is which standard format

Health Status

Health

Integer

The health status of network or device. Range: -1 ~ 100
Good: > 80, Average: 60 ~ 80, Below Average: 40~60, Bad:0~40, -1: Off line or Fault

IPSO Report Data Format:

{
“agentID”:“0123456789012”,
“handlerName”:“custom”,
“commCmd”:123,
“custom_object”:{                      //User defined report data object in IPSO format.
    “bn”:“custom_object”,
    “e”:[
      {“n”:“ custom_data1”, “sv” ”:”string_value”},
      {“n”:“ custom_data2”, “v” ”:123},
      {“n”:“ custom_data3”, “bv” ”:false}
    ]
}
}

DEVOPS MANAGEMENT

Continuous Integration & Continuous Delivery

Version Control

GitHub offers all of the distributed version control and source code management (SCM) functionality of Git. It provides access control and several collaboration features such as bug tracking, feature requests, task management, and wikis for every project. GitHub Flow is a lightweight, branch-based workflow that supports teams and projects where deployments are made regularly.

Build and Test

Add Jenkins script to check out and compile the source code. Then execute on a Docker container to verify the application. The Jenkins will send the report mail to notify pass or not.

Jenkins helps to automate the non-human part of the whole software development process, with now common things like continuous integration, but by further empowering teams to implement the technical part of a Continuous Delivery.

Docker is an open-source project that automates the deployment of applications inside software containers.

Release & Deploy

Add Jenkins script to release and deploy to target VM or upload the install packet to release folder.

CUSTOM SERVICE INTEGRATION

Access Methods

In the common services, there are 3 mechanisms for access these information, ZMQ is the most convenient to communication information between different services. Second, the Java library have a lot’s of interface to access common resources, also you would via Restful APIs to integrate.

Internal Messaging(ZMQ)

In the ZMQ mechanism, we adopt the most robust with load balancing model which called “Paranoid Pirate Pattern”. Based on the ZMQ to handshake message, provide high performance, stability, and cross-languages (Java/C#/C++/C/GO/Python/PHP/…etc.)

The message is adopt JSON format and similar to Restful API, reuse the translator. Format define guide as below.

Example: Get Device Information
Request


   "request":{
      "ProductId":"<xxxx-xxx-yyyy-yyy>",
      "resource":"/wise-paas/common/device/<id>",
      "method":"GET"
   }
}

Reply


   "reply":{
      "did":1,
      "name":"ARK-1155",
      "agentId":"0000BAB374520",
      "isConnected":true,
      "coordinates":"25.069439,121.582485",
      "loginTS":"2017-1-10 12:30",
      "createTS":"2017-1-5 11:00"
   }
}

        Example: Add Account Information

Request


   "request":{
      "ProductId":"<xxxx-xxx-yyyy-yyy>",
      "resource":"/wise-paas/common/account",
      "method":"POST",
      "content":{
      }
   }
}

Reply
{
   "reply":{
      "result":true,
      "aid":2
   }
}

Java Library

[TBD]

Common Restful APIs

[TBD]

RMM/OTA/SCADA/IMM Integration

Through ZMQ/library/Common Restful API get account/group/device information.

Join/Merge custom DB information for advanced usage.

Scenario 1) RMM show more information about account

Due to common account’s fields not enough for more information on RMM, the RMM would have “AccountExt” table to extend and manage.

Step 1) Get common account information via ZMQ then use primary-key to link these tables.

Step 2) Merge the result common/custom account information, then return.

Scenario 2) OTA deploy a package for region devices

The OTA may deploy an update package to devices locate on America, and record the upgrade status. Step 1) Adopt common device information, use advSearch to get these devices on America. Step 2) For these agentId/deviceId to send the request and record the status on OTA database.

Document

WISE-PaaS 2.0 Integration Guide (pdf)

@@ Line 68: / Line 68: @@
 [[File:Device tbl.png|RTENOTITLE]]
+Table MatchGroup
+[[File:MatchGroup tbl.png]]
 === Non-Structure Data (MongoDB/DocumentDB) ===