Difference between revisions of "DataSync V2"
From ESS-WIKI
(Created page with "== Architecture == *DataSync has two tables to save capability data (TB_Cap) and report/event data (TB_Data).<br/> == DataSync.ini == *File example:<br/><pre>[DataSyn...") |
|||
(11 intermediate revisions by the same user not shown) | |||
Line 1: | Line 1: | ||
− | == | + | == DataSync == |
+ | |||
+ | #DataSync will backup capability/report/event data and re-send missing data after connection recover. | ||
+ | #*[[File:DataDync wiki1.png|RTENOTITLE]][[File:DataDync wiki2.png|RTENOTITLE]] | ||
+ | #The data will backup with SQLite3 (datasync.db). | ||
+ | #*Capability is saved in table TB_Cap. | ||
+ | #*Report and Event data are saved in TB_Data. | ||
+ | #DataSync will re-send capability first before the missing report data. | ||
+ | #DataSync always backup data whether connected or disconnected. | ||
+ | #The recover data and report data can send in the same time. | ||
+ | #DataSync doesn't monitor network adapter and pythical layer network status, the disconnected status is reported from mosquitto library (software connection status, ex: TCP connection lost..). | ||
+ | |||
+ | == How to Verify == | ||
+ | |||
+ | You can verify DataSync function by one of following: | ||
+ | |||
+ | #Make a disconnect event by pull out network wire. | ||
+ | #Disable network adapter | ||
+ | #Modify routing table to block WISE-Cloud IP. | ||
+ | |||
+ | == Recover Format == | ||
+ | |||
+ | DataSync will re-send report data (no event data) with batch format, the format is following: | ||
+ | <syntaxhighlight lang="java"> | ||
+ | { | ||
+ | "content":{ // "content" is added by SAManager, handler doesn't need add this header | ||
+ | "data_list":[ | ||
+ | { | ||
+ | "TestHandler":{ // handler report data 1 | ||
+ | ... | ||
+ | } | ||
+ | }, | ||
+ | { | ||
+ | "TestHandler":{ // handler report data 2 | ||
+ | ... | ||
+ | } | ||
+ | }, | ||
+ | { | ||
+ | "TestHandler":{ // handler report data 3 | ||
+ | ... | ||
+ | } | ||
+ | } | ||
+ | ] | ||
+ | }, | ||
+ | "commCmd":2055, | ||
+ | "agentID":"00000001-0000-0000-0000-000C2982D9F6", | ||
+ | "handlerName":"general", | ||
+ | "sendTS":{ | ||
+ | "$date":1533008028483 | ||
+ | } | ||
+ | } | ||
+ | </syntaxhighlight> | ||
− | |||
== DataSync.ini == | == DataSync.ini == | ||
− | *File example:< | + | *File example: |
+ | <pre>[DataSync] | ||
+ | ; database size limit MB | ||
+ | Storage=20 | ||
+ | ; database duration limit HR | ||
+ | Duration=1 | ||
+ | ; keep previous data before # SEC. | ||
+ | RetainDWTP=60 | ||
+ | ; # item in each batch | ||
+ | BatchSize=3 | ||
+ | ; time between each wrap batch send, SEC. | ||
+ | WrapSendInterval=3 | ||
+ | ; preserved storage space. MB | ||
+ | PreserveStorage=200 | ||
+ | </pre> | ||
+ | |||
+ | *Detail: | ||
+ | **Storage | ||
+ | ***A limit of database file size, the database will re-cycle after file size large than this limit. | ||
+ | **Duration | ||
+ | ***A limit of database live time, the database will re-cycle after agent booting time large than this limit. | ||
+ | **RetainDWTP | ||
+ | ***There is a different between disconnected time and disconnected detected time, so DataSync will re-send the data from "disconnected detected time - RetainDWTP" to recover full data. | ||
+ | **BatchSize | ||
+ | ***DataSync will send missing data with batch format, it can customize the number of data in one batch. | ||
+ | **WrapSendInterval | ||
+ | ***The time interval between each wrapped batch data. | ||
+ | **PreserveStorage | ||
+ | ***DataSync will guarantee the free space is large than "PreserveStorage", database will re-cycle in this condition. | ||
+ | |||
+ | *Priority: | ||
+ | **There are 3 limit related config, Storage, Duration, PreserveStorage, if user config all, DataSync has a priority for this condition. | ||
+ | |||
+ | PreserveStorage > Storage > Duration Note: You must config at least one of 3 limit, or DataSync will report error. |
Latest revision as of 05:43, 31 July 2018
DataSync
- DataSync will backup capability/report/event data and re-send missing data after connection recover.
- The data will backup with SQLite3 (datasync.db).
- Capability is saved in table TB_Cap.
- Report and Event data are saved in TB_Data.
- DataSync will re-send capability first before the missing report data.
- DataSync always backup data whether connected or disconnected.
- The recover data and report data can send in the same time.
- DataSync doesn't monitor network adapter and pythical layer network status, the disconnected status is reported from mosquitto library (software connection status, ex: TCP connection lost..).
How to Verify
You can verify DataSync function by one of following:
- Make a disconnect event by pull out network wire.
- Disable network adapter
- Modify routing table to block WISE-Cloud IP.
Recover Format
DataSync will re-send report data (no event data) with batch format, the format is following:
{
"content":{ // "content" is added by SAManager, handler doesn't need add this header
"data_list":[
{
"TestHandler":{ // handler report data 1
...
}
},
{
"TestHandler":{ // handler report data 2
...
}
},
{
"TestHandler":{ // handler report data 3
...
}
}
]
},
"commCmd":2055,
"agentID":"00000001-0000-0000-0000-000C2982D9F6",
"handlerName":"general",
"sendTS":{
"$date":1533008028483
}
}
DataSync.ini
- File example:
[DataSync] ; database size limit MB Storage=20 ; database duration limit HR Duration=1 ; keep previous data before # SEC. RetainDWTP=60 ; # item in each batch BatchSize=3 ; time between each wrap batch send, SEC. WrapSendInterval=3 ; preserved storage space. MB PreserveStorage=200
- Detail:
- Storage
- A limit of database file size, the database will re-cycle after file size large than this limit.
- Duration
- A limit of database live time, the database will re-cycle after agent booting time large than this limit.
- RetainDWTP
- There is a different between disconnected time and disconnected detected time, so DataSync will re-send the data from "disconnected detected time - RetainDWTP" to recover full data.
- BatchSize
- DataSync will send missing data with batch format, it can customize the number of data in one batch.
- WrapSendInterval
- The time interval between each wrapped batch data.
- PreserveStorage
- DataSync will guarantee the free space is large than "PreserveStorage", database will re-cycle in this condition.
- Storage
- Priority:
- There are 3 limit related config, Storage, Duration, PreserveStorage, if user config all, DataSync has a priority for this condition.
PreserveStorage > Storage > Duration Note: You must config at least one of 3 limit, or DataSync will report error.