Difference between revisions of "DataSync V2"

From ESS-WIKI
Jump to: navigation, search
 
(8 intermediate revisions by the same user not shown)
Line 1: Line 1:
 
== DataSync ==
 
== DataSync ==
  
−
#DataSync will cache capability/report/event data and re-send missing data after connection recover.
+
#DataSync will backup capability/report/event data and re-send missing data after connection recover.
−
#The cached data will save with SQLite3 (datasync.db).
+
#*[[File:DataDync wiki1.png|RTENOTITLE]][[File:DataDync wiki2.png|RTENOTITLE]]
−
**Capability is saved in table TB_Cap.
+
#The data will backup with SQLite3 (datasync.db).
−
**Report and Event data are saved in TB_Data.
+
#*Capability is saved in table TB_Cap.
 +
#*Report and Event data are saved in TB_Data.
 
#DataSync will re-send capability first before the missing report data.
 
#DataSync will re-send capability first before the missing report data.
−
#DataSync will send recover progress by auto report data, so you can find recover progress in monitor/DataSync page.
+
#DataSync always backup data whether connected or disconnected.
 +
#The recover data and report data can send in the same time.
 +
#DataSync doesn't monitor network adapter and pythical layer network status, the disconnected status is reported from mosquitto library (software connection status, ex: TCP connection lost..).
  
 
== How to Verify ==
 
== How to Verify ==
 +
 
You can verify DataSync function by one of following:
 
You can verify DataSync function by one of following:
 +
 
#Make a disconnect event by pull out network wire.
 
#Make a disconnect event by pull out network wire.
 
#Disable network adapter
 
#Disable network adapter
 
#Modify routing table to block WISE-Cloud IP.
 
#Modify routing table to block WISE-Cloud IP.
 +
 +
== Recover Format ==
 +
 +
DataSync will re-send report data (no event data) with batch format, the format is following:
 +
<syntaxhighlight lang="java">
 +
{ 
 +
  "content":{  // "content" is added by SAManager, handler doesn't need add this header
 +
      "data_list":[ 
 +
        { 
 +
            "TestHandler":{  // handler report data 1
 +
...
 +
            }
 +
        },
 +
        { 
 +
            "TestHandler":{  // handler report data 2
 +
...
 +
            }
 +
        },
 +
        { 
 +
            "TestHandler":{  // handler report data 3
 +
              ...
 +
            }
 +
        }
 +
      ]
 +
  },
 +
  "commCmd":2055,
 +
  "agentID":"00000001-0000-0000-0000-000C2982D9F6",
 +
  "handlerName":"general",
 +
  "sendTS":{ 
 +
      "$date":1533008028483
 +
  }
 +
}
 +
</syntaxhighlight>
 +
  
 
== DataSync.ini ==
 
== DataSync.ini ==
Line 31: Line 70:
 
PreserveStorage=200
 
PreserveStorage=200
 
</pre>
 
</pre>
 +
 +
*Detail:
 +
**Storage
 +
***A limit of database file size, the database will re-cycle after file size large than this limit.
 +
**Duration
 +
***A limit of database live time, the database will re-cycle after agent booting time large than this limit.
 +
**RetainDWTP
 +
***There is a different between disconnected time and disconnected detected time, so DataSync will re-send the data from "disconnected detected time - RetainDWTP" to recover full data.
 +
**BatchSize
 +
***DataSync will send missing data with batch format, it can customize the number of data in one batch.
 +
**WrapSendInterval
 +
***The time interval between each wrapped batch data.
 +
**PreserveStorage
 +
***DataSync will guarantee the free space is large than "PreserveStorage", database will re-cycle in this condition.
 +
 +
*Priority:
 +
**There are 3 limit related config, Storage, Duration, PreserveStorage, if user config all, DataSync has a priority for this condition.
 +
 +
PreserveStorage > Storage > Duration Note: You must config at least one of 3 limit, or DataSync will report error.

Latest revision as of 04:43, 31 July 2018

DataSync

  1. DataSync will backup capability/report/event data and re-send missing data after connection recover.
    • RTENOTITLERTENOTITLE
  2. The data will backup with SQLite3 (datasync.db).
    • Capability is saved in table TB_Cap.
    • Report and Event data are saved in TB_Data.
  3. DataSync will re-send capability first before the missing report data.
  4. DataSync always backup data whether connected or disconnected.
  5. The recover data and report data can send in the same time.
  6. DataSync doesn't monitor network adapter and pythical layer network status, the disconnected status is reported from mosquitto library (software connection status, ex: TCP connection lost..).

How to Verify

You can verify DataSync function by one of following:

  1. Make a disconnect event by pull out network wire.
  2. Disable network adapter
  3. Modify routing table to block WISE-Cloud IP.

Recover Format

DataSync will re-send report data (no event data) with batch format, the format is following: 

{  
   "content":{  // "content" is added by SAManager, handler doesn't need add this header
      "data_list":[  
         {  
            "TestHandler":{  // handler report data 1
				...
            }
         },
         {  
            "TestHandler":{  // handler report data 2
				...
            }
         },
         {  
            "TestHandler":{  // handler report data 3
               ...
            }
         }
      ]
   },
   "commCmd":2055,
   "agentID":"00000001-0000-0000-0000-000C2982D9F6",
   "handlerName":"general",
   "sendTS":{  
      "$date":1533008028483
   }
}


DataSync.ini

  • File example:
[DataSync]
; database size limit MB
Storage=20
; database duration limit HR
Duration=1
; keep previous data before # SEC.
RetainDWTP=60
; # item in each batch
BatchSize=3
; time between each wrap batch send, SEC.
WrapSendInterval=3
; preserved storage space. MB
PreserveStorage=200
  • Detail:
    • Storage
      • A limit of database file size, the database will re-cycle after file size large than this limit.
    • Duration
      • A limit of database live time, the database will re-cycle after agent booting time large than this limit.
    • RetainDWTP
      • There is a different between disconnected time and disconnected detected time, so DataSync will re-send the data from "disconnected detected time - RetainDWTP" to recover full data.
    • BatchSize
      • DataSync will send missing data with batch format, it can customize the number of data in one batch.
    • WrapSendInterval
      • The time interval between each wrapped batch data.
    • PreserveStorage
      • DataSync will guarantee the free space is large than "PreserveStorage", database will re-cycle in this condition.
  • Priority:
    • There are 3 limit related config, Storage, Duration, PreserveStorage, if user config all, DataSync has a priority for this condition.

PreserveStorage > Storage > Duration Note: You must config at least one of 3 limit, or DataSync will report error.

Retrieved from "https://ess-wiki.advantech.com.tw/wiki/index.php?title=DataSync_V2&oldid=12652"