Blockheight increasing but nknd.exe fails continuously

hello and thank you in advance,

i am just trying to understand if there is a network level block in my way from moving from SYNC_STARTED state to MINING. maybe something upstream at the xfinity provider outside of my control.

while the nknd.exe program is running:

  1. i am able to see my blockheight increase at http://nstatus.org/. last run increased over 1k in basic int value.
  2. localhost:30000/web status shows same sync in progress state.
  3. i paid the 10 nkn via nknx.org add node feature.
  4. router has UPnP on, port forward off, and i put my localhost ip in dmz, both ip4 & ip6 so all ports are open.
  5. full log available for dm
  6. localhost firewall set inbound/outbound accept all tcp/udp for nknd.exe filepath
# win 10, in admin mode, 100+gb free space, i7 & 32gb ddr4.
.\nknd.exe --version
nknd version v2.1.5

.\nknd.exe -p [my_pwd]

2021/08/10 09:39:49.565795 e[0;32m[INFO ]e[m GID 1, My IP address is xx.xx.xxx.xxx
2021/08/10 09:39:50.439793 e[0;32m[INFO ]e[m GID 1, set default cert domain to: xx-xx-xxx-xxx.ipv4.nknlabs.io
2021/08/10 09:39:50.439793 e[0;32m[INFO ]e[m GID 1, use default https certs
2021/08/10 09:39:50.440794 e[0;32m[INFO ]e[m GID 1, use default wss certs
2021/08/10 09:39:50.441792 e[0;32m[INFO ]e[m GID 108, https cert received
2021/08/10 09:39:50.705793 e[0;32m[INFO ]e[m GID 1, database Version: 1
2021/08/10 09:39:51.456874 e[0;32m[INFO ]e[m GID 1, State root: fe724c9273654fb1214e36d71afe8382389a9234cd7e00d5fc8776cb5b4e28b5
2021/08/10 09:39:51.456874 e[0;32m[INFO ]e[m GID 1, Start pruning...
2021/08/10 09:39:51.458880 e[0;32m[INFO ]e[m GID 1, Start compacting database from height 58988 to 102084
2021/08/10 09:40:00.738776 e[0;32m[INFO ]e[m GID 1, Start verifying database at height 102084
2021/08/10 09:40:05.800696 e[0;32m[INFO ]e[m GID 1, get no ID from local ledger

#...
# [start to see connections]

2021/08/10 09:40:07.138697 e[0;32m[INFO ]e[m GID 277, wss cert received
2021/08/10 09:40:07.138697 e[0;32m[INFO ]e[m GID 285, Set min verifiable height to 102089
2021/08/10 09:40:10.468698 e[0;32m[INFO ]e[m GID 261, Remote node connect from 3.239.79.233:44284 to local address 10.0.0.123:30001
2021/08/10 09:40:10.521696 e[0;32m[INFO ]e[m GID 261, Remote node connect from 96.126.113.179:54212 to local address 10.0.0.123:30001
2021/08/10 09:40:10.920696 e[0;32m[INFO ]e[m GID 261, Remote node connect from 143.244.168.112:42658 to local address 10.0.0.123:30001
2021/08/10 09:40:11.120245 e[0;32m[INFO ]e[m GID 261, Remote node connect from 54.221.81.75:51948 to local address 10.0.0.123:30001
2021/08/10 09:40:11.533246 e[0;32m[INFO ]e[m GID 261, Remote node connect from 147.182.189.161:42852 to local address 10.0.0.123:30001
2021/08/10 09:40:11.820246 e[0;32m[INFO ]e[m GID 261, Remote node connect from 104.248.0.50:56848 to local address 10.0.0.123:30001
2021/08/10 09:40:12.434246 e[0;32m[INFO ]e[m GID 261, Remote node connect from 18.217.21.172:51084 to local address 10.0.0.123:30001
2021/08/10 09:40:12.647246 e[0;32m[INFO ]e[m GID 261, Remote node connect from 49.51.188.42:46360 to local address 10.0.0.123:30001
2021/08/10 09:40:12.746246 e[0;32m[INFO ]e[m GID 261, Remote node connect from 3.88.145.36:53372 to local address 10.0.0.123:30001
2021/08/10 09:40:13.462446 e[0;32m[INFO ]e[m GID 256, Error handling msg: receive invalid block hash height 2958597 instead of 102085
2021/08/10 09:40:13.510445 e[0;32m[INFO ]e[m GID 261, Remote node connect from 18.118.25.142:59416 to local address 10.0.0.123:30001
2021/08/10 09:40:13.544447 e[0;32m[INFO ]e[m GID 261, Remote node connect from 45.76.18.151:46250 to local address 10.0.0.123:30001
2021/08/10 09:40:13.558446 e[0;32m[INFO ]e[m GID 256, Error handling msg: receive invalid block hash height 2958597 instead of 102085
2021/08/10 09:40:13.598460 e[0;32m[INFO ]e[m GID 261, Remote node connect from 143.198.176.155:20056 to local address 10.0.0.123:30001
2021/08/10 09:40:13.743445 e[0;32m[INFO ]e[m GID 261, Remote node connect from 43.130.47.249:38448 to local address 10.0.0.123:30001
2021/08/10 09:40:14.050448 e[0;32m[INFO ]e[m GID 261, Remote node connect from 3.234.252.155:33466 to local address 10.0.0.123:30001
2021/08/10 09:40:14.182446 e[0;32m[INFO ]e[m GID 261, Remote node connect from 147.182.198.135:51642 to local address 10.0.0.123:30001
2021/08/10 09:40:14.240445 e[0;32m[INFO ]e[m GID 261, Remote node connect from 173.199.119.200:56432 to local address 10.0.0.123:30001
2021/08/10 09:40:14.342447 e[0;32m[INFO ]e[m GID 261, Remote node connect from 143.198.172.243:52062 to local address 10.0.0.123:30001
2021/08/10 09:40:14.404445 e[0;32m[INFO ]e[m GID 261, Remote node connect from 3.239.42.132:57820 to local address 10.0.0.123:30001
2021/08/10 09:40:14.498446 e[0;32m[INFO ]e[m GID 261, Remote node connect from 142.93.67.8:57456 to local address 10.0.0.123:30001
2021/08/10 09:40:14.535447 e[0;32m[INFO ]e[m GID 261, Remote node connect from 51.81.234.83:57578 to local address 10.0.0.123:30001
2021/08/10 09:40:14.748446 e[0;32m[INFO ]e[m GID 261, Remote node connect from 54.197.80.50:51490 to local address 10.0.0.123:30001
2021/08/10 09:40:14.810445 e[0;32m[INFO ]e[m GID 261, Remote node connect from 18.205.26.63:49700 to local address 10.0.0.123:30001
2021/08/10 09:40:17.139224 e[0;33m[WARN ]e[m GID 282, Handle proposal error: wait for proposal timeout
2021/08/10 09:40:17.325222 e[0;32m[INFO ]e[m GID 261, Remote node connect from 95.179.240.77:34044 to local address 10.0.0.123:30001
2021/08/10 09:40:17.474221 e[0;32m[INFO ]e[m GID 261, Remote node connect from 143.198.175.145:48896 to local address 10.0.0.123:30001
2021/08/10 09:40:18.345805 e[0;32m[INFO ]e[m GID 261, Remote node connect from 44.233.222.145:51344 to local address 10.0.0.123:30001
2021/08/10 09:40:18.582805 e[0;32m[INFO ]e[m GID 261, Remote node connect from 49.51.189.201:33828 to local address 10.0.0.123:30001
2021/08/10 09:40:19.100894 e[0;32m[INFO ]e[m GID 261, Remote node connect from 34.221.115.250:44616 to local address 10.0.0.123:30001
2021/08/10 09:40:19.164894 e[0;32m[INFO ]e[m GID 392, Load remote node <3.87.239.165:30001> from list
2021/08/10 09:40:19.183893 e[0;32m[INFO ]e[m GID 261, Remote node connect from 172.104.240.59:35494 to local address 10.0.0.123:30001
2021/08/10 09:40:19.971906 e[0;32m[INFO ]e[m GID 261, Remote node connect from 35.175.121.38:41180 to local address 10.0.0.123:30001
...

2021/08/10 09:40:21.922920 e[0;32m[INFO ]e[m GID 261, Remote node connect from 45.77.202.84:59660 to local address 10.0.0.123:30001
2021/08/10 09:40:22.247920 e[0;32m[INFO ]e[m GID 261, Remote node connect from 207.148.1.235:34772 to local address 10.0.0.123:30001
2021/08/10 09:40:22.532922 e[0;32m[INFO ]e[m GID 261, Remote node connect from 140.82.51.66:52644 to local address 10.0.0.123:30001
2021/08/10 09:40:22.689919 e[0;32m[INFO ]e[m GID 261, Remote node connect from 3.236.97.133:52746 to local address 10.0.0.123:30001
2021/08/10 09:40:22.701922 e[0;32m[INFO ]e[m GID 261, Remote node connect from 147.182.170.158:40714 to local address 10.0.0.123:30001
2021/08/10 09:40:22.883921 e[0;32m[INFO ]e[m GID 261, Remote node connect from 207.154.222.33:44752 to local address 10.0.0.123:30001
2021/08/10 09:40:23.494100 e[0;32m[INFO ]e[m GID 261, Remote node connect from 143.198.128.117:37398 to local address 10.0.0.123:30001
2021/08/10 09:40:23.574100 e[0;32m[INFO ]e[m GID 392, Load remote node <3.87.239.165:30001> from list
2021/08/10 09:40:23.868108 e[0;32m[INFO ]e[m GID 261, Remote node connect from 108.61.184.100:55322 to local address 10.0.0.123:30001
2021/08/10 09:40:24.014100 e[0;32m[INFO ]e[m GID 261, Remote node connect from 54.193.0.196:34084 to local address 10.0.0.123:30001
2021/08/10 09:40:24.093101 e[0;32m[INFO ]e[m GID 261, Remote node connect from 54.177.25.247:50264 to local address 10.0.0.123:30001
2021/08/10 09:40:24.548100 e[0;32m[INFO ]e[m GID 261, Remote node connect from 147.182.233.150:37298 to local address 10.0.0.123:30001
2021/08/10 09:40:25.562711 e[0;32m[INFO ]e[m GID 261, Remote node connect from 45.77.207.153:60376 to local address 10.0.0.123:30001
2021/08/10 09:40:25.600709 e[0;32m[INFO ]e[m GID 261, Remote node connect from 147.182.203.167:51034 to local address 10.0.0.123:30001
2021/08/10 09:40:26.235711 e[0;32m[INFO ]e[m GID 261, Remote node connect from 147.182.245.37:34400 to local address 10.0.0.123:30001
2021/08/10 09:40:26.337709 e[0;32m[INFO ]e[m GID 261, Remote node connect from 45.63.10.119:46542 to local address 10.0.0.123:30001
2021/08/10 09:40:26.525713 e[0;32m[INFO ]e[m GID 261, Remote node connect from 143.198.180.94:39546 to local address 10.0.0.123:30001
2021/08/10 09:40:26.775710 e[0;32m[INFO ]e[m GID 230, DropTxns: 0 txns (0 bytes) in txpool, no need to drop
2021/08/10 09:40:26.988710 e[0;32m[INFO ]e[m GID 261, Remote node connect from 178.128.150.204:39834 to local address 10.0.0.123:30001
2021/08/10 09:40:27.237826 e[0;33m[WARN ]e[m GID 282, Handle proposal error: wait for neighbor vote timeout
2021/08/10 09:40:28.016824 e[0;32m[INFO ]e[m GID 261, Remote node connect from 35.174.60.242:36308 to local address 10.0.0.123:30001
...
2021/08/10 09:40:34.188206 e[0;32m[INFO ]e[m GID 256, Error handling msg: block 19c3caaf03df773e29ab2d00007e57a8d9a877003f2b8e25e6a9d48bd3c10f70 is not finalized yet
2021/08/10 09:40:34.211205 e[0;32m[INFO ]e[m GID 261, Remote node connect from 34.219.110.204:49936 to local address 10.0.0.123:30001
2021/08/10 09:40:34.306206 e[0;32m[INFO ]e[m GID 256, Error handling msg: block 19c3caaf03df773e29ab2d00007e57a8d9a877003f2b8e25e6a9d48bd3c10f70 is not finalized yet
2021/08/10 09:40:35.691823 e[0;32m[INFO ]e[m GID 256, Error handling msg: receive invalid block hash height 2958598 instead of 102085
2021/08/10 09:40:35.772823 e[0;32m[INFO ]e[m GID 256, Error handling msg: receive invalid block hash height 2958598 instead of 102085
2021/08/10 09:40:35.823824 e[0;32m[INFO ]e[m GID 256, Error handling msg: receive invalid block hash height 2958598 instead of 102085
2021/08/10 09:40:35.894822 e[0;32m[INFO ]e[m GID 256, Error handling msg: receive invalid block hash height 2958598 instead of 102085
2021/08/10 09:40:35.963821 e[0;32m[INFO ]e[m GID 256, Error handling msg: receive invalid block hash height 2958598 instead of 102085
2021/08/10 09:40:36.017823 e[0;32m[INFO ]e[m GID 256, Error handling msg: receive invalid block hash height 2958598 instead of 102085
2021/08/10 09:40:36.037823 e[0;32m[INFO ]e[m GID 256, Error handling msg: receive invalid block hash height 2958598 instead of 102085
2021/08/10 09:40:36.143823 e[0;32m[INFO ]e[m GID 256, Error handling msg: receive invalid block hash height 2958598 instead of 102085
2021/08/10 09:40:36.214822 e[0;32m[INFO ]e[m GID 256, Error handling msg: receive invalid block hash height 2958598 instead of 102085
2021/08/10 09:40:36.304823 e[0;32m[INFO ]e[m GID 397, Remote node b[email protected]tcp://143.198.176.155:30001<143.198.176.155:20056> stops
2021/08/10 09:40:36.304823 e[0;32m[INFO ]e[m GID 397, Remote node b[email protected]tcp://143.198.128.117:30001<143.198.128.117:37398> stops
2021/08/10 09:40:36.307823 e[0;32m[INFO ]e[m GID 397, Remote node 7[email protected]tcp://147.182.189.161:30001<147.182.189.161:42852> stops
...
2021/08/10 09:40:44.933361 e[0;32m[INFO ]e[m GID 253, Handle transactions msg error: [VerifyTransaction] id version 1 is not supported at height 102085
2021/08/10 09:40:45.128360 e[0;32m[INFO ]e[m GID 3050, Remote node 7[email protected]tcp://143.198.174.82:30001<143.198.174.82:30001> stops
2021/08/10 09:40:45.165360 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node 7[email protected]tcp://143.198.174.82:30001<143.198.174.82:30001>
2021/08/10 09:40:45.277361 e[0;32m[INFO ]e[m GID 2642, Connect to random neighbor a[email protected]tcp://104.156.225.95:30001
2021/08/10 09:40:45.449361 e[0;32m[INFO ]e[m GID 2642, Connect to random neighbor b[email protected]tcp://34.125.100.239:30001
2021/08/10 09:40:45.518365 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node 6[email protected]tcp://23.90.33.164:30001<23.90.33.164:30001>
2021/08/10 09:40:45.518365 e[0;32m[INFO ]e[m GID 260, Remote node 6[email protected]tcp://23.90.33.164:30001<23.90.33.164:30001> stops

#... connections start to timeout

2021/08/10 09:44:16.195885 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node <3.223.6.244:30001>
2021/08/10 09:44:16.196884 e[0;32m[INFO ]e[m GID 261, Remote node connect from 54.177.25.247:50288 to local address 10.0.0.123:30001
2021/08/10 09:44:16.829884 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node <54.177.25.247:50288>
2021/08/10 09:44:16.829884 e[0;32m[INFO ]e[m GID 260, Remote node <54.177.25.247:50288> stops
2021/08/10 09:44:16.983886 e[0;32m[INFO ]e[m GID 261, Remote node connect from 161.35.5.244:53048 to local address 10.0.0.123:30001
2021/08/10 09:44:17.057885 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node b[email protected]tcp://3.238.127.2:30001<3.238.127.2:30001>
2021/08/10 09:44:17.058885 e[0;32m[INFO ]e[m GID 260, Remote node b[email protected]tcp://3.238.127.2:30001<3.238.127.2:30001> stops
2021/08/10 09:44:17.114904 e[0;32m[INFO ]e[m GID 261, Remote node connect from 3.239.42.132:58008 to local address 10.0.0.123:30001
...
2021/08/10 09:44:18.210983 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node b[email protected]tcp://207.148.24.232:30001<207.148.24.232:30001>
2021/08/10 09:44:18.359616 e[0;33m[WARN ]e[m GID 10321, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.360628 e[0;33m[WARN ]e[m GID 10407, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.363617 e[0;33m[WARN ]e[m GID 10419, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.363617 e[0;33m[WARN ]e[m GID 10428, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.363617 e[0;33m[WARN ]e[m GID 10403, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.363617 e[0;33m[WARN ]e[m GID 10421, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.364616 e[0;33m[WARN ]e[m GID 10423, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.364616 e[0;33m[WARN ]e[m GID 10420, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.364616 e[0;33m[WARN ]e[m GID 10415, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.364616 e[0;33m[WARN ]e[m GID 10405, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.364616 e[0;33m[WARN ]e[m GID 10318, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.365617 e[0;33m[WARN ]e[m GID 10409, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.365617 e[0;33m[WARN ]e[m GID 10426, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.365617 e[0;33m[WARN ]e[m GID 10321, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.365617 e[0;33m[WARN ]e[m GID 10407, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.365617 e[0;33m[WARN ]e[m GID 10419, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.365617 e[0;33m[WARN ]e[m GID 10428, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.365617 e[0;33m[WARN ]e[m GID 10403, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.365617 e[0;33m[WARN ]e[m GID 10423, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.365617 e[0;33m[WARN ]e[m GID 10421, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.365617 e[0;33m[WARN ]e[m GID 10420, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.365617 e[0;33m[WARN ]e[m GID 10415, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.365617 e[0;33m[WARN ]e[m GID 10405, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.365617 e[0;33m[WARN ]e[m GID 10318, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.366617 e[0;33m[WARN ]e[m GID 10409, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.366617 e[0;33m[WARN ]e[m GID 10407, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.366617 e[0;33m[WARN ]e[m GID 10426, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.366617 e[0;33m[WARN ]e[m GID 10321, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.366617 e[0;33m[WARN ]e[m GID 10419, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.366617 e[0;33m[WARN ]e[m GID 10428, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.366617 e[0;33m[WARN ]e[m GID 10403, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.366617 e[0;33m[WARN ]e[m GID 10423, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.366617 e[0;33m[WARN ]e[m GID 10421, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.366617 e[0;33m[WARN ]e[m GID 10420, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.366617 e[0;33m[WARN ]e[m GID 10405, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.366617 e[0;33m[WARN ]e[m GID 10409, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.366617 e[0;33m[WARN ]e[m GID 10318, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.366617 e[0;33m[WARN ]e[m GID 10407, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.367621 e[0;33m[WARN ]e[m GID 10426, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.367621 e[0;33m[WARN ]e[m GID 10321, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.367621 e[0;33m[WARN ]e[m GID 10419, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.367621 e[0;33m[WARN ]e[m GID 10428, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.367621 e[0;33m[WARN ]e[m GID 10403, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.367621 e[0;33m[WARN ]e[m GID 10421, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.413616 e[0;31m[ERROR]e[m GID 392, Update successors error: Wait for reply timeout
2021/08/10 09:44:18.489619 e[0;32m[INFO ]e[m GID 2643, Connect to random neighbor 0[email protected]tcp://140.82.42.199:30001
2021/08/10 09:44:18.630617 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node <54.177.25.247:50290>
2021/08/10 09:44:18.630617 e[0;32m[INFO ]e[m GID 260, Remote node <54.177.25.247:50290> stops
2021/08/10 09:44:19.157132 e[0;33m[WARN ]e[m GID 15118, Remote node d[email protected]tcp://35.165.200.164:30001<35.165.200.164:30001> stops because of error: Read msg len error: io: read/write on closed pipe
2021/08/10 09:44:19.189130 e[0;32m[INFO ]e[m GID 261, Remote node connect from 3.239.42.132:58010 to local address 10.0.0.123:30001
2021/08/10 09:44:19.279132 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node <3.239.42.132:58010>
2021/08/10 09:44:19.280133 e[0;32m[INFO ]e[m GID 260, Remote node b[email protected]tcp://3.239.42.132:30001<3.239.42.132:58010> stops
2021/08/10 09:44:19.382131 e[0;32m[INFO ]e[m GID 2643, Connect to random neighbor c[email protected]tcp://44.234.63.118:30001
2021/08/10 09:44:19.733135 e[0;32m[INFO ]e[m GID 15668, Remote node c[email protected]tcp://44.234.63.118:30001<44.234.63.118:30001> stops
2021/08/10 09:44:19.813133 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node c[email protected]tcp://44.234.63.118:30001<44.234.63.118:30001>
2021/08/10 09:44:19.909133 e[0;32m[INFO ]e[m GID 15655, Remote node c[email protected]tcp://3.141.104.18:30001<3.141.104.18:30001> stops
2021/08/10 09:44:19.941132 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node c[email protected]tcp://3.141.104.18:30001<3.141.104.18:30001>
2021/08/10 09:44:20.301896 e[0;32m[INFO ]e[m GID 2643, Connect to random neighbor 4[email protected]tcp://18.210.24.71:30001
2021/08/10 09:44:20.433898 e[0;32m[INFO ]e[m GID 2643, Connect to random neighbor 6[email protected]tcp://3.101.131.78:30001
2021/08/10 09:44:21.215926 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node a[email protected]tcp://45.33.77.247:30001<45.33.77.247:30001>
2021/08/10 09:44:21.215926 e[0;32m[INFO ]e[m GID 260, Remote node a[email protected]tcp://45.33.77.247:30001<45.33.77.247:30001> stops
2021/08/10 09:44:21.273925 e[0;32m[INFO ]e[m GID 261, Remote node connect from 54.177.25.247:50292 to local address 10.0.0.123:30001
2021/08/10 09:44:21.571936 e[0;32m[INFO ]e[m GID 261, Remote node connect from 3.239.42.132:58012 to local address 10.0.0.123:30001
2021/08/10 09:44:21.595927 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node <3.239.42.132:58012>
2021/08/10 09:44:21.595927 e[0;32m[INFO ]e[m GID 260, Remote node b[email protected]tcp://3.239.42.132:30001<3.239.42.132:58012> stops
2021/08/10 09:44:21.595927 e[0;32m[INFO ]e[m GID 2643, Connect to random neighbor 6[email protected]tcp://207.148.19.128:30001
2021/08/10 09:44:21.781925 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node <54.177.25.247:50292>
2021/08/10 09:44:21.781925 e[0;32m[INFO ]e[m GID 260, Remote node <54.177.25.247:50292> stops
2021/08/10 09:44:22.020028 e[0;33m[WARN ]e[m GID 12991, Remote node b[email protected]tcp://34.230.58.144:30001<34.230.58.144:30001> stops because of error: keepalive timeout
2021/08/10 09:44:22.551030 e[0;32m[INFO ]e[m GID 15834, Remote node c[email protected]tcp://104.248.233.99:30001<104.248.233.99:30001> stops
2021/08/10 09:44:22.583030 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node c[email protected]tcp://104.248.233.99:30001<104.248.233.99:30001>
2021/08/10 09:44:22.867029 e[0;32m[INFO ]e[m GID 261, Remote node connect from 3.238.127.2:59466 to local address 10.0.0.123:30001
2021/08/10 09:44:23.085030 e[0;33m[WARN ]e[m GID 13398, Remote node b[email protected]tcp://147.182.170.158:30001<147.182.170.158:30001> stops because of error: keepalive timeout
2021/08/10 09:44:23.088028 e[0;33m[WARN ]e[m GID 13729, Remote node b[email protected]tcp://147.182.233.150:30001<147.182.233.150:37444> stops because of error: keepalive timeout
2021/08/10 09:44:23.117034 e[0;33m[WARN ]e[m GID 12934, Remote node b[email protected]tcp://143.198.180.94:30001<143.198.180.94:30001> stops because of error: Read msg len error: io: read/write on closed pipe
2021/08/10 09:44:23.150028 e[0;33m[WARN ]e[m GID 1308, Remote node b[email protected]tcp://165.227.196.116:30001<165.227.196.116:38482> stops because of error: keepalive timeout
2021/08/10 09:44:23.174028 e[0;33m[WARN ]e[m GID 12002, Remote node b[email protected]tcp://51.81.51.252:30001<51.81.51.252:45220> stops because of error: keepalive timeout
2021/08/10 09:44:23.182027 e[0;32m[INFO ]e[m GID 261, Remote node connect from 165.227.196.116:38498 to local address 10.0.0.123:30001
2021/08/10 09:44:23.247027 e[0;32m[INFO ]e[m GID 261, Remote node connect from 143.198.180.94:39562 to local address 10.0.0.123:30001
2021/08/10 09:44:23.247027 e[0;33m[WARN ]e[m GID 15869, Remote node <165.227.196.116:38498> stops because of error: Node with id b797d90eaec3cc3083610365944b00aa1bf45a4620218c8b7a764ed6f9ff2780 is already connected at addr 165.227.196.116:38482
2021/08/10 09:44:23.258029 e[0;32m[INFO ]e[m GID 2642, Connect to random neighbor 9[email protected]tcp://143.198.59.112:30001
2021/08/10 09:44:23.277029 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node <165.227.196.116:38498>
2021/08/10 09:44:23.338030 e[0;31m[ERROR]e[m GID 393, Update predecessor error: Wait for reply timeout
2021/08/10 09:44:23.463029 e[0;32m[INFO ]e[m GID 261, Remote node connect from 95.179.240.77:34066 to local address 10.0.0.123:30001
2021/08/10 09:44:23.589029 e[0;33m[WARN ]e[m GID 15976, Remote node <95.179.240.77:34066> stops because of error: Node with id b79dfa7f3861c898711d8a258e027de5cba45758573b4f1d4e841fdc80ae1dc2 is already connected at addr 95.179.240.77:34064
2021/08/10 09:44:23.643028 e[0;33m[WARN ]e[m GID 8211, Remote node 4[email protected]tcp://34.217.121.201:30001<34.217.121.201:30001> stops because of error: keepalive timeout
2021/08/10 09:44:23.657028 e[0;33m[WARN ]e[m GID 13647, Remote node b[email protected]tcp://45.63.10.119:30001<45.63.10.119:46574> stops because of error: keepalive timeout
2021/08/10 09:44:23.676029 e[0;33m[WARN ]e[m GID 14084, Remote node b[email protected]tcp://95.179.240.77:30001<95.179.240.77:34064> stops because of error: keepalive timeout
2021/08/10 09:44:23.759027 e[0;32m[INFO ]e[m GID 261, Remote node connect from 147.182.170.158:40730 to local address 10.0.0.123:30001
2021/08/10 09:44:23.782031 e[0;32m[INFO ]e[m GID 261, Remote node connect from 3.87.239.165:49064 to local address 10.0.0.123:30001
2021/08/10 09:44:23.792029 e[0;32m[INFO ]e[m GID 261, Remote node connect from 147.182.204.251:45384 to local address 10.0.0.123:30001
2021/08/10 09:44:23.850030 e[0;33m[WARN ]e[m GID 11407, Remote node e[email protected]tcp://45.77.96.118:30001<45.77.96.118:30001> stops because of error: keepalive timeout
2021/08/10 09:44:23.884030 e[0;33m[WARN ]e[m GID 13761, Remote node b[email protected]tcp://147.182.204.251:30001<147.182.204.251:30001> stops because of error: keepalive timeout
2021/08/10 09:44:23.904027 e[0;33m[WARN ]e[m GID 15996, Remote node <147.182.204.251:45384> stops because of error: Node with id b7a5a308f214c1f4392ba40454bc534528686d5da83b28109e1856f2f66dbef5 is already connected at addr 147.182.204.251:30001
2021/08/10 09:44:24.010029 e[0;32m[INFO ]e[m GID 261, Remote node connect from 3.239.42.132:58014 to local address 10.0.0.123:30001
2021/08/10 09:44:24.030030 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node b[email protected]tcp://3.239.42.132:30001<3.239.42.132:58014>
2021/08/10 09:44:24.030030 e[0;32m[INFO ]e[m GID 260, Remote node b[email protected]tcp://3.239.42.132:30001<3.239.42.132:58014> stops
2021/08/10 09:44:24.441028 e[0;33m[WARN ]e[m GID 14164, Remote node b[email protected]tcp://45.77.207.153:30001<45.77.207.153:60528> stops because of error: keepalive timeout
2021/08/10 09:44:24.444028 e[0;32m[INFO ]e[m GID 16141, Remote node e[email protected]tcp://206.189.233.206:30001<206.189.233.206:30001> stops
2021/08/10 09:44:24.471028 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node e[email protected]tcp://206.189.233.206:30001<206.189.233.206:30001>
2021/08/10 09:44:24.483031 e[0;32m[INFO ]e[m GID 397, Remote node b[email protected]tcp://18.117.96.156:30001<18.117.96.156:40812> stops
2021/08/10 09:44:24.483031 e[0;32m[INFO ]e[m GID 397, Remote node b[email protected]tcp://161.35.5.244:30001<161.35.5.244:53048> stops
2021/08/10 09:44:24.483031 e[0;32m[INFO ]e[m GID 397, Remote node b[email protected]tcp://143.198.175.49:30001<143.198.175.49:45948> stops
2021/08/10 09:44:24.484030 e[0;32m[INFO ]e[m GID 397, Remote node b[email protected]tcp://3.223.6.244:30001<3.223.6.244:59348> stops
2021/08/10 09:44:24.484030 e[0;32m[INFO ]e[m GID 397, Remote node b[email protected]tcp://207.154.222.33:30001<207.154.222.33:44902> stops
2021/08/10 09:44:24.509030 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node b[email protected]tcp://3.223.6.244:30001<3.223.6.244:59348>
2021/08/10 09:44:24.511031 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node b[email protected]tcp://161.35.5.244:30001<161.35.5.244:53048>
2021/08/10 09:44:24.515037 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node b[email protected]tcp://18.117.96.156:30001<18.117.96.156:40812>
2021/08/10 09:44:24.516032 e[0;32m[INFO ]e[m GID 2643, Connect to random neighbor e[email protected]tcp://147.182.183.19:30001
2021/08/10 09:44:24.521029 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node b[email protected]tcp://143.198.175.49:30001<143.198.175.49:45948>
2021/08/10 09:44:24.624028 e[0;32m[INFO ]e[m GID 282, Elected block hash e8db7609f31ee8073bb39d8c336b65a40b8458f318a89f7599204376f77af259 got 103/106 neighbor votes, weight: 26 (96.30%)
2021/08/10 09:44:24.624028 e[0;32m[INFO ]e[m GID 282, Accept block e8db7609f31ee8073bb39d8c336b65a40b8458f318a89f7599204376f77af259 at height 2958607
2021/08/10 09:44:24.728029 e[0;33m[WARN ]e[m GID 12214, Remote node b[email protected]tcp://54.221.81.75:30001<54.221.81.75:52012> stops because of error: Read msg len error: io: read/write on closed pipe
2021/08/10 09:44:24.760026 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node b[email protected]tcp://159.65.35.196:30001<159.65.35.196:30001>
2021/08/10 09:44:24.766028 e[0;32m[INFO ]e[m GID 260, Remote node b[email protected]tcp://159.65.35.196:30001<159.65.35.196:30001> stops
2021/08/10 09:44:25.590029 e[0;32m[INFO ]e[m GID 256, Error handling msg: block 88f7e91bea00a9cda0e5390dc0ce44c4a4982faa604fe572f8b512b2b5073510 is not finalized yet
2021/08/10 09:44:25.682029 e[0;31m[ERROR]e[m GID 392, Update successors error: Wait for reply timeout
2021/08/10 09:44:26.590170 e[0;32m[INFO ]e[m GID 256, Error handling msg: block 88f7e91bea00a9cda0e5390dc0ce44c4a4982faa604fe572f8b512b2b5073510 is not finalized yet
2021/08/10 09:44:26.784169 e[0;32m[INFO ]e[m GID 230, DropTxns: 4 txns (1221 bytes) in txpool, no need to drop
2021/08/10 09:44:26.942169 e[0;32m[INFO ]e[m GID 261, Remote node connect from 143.198.175.49:45952 to local address 10.0.0.123:30001
2021/08/10 09:44:27.106170 e[0;32m[INFO ]e[m GID 16293, Remote node 0[email protected]tcp://3.142.151.16:30001<3.142.151.16:30001> stops
2021/08/10 09:44:27.185169 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node 0[email protected]tcp://3.142.151.16:30001<3.142.151.16:30001>
...
2021/08/10 09:44:31.066160 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node <34.230.58.144:30001>
2021/08/10 09:44:31.067160 e[0;32m[INFO ]e[m GID 260, Remote node b[email protected]tcp://34.230.58.144:30001<34.230.58.144:30001> stops
2021/08/10 09:44:31.208161 e[0;32m[INFO ]e[m GID 261, Remote node connect from 18.117.96.156:40814 to local address 10.0.0.123:30001
2021/08/10 09:44:32.407889 e[0;33m[WARN ]e[m GID 14006, Remote node b[email protected]tcp://207.154.194.250:30001<207.154.194.250:30001> stops because of error: keepalive timeout
2021/08/10 09:44:32.451887 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node <34.230.58.144:30001>
2021/08/10 09:44:32.451887 e[0;32m[INFO ]e[m GID 260, Remote node <34.230.58.144:30001> stops
2021/08/10 09:44:32.724887 e[0;32m[INFO ]e[m GID 2642, Connect to random neighbor 7[email protected]tcp://52.73.230.232:30001
2021/08/10 09:44:32.905887 e[0;32m[INFO ]e[m GID 2643, Connect to random neighbor d[email protected]tcp://198.50.252.32:30001
2021/08/10 09:44:34.382633 e[0;32m[INFO ]e[m GID 2643, Connect to random neighbor 2[email protected]tcp://147.182.227.21:30001
2021/08/10 09:44:34.653633 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node <34.230.58.144:30001>
2021/08/10 09:44:34.653633 e[0;32m[INFO ]e[m GID 260, Remote node <34.230.58.144:30001> stops
2021/08/10 09:44:34.686633 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node b[email protected]tcp://207.154.194.250:30001<207.154.194.250:30001>
...
2021/08/10 09:44:35.246635 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node <54.177.25.247:50296>
2021/08/10 09:44:35.246635 e[0;32m[INFO ]e[m GID 260, Remote node <54.177.25.247:50296> stops
2021/08/10 09:44:35.330632 e[0;32m[INFO ]e[m GID 261, Remote node connect from 161.35.221.158:59524 to local address 10.0.0.123:30001
2021/08/10 09:44:35.455633 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node <34.230.58.144:36700>
2021/08/10 09:44:35.456634 e[0;32m[INFO ]e[m GID 260, Remote node <34.230.58.144:36700> stops
2021/08/10 09:44:36.856633 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node <34.230.58.144:30001>
2021/08/10 09:44:36.856633 e[0;32m[INFO ]e[m GID 260, Remote node <34.230.58.144:30001> stops
2021/08/10 09:44:37.730634 e[0;32m[INFO ]e[m GID 261, Remote node connect from 147.182.245.37:34552 to local address 10.0.0.123:30001
2021/08/10 09:44:37.759633 e[0;32m[INFO ]e[m GID 282, Change expected block height to 2958609

#...
# [prune always followed by failure]

2021/08/10 09:44:38.966710 e[0;33m[WARN ]e[m GID 17148, Get blocks error: Remote node has stopped
2021/08/10 09:44:38.966710 e[0;33m[WARN ]e[m GID 17148, Get blocks error: Remote node has stopped
2021/08/10 09:44:38.969719 e[0;33m[WARN ]e[m GID 17148, Get blocks error: Remote node has stopped
2021/08/10 09:44:38.969719 e[0;33m[WARN ]e[m GID 17148, Get blocks error: Remote node has stopped
2021/08/10 09:44:39.032709 e[0;32m[INFO ]e[m GID 17088, RefCount height: 102089, length of refCounts: 332
2021/08/10 09:44:39.035709 e[0;32m[INFO ]e[m GID 17088, Pruning height: 101065, length of refCounts: 154
2021/08/10 09:44:39.035709 e[0;32m[INFO ]e[m GID 17088, # current block height: 102089, block hash: 37045b38fe587fab2701cfdac5595433d79625998e918f218fdbbf773b5b1f36
2021/08/10 09:44:39.043710 e[0;32m[INFO ]e[m GID 17088, RefCount height: 102090, length of refCounts: 129
2021/08/10 09:44:39.048719 e[0;32m[INFO ]e[m GID 17088, Pruning height: 101066, length of refCounts: 111
2021/08/10 09:44:39.049712 e[0;32m[INFO ]e[m GID 17088, # current block height: 102090, block hash: 7bf15c817e663c8b26b5d15e0e107eb0dfc82afde2724248a79cface2be65a1e
2021/08/10 09:44:39.057712 e[0;32m[INFO ]e[m GID 17088, RefCount height: 102091, length of refCounts: 124
2021/08/10 09:44:39.064711 e[0;32m[INFO ]e[m GID 17088, Pruning height: 101067, length of refCounts: 230
2021/08/10 09:44:39.064711 e[0;32m[INFO ]e[m GID 17088, # current block height: 102091, block hash: b759bd215180715c2703547d625972e6cdd0382c3a5d03f1b434d8a6edffce09
2021/08/10 09:44:39.071709 e[0;32m[INFO ]e[m GID 17088, RefCount height: 102092, length of refCounts: 130
2021/08/10 09:44:39.074711 e[0;32m[INFO ]e[m GID 17088, Pruning height: 101068, length of refCounts: 114
2021/08/10 09:44:39.076712 e[0;32m[INFO ]e[m GID 17088, # current block height: 102092, block hash: 61ae1a79fe64afc4035381c826fcf0571f016bdc92a40f6b8f9677b521eb75e8
2021/08/10 09:44:39.084719 e[0;32m[INFO ]e[m GID 17088, RefCount height: 102093, length of refCounts: 129
2021/08/10 09:44:39.087709 e[0;32m[INFO ]e[m GID 17088, Pruning height: 101069, length of refCounts: 112
2021/08/10 09:44:39.087709 e[0;32m[INFO ]e[m GID 17088, # current block height: 102093, block hash: f4ff2cc84b60883148eb310c0b234525b0dbf048616f86b0ce4168b128832fee
2021/08/10 09:44:39.094708 e[0;32m[INFO ]e[m GID 17088, RefCount height: 102094, length of refCounts: 224
2021/08/10 09:44:39.100709 e[0;32m[INFO ]e[m GID 17088, Pruning height: 101070, length of refCounts: 195
2021/08/10 09:44:39.100709 e[0;32m[INFO ]e[m GID 17088, # current block height: 102094, block hash: aa02d7527e0590725118628bff7813132bd9f62e51efeae105e54adb379a4c82
2021/08/10 09:44:39.108710 e[0;32m[INFO ]e[m GID 17088, RefCount height: 102095, length of refCounts: 181
2021/08/10 09:44:39.115716 e[0;32m[INFO ]e[m GID 17088, Pruning height: 101071, length of refCounts: 240
2021/08/10 09:44:39.115716 e[0;32m[INFO ]e[m GID 17088, # current block height: 102095, block hash: 63592d78ed7d2ebe5c49098740f0ddebe84a268a58ad287870b9b7affb738130
2021/08/10 09:44:39.122714 e[0;32m[INFO ]e[m GID 17088, RefCount height: 102096, length of refCounts: 129
2021/08/10 09:44:39.125709 e[0;32m[INFO ]e[m GID 17088, Pruning height: 101072, length of refCounts: 97
2021/08/10 09:44:39.125709 e[0;32m[INFO ]e[m GID 17088, # current block height: 102096, block hash: d7741f301101ab8d5863442e453aa8e0b414bfc24a3bcf8ce9c37adb226f1bf0
2021/08/10 09:44:39.134711 e[0;32m[INFO ]e[m GID 17088, RefCount height: 102097, length of refCounts: 232
2021/08/10 09:44:39.138710 e[0;32m[INFO ]e[m GID 17088, Pruning height: 101073, length of refCounts: 134
2021/08/10 09:44:39.138710 e[0;32m[INFO ]e[m GID 17088, # current block height: 102097, block hash: 1e333649ec48a7ac44c7eb148c1f73d784f0ecece20579524465664978f7be32
2021/08/10 09:44:39.145708 e[0;32m[INFO ]e[m GID 17088, RefCount height: 102098, length of refCounts: 131
2021/08/10 09:44:39.149734 e[0;32m[INFO ]e[m GID 17088, Pruning height: 101074, length of refCounts: 155
2021/08/10 09:44:39.149734 e[0;32m[INFO ]e[m GID 17088, # current block height: 102098, block hash: 6c28b105161ceec53dcbbd087d258ba5ada92110bc9adde400ee750081897bbb
2021/08/10 09:44:39.157713 e[0;32m[INFO ]e[m GID 17088, RefCount height: 102099, length of refCounts: 127
2021/08/10 09:44:39.163711 e[0;32m[INFO ]e[m GID 17088, Pruning height: 101075, length of refCounts: 230
2021/08/10 09:44:39.163711 e[0;32m[INFO ]e[m GID 17088, # current block height: 102099, block hash: c1a80f7849387b45f0531e4beb10d6f6c8efe915ed018566bcc697f82178b2e8
2021/08/10 09:44:39.172709 e[0;32m[INFO ]e[m GID 17088, RefCount height: 102100, length of refCounts: 176
2021/08/10 09:44:39.175709 e[0;32m[INFO ]e[m GID 17088, Pruning height: 101076, length of refCounts: 111
2021/08/10 09:44:39.175709 e[0;32m[INFO ]e[m GID 17088, # current block height: 102100, block hash: 3044327373354207e5c17e85aad65f4a044533e9b9254c7b7769fed1d89a82b3
2021/08/10 09:44:39.183709 e[0;32m[INFO ]e[m GID 17088, RefCount height: 102101, length of refCounts: 130
2021/08/10 09:44:39.199730 e[0;32m[INFO ]e[m GID 17088, Pruning height: 101077, length of refCounts: 456
2021/08/10 09:44:39.201711 e[0;32m[INFO ]e[m GID 17088, # current block height: 102101, block hash: 41b2011a03c65e9556082c7d2c2c7c44bc9332526791740f1d15473de4cd3894
2021/08/10 09:44:39.208710 e[0;32m[INFO ]e[m GID 17088, RefCount height: 102102, length of refCounts: 129
2021/08/10 09:44:39.212709 e[0;32m[INFO ]e[m GID 17088, Pruning height: 101078, length of refCounts: 182
2021/08/10 09:44:39.212709 e[0;32m[INFO ]e[m GID 17088, # current block height: 102102, block hash: febcb61c271ced02cb0274b758277eb456b3626c838fa212faaeed46cf80701b
2021/08/10 09:44:39.219709 e[0;32m[INFO ]e[m GID 17088, RefCount height: 102103, length of refCounts: 166
2021/08/10 09:44:39.224709 e[0;32m[INFO ]e[m GID 17088, Pruning height: 101079, length of refCounts: 241
2021/08/10 09:44:39.224709 e[0;32m[INFO ]e[m GID 17088, # current block height: 102103, block hash: b1588d52487ff93380343f4a551196c8c5f72105c3eecc5304008c7eca310649
2021/08/10 09:44:39.231715 e[0;32m[INFO ]e[m GID 17088, RefCount height: 102104, length of refCounts: 129
2021/08/10 09:44:39.238709 e[0;32m[INFO ]e[m GID 17088, Pruning height: 101080, length of refCounts: 137
2021/08/10 09:44:39.238709 e[0;32m[INFO ]e[m GID 17088, # current block height: 102104, block hash: fc4b3bbab3ab7db0c8c1a6465c35bd5451b6e65e62c76bce080ba8e453d4683e
2021/08/10 09:44:39.245708 e[0;32m[INFO ]e[m GID 17088, RefCount height: 102105, length of refCounts: 132
2021/08/10 09:44:39.251712 e[0;32m[INFO ]e[m GID 17088, Pruning height: 101081, length of refCounts: 139
2021/08/10 09:44:39.251712 e[0;32m[INFO ]e[m GID 17088, # current block height: 102105, block hash: 3b58ff7d9bbfda38d4c23d4ff42f3a6c76de254e79f3347f10ebd7178da9620e
2021/08/10 09:44:39.257710 e[0;32m[INFO ]e[m GID 17088, RefCount height: 102106, length of refCounts: 131
2021/08/10 09:44:39.261710 e[0;32m[INFO ]e[m GID 17088, Pruning height: 101082, length of refCounts: 156
2021/08/10 09:44:39.261710 e[0;32m[INFO ]e[m GID 17088, # current block height: 102106, block hash: 31b74387bffa77dc864c3c73c3a40447983588da75c03281e1f4ff5e5c7295cc
2021/08/10 09:44:39.270709 e[0;32m[INFO ]e[m GID 17088, RefCount height: 102107, length of refCounts: 179
2021/08/10 09:44:39.274708 e[0;32m[INFO ]e[m GID 17088, Pruning height: 101083, length of refCounts: 198
2021/08/10 09:44:39.274708 e[0;32m[INFO ]e[m GID 17088, # current block height: 102107, block hash: 737bc1ab792d43922b15ef2b6406c7bbfd6d02b7579deefa0be771940e082c5e
2021/08/10 09:44:39.283710 e[0;32m[INFO ]e[m GID 17088, RefCount height: 102108, length of refCounts: 159
2021/08/10 09:44:39.287709 e[0;32m[INFO ]e[m GID 17088, Pruning height: 101084, length of refCounts: 138
2021/08/10 09:44:39.287709 e[0;32m[INFO ]e[m GID 17088, # current block height: 102108, block hash: 41de60d9f04f2ec8b226140b980956a1667126ee3e2a5e2012fe4222b83d80c2
2021/08/10 09:44:39.294709 e[0;32m[INFO ]e[m GID 17137, RefCount height: 102109, length of refCounts: 130
2021/08/10 09:44:39.305708 e[0;32m[INFO ]e[m GID 17137, Pruning height: 101085, length of refCounts: 456
2021/08/10 09:44:39.305708 e[0;32m[INFO ]e[m GID 17137, # current block height: 102109, block hash: 58a57bab5c5cc8d1c96eecff7ab7a5090beba73bdaff8810c8832a48a5adfe52
2021/08/10 09:44:39.312709 e[0;32m[INFO ]e[m GID 17137, RefCount height: 102110, length of refCounts: 131
2021/08/10 09:44:39.316711 e[0;32m[INFO ]e[m GID 17137, Pruning height: 101086, length of refCounts: 156
2021/08/10 09:44:39.317709 e[0;32m[INFO ]e[m GID 17137, # current block height: 102110, block hash: eea58f5bf0c634169909bc56ea0f41bb44904fc12d9464e5c3600e1dbc7f8276
2021/08/10 09:44:39.325715 e[0;32m[INFO ]e[m GID 17137, RefCount height: 102111, length of refCounts: 131
2021/08/10 09:44:39.331711 e[0;32m[INFO ]e[m GID 17137, Pruning height: 101087, length of refCounts: 155
2021/08/10 09:44:39.331711 e[0;32m[INFO ]e[m GID 17137, # current block height: 102111, block hash: 6189619a730abe688668a6e50f7dfcda9144dc2bfdbb525f08f1ae39101dbb36
2021/08/10 09:44:39.338709 e[0;32m[INFO ]e[m GID 17137, RefCount height: 102112, length of refCounts: 129
2021/08/10 09:44:39.342708 e[0;32m[INFO ]e[m GID 17137, Pruning height: 101088, length of refCounts: 166
2021/08/10 09:44:39.342708 e[0;32m[INFO ]e[m GID 17137, # current block height: 102112, block hash: 89c901848f477589675dd8667c210c90ce14503627e80f6b9fcaf4b0ace81df0
2021/08/10 09:44:39.350711 e[0;32m[INFO ]e[m GID 17137, RefCount height: 102113, length of refCounts: 130
2021/08/10 09:44:39.353711 e[0;32m[INFO ]e[m GID 17137, Pruning height: 101089, length of refCounts: 111
2021/08/10 09:44:39.353711 e[0;32m[INFO ]e[m GID 17137, # current block height: 102113, block hash: c4107fefc2c231964166ccaa6d14a435885e7f73e41ff18986cd5ea9a5948752
2021/08/10 09:44:39.361708 e[0;32m[INFO ]e[m GID 17137, RefCount height: 102114, length of refCounts: 132
2021/08/10 09:44:39.366713 e[0;32m[INFO ]e[m GID 17137, Pruning height: 101090, length of refCounts: 157
2021/08/10 09:44:39.367710 e[0;32m[INFO ]e[m GID 17137, # current block height: 102114, block hash: 160c78c461a9c50e472f1521528be6ab46f8c814e57c8a9026a7224f14978b58
2021/08/10 09:44:39.374711 e[0;32m[INFO ]e[m GID 17137, RefCount height: 102115, length of refCounts: 178
2021/08/10 09:44:39.376709 e[0;32m[INFO ]e[m GID 17137, Pruning height: 101091, length of refCounts: 95
2021/08/10 09:44:39.376709 e[0;32m[INFO ]e[m GID 17137, # current block height: 102115, block hash: b0978a0d9bab2e3002c4198fe9e45eaf9180a6d0176f8670cc57e2cb084a74c4
2021/08/10 09:44:39.384709 e[0;32m[INFO ]e[m GID 17137, RefCount height: 102116, length of refCounts: 112
2021/08/10 09:44:39.399710 e[0;32m[INFO ]e[m GID 17137, Pruning height: 101092, length of refCounts: 441

...

2021/08/10 09:53:45.740181 e[0;33m[WARN ]e[m GID 72218, Get blocks error: Remote node has stopped
2021/08/10 09:53:45.761194 e[0;33m[WARN ]e[m GID 72219, Get blocks error: Remote node has stopped
2021/08/10 09:53:45.761194 e[0;33m[WARN ]e[m GID 72219, Get blocks error: Remote node has stopped
2021/08/10 09:53:45.764183 e[0;33m[WARN ]e[m GID 72219, Get blocks error: Remote node has stopped
2021/08/10 09:53:45.764183 e[0;33m[WARN ]e[m GID 72219, Get blocks error: Remote node has stopped
2021/08/10 09:53:45.780185 e[0;33m[WARN ]e[m GID 72220, Get blocks error: Remote node has stopped
2021/08/10 09:53:45.780185 e[0;33m[WARN ]e[m GID 72220, Get blocks error: Remote node has stopped
2021/08/10 09:53:45.780185 e[0;33m[WARN ]e[m GID 72220, Get blocks error: Remote node has stopped
2021/08/10 09:53:45.780185 e[0;33m[WARN ]e[m GID 72220, Get blocks error: Remote node has stopped
2021/08/10 09:53:45.780185 e[0;31m[ERROR]e[m GID 9903, Sync blocks error with batch size 1: all workers failed
2021/08/10 09:53:45.780185 e[0;35m[FATAL]e[m GID 9903, [consensus.go:L341 func2()] Error syncing blocks: sync blocks error: all workers failed

i ended up deleting the chaindb folder and started over using the --no-nat option and its run longer then any previous attempt. so far the block height is almost back to 10k.

just following up in case anyone has any suggestions. it always ends up in the same error. it will climb 4k to 10k in block height per run it seems. i am on attempt 5 and approaching 100k at this point.

[consensus.go:L341 func2()] Error syncing blocks: sync blocks error: all workers failed

So just to answer my own question after two days of experience with the sync system I have come to believe the service has the ability to virtually saturate this xfinity router, or some other upstream network component to the point where it gives up.

I wrote a simple recursive start.ps1 script to relaunch the script when it exits. Just going off the number of log files, one per startup, its restarted 26 times today.

It may be worth adding an exponential backoff to the system when its run into the situation where it detects everything has disconnected, or at least a parameter option we could use to signify non-enterprise grade routers and to expect this situation.

Another idea that could work in combination with the above would be to include an automatic throttle rather then a backoff, or both, that increased every time the error condition occurs. Something non-parameterized in this case to prevent slow ddos attacks.

function startMiner() {
  echo "Starting NKN Daemon"
  .\nknd.exe -p abcdefg --no-nat 

  echo "Restarting NKN Daemon"
  
  startMiner
}

startMiner

image