Blockheight increasing but nknd.exe fails continuously

wanabeminer · August 11, 2021, 3:25am

hello and thank you in advance,

i am just trying to understand if there is a network level block in my way from moving from SYNC_STARTED state to MINING. maybe something upstream at the xfinity provider outside of my control.

while the nknd.exe program is running:

i am able to see my blockheight increase at http://nstatus.org/. last run increased over 1k in basic int value.
localhost:30000/web status shows same sync in progress state.
i paid the 10 nkn via nknx.org add node feature.
router has UPnP on, port forward off, and i put my localhost ip in dmz, both ip4 & ip6 so all ports are open.
full log available for dm
localhost firewall set inbound/outbound accept all tcp/udp for nknd.exe filepath

# win 10, in admin mode, 100+gb free space, i7 & 32gb ddr4.
.\nknd.exe --version
nknd version v2.1.5

.\nknd.exe -p [my_pwd]

2021/08/10 09:39:49.565795 e[0;32m[INFO ]e[m GID 1, My IP address is xx.xx.xxx.xxx
2021/08/10 09:39:50.439793 e[0;32m[INFO ]e[m GID 1, set default cert domain to: xx-xx-xxx-xxx.ipv4.nknlabs.io
2021/08/10 09:39:50.439793 e[0;32m[INFO ]e[m GID 1, use default https certs
2021/08/10 09:39:50.440794 e[0;32m[INFO ]e[m GID 1, use default wss certs
2021/08/10 09:39:50.441792 e[0;32m[INFO ]e[m GID 108, https cert received
2021/08/10 09:39:50.705793 e[0;32m[INFO ]e[m GID 1, database Version: 1
2021/08/10 09:39:51.456874 e[0;32m[INFO ]e[m GID 1, State root: fe724c9273654fb1214e36d71afe8382389a9234cd7e00d5fc8776cb5b4e28b5
2021/08/10 09:39:51.456874 e[0;32m[INFO ]e[m GID 1, Start pruning...
2021/08/10 09:39:51.458880 e[0;32m[INFO ]e[m GID 1, Start compacting database from height 58988 to 102084
2021/08/10 09:40:00.738776 e[0;32m[INFO ]e[m GID 1, Start verifying database at height 102084
2021/08/10 09:40:05.800696 e[0;32m[INFO ]e[m GID 1, get no ID from local ledger

#...
# [start to see connections]

2021/08/10 09:40:07.138697 e[0;32m[INFO ]e[m GID 277, wss cert received
2021/08/10 09:40:07.138697 e[0;32m[INFO ]e[m GID 285, Set min verifiable height to 102089
2021/08/10 09:40:10.468698 e[0;32m[INFO ]e[m GID 261, Remote node connect from 3.239.79.233:44284 to local address 10.0.0.123:30001
2021/08/10 09:40:10.521696 e[0;32m[INFO ]e[m GID 261, Remote node connect from 96.126.113.179:54212 to local address 10.0.0.123:30001
2021/08/10 09:40:10.920696 e[0;32m[INFO ]e[m GID 261, Remote node connect from 143.244.168.112:42658 to local address 10.0.0.123:30001
2021/08/10 09:40:11.120245 e[0;32m[INFO ]e[m GID 261, Remote node connect from 54.221.81.75:51948 to local address 10.0.0.123:30001
2021/08/10 09:40:11.533246 e[0;32m[INFO ]e[m GID 261, Remote node connect from 147.182.189.161:42852 to local address 10.0.0.123:30001
2021/08/10 09:40:11.820246 e[0;32m[INFO ]e[m GID 261, Remote node connect from 104.248.0.50:56848 to local address 10.0.0.123:30001
2021/08/10 09:40:12.434246 e[0;32m[INFO ]e[m GID 261, Remote node connect from 18.217.21.172:51084 to local address 10.0.0.123:30001
2021/08/10 09:40:12.647246 e[0;32m[INFO ]e[m GID 261, Remote node connect from 49.51.188.42:46360 to local address 10.0.0.123:30001
2021/08/10 09:40:12.746246 e[0;32m[INFO ]e[m GID 261, Remote node connect from 3.88.145.36:53372 to local address 10.0.0.123:30001
2021/08/10 09:40:13.462446 e[0;32m[INFO ]e[m GID 256, Error handling msg: receive invalid block hash height 2958597 instead of 102085
2021/08/10 09:40:13.510445 e[0;32m[INFO ]e[m GID 261, Remote node connect from 18.118.25.142:59416 to local address 10.0.0.123:30001
2021/08/10 09:40:13.544447 e[0;32m[INFO ]e[m GID 261, Remote node connect from 45.76.18.151:46250 to local address 10.0.0.123:30001
2021/08/10 09:40:13.558446 e[0;32m[INFO ]e[m GID 256, Error handling msg: receive invalid block hash height 2958597 instead of 102085
2021/08/10 09:40:13.598460 e[0;32m[INFO ]e[m GID 261, Remote node connect from 143.198.176.155:20056 to local address 10.0.0.123:30001
2021/08/10 09:40:13.743445 e[0;32m[INFO ]e[m GID 261, Remote node connect from 43.130.47.249:38448 to local address 10.0.0.123:30001
2021/08/10 09:40:14.050448 e[0;32m[INFO ]e[m GID 261, Remote node connect from 3.234.252.155:33466 to local address 10.0.0.123:30001
2021/08/10 09:40:14.182446 e[0;32m[INFO ]e[m GID 261, Remote node connect from 147.182.198.135:51642 to local address 10.0.0.123:30001
2021/08/10 09:40:14.240445 e[0;32m[INFO ]e[m GID 261, Remote node connect from 173.199.119.200:56432 to local address 10.0.0.123:30001
2021/08/10 09:40:14.342447 e[0;32m[INFO ]e[m GID 261, Remote node connect from 143.198.172.243:52062 to local address 10.0.0.123:30001
2021/08/10 09:40:14.404445 e[0;32m[INFO ]e[m GID 261, Remote node connect from 3.239.42.132:57820 to local address 10.0.0.123:30001
2021/08/10 09:40:14.498446 e[0;32m[INFO ]e[m GID 261, Remote node connect from 142.93.67.8:57456 to local address 10.0.0.123:30001
2021/08/10 09:40:14.535447 e[0;32m[INFO ]e[m GID 261, Remote node connect from 51.81.234.83:57578 to local address 10.0.0.123:30001
2021/08/10 09:40:14.748446 e[0;32m[INFO ]e[m GID 261, Remote node connect from 54.197.80.50:51490 to local address 10.0.0.123:30001
2021/08/10 09:40:14.810445 e[0;32m[INFO ]e[m GID 261, Remote node connect from 18.205.26.63:49700 to local address 10.0.0.123:30001
2021/08/10 09:40:17.139224 e[0;33m[WARN ]e[m GID 282, Handle proposal error: wait for proposal timeout
2021/08/10 09:40:17.325222 e[0;32m[INFO ]e[m GID 261, Remote node connect from 95.179.240.77:34044 to local address 10.0.0.123:30001
2021/08/10 09:40:17.474221 e[0;32m[INFO ]e[m GID 261, Remote node connect from 143.198.175.145:48896 to local address 10.0.0.123:30001
2021/08/10 09:40:18.345805 e[0;32m[INFO ]e[m GID 261, Remote node connect from 44.233.222.145:51344 to local address 10.0.0.123:30001
2021/08/10 09:40:18.582805 e[0;32m[INFO ]e[m GID 261, Remote node connect from 49.51.189.201:33828 to local address 10.0.0.123:30001
2021/08/10 09:40:19.100894 e[0;32m[INFO ]e[m GID 261, Remote node connect from 34.221.115.250:44616 to local address 10.0.0.123:30001
2021/08/10 09:40:19.164894 e[0;32m[INFO ]e[m GID 392, Load remote node <3.87.239.165:30001> from list
2021/08/10 09:40:19.183893 e[0;32m[INFO ]e[m GID 261, Remote node connect from 172.104.240.59:35494 to local address 10.0.0.123:30001
2021/08/10 09:40:19.971906 e[0;32m[INFO ]e[m GID 261, Remote node connect from 35.175.121.38:41180 to local address 10.0.0.123:30001
...

2021/08/10 09:40:21.922920 e[0;32m[INFO ]e[m GID 261, Remote node connect from 45.77.202.84:59660 to local address 10.0.0.123:30001
2021/08/10 09:40:22.247920 e[0;32m[INFO ]e[m GID 261, Remote node connect from 207.148.1.235:34772 to local address 10.0.0.123:30001
2021/08/10 09:40:22.532922 e[0;32m[INFO ]e[m GID 261, Remote node connect from 140.82.51.66:52644 to local address 10.0.0.123:30001
2021/08/10 09:40:22.689919 e[0;32m[INFO ]e[m GID 261, Remote node connect from 3.236.97.133:52746 to local address 10.0.0.123:30001
2021/08/10 09:40:22.701922 e[0;32m[INFO ]e[m GID 261, Remote node connect from 147.182.170.158:40714 to local address 10.0.0.123:30001
2021/08/10 09:40:22.883921 e[0;32m[INFO ]e[m GID 261, Remote node connect from 207.154.222.33:44752 to local address 10.0.0.123:30001
2021/08/10 09:40:23.494100 e[0;32m[INFO ]e[m GID 261, Remote node connect from 143.198.128.117:37398 to local address 10.0.0.123:30001
2021/08/10 09:40:23.574100 e[0;32m[INFO ]e[m GID 392, Load remote node <3.87.239.165:30001> from list
2021/08/10 09:40:23.868108 e[0;32m[INFO ]e[m GID 261, Remote node connect from 108.61.184.100:55322 to local address 10.0.0.123:30001
2021/08/10 09:40:24.014100 e[0;32m[INFO ]e[m GID 261, Remote node connect from 54.193.0.196:34084 to local address 10.0.0.123:30001
2021/08/10 09:40:24.093101 e[0;32m[INFO ]e[m GID 261, Remote node connect from 54.177.25.247:50264 to local address 10.0.0.123:30001
2021/08/10 09:40:24.548100 e[0;32m[INFO ]e[m GID 261, Remote node connect from 147.182.233.150:37298 to local address 10.0.0.123:30001
2021/08/10 09:40:25.562711 e[0;32m[INFO ]e[m GID 261, Remote node connect from 45.77.207.153:60376 to local address 10.0.0.123:30001
2021/08/10 09:40:25.600709 e[0;32m[INFO ]e[m GID 261, Remote node connect from 147.182.203.167:51034 to local address 10.0.0.123:30001
2021/08/10 09:40:26.235711 e[0;32m[INFO ]e[m GID 261, Remote node connect from 147.182.245.37:34400 to local address 10.0.0.123:30001
2021/08/10 09:40:26.337709 e[0;32m[INFO ]e[m GID 261, Remote node connect from 45.63.10.119:46542 to local address 10.0.0.123:30001
2021/08/10 09:40:26.525713 e[0;32m[INFO ]e[m GID 261, Remote node connect from 143.198.180.94:39546 to local address 10.0.0.123:30001
2021/08/10 09:40:26.775710 e[0;32m[INFO ]e[m GID 230, DropTxns: 0 txns (0 bytes) in txpool, no need to drop
2021/08/10 09:40:26.988710 e[0;32m[INFO ]e[m GID 261, Remote node connect from 178.128.150.204:39834 to local address 10.0.0.123:30001
2021/08/10 09:40:27.237826 e[0;33m[WARN ]e[m GID 282, Handle proposal error: wait for neighbor vote timeout
2021/08/10 09:40:28.016824 e[0;32m[INFO ]e[m GID 261, Remote node connect from 35.174.60.242:36308 to local address 10.0.0.123:30001
...
2021/08/10 09:40:34.188206 e[0;32m[INFO ]e[m GID 256, Error handling msg: block 19c3caaf03df773e29ab2d00007e57a8d9a877003f2b8e25e6a9d48bd3c10f70 is not finalized yet
2021/08/10 09:40:34.211205 e[0;32m[INFO ]e[m GID 261, Remote node connect from 34.219.110.204:49936 to local address 10.0.0.123:30001
2021/08/10 09:40:34.306206 e[0;32m[INFO ]e[m GID 256, Error handling msg: block 19c3caaf03df773e29ab2d00007e57a8d9a877003f2b8e25e6a9d48bd3c10f70 is not finalized yet
2021/08/10 09:40:35.691823 e[0;32m[INFO ]e[m GID 256, Error handling msg: receive invalid block hash height 2958598 instead of 102085
2021/08/10 09:40:35.772823 e[0;32m[INFO ]e[m GID 256, Error handling msg: receive invalid block hash height 2958598 instead of 102085
2021/08/10 09:40:35.823824 e[0;32m[INFO ]e[m GID 256, Error handling msg: receive invalid block hash height 2958598 instead of 102085
2021/08/10 09:40:35.894822 e[0;32m[INFO ]e[m GID 256, Error handling msg: receive invalid block hash height 2958598 instead of 102085
2021/08/10 09:40:35.963821 e[0;32m[INFO ]e[m GID 256, Error handling msg: receive invalid block hash height 2958598 instead of 102085
2021/08/10 09:40:36.017823 e[0;32m[INFO ]e[m GID 256, Error handling msg: receive invalid block hash height 2958598 instead of 102085
2021/08/10 09:40:36.037823 e[0;32m[INFO ]e[m GID 256, Error handling msg: receive invalid block hash height 2958598 instead of 102085
2021/08/10 09:40:36.143823 e[0;32m[INFO ]e[m GID 256, Error handling msg: receive invalid block hash height 2958598 instead of 102085
2021/08/10 09:40:36.214822 e[0;32m[INFO ]e[m GID 256, Error handling msg: receive invalid block hash height 2958598 instead of 102085
2021/08/10 09:40:36.304823 e[0;32m[INFO ]e[m GID 397, Remote node b7ae396672396dff75aead4fe3573d1047beed5b9f01b431034b6e7f96e1f8fe@tcp://143.198.176.155:30001<143.198.176.155:20056> stops
2021/08/10 09:40:36.304823 e[0;32m[INFO ]e[m GID 397, Remote node b7b5cc91606354c2b8ceec7a904a9d699f75a619d2296f809363f315fb43a195@tcp://143.198.128.117:30001<143.198.128.117:37398> stops
2021/08/10 09:40:36.307823 e[0;32m[INFO ]e[m GID 397, Remote node 77a4a3ec05cfcd955b8442ce18a2a98d6da574639988ffe07446df28ca75db12@tcp://147.182.189.161:30001<147.182.189.161:42852> stops
...
2021/08/10 09:40:44.933361 e[0;32m[INFO ]e[m GID 253, Handle transactions msg error: [VerifyTransaction] id version 1 is not supported at height 102085
2021/08/10 09:40:45.128360 e[0;32m[INFO ]e[m GID 3050, Remote node 7654a765e2570fac10f80d193ea0dd7d3f854b7b55e695558f5d0409a1651577@tcp://143.198.174.82:30001<143.198.174.82:30001> stops
2021/08/10 09:40:45.165360 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node 7654a765e2570fac10f80d193ea0dd7d3f854b7b55e695558f5d0409a1651577@tcp://143.198.174.82:30001<143.198.174.82:30001>
2021/08/10 09:40:45.277361 e[0;32m[INFO ]e[m GID 2642, Connect to random neighbor ab8c248197612c738b1bce7f68957b8225354c57ae5a7f9daa3202afcf117381@tcp://104.156.225.95:30001
2021/08/10 09:40:45.449361 e[0;32m[INFO ]e[m GID 2642, Connect to random neighbor b8f887fffe632ce759c31dcebc941242a16935d5bac134257a54ab06335f2e6c@tcp://34.125.100.239:30001
2021/08/10 09:40:45.518365 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node 68826f47fc74a1442f00dd9655072571317f98e533fba920661c74db8051e4d4@tcp://23.90.33.164:30001<23.90.33.164:30001>
2021/08/10 09:40:45.518365 e[0;32m[INFO ]e[m GID 260, Remote node 68826f47fc74a1442f00dd9655072571317f98e533fba920661c74db8051e4d4@tcp://23.90.33.164:30001<23.90.33.164:30001> stops

#... connections start to timeout

2021/08/10 09:44:16.195885 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node <3.223.6.244:30001>
2021/08/10 09:44:16.196884 e[0;32m[INFO ]e[m GID 261, Remote node connect from 54.177.25.247:50288 to local address 10.0.0.123:30001
2021/08/10 09:44:16.829884 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node <54.177.25.247:50288>
2021/08/10 09:44:16.829884 e[0;32m[INFO ]e[m GID 260, Remote node <54.177.25.247:50288> stops
2021/08/10 09:44:16.983886 e[0;32m[INFO ]e[m GID 261, Remote node connect from 161.35.5.244:53048 to local address 10.0.0.123:30001
2021/08/10 09:44:17.057885 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node b7959d0d4bde70b62245fc07d5fca551d8fdbd0c9dfc688396697a2331ba7237@tcp://3.238.127.2:30001<3.238.127.2:30001>
2021/08/10 09:44:17.058885 e[0;32m[INFO ]e[m GID 260, Remote node b7959d0d4bde70b62245fc07d5fca551d8fdbd0c9dfc688396697a2331ba7237@tcp://3.238.127.2:30001<3.238.127.2:30001> stops
2021/08/10 09:44:17.114904 e[0;32m[INFO ]e[m GID 261, Remote node connect from 3.239.42.132:58008 to local address 10.0.0.123:30001
...
2021/08/10 09:44:18.210983 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node bb032bd30a4cbf618e691489aa295e667040aa76b5889830a03e8f9c3913bcb8@tcp://207.148.24.232:30001<207.148.24.232:30001>
2021/08/10 09:44:18.359616 e[0;33m[WARN ]e[m GID 10321, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.360628 e[0;33m[WARN ]e[m GID 10407, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.363617 e[0;33m[WARN ]e[m GID 10419, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.363617 e[0;33m[WARN ]e[m GID 10428, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.363617 e[0;33m[WARN ]e[m GID 10403, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.363617 e[0;33m[WARN ]e[m GID 10421, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.364616 e[0;33m[WARN ]e[m GID 10423, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.364616 e[0;33m[WARN ]e[m GID 10420, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.364616 e[0;33m[WARN ]e[m GID 10415, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.364616 e[0;33m[WARN ]e[m GID 10405, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.364616 e[0;33m[WARN ]e[m GID 10318, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.365617 e[0;33m[WARN ]e[m GID 10409, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.365617 e[0;33m[WARN ]e[m GID 10426, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.365617 e[0;33m[WARN ]e[m GID 10321, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.365617 e[0;33m[WARN ]e[m GID 10407, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.365617 e[0;33m[WARN ]e[m GID 10419, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.365617 e[0;33m[WARN ]e[m GID 10428, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.365617 e[0;33m[WARN ]e[m GID 10403, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.365617 e[0;33m[WARN ]e[m GID 10423, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.365617 e[0;33m[WARN ]e[m GID 10421, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.365617 e[0;33m[WARN ]e[m GID 10420, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.365617 e[0;33m[WARN ]e[m GID 10415, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.365617 e[0;33m[WARN ]e[m GID 10405, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.365617 e[0;33m[WARN ]e[m GID 10318, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.366617 e[0;33m[WARN ]e[m GID 10409, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.366617 e[0;33m[WARN ]e[m GID 10407, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.366617 e[0;33m[WARN ]e[m GID 10426, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.366617 e[0;33m[WARN ]e[m GID 10321, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.366617 e[0;33m[WARN ]e[m GID 10419, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.366617 e[0;33m[WARN ]e[m GID 10428, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.366617 e[0;33m[WARN ]e[m GID 10403, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.366617 e[0;33m[WARN ]e[m GID 10423, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.366617 e[0;33m[WARN ]e[m GID 10421, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.366617 e[0;33m[WARN ]e[m GID 10420, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.366617 e[0;33m[WARN ]e[m GID 10405, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.366617 e[0;33m[WARN ]e[m GID 10409, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.366617 e[0;33m[WARN ]e[m GID 10318, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.366617 e[0;33m[WARN ]e[m GID 10407, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.367621 e[0;33m[WARN ]e[m GID 10426, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.367621 e[0;33m[WARN ]e[m GID 10321, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.367621 e[0;33m[WARN ]e[m GID 10419, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.367621 e[0;33m[WARN ]e[m GID 10428, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.367621 e[0;33m[WARN ]e[m GID 10403, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.367621 e[0;33m[WARN ]e[m GID 10421, Get block headers error: Remote node has stopped
2021/08/10 09:44:18.413616 e[0;31m[ERROR]e[m GID 392, Update successors error: Wait for reply timeout
2021/08/10 09:44:18.489619 e[0;32m[INFO ]e[m GID 2643, Connect to random neighbor 0f6b4a58884f60ab1bbea1ad6c7427e2ae038a4df69eb992097f43359c0be5f1@tcp://140.82.42.199:30001
2021/08/10 09:44:18.630617 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node <54.177.25.247:50290>
2021/08/10 09:44:18.630617 e[0;32m[INFO ]e[m GID 260, Remote node <54.177.25.247:50290> stops
2021/08/10 09:44:19.157132 e[0;33m[WARN ]e[m GID 15118, Remote node da0120963d06628ccc945d11344379f0e99a298de1902fc325bcbf9421909dd6@tcp://35.165.200.164:30001<35.165.200.164:30001> stops because of error: Read msg len error: io: read/write on closed pipe
2021/08/10 09:44:19.189130 e[0;32m[INFO ]e[m GID 261, Remote node connect from 3.239.42.132:58010 to local address 10.0.0.123:30001
2021/08/10 09:44:19.279132 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node <3.239.42.132:58010>
2021/08/10 09:44:19.280133 e[0;32m[INFO ]e[m GID 260, Remote node b793e4df2dbf35992cebbee0df5d49905e41119539d42aa9ee23db768f9a0f30@tcp://3.239.42.132:30001<3.239.42.132:58010> stops
2021/08/10 09:44:19.382131 e[0;32m[INFO ]e[m GID 2643, Connect to random neighbor c20cfc8ae6f8f9edede4931585c16d51ac213affa1fd8486e2308ca762542237@tcp://44.234.63.118:30001
2021/08/10 09:44:19.733135 e[0;32m[INFO ]e[m GID 15668, Remote node c20cfc8ae6f8f9edede4931585c16d51ac213affa1fd8486e2308ca762542237@tcp://44.234.63.118:30001<44.234.63.118:30001> stops
2021/08/10 09:44:19.813133 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node c20cfc8ae6f8f9edede4931585c16d51ac213affa1fd8486e2308ca762542237@tcp://44.234.63.118:30001<44.234.63.118:30001>
2021/08/10 09:44:19.909133 e[0;32m[INFO ]e[m GID 15655, Remote node c00ab3fd2868e300cbea8734e8d68ce6be2424a6af7afb2d7b85c4fbb426cd87@tcp://3.141.104.18:30001<3.141.104.18:30001> stops
2021/08/10 09:44:19.941132 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node c00ab3fd2868e300cbea8734e8d68ce6be2424a6af7afb2d7b85c4fbb426cd87@tcp://3.141.104.18:30001<3.141.104.18:30001>
2021/08/10 09:44:20.301896 e[0;32m[INFO ]e[m GID 2643, Connect to random neighbor 41991934a317315069d1bef0bbc27d6bba0de02dc5168f0e532439c7cc85183b@tcp://18.210.24.71:30001
2021/08/10 09:44:20.433898 e[0;32m[INFO ]e[m GID 2643, Connect to random neighbor 69d55dfdc0e8e8c0766c58abe2a12ebca574f923c4657814d6daa09c3130c8cd@tcp://3.101.131.78:30001
2021/08/10 09:44:21.215926 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node adf779fa9cb880bc99971333c40de86638af111c0a87f6c5a2a529f39465ce20@tcp://45.33.77.247:30001<45.33.77.247:30001>
2021/08/10 09:44:21.215926 e[0;32m[INFO ]e[m GID 260, Remote node adf779fa9cb880bc99971333c40de86638af111c0a87f6c5a2a529f39465ce20@tcp://45.33.77.247:30001<45.33.77.247:30001> stops
2021/08/10 09:44:21.273925 e[0;32m[INFO ]e[m GID 261, Remote node connect from 54.177.25.247:50292 to local address 10.0.0.123:30001
2021/08/10 09:44:21.571936 e[0;32m[INFO ]e[m GID 261, Remote node connect from 3.239.42.132:58012 to local address 10.0.0.123:30001
2021/08/10 09:44:21.595927 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node <3.239.42.132:58012>
2021/08/10 09:44:21.595927 e[0;32m[INFO ]e[m GID 260, Remote node b793e4df2dbf35992cebbee0df5d49905e41119539d42aa9ee23db768f9a0f30@tcp://3.239.42.132:30001<3.239.42.132:58012> stops
2021/08/10 09:44:21.595927 e[0;32m[INFO ]e[m GID 2643, Connect to random neighbor 65ddded132ec7b518c87a8831c7af3beed564afe53f53bb3f75175480da3b14b@tcp://207.148.19.128:30001
2021/08/10 09:44:21.781925 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node <54.177.25.247:50292>
2021/08/10 09:44:21.781925 e[0;32m[INFO ]e[m GID 260, Remote node <54.177.25.247:50292> stops
2021/08/10 09:44:22.020028 e[0;33m[WARN ]e[m GID 12991, Remote node b7aac1d6a8947a2da38877ead146ac47abc97ab3cef4c1455ea2145ab83ef4f8@tcp://34.230.58.144:30001<34.230.58.144:30001> stops because of error: keepalive timeout
2021/08/10 09:44:22.551030 e[0;32m[INFO ]e[m GID 15834, Remote node cdd81ef549a35e9b6f44177bcff5b19d1bd22917465b07081c0b93a8bd48cd65@tcp://104.248.233.99:30001<104.248.233.99:30001> stops
2021/08/10 09:44:22.583030 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node cdd81ef549a35e9b6f44177bcff5b19d1bd22917465b07081c0b93a8bd48cd65@tcp://104.248.233.99:30001<104.248.233.99:30001>
2021/08/10 09:44:22.867029 e[0;32m[INFO ]e[m GID 261, Remote node connect from 3.238.127.2:59466 to local address 10.0.0.123:30001
2021/08/10 09:44:23.085030 e[0;33m[WARN ]e[m GID 13398, Remote node b797ec2cf5e3d39d403f7f08a76aad61d5509599ba54f3e90c91ac09f53ef4c1@tcp://147.182.170.158:30001<147.182.170.158:30001> stops because of error: keepalive timeout
2021/08/10 09:44:23.088028 e[0;33m[WARN ]e[m GID 13729, Remote node b764b39b29e85f00be031236481edd55ef5b7d05f9af809635de4e8756909825@tcp://147.182.233.150:30001<147.182.233.150:37444> stops because of error: keepalive timeout
2021/08/10 09:44:23.117034 e[0;33m[WARN ]e[m GID 12934, Remote node b7b55dbf8ce45b808a3186fbd7986f48029c88b89406348307e1a733478753c1@tcp://143.198.180.94:30001<143.198.180.94:30001> stops because of error: Read msg len error: io: read/write on closed pipe
2021/08/10 09:44:23.150028 e[0;33m[WARN ]e[m GID 1308, Remote node b797d90eaec3cc3083610365944b00aa1bf45a4620218c8b7a764ed6f9ff2780@tcp://165.227.196.116:30001<165.227.196.116:38482> stops because of error: keepalive timeout
2021/08/10 09:44:23.174028 e[0;33m[WARN ]e[m GID 12002, Remote node b724bef3be833d752dce6f04550ac339141121a2296fdd78f88bdf56d7af159d@tcp://51.81.51.252:30001<51.81.51.252:45220> stops because of error: keepalive timeout
2021/08/10 09:44:23.182027 e[0;32m[INFO ]e[m GID 261, Remote node connect from 165.227.196.116:38498 to local address 10.0.0.123:30001
2021/08/10 09:44:23.247027 e[0;32m[INFO ]e[m GID 261, Remote node connect from 143.198.180.94:39562 to local address 10.0.0.123:30001
2021/08/10 09:44:23.247027 e[0;33m[WARN ]e[m GID 15869, Remote node <165.227.196.116:38498> stops because of error: Node with id b797d90eaec3cc3083610365944b00aa1bf45a4620218c8b7a764ed6f9ff2780 is already connected at addr 165.227.196.116:38482
2021/08/10 09:44:23.258029 e[0;32m[INFO ]e[m GID 2642, Connect to random neighbor 9c06a165957810541866cb5a5317c98f7298e60842dc3c63fdb633fc617a335c@tcp://143.198.59.112:30001
2021/08/10 09:44:23.277029 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node <165.227.196.116:38498>
2021/08/10 09:44:23.338030 e[0;31m[ERROR]e[m GID 393, Update predecessor error: Wait for reply timeout
2021/08/10 09:44:23.463029 e[0;32m[INFO ]e[m GID 261, Remote node connect from 95.179.240.77:34066 to local address 10.0.0.123:30001
2021/08/10 09:44:23.589029 e[0;33m[WARN ]e[m GID 15976, Remote node <95.179.240.77:34066> stops because of error: Node with id b79dfa7f3861c898711d8a258e027de5cba45758573b4f1d4e841fdc80ae1dc2 is already connected at addr 95.179.240.77:34064
2021/08/10 09:44:23.643028 e[0;33m[WARN ]e[m GID 8211, Remote node 49bf531dd3ed1737d9ed8de116a35fae46dccce173ac565337eb9a1ac576d5a4@tcp://34.217.121.201:30001<34.217.121.201:30001> stops because of error: keepalive timeout
2021/08/10 09:44:23.657028 e[0;33m[WARN ]e[m GID 13647, Remote node b5a5378366bdc4de58e1f5461062ec9505b00978b0b26d1f4f566ae1807ac284@tcp://45.63.10.119:30001<45.63.10.119:46574> stops because of error: keepalive timeout
2021/08/10 09:44:23.676029 e[0;33m[WARN ]e[m GID 14084, Remote node b79dfa7f3861c898711d8a258e027de5cba45758573b4f1d4e841fdc80ae1dc2@tcp://95.179.240.77:30001<95.179.240.77:34064> stops because of error: keepalive timeout
2021/08/10 09:44:23.759027 e[0;32m[INFO ]e[m GID 261, Remote node connect from 147.182.170.158:40730 to local address 10.0.0.123:30001
2021/08/10 09:44:23.782031 e[0;32m[INFO ]e[m GID 261, Remote node connect from 3.87.239.165:49064 to local address 10.0.0.123:30001
2021/08/10 09:44:23.792029 e[0;32m[INFO ]e[m GID 261, Remote node connect from 147.182.204.251:45384 to local address 10.0.0.123:30001
2021/08/10 09:44:23.850030 e[0;33m[WARN ]e[m GID 11407, Remote node ed9272f7d28015a12fb6862a9ffbb6f083a1b91f92acaee816dca5a2097aff1d@tcp://45.77.96.118:30001<45.77.96.118:30001> stops because of error: keepalive timeout
2021/08/10 09:44:23.884030 e[0;33m[WARN ]e[m GID 13761, Remote node b7a5a308f214c1f4392ba40454bc534528686d5da83b28109e1856f2f66dbef5@tcp://147.182.204.251:30001<147.182.204.251:30001> stops because of error: keepalive timeout
2021/08/10 09:44:23.904027 e[0;33m[WARN ]e[m GID 15996, Remote node <147.182.204.251:45384> stops because of error: Node with id b7a5a308f214c1f4392ba40454bc534528686d5da83b28109e1856f2f66dbef5 is already connected at addr 147.182.204.251:30001
2021/08/10 09:44:24.010029 e[0;32m[INFO ]e[m GID 261, Remote node connect from 3.239.42.132:58014 to local address 10.0.0.123:30001
2021/08/10 09:44:24.030030 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node b793e4df2dbf35992cebbee0df5d49905e41119539d42aa9ee23db768f9a0f30@tcp://3.239.42.132:30001<3.239.42.132:58014>
2021/08/10 09:44:24.030030 e[0;32m[INFO ]e[m GID 260, Remote node b793e4df2dbf35992cebbee0df5d49905e41119539d42aa9ee23db768f9a0f30@tcp://3.239.42.132:30001<3.239.42.132:58014> stops
2021/08/10 09:44:24.441028 e[0;33m[WARN ]e[m GID 14164, Remote node b7bdbcbcdfc24fad09182d2ca7be1c57594c1ad5a1a13803495d1c5f4f8493f6@tcp://45.77.207.153:30001<45.77.207.153:60528> stops because of error: keepalive timeout
2021/08/10 09:44:24.444028 e[0;32m[INFO ]e[m GID 16141, Remote node e5af745bfde5ad2738fad8eb3e8fcc5dad4c516d3253ee1c49263003c6b784e2@tcp://206.189.233.206:30001<206.189.233.206:30001> stops
2021/08/10 09:44:24.471028 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node e5af745bfde5ad2738fad8eb3e8fcc5dad4c516d3253ee1c49263003c6b784e2@tcp://206.189.233.206:30001<206.189.233.206:30001>
2021/08/10 09:44:24.483031 e[0;32m[INFO ]e[m GID 397, Remote node b7a68c3adcabaa25020fe9cc0200f1b8bbc6e0d680de3c7450d9804af2a39bfa@tcp://18.117.96.156:30001<18.117.96.156:40812> stops
2021/08/10 09:44:24.483031 e[0;32m[INFO ]e[m GID 397, Remote node b7a7acc29879e81915cd82dd85528f0f4746f1897bd3bccc1884b4221c083800@tcp://161.35.5.244:30001<161.35.5.244:53048> stops
2021/08/10 09:44:24.483031 e[0;32m[INFO ]e[m GID 397, Remote node b7aa3f25a0875469529cda734faddc13699e0935ad398d5ee0cf5a9559e0ecb3@tcp://143.198.175.49:30001<143.198.175.49:45948> stops
2021/08/10 09:44:24.484030 e[0;32m[INFO ]e[m GID 397, Remote node b7adf54c4ad983ca2e7acc9074c9a6cdb04a8db41cf7ed362e9d582ce14cd2a2@tcp://3.223.6.244:30001<3.223.6.244:59348> stops
2021/08/10 09:44:24.484030 e[0;32m[INFO ]e[m GID 397, Remote node b7b67c2aa49072aa98028bef06a96b7e11982f21a3a5ed716e2d890f6c2a3640@tcp://207.154.222.33:30001<207.154.222.33:44902> stops
2021/08/10 09:44:24.509030 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node b7adf54c4ad983ca2e7acc9074c9a6cdb04a8db41cf7ed362e9d582ce14cd2a2@tcp://3.223.6.244:30001<3.223.6.244:59348>
2021/08/10 09:44:24.511031 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node b7a7acc29879e81915cd82dd85528f0f4746f1897bd3bccc1884b4221c083800@tcp://161.35.5.244:30001<161.35.5.244:53048>
2021/08/10 09:44:24.515037 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node b7a68c3adcabaa25020fe9cc0200f1b8bbc6e0d680de3c7450d9804af2a39bfa@tcp://18.117.96.156:30001<18.117.96.156:40812>
2021/08/10 09:44:24.516032 e[0;32m[INFO ]e[m GID 2643, Connect to random neighbor ec8eda73e54dd072c81590210511f9c2e2f957d05e14641f629f43117e174733@tcp://147.182.183.19:30001
2021/08/10 09:44:24.521029 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node b7aa3f25a0875469529cda734faddc13699e0935ad398d5ee0cf5a9559e0ecb3@tcp://143.198.175.49:30001<143.198.175.49:45948>
2021/08/10 09:44:24.624028 e[0;32m[INFO ]e[m GID 282, Elected block hash e8db7609f31ee8073bb39d8c336b65a40b8458f318a89f7599204376f77af259 got 103/106 neighbor votes, weight: 26 (96.30%)
2021/08/10 09:44:24.624028 e[0;32m[INFO ]e[m GID 282, Accept block e8db7609f31ee8073bb39d8c336b65a40b8458f318a89f7599204376f77af259 at height 2958607
2021/08/10 09:44:24.728029 e[0;33m[WARN ]e[m GID 12214, Remote node b799599d5f8f3a97d491f193e6a142062b256281bfb7b614a37c2b9025a68cc8@tcp://54.221.81.75:30001<54.221.81.75:52012> stops because of error: Read msg len error: io: read/write on closed pipe
2021/08/10 09:44:24.760026 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node b7a09f676b9df1308bdfc758a5fe523a9f1f027eb19640ac7b98a1a693d18d29@tcp://159.65.35.196:30001<159.65.35.196:30001>
2021/08/10 09:44:24.766028 e[0;32m[INFO ]e[m GID 260, Remote node b7a09f676b9df1308bdfc758a5fe523a9f1f027eb19640ac7b98a1a693d18d29@tcp://159.65.35.196:30001<159.65.35.196:30001> stops
2021/08/10 09:44:25.590029 e[0;32m[INFO ]e[m GID 256, Error handling msg: block 88f7e91bea00a9cda0e5390dc0ce44c4a4982faa604fe572f8b512b2b5073510 is not finalized yet
2021/08/10 09:44:25.682029 e[0;31m[ERROR]e[m GID 392, Update successors error: Wait for reply timeout
2021/08/10 09:44:26.590170 e[0;32m[INFO ]e[m GID 256, Error handling msg: block 88f7e91bea00a9cda0e5390dc0ce44c4a4982faa604fe572f8b512b2b5073510 is not finalized yet
2021/08/10 09:44:26.784169 e[0;32m[INFO ]e[m GID 230, DropTxns: 4 txns (1221 bytes) in txpool, no need to drop
2021/08/10 09:44:26.942169 e[0;32m[INFO ]e[m GID 261, Remote node connect from 143.198.175.49:45952 to local address 10.0.0.123:30001
2021/08/10 09:44:27.106170 e[0;32m[INFO ]e[m GID 16293, Remote node 094c377ac1b7ccad16ecb6bdf1825bae8c30c336f77625c00d85ea42fae2f966@tcp://3.142.151.16:30001<3.142.151.16:30001> stops
2021/08/10 09:44:27.185169 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node 094c377ac1b7ccad16ecb6bdf1825bae8c30c336f77625c00d85ea42fae2f966@tcp://3.142.151.16:30001<3.142.151.16:30001>
...
2021/08/10 09:44:31.066160 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node <34.230.58.144:30001>
2021/08/10 09:44:31.067160 e[0;32m[INFO ]e[m GID 260, Remote node b7aac1d6a8947a2da38877ead146ac47abc97ab3cef4c1455ea2145ab83ef4f8@tcp://34.230.58.144:30001<34.230.58.144:30001> stops
2021/08/10 09:44:31.208161 e[0;32m[INFO ]e[m GID 261, Remote node connect from 18.117.96.156:40814 to local address 10.0.0.123:30001
2021/08/10 09:44:32.407889 e[0;33m[WARN ]e[m GID 14006, Remote node b7ae84e1528c5002be7aa86a91b5215f05d533dcf640c99d872a87ba1539580c@tcp://207.154.194.250:30001<207.154.194.250:30001> stops because of error: keepalive timeout
2021/08/10 09:44:32.451887 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node <34.230.58.144:30001>
2021/08/10 09:44:32.451887 e[0;32m[INFO ]e[m GID 260, Remote node <34.230.58.144:30001> stops
2021/08/10 09:44:32.724887 e[0;32m[INFO ]e[m GID 2642, Connect to random neighbor 7f2122e412397de7749c2d7c44fe750f5eb0bf379ca10e99ca4d9d30f29204cb@tcp://52.73.230.232:30001
2021/08/10 09:44:32.905887 e[0;32m[INFO ]e[m GID 2643, Connect to random neighbor dba55c5237282e4113039cfe8a5b83d6eedfc88185d13b3a4adf5a29aae3f051@tcp://198.50.252.32:30001
2021/08/10 09:44:34.382633 e[0;32m[INFO ]e[m GID 2643, Connect to random neighbor 20adcfa16efeebf6fa8d8b85638578668b1ec09c07b74e9731fda0cb27087e8c@tcp://147.182.227.21:30001
2021/08/10 09:44:34.653633 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node <34.230.58.144:30001>
2021/08/10 09:44:34.653633 e[0;32m[INFO ]e[m GID 260, Remote node <34.230.58.144:30001> stops
2021/08/10 09:44:34.686633 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node b7ae84e1528c5002be7aa86a91b5215f05d533dcf640c99d872a87ba1539580c@tcp://207.154.194.250:30001<207.154.194.250:30001>
...
2021/08/10 09:44:35.246635 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node <54.177.25.247:50296>
2021/08/10 09:44:35.246635 e[0;32m[INFO ]e[m GID 260, Remote node <54.177.25.247:50296> stops
2021/08/10 09:44:35.330632 e[0;32m[INFO ]e[m GID 261, Remote node connect from 161.35.221.158:59524 to local address 10.0.0.123:30001
2021/08/10 09:44:35.455633 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node <34.230.58.144:36700>
2021/08/10 09:44:35.456634 e[0;32m[INFO ]e[m GID 260, Remote node <34.230.58.144:36700> stops
2021/08/10 09:44:36.856633 e[0;32m[INFO ]e[m GID 260, Received stop message from remote node <34.230.58.144:30001>
2021/08/10 09:44:36.856633 e[0;32m[INFO ]e[m GID 260, Remote node <34.230.58.144:30001> stops
2021/08/10 09:44:37.730634 e[0;32m[INFO ]e[m GID 261, Remote node connect from 147.182.245.37:34552 to local address 10.0.0.123:30001
2021/08/10 09:44:37.759633 e[0;32m[INFO ]e[m GID 282, Change expected block height to 2958609

#...
# [prune always followed by failure]

2021/08/10 09:44:38.966710 e[0;33m[WARN ]e[m GID 17148, Get blocks error: Remote node has stopped
2021/08/10 09:44:38.966710 e[0;33m[WARN ]e[m GID 17148, Get blocks error: Remote node has stopped
2021/08/10 09:44:38.969719 e[0;33m[WARN ]e[m GID 17148, Get blocks error: Remote node has stopped
2021/08/10 09:44:38.969719 e[0;33m[WARN ]e[m GID 17148, Get blocks error: Remote node has stopped
2021/08/10 09:44:39.032709 e[0;32m[INFO ]e[m GID 17088, RefCount height: 102089, length of refCounts: 332
2021/08/10 09:44:39.035709 e[0;32m[INFO ]e[m GID 17088, Pruning height: 101065, length of refCounts: 154
2021/08/10 09:44:39.035709 e[0;32m[INFO ]e[m GID 17088, # current block height: 102089, block hash: 37045b38fe587fab2701cfdac5595433d79625998e918f218fdbbf773b5b1f36
2021/08/10 09:44:39.043710 e[0;32m[INFO ]e[m GID 17088, RefCount height: 102090, length of refCounts: 129
2021/08/10 09:44:39.048719 e[0;32m[INFO ]e[m GID 17088, Pruning height: 101066, length of refCounts: 111
2021/08/10 09:44:39.049712 e[0;32m[INFO ]e[m GID 17088, # current block height: 102090, block hash: 7bf15c817e663c8b26b5d15e0e107eb0dfc82afde2724248a79cface2be65a1e
2021/08/10 09:44:39.057712 e[0;32m[INFO ]e[m GID 17088, RefCount height: 102091, length of refCounts: 124
2021/08/10 09:44:39.064711 e[0;32m[INFO ]e[m GID 17088, Pruning height: 101067, length of refCounts: 230
2021/08/10 09:44:39.064711 e[0;32m[INFO ]e[m GID 17088, # current block height: 102091, block hash: b759bd215180715c2703547d625972e6cdd0382c3a5d03f1b434d8a6edffce09
2021/08/10 09:44:39.071709 e[0;32m[INFO ]e[m GID 17088, RefCount height: 102092, length of refCounts: 130
2021/08/10 09:44:39.074711 e[0;32m[INFO ]e[m GID 17088, Pruning height: 101068, length of refCounts: 114
2021/08/10 09:44:39.076712 e[0;32m[INFO ]e[m GID 17088, # current block height: 102092, block hash: 61ae1a79fe64afc4035381c826fcf0571f016bdc92a40f6b8f9677b521eb75e8
2021/08/10 09:44:39.084719 e[0;32m[INFO ]e[m GID 17088, RefCount height: 102093, length of refCounts: 129
2021/08/10 09:44:39.087709 e[0;32m[INFO ]e[m GID 17088, Pruning height: 101069, length of refCounts: 112
2021/08/10 09:44:39.087709 e[0;32m[INFO ]e[m GID 17088, # current block height: 102093, block hash: f4ff2cc84b60883148eb310c0b234525b0dbf048616f86b0ce4168b128832fee
2021/08/10 09:44:39.094708 e[0;32m[INFO ]e[m GID 17088, RefCount height: 102094, length of refCounts: 224
2021/08/10 09:44:39.100709 e[0;32m[INFO ]e[m GID 17088, Pruning height: 101070, length of refCounts: 195
2021/08/10 09:44:39.100709 e[0;32m[INFO ]e[m GID 17088, # current block height: 102094, block hash: aa02d7527e0590725118628bff7813132bd9f62e51efeae105e54adb379a4c82
2021/08/10 09:44:39.108710 e[0;32m[INFO ]e[m GID 17088, RefCount height: 102095, length of refCounts: 181
2021/08/10 09:44:39.115716 e[0;32m[INFO ]e[m GID 17088, Pruning height: 101071, length of refCounts: 240
2021/08/10 09:44:39.115716 e[0;32m[INFO ]e[m GID 17088, # current block height: 102095, block hash: 63592d78ed7d2ebe5c49098740f0ddebe84a268a58ad287870b9b7affb738130
2021/08/10 09:44:39.122714 e[0;32m[INFO ]e[m GID 17088, RefCount height: 102096, length of refCounts: 129
2021/08/10 09:44:39.125709 e[0;32m[INFO ]e[m GID 17088, Pruning height: 101072, length of refCounts: 97
2021/08/10 09:44:39.125709 e[0;32m[INFO ]e[m GID 17088, # current block height: 102096, block hash: d7741f301101ab8d5863442e453aa8e0b414bfc24a3bcf8ce9c37adb226f1bf0
2021/08/10 09:44:39.134711 e[0;32m[INFO ]e[m GID 17088, RefCount height: 102097, length of refCounts: 232
2021/08/10 09:44:39.138710 e[0;32m[INFO ]e[m GID 17088, Pruning height: 101073, length of refCounts: 134
2021/08/10 09:44:39.138710 e[0;32m[INFO ]e[m GID 17088, # current block height: 102097, block hash: 1e333649ec48a7ac44c7eb148c1f73d784f0ecece20579524465664978f7be32
2021/08/10 09:44:39.145708 e[0;32m[INFO ]e[m GID 17088, RefCount height: 102098, length of refCounts: 131
2021/08/10 09:44:39.149734 e[0;32m[INFO ]e[m GID 17088, Pruning height: 101074, length of refCounts: 155
2021/08/10 09:44:39.149734 e[0;32m[INFO ]e[m GID 17088, # current block height: 102098, block hash: 6c28b105161ceec53dcbbd087d258ba5ada92110bc9adde400ee750081897bbb
2021/08/10 09:44:39.157713 e[0;32m[INFO ]e[m GID 17088, RefCount height: 102099, length of refCounts: 127
2021/08/10 09:44:39.163711 e[0;32m[INFO ]e[m GID 17088, Pruning height: 101075, length of refCounts: 230
2021/08/10 09:44:39.163711 e[0;32m[INFO ]e[m GID 17088, # current block height: 102099, block hash: c1a80f7849387b45f0531e4beb10d6f6c8efe915ed018566bcc697f82178b2e8
2021/08/10 09:44:39.172709 e[0;32m[INFO ]e[m GID 17088, RefCount height: 102100, length of refCounts: 176
2021/08/10 09:44:39.175709 e[0;32m[INFO ]e[m GID 17088, Pruning height: 101076, length of refCounts: 111
2021/08/10 09:44:39.175709 e[0;32m[INFO ]e[m GID 17088, # current block height: 102100, block hash: 3044327373354207e5c17e85aad65f4a044533e9b9254c7b7769fed1d89a82b3
2021/08/10 09:44:39.183709 e[0;32m[INFO ]e[m GID 17088, RefCount height: 102101, length of refCounts: 130
2021/08/10 09:44:39.199730 e[0;32m[INFO ]e[m GID 17088, Pruning height: 101077, length of refCounts: 456
2021/08/10 09:44:39.201711 e[0;32m[INFO ]e[m GID 17088, # current block height: 102101, block hash: 41b2011a03c65e9556082c7d2c2c7c44bc9332526791740f1d15473de4cd3894
2021/08/10 09:44:39.208710 e[0;32m[INFO ]e[m GID 17088, RefCount height: 102102, length of refCounts: 129
2021/08/10 09:44:39.212709 e[0;32m[INFO ]e[m GID 17088, Pruning height: 101078, length of refCounts: 182
2021/08/10 09:44:39.212709 e[0;32m[INFO ]e[m GID 17088, # current block height: 102102, block hash: febcb61c271ced02cb0274b758277eb456b3626c838fa212faaeed46cf80701b
2021/08/10 09:44:39.219709 e[0;32m[INFO ]e[m GID 17088, RefCount height: 102103, length of refCounts: 166
2021/08/10 09:44:39.224709 e[0;32m[INFO ]e[m GID 17088, Pruning height: 101079, length of refCounts: 241
2021/08/10 09:44:39.224709 e[0;32m[INFO ]e[m GID 17088, # current block height: 102103, block hash: b1588d52487ff93380343f4a551196c8c5f72105c3eecc5304008c7eca310649
2021/08/10 09:44:39.231715 e[0;32m[INFO ]e[m GID 17088, RefCount height: 102104, length of refCounts: 129
2021/08/10 09:44:39.238709 e[0;32m[INFO ]e[m GID 17088, Pruning height: 101080, length of refCounts: 137
2021/08/10 09:44:39.238709 e[0;32m[INFO ]e[m GID 17088, # current block height: 102104, block hash: fc4b3bbab3ab7db0c8c1a6465c35bd5451b6e65e62c76bce080ba8e453d4683e
2021/08/10 09:44:39.245708 e[0;32m[INFO ]e[m GID 17088, RefCount height: 102105, length of refCounts: 132
2021/08/10 09:44:39.251712 e[0;32m[INFO ]e[m GID 17088, Pruning height: 101081, length of refCounts: 139
2021/08/10 09:44:39.251712 e[0;32m[INFO ]e[m GID 17088, # current block height: 102105, block hash: 3b58ff7d9bbfda38d4c23d4ff42f3a6c76de254e79f3347f10ebd7178da9620e
2021/08/10 09:44:39.257710 e[0;32m[INFO ]e[m GID 17088, RefCount height: 102106, length of refCounts: 131
2021/08/10 09:44:39.261710 e[0;32m[INFO ]e[m GID 17088, Pruning height: 101082, length of refCounts: 156
2021/08/10 09:44:39.261710 e[0;32m[INFO ]e[m GID 17088, # current block height: 102106, block hash: 31b74387bffa77dc864c3c73c3a40447983588da75c03281e1f4ff5e5c7295cc
2021/08/10 09:44:39.270709 e[0;32m[INFO ]e[m GID 17088, RefCount height: 102107, length of refCounts: 179
2021/08/10 09:44:39.274708 e[0;32m[INFO ]e[m GID 17088, Pruning height: 101083, length of refCounts: 198
2021/08/10 09:44:39.274708 e[0;32m[INFO ]e[m GID 17088, # current block height: 102107, block hash: 737bc1ab792d43922b15ef2b6406c7bbfd6d02b7579deefa0be771940e082c5e
2021/08/10 09:44:39.283710 e[0;32m[INFO ]e[m GID 17088, RefCount height: 102108, length of refCounts: 159
2021/08/10 09:44:39.287709 e[0;32m[INFO ]e[m GID 17088, Pruning height: 101084, length of refCounts: 138
2021/08/10 09:44:39.287709 e[0;32m[INFO ]e[m GID 17088, # current block height: 102108, block hash: 41de60d9f04f2ec8b226140b980956a1667126ee3e2a5e2012fe4222b83d80c2
2021/08/10 09:44:39.294709 e[0;32m[INFO ]e[m GID 17137, RefCount height: 102109, length of refCounts: 130
2021/08/10 09:44:39.305708 e[0;32m[INFO ]e[m GID 17137, Pruning height: 101085, length of refCounts: 456
2021/08/10 09:44:39.305708 e[0;32m[INFO ]e[m GID 17137, # current block height: 102109, block hash: 58a57bab5c5cc8d1c96eecff7ab7a5090beba73bdaff8810c8832a48a5adfe52
2021/08/10 09:44:39.312709 e[0;32m[INFO ]e[m GID 17137, RefCount height: 102110, length of refCounts: 131
2021/08/10 09:44:39.316711 e[0;32m[INFO ]e[m GID 17137, Pruning height: 101086, length of refCounts: 156
2021/08/10 09:44:39.317709 e[0;32m[INFO ]e[m GID 17137, # current block height: 102110, block hash: eea58f5bf0c634169909bc56ea0f41bb44904fc12d9464e5c3600e1dbc7f8276
2021/08/10 09:44:39.325715 e[0;32m[INFO ]e[m GID 17137, RefCount height: 102111, length of refCounts: 131
2021/08/10 09:44:39.331711 e[0;32m[INFO ]e[m GID 17137, Pruning height: 101087, length of refCounts: 155
2021/08/10 09:44:39.331711 e[0;32m[INFO ]e[m GID 17137, # current block height: 102111, block hash: 6189619a730abe688668a6e50f7dfcda9144dc2bfdbb525f08f1ae39101dbb36
2021/08/10 09:44:39.338709 e[0;32m[INFO ]e[m GID 17137, RefCount height: 102112, length of refCounts: 129
2021/08/10 09:44:39.342708 e[0;32m[INFO ]e[m GID 17137, Pruning height: 101088, length of refCounts: 166
2021/08/10 09:44:39.342708 e[0;32m[INFO ]e[m GID 17137, # current block height: 102112, block hash: 89c901848f477589675dd8667c210c90ce14503627e80f6b9fcaf4b0ace81df0
2021/08/10 09:44:39.350711 e[0;32m[INFO ]e[m GID 17137, RefCount height: 102113, length of refCounts: 130
2021/08/10 09:44:39.353711 e[0;32m[INFO ]e[m GID 17137, Pruning height: 101089, length of refCounts: 111
2021/08/10 09:44:39.353711 e[0;32m[INFO ]e[m GID 17137, # current block height: 102113, block hash: c4107fefc2c231964166ccaa6d14a435885e7f73e41ff18986cd5ea9a5948752
2021/08/10 09:44:39.361708 e[0;32m[INFO ]e[m GID 17137, RefCount height: 102114, length of refCounts: 132
2021/08/10 09:44:39.366713 e[0;32m[INFO ]e[m GID 17137, Pruning height: 101090, length of refCounts: 157
2021/08/10 09:44:39.367710 e[0;32m[INFO ]e[m GID 17137, # current block height: 102114, block hash: 160c78c461a9c50e472f1521528be6ab46f8c814e57c8a9026a7224f14978b58
2021/08/10 09:44:39.374711 e[0;32m[INFO ]e[m GID 17137, RefCount height: 102115, length of refCounts: 178
2021/08/10 09:44:39.376709 e[0;32m[INFO ]e[m GID 17137, Pruning height: 101091, length of refCounts: 95
2021/08/10 09:44:39.376709 e[0;32m[INFO ]e[m GID 17137, # current block height: 102115, block hash: b0978a0d9bab2e3002c4198fe9e45eaf9180a6d0176f8670cc57e2cb084a74c4
2021/08/10 09:44:39.384709 e[0;32m[INFO ]e[m GID 17137, RefCount height: 102116, length of refCounts: 112
2021/08/10 09:44:39.399710 e[0;32m[INFO ]e[m GID 17137, Pruning height: 101092, length of refCounts: 441

...

2021/08/10 09:53:45.740181 e[0;33m[WARN ]e[m GID 72218, Get blocks error: Remote node has stopped
2021/08/10 09:53:45.761194 e[0;33m[WARN ]e[m GID 72219, Get blocks error: Remote node has stopped
2021/08/10 09:53:45.761194 e[0;33m[WARN ]e[m GID 72219, Get blocks error: Remote node has stopped
2021/08/10 09:53:45.764183 e[0;33m[WARN ]e[m GID 72219, Get blocks error: Remote node has stopped
2021/08/10 09:53:45.764183 e[0;33m[WARN ]e[m GID 72219, Get blocks error: Remote node has stopped
2021/08/10 09:53:45.780185 e[0;33m[WARN ]e[m GID 72220, Get blocks error: Remote node has stopped
2021/08/10 09:53:45.780185 e[0;33m[WARN ]e[m GID 72220, Get blocks error: Remote node has stopped
2021/08/10 09:53:45.780185 e[0;33m[WARN ]e[m GID 72220, Get blocks error: Remote node has stopped
2021/08/10 09:53:45.780185 e[0;33m[WARN ]e[m GID 72220, Get blocks error: Remote node has stopped
2021/08/10 09:53:45.780185 e[0;31m[ERROR]e[m GID 9903, Sync blocks error with batch size 1: all workers failed
2021/08/10 09:53:45.780185 e[0;35m[FATAL]e[m GID 9903, [consensus.go:L341 func2()] Error syncing blocks: sync blocks error: all workers failed

wanabeminer · August 10, 2021, 3:20pm

i ended up deleting the chaindb folder and started over using the --no-nat option and its run longer then any previous attempt. so far the block height is almost back to 10k.

wanabeminer · August 11, 2021, 1:45am

just following up in case anyone has any suggestions. it always ends up in the same error. it will climb 4k to 10k in block height per run it seems. i am on attempt 5 and approaching 100k at this point.

[consensus.go:L341 func2()] Error syncing blocks: sync blocks error: all workers failed

wanabeminer · August 12, 2021, 2:46am

So just to answer my own question after two days of experience with the sync system I have come to believe the service has the ability to virtually saturate this xfinity router, or some other upstream network component to the point where it gives up.

I wrote a simple recursive start.ps1 script to relaunch the script when it exits. Just going off the number of log files, one per startup, its restarted 26 times today.

It may be worth adding an exponential backoff to the system when its run into the situation where it detects everything has disconnected, or at least a parameter option we could use to signify non-enterprise grade routers and to expect this situation.

Another idea that could work in combination with the above would be to include an automatic throttle rather then a backoff, or both, that increased every time the error condition occurs. Something non-parameterized in this case to prevent slow ddos attacks.

function startMiner() {
  echo "Starting NKN Daemon"
  .\nknd.exe -p abcdefg --no-nat 

  echo "Restarting NKN Daemon"
  
  startMiner
}

startMiner