Panic: Rollback error: local ledger has forked

Nodes & Mining
#1

So this happened about 24 hours ago with 2 of my nodes. They have been unable to recover.
This morning I manually ran an nknupdate and rebooted. Still doesn’t seem to want to recover.
Do I need to just let it run? Is there an issue? Anything I can do? See logs below…

Dec 13 14:18:26 nyc3-n6 nknd[2407]: 2019/12/13 14:18:26.498593 #033[0;32m[INFO ]#033[m GID 1881, Set sync state to SYNC_STARTED                                                                                                                          
Dec 13 14:18:29 nyc3-n6 nknd[2407]: 2019/12/13 14:18:29.817042 #033[0;33m[WARN ]#033[m GID 1881, Local block hash 7746ab1486c6e232685c4c265684876e098bcbce1778297a25750737c6f3e7eb is different from neighbors' majority block hash 34ec593f79350dda4d27b
9f158fc531957fcc3adf8a79e4b1f85dc69f78126ff at height 623905, rollback needed                                                                                                     
Dec 13 14:18:33 nyc3-n6 nknd[2407]: panic: Rollback error: local ledger has forked for more than 180 blocks                                                
Dec 13 14:18:33 nyc3-n6 nknd[2407]: goroutine 1881 [running]:                                                                                                                                                Dec 13 14:18:33 nyc3-n6 nknd[2407]: github.com/nknorg/nkn/node.(*LocalNode).StartSyncing.func1()                                                             
Dec 13 14:18:33 nyc3-n6 nknd[2407]: #011/home/nknx/go/src/github.com/nknorg/nkn/node/syncblock.go:334 +0xada                              
Dec 13 14:18:33 nyc3-n6 nknd[2407]: sync.(*Once).doSlow(0xc00039f940, 0xc0022fb638)                                                                                       
Dec 13 14:18:33 nyc3-n6 nknd[2407]: #011/usr/local/go/src/sync/once.go:66 +0xe3                                                                                                      
Dec 13 14:18:33 nyc3-n6 nknd[2407]: sync.(*Once).Do(...)                                                                                                                                                            
Dec 13 14:18:33 nyc3-n6 nknd[2407]: #011/usr/local/go/src/sync/once.go:57                                                                                                                    
Dec 13 14:18:33 nyc3-n6 nknd[2407]: github.com/nknorg/nkn/node.(*LocalNode).StartSyncing(0xc000250380, 0x3338099807c11fff, 0xb2e5640d8f955e4c, 0x1a5dd0d92cff9ff6, 0x770128188f8af6b, 0x9ac44, 0xc001ce2c00, 0x24, 0x40, 0xc0022fb7b0, ...)
Dec 13 14:18:33 nyc3-n6 nknd[2407]: #011/home/nknx/go/src/github.com/nknorg/nkn/node/syncblock.go:315 +0x159                              
Dec 13 14:18:33 nyc3-n6 nknd[2407]: github.com/nknorg/nkn/consensus.(*Consensus).saveAcceptedBlock.func2(0xc002bb45a0, 0xc0000f2f70, 0xc001ce2c00, 0x24, 0x40)                                                                                           
Dec 13 14:18:33 nyc3-n6 nknd[2407]: #011/home/nknx/go/src/github.com/nknorg/nkn/consensus/consensus.go:338 +0xfe                    
Dec 13 14:18:33 nyc3-n6 nknd[2407]: created by github.com/nknorg/nkn/consensus.(*Consensus).saveAcceptedBlock                          
Dec 13 14:18:33 nyc3-n6 nknd[2407]: #011/home/nknx/go/src/github.com/nknorg/nkn/consensus/consensus.go:336 +0x62a                
Dec 13 14:18:36 nyc3-n6 nknd[2495]: 2019/12/13 14:18:36 Set SyncBatchWindowSize to 61                                                                                    
Dec 13 14:18:36 nyc3-n6 nknd[2495]: 2019/12/13 14:18:36 Set TxPoolMaxMemorySize to 3                                                                                       
Dec 13 14:18:36 nyc3-n6 nknd[2495]: 2019/12/13 14:18:36 Discovering NAT gateway...

on another note … all of my miners are now aroun block 629### so currently 4000 blocks behind!
up until a couple days ago they’ve never been more than a 100 blocks behind but never live. What gives?

#2

In regards to last note: looks like nknx is just not receiving the most recent metrics.

root@ams3-n1:/home/nknx/go/src/github.com/nknorg/nkn# ./nknc info -s
{
“id”: “1”,
“jsonrpc”: “2.0”,
“result”: {
“addr”: “tcp://142.93.234.7:30001”,
“currTimeStamp”: 1576265047,
“height”: 634761,
“id”: “b7008fae91605608e9d41bc71be2c77049e2abd28dbd1d4317855600f770a1c3”,
“jsonRpcPort”: 30003,
“proposalSubmitted”: 0,
“protocolVersion”: 30,
“publicKey”: “0c81e263b16d39b6bec8c9bc68557f3910087520f586bfe864341ba135d9e52c”,
“relayMessageCount”: 4494,
“syncState”: “PERSIST_FINISHED”,
“tlsWebsocketDomain”: “142-93-234-7.ipv4.nknlabs.io”,
“uptime”: 11421,
“version”: “v1.1.1-beta-11-gcd058”,
“websocketPort”: 30002
}

#3

This text will be hiddenlatest log summary

  • Set sync state to SYNC_STARTED
  • Change expected block height to 634841
- Local block hash 7746ab1486c6e232685c4c265684876e098bcbce1778297a25750737c6f3e7eb is different from neighbors' majority block hash 34ec593f79350dda4d27b9f158fc531957fcc3adf8a79e4b1f85dc69f78126ff at height 623905, rollback needed
- panic: Rollback error: local ledger has forked for more than 180 blocks
- goroutine 1895 [running]:
- github.com/nknorg/nkn/node.(*LocalNode).StartSyncing.func1()
- #011/home/nknx/go/src/github.com/nknorg/nkn/node/syncblock.go:334 +0xada
- sync.(*Once).doSlow(0xc00027d0c0, 0xc0005b1e38)
- #011/usr/local/go/src/sync/once.go:66 +0xe3
- sync.(*Once).Do(...)
- #011/usr/local/go/src/sync/once.go:57
- github.com/nknorg/nkn/node.(*LocalNode).StartSyncing(0xc0002aa380, 0xd4a7972757bc42e4, 0x96300934b99a7135, 0x5668d58d4f7f59c0, 0x8d712e36d362ea22, 0x9afd6, 0xc002660a00, 0x28, 0x40, 0xc0005b1fb0, ...)
- #011/home/nknx/go/src/github.com/nknorg/nkn/node/syncblock.go:315 +0x159
- github.com/nknorg/nkn/consensus.(*Consensus).saveAcceptedBlock.func2(0xc002ca58f0, 0xc0001cf6c0, 0xc002660a00, 0x28, 0x40)
- #011/home/nknx/go/src/github.com/nknorg/nkn/consensus/consensus.go:338 +0xfe
- created by github.com/nknorg/nkn/consensus.(*Consensus).saveAcceptedBlock
- #011/home/nknx/go/src/github.com/nknorg/nkn/consensus/consensus.go:336 +0x62a
- 2019/12/13 19:52:31 Set SyncBatchWindowSize to 61
- 2019/12/13 19:52:31 Set TxPoolMaxMemorySize to 3
  • 2019/12/13 19:52:31 Discovering NAT gateway…
    But then after getting to SynStart again the process starts all over
    Never actually syncs

Entire log below:

Dec 13 19:51:28 nyc3-n6 nknd[11838]: 2019/12/13 19:51:28.463399 #033[0;32m[INFO ]#033[m GID 254, Set min verifiable height to 634844
Dec 13 19:51:29 nyc3-n6 nknd[11838]: 2019/12/13 19:51:29.062139 #033[0;32m[INFO ]#033[m GID 72, DropTxns: 0 txns (0 bytes) in txpool, no need to drop
Dec 13 19:51:29 nyc3-n6 nknd[11838]: 2019/12/13 19:51:29.970557 #033[0;33m[WARN ]#033[m GID 251, Handle proposal error: Wait for neighbor vote timeout
Dec 13 19:51:30 nyc3-n6 nknd[11838]: 2019/12/13 19:51:30.021952 #033[0;32m[INFO ]#033[m GID 251, Change expected block height to 634839
Dec 13 19:51:36 nyc3-n6 nknd[11838]: 2019/12/13 19:51:36.782377 #033[0;32m[INFO ]#033[m GID 94, Remote node connect from 104.248.144.210:40236 to local address 159.203.87.251:30001
Dec 13 19:51:38 nyc3-n6 nknd[11838]: 2019/12/13 19:51:38.248058 #033[0;32m[INFO ]#033[m GID 1612, Remote node c6236915e9750a6c89ae2d7f54c0671657e82f42f722fe4c7bdb99d291a34a3c@tcp://149.28.225.32:30001<149.28.225.32:30001> stops
Dec 13 19:51:38 nyc3-n6 nknd[11838]: 2019/12/13 19:51:38.254375 #033[0;32m[INFO ]#033[m GID 93, Received stop message from remote node c6236915e9750a6c89ae2d7f54c0671657e82f42f722fe4c7bdb99d291a34a3c@tcp://149.28.225.32:30001<149.28.225.32:30001>
Dec 13 19:51:39 nyc3-n6 nknd[11838]: 2019/12/13 19:51:39.043385 #033[0;33m[WARN ]#033[m GID 85, Handle transactions msg error: nonce is not continuous
Dec 13 19:51:40 nyc3-n6 nknd[11838]: 2019/12/13 19:51:40.061111 #033[0;33m[WARN ]#033[m GID 251, Handle proposal error: Wait for neighbor vote timeout
Dec 13 19:51:41 nyc3-n6 nknd[11838]: 2019/12/13 19:51:41.534710 #033[0;32m[INFO ]#033[m GID 253, Request block ce48069d2275a56e5486c1faa94f57c381bc3016f68173f15b7341434ef03ffe from neighbor d883deb1f5d342753ec22661e548cc9eebc6a02f32246e95de032cf2ac0e1467
Dec 13 19:51:41 nyc3-n6 nknd[11838]: 2019/12/13 19:51:41.537115 #033[0;32m[INFO ]#033[m GID 253, Receive block info ce48069d2275a56e5486c1faa94f57c381bc3016f68173f15b7341434ef03ffe, 0 txn found in pool, 3 txn to request
Dec 13 19:51:41 nyc3-n6 nknd[11838]: 2019/12/13 19:51:41.539329 #033[0;32m[INFO ]#033[m GID 253, Receive block proposal ce48069d2275a56e5486c1faa94f57c381bc3016f68173f15b7341434ef03ffe (3 txn, 3345 bytes) by d8d8638f152e8eb6137bea342120d1bdeca1f5ea19aa6156efe6b61d9cfec4eb
Dec 13 19:51:48 nyc3-n6 nknd[11838]: 2019/12/13 19:51:48.899565 #033[0;32m[INFO ]#033[m GID 1730, Remote node c61f3b938618896ee18373e1f0e242cb8227179e1b1df2022a50c8b3b67fc300@tcp://35.171.193.111:30001<35.171.193.111:30001> stops
Dec 13 19:51:48 nyc3-n6 nknd[11838]: 2019/12/13 19:51:48.927357 #033[0;32m[INFO ]#033[m GID 93, Received stop message from remote node c61f3b938618896ee18373e1f0e242cb8227179e1b1df2022a50c8b3b67fc300@tcp://35.171.193.111:30001<35.171.193.111:30001>
Dec 13 19:51:49 nyc3-n6 nknd[11838]: 2019/12/13 19:51:49.063277 #033[0;32m[INFO ]#033[m GID 72, DropTxns: 0 txns (0 bytes) in txpool, no need to drop
Dec 13 19:51:51 nyc3-n6 nknd[11838]: 2019/12/13 19:51:51.818367 #033[0;32m[INFO ]#033[m GID 251, Change expected block height to 634840
Dec 13 19:51:54 nyc3-n6 nknd[11838]: 2019/12/13 19:51:54.165225 #033[0;32m[INFO ]#033[m GID 93, Received stop message from remote node c64b03947739d7b8163b68b0460a8d52bddaef00111755fe580401191dc6301d@tcp://173.199.119.231:30001<173.199.119.231:53378>
Dec 13 19:51:54 nyc3-n6 nknd[11838]: 2019/12/13 19:51:54.167415 #033[0;32m[INFO ]#033[m GID 93, Remote node c64b03947739d7b8163b68b0460a8d52bddaef00111755fe580401191dc6301d@tcp://173.199.119.231:30001<173.199.119.231:53378> stops
Dec 13 19:51:57 nyc3-n6 nknd[11838]: 2019/12/13 19:51:57.329316 #033[0;32m[INFO ]#033[m GID 94, Remote node connect from 50.116.41.154:38682 to local address 159.203.87.251:30001
Dec 13 19:51:58 nyc3-n6 nknd[11838]: 2019/12/13 19:51:58.660122 #033[0;33m[WARN ]#033[m GID 85, Handle transactions msg error: nonce is not continuous
Dec 13 19:52:00 nyc3-n6 nknd[11838]: 2019/12/13 19:52:00.761652 #033[0;32m[INFO ]#033[m GID 1874, Remote node c61aca5c4b23d805d72c9f76315e1f4b4f8d4b47e98acc55f2392f0f5ba4540c@tcp://104.156.254.23:30001<104.156.254.23:30001> stops
Dec 13 19:52:00 nyc3-n6 nknd[11838]: 2019/12/13 19:52:00.788205 #033[0;32m[INFO ]#033[m GID 93, Received stop message from remote node c61aca5c4b23d805d72c9f76315e1f4b4f8d4b47e98acc55f2392f0f5ba4540c@tcp://104.156.254.23:30001<104.156.254.23:30001>
Dec 13 19:52:01 nyc3-n6 nknd[11838]: 2019/12/13 19:52:01.819763 #033[0;32m[INFO ]#033[m GID 251, Elected block hash ce48069d2275a56e5486c1faa94f57c381bc3016f68173f15b7341434ef03ffe got 42/43 neighbor votes, weight: 13 (100.00%)
Dec 13 19:52:01 nyc3-n6 nknd[11838]: 2019/12/13 19:52:01.820807 #033[0;32m[INFO ]#033[m GID 251, Accept block ce48069d2275a56e5486c1faa94f57c381bc3016f68173f15b7341434ef03ffe at height 634839
Dec 13 19:52:01 nyc3-n6 nknd[11838]: 2019/12/13 19:52:01.821696 #033[0;32m[INFO ]#033[m GID 251, Accepted block height: 634839, local ledger block height: 623905, sync needed.
Dec 13 19:52:01 nyc3-n6 nknd[11838]: 2019/12/13 19:52:01.823812 #033[0;32m[INFO ]#033[m GID 1895, Set sync state to SYNC_STARTED
Dec 13 19:52:02 nyc3-n6 nknd[11838]: 2019/12/13 19:52:02.844012 #033[0;32m[INFO ]#033[m GID 253, Request block 6260d423c280a6dea64c32788649599972fb663b4d9a906ccdbb67f07d253668 from neighbor e8500c506f9843ee524c4c7dcef4c2dc904c978ee63e077816f9686b4ef19b50
Dec 13 19:52:02 nyc3-n6 nknd[11838]: 2019/12/13 19:52:02.846434 #033[0;32m[INFO ]#033[m GID 253, Receive block info 6260d423c280a6dea64c32788649599972fb663b4d9a906ccdbb67f07d253668, 0 txn found in pool, 2 txn to request
Dec 13 19:52:02 nyc3-n6 nknd[11838]: 2019/12/13 19:52:02.848281 #033[0;32m[INFO ]#033[m GID 253, Receive block proposal 6260d423c280a6dea64c32788649599972fb663b4d9a906ccdbb67f07d253668 (2 txn, 2751 bytes) by e8fdc9edcdc72dfd7371ed6730ac818fca8c2e005543e12d9ff79acfab12f924
Dec 13 19:52:04 nyc3-n6 nknd[11838]: 2019/12/13 19:52:04.834835 #033[0;32m[INFO ]#033[m GID 93, Received stop message from remote node c64e57308af9d5eab511b4ba949364839d0c8e80126cffa5c54a222cf65847c7@tcp://104.248.144.210:30001<104.248.144.210:40236>
Dec 13 19:52:04 nyc3-n6 nknd[11838]: 2019/12/13 19:52:04.837468 #033[0;32m[INFO ]#033[m GID 93, Remote node c64e57308af9d5eab511b4ba949364839d0c8e80126cffa5c54a222cf65847c7@tcp://104.248.144.210:30001<104.248.144.210:40236> stops
Dec 13 19:52:09 nyc3-n6 nknd[11838]: 2019/12/13 19:52:09.064555 #033[0;32m[INFO ]#033[m GID 72, DropTxns: 1 txns (154 bytes) in txpool, no need to drop
Dec 13 19:52:09 nyc3-n6 nknd[11838]: 2019/12/13 19:52:09.596273 #033[0;32m[INFO ]#033[m GID 1990, Remote node c61971cb63f2a8ea70d2fcce9eb79bf55926e630709587c19a6de1833894bec3@tcp://35.185.248.192:30001<35.185.248.192:30001> stops
Dec 13 19:52:09 nyc3-n6 nknd[11838]: 2019/12/13 19:52:09.660938 #033[0;32m[INFO ]#033[m GID 93, Received stop message from remote node c61971cb63f2a8ea70d2fcce9eb79bf55926e630709587c19a6de1833894bec3@tcp://35.185.248.192:30001<35.185.248.192:30001>
Dec 13 19:52:13 nyc3-n6 nknd[11838]: 2019/12/13 19:52:13.077879 #033[0;32m[INFO ]#033[m GID 251, Change expected block height to 634841
Dec 13 19:52:18 nyc3-n6 nknd[11838]: 2019/12/13 19:52:18.460048 #033[0;32m[INFO ]#033[m GID 2101, Remote node c60e5e2624353ace123b6203c33a7ea50e992fc61226afe6861e7f18a05b2b0a@tcp://172.104.31.117:30001<172.104.31.117:30001> stops
Dec 13 19:52:18 nyc3-n6 nknd[11838]: 2019/12/13 19:52:18.470684 #033[0;32m[INFO ]#033[m GID 93, Received stop message from remote node c60e5e2624353ace123b6203c33a7ea50e992fc61226afe6861e7f18a05b2b0a@tcp://172.104.31.117:30001<172.104.31.117:30001>
Dec 13 19:52:23 nyc3-n6 nknd[11838]: 2019/12/13 19:52:23.079246 #033[0;32m[INFO ]#033[m GID 251, Elected block hash 6260d423c280a6dea64c32788649599972fb663b4d9a906ccdbb67f07d253668 got 42/43 neighbor votes, weight: 15 (100.00%)
Dec 13 19:52:23 nyc3-n6 nknd[11838]: 2019/12/13 19:52:23.080531 #033[0;32m[INFO ]#033[m GID 251, Accept block 6260d423c280a6dea64c32788649599972fb663b4d9a906ccdbb67f07d253668 at height 634840
Dec 13 19:52:24 nyc3-n6 nknd[11838]: 2019/12/13 19:52:24.668311 #033[0;32m[INFO ]#033[m GID 253, Request block 4fdf358834b3a97ec1ffe1228156d1ffdcd56aff58b92bba5669b88df57e66df from neighbor c5ee377ed241d817f466c93cf4ea27824403149249b5d3efb4bbe20a4d72fd63
Dec 13 19:52:24 nyc3-n6 nknd[11838]: 2019/12/13 19:52:24.692562 #033[0;32m[INFO ]#033[m GID 253, Receive block info 4fdf358834b3a97ec1ffe1228156d1ffdcd56aff58b92bba5669b88df57e66df, 1 txn found in pool, 4 txn to request
Dec 13 19:52:24 nyc3-n6 nknd[11838]: 2019/12/13 19:52:24.698827 #033[0;32m[INFO ]#033[m GID 253, Receive block proposal 4fdf358834b3a97ec1ffe1228156d1ffdcd56aff58b92bba5669b88df57e66df (5 txn, 3613 bytes) by bf9def3a7ba75055ffff086b54a4ad55fb9747acdf17ff7194ebdc393ecffc8f
Dec 13 19:52:24 nyc3-n6 nknd[11838]: 2019/12/13 19:52:24.828882 #033[0;31m[ERROR]#033[m GID 1975, Error sending sync messge to node: Wait for reply timeout
Dec 13 19:52:24 nyc3-n6 nknd[11838]: 2019/12/13 19:52:24.829626 #033[0;33m[WARN ]#033[m GID 1975, Get block header at height 623905 from neighbor c64e57308af9d5eab511b4ba949364839d0c8e80126cffa5c54a222cf65847c7 error: Wait for reply timeout
Dec 13 19:52:24 nyc3-n6 nknd[11838]: 2019/12/13 19:52:24.830265 #033[0;33m[WARN ]#033[m GID 1895, Local block hash 7746ab1486c6e232685c4c265684876e098bcbce1778297a25750737c6f3e7eb is different from neighbors' majority block hash 34ec593f79350dda4d27b9f158fc531957fcc3adf8a79e4b1f85dc69f78126ff at height 623905, rollback needed
Dec 13 19:52:27 nyc3-n6 nknd[11838]: 2019/12/13 19:52:27.831742 #033[0;31m[ERROR]#033[m GID 2172, Error sending sync messge to node: Remote node has stopped
Dec 13 19:52:27 nyc3-n6 nknd[11838]: 2019/12/13 19:52:27.837054 #033[0;33m[WARN ]#033[m GID 2172, Get block header at height 623725 from neighbor c61971cb63f2a8ea70d2fcce9eb79bf55926e630709587c19a6de1833894bec3 error: Remote node has stopped
Dec 13 19:52:27 nyc3-n6 nknd[11838]: 2019/12/13 19:52:27.839266 #033[0;31m[ERROR]#033[m GID 2200, Error sending sync messge to node: Remote node has stopped
Dec 13 19:52:27 nyc3-n6 nknd[11838]: 2019/12/13 19:52:27.840132 #033[0;33m[WARN ]#033[m GID 2200, Get block header at height 623725 from neighbor c64e57308af9d5eab511b4ba949364839d0c8e80126cffa5c54a222cf65847c7 error: Remote node has stopped
Dec 13 19:52:28 nyc3-n6 nknd[11838]: panic: Rollback error: local ledger has forked for more than 180 blocks
Dec 13 19:52:28 nyc3-n6 nknd[11838]: goroutine 1895 [running]:
Dec 13 19:52:28 nyc3-n6 nknd[11838]: github.com/nknorg/nkn/node.(*LocalNode).StartSyncing.func1()
Dec 13 19:52:28 nyc3-n6 nknd[11838]: #011/home/nknx/go/src/github.com/nknorg/nkn/node/syncblock.go:334 +0xada
Dec 13 19:52:28 nyc3-n6 nknd[11838]: sync.(*Once).doSlow(0xc00027d0c0, 0xc0005b1e38)
Dec 13 19:52:28 nyc3-n6 nknd[11838]: #011/usr/local/go/src/sync/once.go:66 +0xe3
Dec 13 19:52:28 nyc3-n6 nknd[11838]: sync.(*Once).Do(...)
Dec 13 19:52:28 nyc3-n6 nknd[11838]: #011/usr/local/go/src/sync/once.go:57
Dec 13 19:52:28 nyc3-n6 nknd[11838]: github.com/nknorg/nkn/node.(*LocalNode).StartSyncing(0xc0002aa380, 0xd4a7972757bc42e4, 0x96300934b99a7135, 0x5668d58d4f7f59c0, 0x8d712e36d362ea22, 0x9afd6, 0xc002660a00, 0x28, 0x40, 0xc0005b1fb0, ...)
Dec 13 19:52:28 nyc3-n6 nknd[11838]: #011/home/nknx/go/src/github.com/nknorg/nkn/node/syncblock.go:315 +0x159
Dec 13 19:52:28 nyc3-n6 nknd[11838]: github.com/nknorg/nkn/consensus.(*Consensus).saveAcceptedBlock.func2(0xc002ca58f0, 0xc0001cf6c0, 0xc002660a00, 0x28, 0x40)
Dec 13 19:52:28 nyc3-n6 nknd[11838]: #011/home/nknx/go/src/github.com/nknorg/nkn/consensus/consensus.go:338 +0xfe
Dec 13 19:52:28 nyc3-n6 nknd[11838]: created by github.com/nknorg/nkn/consensus.(*Consensus).saveAcceptedBlock
Dec 13 19:52:28 nyc3-n6 nknd[11838]: #011/home/nknx/go/src/github.com/nknorg/nkn/consensus/consensus.go:336 +0x62a
Dec 13 19:52:31 nyc3-n6 nknd[12036]: 2019/12/13 19:52:31 Set SyncBatchWindowSize to 61
Dec 13 19:52:31 nyc3-n6 nknd[12036]: 2019/12/13 19:52:31 Set TxPoolMaxMemorySize to 3
Dec 13 19:52:31 nyc3-n6 nknd[12036]: 2019/12/13 19:52:31 Discovering NAT gateway...
#4

Where are your nodes hosted? Have your nodes (typically its VPS) lost connections to the outside. The problem typically happens when a small group of nodes lost connection to the outside for quite a while and started produced blocks internally for too long (longer than the safety recover threshold hard coded in the code).

There are a few ways to recover the nodes:

  • Remove the ChainDB folder, either use a snapshot or let it sync from start
  • Change the config and recompile nknd to allow it recover from a forked state
#5

@yilun
Digitalocean.
I removed chain/db folder.
What do I change in the config and how do I recompile? Sorry not that familiar with go.

#6

If you have removed the ChainDB folder, you don’t need to do anything else. Restarting the node should be enough

#7

@yilun Issues persists. Everytime it gets to SYNC_START it realizes fork and starts over.

#8

@yilun I deleted the db folder inside chain folder by mistake. I now deleted the ChainDB folder and restarted service. Won’t that sync the entire chain though? I installed it as fast deploy I think with a pruned chain. How can I just download what’s needed to move forward?

#9

@yilun looks like it syncing blocks now - started at 1 and up to 2041. Thank you!

#10

If you don’t want to wait for syncing, or want to use the pruned snapshot, you can do the following:

  1. Stop nknd and remove ChainDB
  2. Download pruned snapshot: wget https://nkn.org/ChainDB_pruned_latest.tar.gz
  3. Decompress snapshot: tar -zxvf ChainDB_pruned_latest.tar.gz
  4. Move the ChainDB directory you got to the ChainDB location you removed in step 1
  5. Start nknd again
1 Like
#11

@yilun Thank you very much for your help. It did continue to sync up to block 200,000 or so before I decided to download the pruned chain. It is continuing to sync now up to 630,000 consecutively and quickly.

If originally removing ChainDB and restarting the service caused the node to actually start re-syncing from block 0, how come it couldn’t just continue syncing from its latest block without that action? Can a failsafe be built-in since this is bound to happen?

#12

Because all the original block data is stored in ChainDB folder, which is removed. It is possible to do it without removing the folder and resync from block 0, but it’s quite technical as I mentioned above, so the easiest way is simply start from scratch :joy: But it’s definitely possible if you are interested!

1 Like