Nodes keep restarting even after latest update to 1.0.6

Nodes & Mining Problems
#1

I’m constantly keeping an eye on my nodes after realizing that they were restarting quite often. After checking in syslog, I found following:

[22991] 2019/08/06 11:11:11.511375 #033[0;33m[WARN ]#033[m GID 77, Error handling msg: reveive vote at 139298 for 0000000000000000000000000000000000000000000000000000000000000000 error: Election has already stopped
[22991] 2019/08/06 11:11:11.562320 #033[0;32m[INFO ]#033[m GID 229, Change expected block height to 139300
[22991] fatal error: runtime: out of memory
[22991] runtime stack:
[22991] runtime.throw(0xca0a6b, 0x16)
[22991] #011/usr/local/go/src/runtime/panic.go:617 +0x72
[22991] runtime.sysMap(0xc030000000, 0x4000000, 0x1422b58)
[22991] #011/usr/local/go/src/runtime/mem_linux.go:170 +0xc7

I need more information to debug this problem, kindly suggest further.

I’m seeing these two error frequently

  1. Out of memory
  2. Concurrent read write on map

Aug 6 13:31:55 – nknd[27501]: 2019/08/06 13:31:55.587982 #033[0;32m[INFO ]#033[m GID 236, Receive block info 5ba4cefb331487fd3a231facaeaf334232b9a33528272778aba98583c75f5278, 145 txn found in pool, 111 txn to request
Aug 6 13:31:55 – nknd[27501]: 2019/08/06 13:31:55.639029 #033[0;32m[INFO ]#033[m GID 236, Receive block proposal 5ba4cefb331487fd3a231facaeaf334232b9a33528272778aba98583c75f5278 (256 txn, 390199 bytes) by a5dbe02a766b59b2e6710c85d5f17675d17353c15c46b257132374851e9b6fa1
Aug 6 13:31:56 – nknd[27501]: fatal error: concurrent map read and map write
Aug 6 13:31:56 – nknd[27501]: goroutine 74 [running]:
Aug 6 13:31:56 – nknd[27501]: runtime.throw(0xcaa721, 0x21)
Aug 6 13:31:56 – nknd[27501]: #011/usr/local/go/src/runtime/panic.go:617 +0x72 fp=0xc00f4418d8 sp=0xc00f4418a8 pc=0x42da52

#2

Once vast majority of nodes upgrade to v 1.0.6, then the system will return to normal. Currently there are still nodes on older v 1.0.5.

1 Like
#3

Can you also post your node spec (e.g. RAM)?

#4

I’m using standard droplet settings from digitalocean
1 GB Memory / 25 GB Disk / BLR1 - Debian 9.7 x64
Let me know if you need more information and how can i collect it.

I could only check syslog after noticing any node’s reboot, if there’s a way I can check the problem from nknc debug information, that’d be helpful to you too I guess

Thanks.

#5

Update

I haven’t observed the issue in last 5-6 hours. I think it was as @zbruceli said, once majority of nodes switched to 1.0.6, network seem to be stabilized compared to before.

#6

Looks like the problem came up again. Half of my nodes restarted… Can we check some logs and debug this?

#7

Yes, our devs are testing a fix. To be released as v1.0.7.

1 Like
#8

Thank you.
Just updated all my nodes to 1.0.7
I’ll keep an eye on the performance and update here incase I see that restart problem again.

#9

Out of memory issue in 1.0.8
Aug 15 14:25:16 nknd[26234]: fatal error: runtime: out of memory
Aug 15 14:25:16 nknd[26234]: runtime stack:
Aug 15 14:25:16 nknd[26234]: runtime.throw(0xca2cad, 0x16)
Aug 15 14:25:21 systemd[1]: nkn.service: Service hold-off time over, scheduling restart.
Aug 15 14:25:21 systemd[1]: Stopped nkn.
Aug 15 14:25:21 systemd[1]: Started nkn.

#10

Hi, the developer team is aware of the issue and looking into it.

1 Like
#12

You can try to set TxPoolMaxMemorySize to lower value (e.g. 4) in config.json and see whether it resolves the problem