Error corruption on data-block / checksum mismatch

Nodes & Mining Problems
#1

rPI 2 CLI dedicated just to NKN mining. v0.8.2-alpha
SD SanDisk Extreme 64 GB Class 10, UHS-I, U3, V30
wifi ac (dongle connected to hub w/ power) Bit rate 433 MB/s / link quality +80% / signal level -50 dBm
ping to 8.8.8.8 10ms
ping from UK/Germany/Netherlands about 35 ms
ping from US about 100ms

mining for 24h with 53 connections and +350 relayed messages/hour

Suddenly getting :
2019/05/15 15:00:35.498213 e[0;31m[ERROR]e[m GID 67093, No error code
2019/05/15 15:00:35.629916 e[0;32m[INFO ]e[m GID 67037, Add sigchain from transaction error: signature chain path is invalid
2019/05/15 15:00:35.630254 e[0;31m[ERROR]e[m GID 67037, No error code
2019/05/15 15:00:38.147444 e[0;31m[ERROR]e[m GID 83, Error handling message: leveldb/storage: invalid file for argument
2019/05/15 15:00:41.740634 e[0;32m[INFO ]e[m GID 142, Change expected block height to 943206
2019/05/15 15:00:51.742308 e[0;32m[INFO ]e[m GID 142, Accept block 0c4af3d9ad864387524e3a492bcb38af0bc619927fcc43f0191c346c09f1b146 at height 943205
.
.
.
2019/05/15 15:05:25.820669 e[0;33m[WARN ]e[m GID 67664, Get latest block info from neighbor 4f211b2633b64e62223979a3bc12ab9c0eee67ed51e933a945892079d9afe851 error: Wait for reply timeout
2019/05/15 15:05:25.825949 e[0;31m[ERROR]e[m GID 67665, Error sending sync messge to node: Wait for reply timeout
2019/05/15 15:05:25.827918 e[0;33m[WARN ]e[m GID 67665, Get latest block info from neighbor 50d4c3ef4224ccf4743356194d2d9eef154e91493505ad3071e0e791fb06bb2d error: Wait for reply timeout
2019/05/15 15:05:25.831646 e[0;31m[ERROR]e[m GID 67695, Error sending sync messge to node: Wait for reply timeout
.
.
.
2019/05/15 15:11:09.251174 e[0;32m[INFO ]e[m GID 144, Request block 361866fc44326183b29f29d49c5a244e0647f5ae000219755342b3a0407a5841 from neighbor 50d910f480fc64fe5d80f3729b5ce6e2cf5fbe651bce95d66884e79d1d60c4f3
2019/05/15 15:11:09.264842 e[0;32m[INFO ]e[m GID 142, Accept block fdc707b0ed1bc834452f1f67c3236376ce2a912c7a90716dd5e927a78a948b4c at height 943235
2019/05/15 15:11:09.268667 e[0;31m[ERROR]e[m GID 24, [persistBlocks]: error to persist block: leveldb/table: corruption on data-block (pos=1350891): checksum mismatch, want=0x67a0e747 got=0xa3febee1 [file=046407.ldb]
2019/05/15 15:11:19.566197 e[0;32m[INFO ]e[m GID 142, Change expected block height to 943237
2019/05/15 15:11:29.568148 e[0;32m[INFO ]e[m GID 142, Accept block 361866fc44326183b29f29d49c5a244e0647f5ae000219755342b3a0407a5841 at height 943236
2019/05/15 15:11:29.568764 e[0;32m[INFO ]e[m GID 142, Accepted block height: 943236, local ledger block height: 943234, sync needed.
2019/05/15 15:11:29.575761 e[0;31m[ERROR]e[m GID 142, Error saving accepted block: Cannot get neighbors voted for block hash 361866fc44326183b29f29d49c5a244e0647f5ae000219755342b3a0407a5841
2019/05/15 15:11:29.576255 e[0;32m[INFO ]e[m GID 142, Change expected block height to 943236
2019/05/15 15:11:29.672889 e[0;33m[WARN ]e[m GID 83, Receive block hash error when receive vote: Receive invalid block hash height 943237 instead of 943236
2019/05/15 15:11:29.689241 e[0;33m[WARN ]e[m GID 83, Receive block hash error when receive vote: Receive invalid block hash height 943237 instead of 943236
2019/05/15 15:11:29.705341 e[0;33m[WARN ]e[m GID 83, Receive block hash error when receive vote: Receive invalid block hash height 943237 instead of 943236
2019/05/15 15:11:29.721671 e[0;33m[WARN ]e[m GID 83, Receive block hash error when receive vote: Receive invalid block hash height 943237 instead of 943236
2019/05/15 15:11:29.737902 e[0;33m[WARN ]e[m GID 83, Receive block hash error when receive vote: Receive invalid block hash height 943237 instead of 943236
2019/05/15 15:11:29.754451 e[0;33m[WARN ]e[m GID 83, Receive block hash error when receive vote: Receive invalid block hash height 943237 instead of 943236
2019/05/15 15:11:29.770613 e[0;33m[WARN ]e[m GID 83, Receive block hash error when receive vote: Receive invalid block hash height 943237 instead of 943236
.
.
.
2019/05/15 15:11:52.304354 e[0;33m[WARN ]e[m GID 83, Receive block hash error when receive vote: Receive invalid block hash height 943238 instead of 943236
2019/05/15 15:11:52.488658 e[0;32m[INFO ]e[m GID 65, Add sigchain from transaction error: leveldb: not found
2019/05/15 15:11:52.489232 e[0;33m[WARN ]e[m GID 65, Verify transaction failed with No error code when append to txn pool
2019/05/15 15:11:52.489635 e[0;31m[ERROR]e[m GID 65, Error handling message: all transactions in msg are invalid
2019/05/15 15:11:52.661122 e[0;32m[INFO ]e[m GID 65, Add sigchain from transaction error: leveldb: not found
2019/05/15 15:11:52.661630 e[0;33m[WARN ]e[m GID 65, Verify transaction failed with No error code when append to txn pool
.
.
.
2019/05/15 15:12:11.248856 e[0;32m[INFO ]e[m GID 65, Add sigchain from transaction error: leveldb: not found
2019/05/15 15:12:11.249385 e[0;33m[WARN ]e[m GID 65, Verify transaction failed with No error code when append to txn pool
2019/05/15 15:12:11.249775 e[0;31m[ERROR]e[m GID 65, Error handling message: all transactions in msg are invalid
2019/05/15 15:12:11.251580 e[0;33m[WARN ]e[m GID 83, Receive block hash error when receive vote: Receive invalid block hash height 943239 instead of 943236
2019/05/15 15:12:11.268251 e[0;33m[WARN ]e[m GID 83, Receive block hash error when receive vote: Receive invalid block hash height 943239 instead of 943236
2019/05/15 15:12:11.284921 e[0;33m[WARN ]e[m GID 83, Receive block hash error when receive vote: Receive invalid block hash height 943239 instead of 943236
2019/05/15 15:12:11.301503 e[0;33m[WARN ]e[m GID 83, Receive block hash error when receive vote: Receive invalid block hash height 943239 instead of 943236
2019/05/15 15:12:11.318580 e[0;33m[WARN ]e[m GID 83, Receive block hash error when receive vote: Receive invalid block hash height 943239 instead of 943236
2019/05/15 15:12:11.335441 e[0;33m[WARN ]e[m GID 83, Receive block hash error when receive vote: Receive invalid block hash height 943239 instead of 943236
.
.
.
2019/05/15 15:12:52.863223 e[0;32m[INFO ]e[m GID 65, Add sigchain from transaction error: leveldb: not found
2019/05/15 15:12:52.863727 e[0;33m[WARN ]e[m GID 65, Verify transaction failed with No error code when append to txn pool
2019/05/15 15:12:52.864149 e[0;31m[ERROR]e[m GID 65, Error handling message: all transactions in msg are invalid
2019/05/15 15:12:55.104712 e[0;31m[ERROR]e[m GID 83, Error handling message: Election has already stopped
2019/05/15 15:12:57.635388 e[0;32m[INFO ]e[m GID 68691, Synced 5 block headers in 1.024823072s
2019/05/15 15:12:57.684070 e[0;31m[ERROR]e[m GID 24, [persistBlocks]: error to persist block: leveldb/table: corruption on data-block (pos=1350891): checksum mismatch, want=0x67a0e747 got=0xa3febee1 [file=046407.ldb]
2019/05/15 15:12:57.686229 e[0;31m[ERROR]e[m GID 24, [persistBlocks]: error to persist block: leveldb/table: corruption on data-block (pos=1350891): checksum mismatch, want=0x67a0e747 got=0xa3febee1 [file=046407.ldb]
2019/05/15 15:12:57.688228 e[0;31m[ERROR]e[m GID 24, [persistBlocks]: error to persist block: leveldb/table: corruption on data-block (pos=1350891): checksum mismatch, want=0x67a0e747 got=0xa3febee1 [file=046407.ldb]
2019/05/15 15:12:57.691140 e[0;31m[ERROR]e[m GID 24, [persistBlocks]: error to persist block: leveldb/table: corruption on data-block (pos=1350891): checksum mismatch, want=0x67a0e747 got=0xa3febee1 [file=046407.ldb]
2019/05/15 15:12:57.693764 e[0;31m[ERROR]e[m GID 24, [persistBlocks]: error to persist block: leveldb/table: corruption on data-block (pos=1350891): checksum mismatch, want=0x67a0e747 got=0xa3febee1 [file=046407.ldb]
2019/05/15 15:12:58.632422 e[0;33m[WARN ]e[m GID 67262, Ping error: Wait for reply timeout
2019/05/15 15:12:58.656959 e[0;32m[INFO ]e[m GID 68691, Synced 5 blocks in 1.0208096s
2019/05/15 15:12:58.657530 e[0;32m[INFO ]e[m GID 68691, Set sync state to SyncFinished
2019/05/15 15:12:58.657905 e[0;32m[INFO ]e[m GID 68691, Start saving blocks accepted during sync
2019/05/15 15:12:58.959327 e[0;32m[INFO ]e[m GID 68691, Saved 1 blocks accepted during sync
2019/05/15 15:12:58.959818 e[0;32m[INFO ]e[m GID 68691, Set sync state to PersistFinished
2019/05/15 15:12:58.962134 e[0;31m[ERROR]e[m GID 24, [persistBlocks]: error to persist block: leveldb/table: corruption on data-block (pos=1350891): checksum mismatch, want=0x67a0e747 got=0xa3febee1 [file=046407.ldb]
…

and stuck for hours with corruption on data-block (pos=1350891): checksum mismatch, want=0x67a0e747 got=0xa3febee1 [file=046407.ldb]

Thanks in advance.

1 Like
#2

It seems DB had corrupted before 2019/05/15 15:00:38. Could you post more log details before that moment?

#3

Thanks. I uploaded 2 log files. the arrow points to the log file where “checksum error” appeared for first time. All subsequent log files have only the same error repeated.
https://mega.nz/#F!cWxHQY6J!qXbis-p27RG3C9IQQDbLCQ
log%20error

This is the first ERROR leveldb/storage
image

#4

48h later…+650 messages-relayed /hour and 71 connections, although what was happening in the background was…
image

log file

image
rest of log files during these 48 h. just with the same error

#5

Restarted the service (without rebooting rPi). Apparently it’s mining again.
53 connections and a ratio of +1300 relayed messages/hour (during an hour so far). Is there any way to slow down the speed of relayed messages? In case my rPi couldn’t cope with the speed while writing on the sdcard the .ldb and .log files. sdcard is fast(write @ +60 MB/sec on PC), just thinking on rPi capabilities…

ran a benchmark on rPi
image

#6

Relaying message is purely offchain, which means nothing will be written to your database :slight_smile:

#7

Thanks for clarification. I still trying to understand it.:smiley: I’d say its closer enough to a neuronal network. Quite complex.
If my node has a # connections, and the average of the “roundTripTime” with each neighbor is lower than other node with same # of connections…would get a higher ratio Relayed messages /hour? Or will also depends on the connections of each neighbor?

After 15 hours, my rPi is getting around 1900 Relayed messages/hour, so is getting closer to my average performance of my VPS.

image

and this time is mining fine, so lets see how goes
image

#8

Yes! It’s like a a recurrent neural net.

The relay message count depends on topology, latency and more. For the latency part, when a node choose your pi as next hop, that means his latency to your pi is lower than the other (up to) 2 alternative candidates he has for this hop. So latency plays a role here, but only to a degree.