Solution: AWS T2 Micro running out of disk and stopped mining

Nodes & Mining
#1

Today I was checking on NKNx.org (public beta on August 6th) and found that two of my AWS nodes stopped working. And after ssh logged in, I found the reason: these AWS T2-Micro (8GB) instances run out of disk. (By the way, this is partially caused by the recent large blocks due to Pub/Sub metadata, and a fix is on the way.)

Upon checking my disk, here are the main directories that uses up space:

  • ChainDB: >3.6GB
  • Log: about 1.6GB

So a short term fix is to free up some space by cleaning up the Log files. And you can do like this:

ssh -i YOU_SSH.PEM ubuntu@YOU_NODE_IP
sudo rm -f /home/nkn/go/src/github.com/nknorg/nkn/Log/*

But better approach is to resize your instance to have more storage space, because the NKN blockchain will grow over time. It’s better to have enough space (e.g. 25GB) so you don’t have to worry about it for a while. It’s actually quite simple, and you can do it in a few steps like this:

After this, please do remember to reboot your node from EC2 console. And now your node should have enough storage space for quite a while.

You might also need to adjust the SyncBatchWindowSize parameter in if your node is still stuck during initial synchronization of the blockchain. For 1GB RAM machine, use 64. For machines with smaller RAM size, 16 or 32 works. It will slow down the initial sync, but also avoid node gets stuck due to out-of-memory errors. Adding the following line to config.json:

"SyncBatchWindowSize":64,

Footnote

  1. We are going to optimize the blockchain database, so only latest balance will be kept and older balance purged.
  2. We will develop a light node that do need to keep the entire blockchain data, similar to Ethereum.
1 Like