Cardano node metrics monitoring

We have our monitoring solution in place. We are now in control of our cardano-node services, that are running as a background process.

The most important reason to get this up and running quickly, is the expiry of the KES period. KES is a key generation algorithm that is used in cryptography and network security. In the case of the Cardano Node, it is represented by “hot keys” a signing key and operational certificate that are applied to the stakingpools. The KES period is the time frame in which the signing key and operation certificate are valid, in other words, they expire. As a pool operator, you want your stakepool “hot keys” updated before the KES period expires. If you are too late with updating these files on your stakepool, you won’t be able to forge(mint) blocks with your stakepool nodes. Which will be bad for the performance of your pool and for the delegators in it.

We are making use of alerts, to detect the KES period expiration, and notify ourselves through Telegram in such events. The notifications will repeat every 30 min before the KES period expires so they can’t be missed.

The dashboard above, shows us the most relevant information:

• Chain status; current epoch, slots in epoch and the current height
• Server resource utilization for all our nodes over a period of time
• KES period and expiry period of our “hot keys” (this is where we put an alert)
• Numbers of blocks we forged as stakepool over a period of time

Grafana, the monitoring tool we use, to setup alerts that sends them out to channels. We are using the Telegram channel to stay in touch with the Cardano community. It’s an obvious choice to get alerts through a channel already in use.

Above is an example scenario, where we defined an alert, in case the KES period gets less than 100. We received an alert about this in our Telegram ₳4PROFIT stakepool alerter group. In such case, we can generate a new set of “hot keys” in time, to restart the stakepool before the stakepool gets in a state where it is unable to forge blocks.

Of course there are other alternatives instead of using Grafana and Telegram for alerting. For example you could also use the ELK stack (a popular monitoring stack). For us using Grafana and Telegram is sufficient to do the job. Depending on your needs, you should choose the tools that work best for you.

Cheers until our next blog post.