Explained: Inside the Tendermint DoS Vulnerabilities

In January 2021, the Tendermint security team received a report of a Denial of Service (DoS) vulnerability from Crypto.com.  After releasing a patch for this vulnerability and performing further analysis, another vulnerability was discovered and patched.  The first vulnerability was scored as medium severity on the CVSS scale, and the second was labeled as a high severity vulnerability.

Tendermint and Misbehaving Nodes

The two Tendermint DoS vulnerabilities were caused by updates to the Tendermint evidence management system.  This system is designed to allow nodes in the network to call out misbehaving validators.

A validator in the Tendermint network should only sign one version of a block at each block height in the blockchain.  This is essential to consensus finalization in Tendermint, which requires a ⅔ majority for a block to be officially “committed”.

If a validator node signs multiple versions of a block, it is possible that – with enough misbehaving validators – two versions of the blockchain could exist, each with a ⅔ vote.  For this reason, Tendermint includes the capability to publicly report “evidence” of a misbehaving node (i.e. one signing multiple versions of a block).

The Tendermint DoS Vulnerabilities

When Tendermint released the update to the evidence protocol, a couple of corner cases were overlooked.  This resulted in two potential DoS vulnerabilities that could knock nodes offline if a validator engaged in double-signing.

Vulnerability 1

Like any blockchain, Tendermint is built on a peer-to-peer network for communications.  This means that it is natural for different nodes in the network to receive transactions and blocks at different times.  When one node creates a transaction or block, it sends it to its direct peers, who send it to their direct peers, and so on.  In the end, every node receives the message, but there can be a noticeable delay between the earliest and latest recipients.

Additionally, Tendermint works on a consensus algorithm where ⅔ of the network’s validators need to inspect and validate a block before it is officially committed and part of the distributed ledger.  This means that some blocks will exist that are still “in-flight”, where part (or all) of the network has received them but they haven’t been officially added to the blockchain yet.

These two factors combine to create the first DoS vulnerability.  This vulnerability exists because a Tendermint validator could propose evidence of node misbehavior for an “in-flight” block. In Tendermint Core v0.34.0 and v0.34.1. This meant that nodes could receive evidence for a block when they hadn’t seen the block itself.  Under these circumstances, the node receiving the evidence would panic and be unable to continue operating, making this a DoS.

Vulnerability 2

The second Tendermint DoS attack dealt with timestamps and evidence in Tendermint.  When a node creates evidence of misbehavior, a timestamp is included in the evidence.  Part of this timestamp calculation is the timestamp of the “last commit” for a specific block.

When a block is “in-flight” different nodes may have different values for the “last commit” depending on when they create their evidence and their visibility of the network.  This means that two validator nodes attempting to create the same evidence may have different timestamp values for this evidence.

Later on, one of these pieces of evidence will be accepted and added to a block, but the other will be dropped (to avoid duplication).  The dropped piece of evidence will then be labeled as “invalid” (even though it was valid at time of creation).  This means that any future evidence created by the same validator in the future and building on this evidence will also be labeled as invalid.

As a result, nodes in the Tendermint network may choose to disconnect from it, leaving it unable to participate in the blockchain’s operations.  As a result, creating duplicate signatures enables an attacker to perform an untargeted DoS attack because any node whose evidence “loses” may be kicked out for passing invalid information.

Lessons Learned from the Tendermint DoS Vulnerabilities

Both of the DoS vulnerabilities in Tendermint Core resulted from a failure to test all potential corner cases before releasing the code update.  In both cases, the issue comes from validators “jumping the gun” to start acting on evidence of misbehavior in a block while it was still “in flight” rather than fully committed.

The discovery of these vulnerabilities demonstrates the value of external security testing as they were brought to Tendermint’s attention by a team from Crypto.com.  However, it also underscores the importance of performing robust security testing before releasing software to production.