CVE Vulnerabilities

CVE-2023-34450

Missing Release of Memory after Effective Lifetime

Published: Jul 03, 2023 | Modified: Jul 17, 2023
CVSS 3.x
5.3
MEDIUM
Source:
NVD
CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:L
CVSS 2.x
RedHat/V2
RedHat/V3
Ubuntu

CometBFT is a Byzantine Fault Tolerant (BFT) middleware that takes a state transition machine and replicates it on many machines. An internal modification made in versions 0.34.28 and 0.37.1 to the way struct PeerState is serialized to JSON introduced a deadlock when new function MarshallJSON is called. This function can be called from two places. The first is via logs, setting the consensus logging module to debug level (should not happen in production), and setting the log output format to JSON. The second is via RPC dump_consensus_state.

Case 1, which should not be hit in production, will eventually hit the deadlock in most goroutines, effectively halting the node.

In case 2, only the data structures related to the first peer will be deadlocked, together with the thread(s) dealing with the RPC request(s). This means that only one of the channels of communication to the nodes peers will be blocked. Eventually the peer will timeout and excluded from the list (typically after 2 minutes). The goroutines involved in the deadlock will not be garbage collected, but they will not interfere with the system after the peer is excluded.

The theoretical worst case for case 2, is a network with only two validator nodes. In this case, each of the nodes only has one PeerState struct. If dump_consensus_state is called in either node (or both), the chain will halt until the peer connections time out, after which the nodes will reconnect (with different PeerState structs) and the chain will progress again. Then, the same process can be repeated.

As the number of nodes in a network increases, and thus, the number of peer struct each node maintains, the possibility of reproducing the perturbation visible with two nodes decreases. Only the first PeerState struct will deadlock, and not the others (RPC dump_consensus_state accesses them in a for loop, so the deadlock at the first iteration causes the rest of the iterations of that for loop to never be reached).

This regression was fixed in versions 0.34.29 and 0.37.2. Some workarounds are available. For case 1 (hitting the deadlock via logs), either dont set the log output to json, leave at plain, or dont set the consensus logging module to debug, leave it at info or higher. For case 2 (hitting the deadlock via RPC dump_consensus_state), do not expose dump_consensus_state RPC endpoint to the public internet (e.g., via rules in ones nginx setup).

Weakness

The product does not sufficiently track and release allocated memory after it has been used, which slowly consumes remaining memory.

Affected Software

Name Vendor Start Version End Version
Cometbft Cometbft 0.34.28 (including) 0.34.29 (excluding)
Cometbft Cometbft 0.37.1 (including) 0.37.2 (excluding)

Potential Mitigations

  • Choose a language or tool that provides automatic memory management, or makes manual memory management less error-prone.
  • For example, glibc in Linux provides protection against free of invalid pointers.
  • When using Xcode to target OS X or iOS, enable automatic reference counting (ARC) [REF-391].
  • To help correctly and consistently manage memory when programming in C++, consider using a smart pointer class such as std::auto_ptr (defined by ISO/IEC ISO/IEC 14882:2003), std::shared_ptr and std::unique_ptr (specified by an upcoming revision of the C++ standard, informally referred to as C++ 1x), or equivalent solutions such as Boost.

References