CVE-2022-21698

client_golang is the instrumentation library for Go applications in Prometheus, and the promhttp package in client_golang provides tooling around HTTP servers and clients. In client_golang prior to version 1.11.1, HTTP server is susceptible to a Denial of Service through unbounded cardinality, and potential memory exhaustion, when handling requests with non-standard HTTP methods. In order to be affected, an instrumented software must use any of promhttp.InstrumentHandler* middleware except RequestsInFlight; not filter any specific methods (e.g GET) before middleware; pass metric with method label name to our middleware; and not have any firewall/LB/proxy that filters away requests with unknown method. client_golang version 1.11.1 contains a patch for this issue. Several workarounds are available, including removing the method label name from counter/gauge used in the InstrumentHandler; turning off affected promhttp handlers; adding custom middleware before promhttp handler that will sanitize the request method given by Go http.Request; and using a reverse proxy or web application firewall, configured to only allow a limited set of methods.

Weakness

The product allocates a reusable resource or group of resources on behalf of an actor without imposing any restrictions on the size or number of resources that can be allocated, in violation of the intended security policy for that actor.

Affected Software

Extended Description

Code frequently has to work with limited resources, so programmers must be careful to ensure that resources are not consumed too quickly, or too easily. Without use of quotas, resource limits, or other protection mechanisms, it can be easy for an attacker to consume many resources by rapidly making many requests, or causing larger resources to be used than is needed. When too many resources are allocated, or if a single resource is too large, then it can prevent the code from working correctly, possibly leading to a denial of service.

Potential Mitigations

• Assume all input is malicious. Use an “accept known good” input validation strategy, i.e., use a list of acceptable inputs that strictly conform to specifications. Reject any input that does not strictly conform to specifications, or transform it into something that does.

• When performing input validation, consider all potentially relevant properties, including length, type of input, the full range of acceptable values, missing or extra inputs, syntax, consistency across related fields, and conformance to business rules. As an example of business rule logic, “boat” may be syntactically valid because it only contains alphanumeric characters, but it is not valid if the input is only expected to contain colors such as “red” or “blue.”

• Do not rely exclusively on looking for malicious or malformed inputs. This is likely to miss at least one undesirable input, especially if the code’s environment changes. This can give attackers enough room to bypass the intended validation. However, denylists can be useful for detecting potential attacks or determining which inputs are so malformed that they should be rejected outright.

• Mitigation of resource exhaustion attacks requires that the target system either:

• The first of these solutions is an issue in itself though, since it may allow attackers to prevent the use of the system by a particular valid user. If the attacker impersonates the valid user, they may be able to prevent the user from accessing the server in question.

• The second solution can be difficult to effectively institute – and even when properly done, it does not provide a full solution. It simply requires more resources on the part of the attacker.

• If the program must fail, ensure that it fails gracefully (fails closed). There may be a temptation to simply let the program fail poorly in cases such as low memory conditions, but an attacker may be able to assert control before the software has fully exited. Alternately, an uncontrolled failure could cause cascading problems with other downstream components; for example, the program could send a signal to a downstream process so the process immediately knows that a problem has occurred and has a better chance of recovery.

• Ensure that all failures in resource allocation place the system into a safe posture.

• Use resource-limiting settings provided by the operating system or environment. For example, when managing system resources in POSIX, setrlimit() can be used to set limits for certain types of resources, and getrlimit() can determine how many resources are available. However, these functions are not available on all operating systems.

• When the current levels get close to the maximum that is defined for the application (see CWE-770), then limit the allocation of further resources to privileged users; alternately, begin releasing resources for less-privileged users. While this mitigation may protect the system from attack, it will not necessarily stop attackers from adversely impacting other users.

• Ensure that the application performs the appropriate error checks and error handling in case resources become unavailable (CWE-703).

Related Attack Patterns

• https://cwe.mitre.org/data/definitions/125.html
• https://cwe.mitre.org/data/definitions/130.html
• https://cwe.mitre.org/data/definitions/147.html
• https://cwe.mitre.org/data/definitions/197.html
• https://cwe.mitre.org/data/definitions/229.html
• https://cwe.mitre.org/data/definitions/230.html
• https://cwe.mitre.org/data/definitions/231.html
• https://cwe.mitre.org/data/definitions/469.html
• https://cwe.mitre.org/data/definitions/482.html
• https://cwe.mitre.org/data/definitions/486.html
• https://cwe.mitre.org/data/definitions/487.html
• https://cwe.mitre.org/data/definitions/488.html
• https://cwe.mitre.org/data/definitions/489.html
• https://cwe.mitre.org/data/definitions/490.html
• https://cwe.mitre.org/data/definitions/491.html
• https://cwe.mitre.org/data/definitions/493.html
• https://cwe.mitre.org/data/definitions/494.html
• https://cwe.mitre.org/data/definitions/495.html
• https://cwe.mitre.org/data/definitions/496.html
• https://cwe.mitre.org/data/definitions/528.html

References

• https://github.com/prometheus/client_golang/pull/962
• https://github.com/prometheus/client_golang/pull/987
• https://github.com/prometheus/client_golang/releases/tag/v1.11.1
• https://github.com/prometheus/client_golang/security/advisories/GHSA-cg3q-j54f-5p7p
• https://lists.fedoraproject.org/archives/list/package-announce%40lists.fedoraproject.org/message/2IK53GWZ475OQ6ENABKMJMTOBZG6LXUR/
• https://lists.fedoraproject.org/archives/list/package-announce%40lists.fedoraproject.org/message/2PFW6Q2LXXWTFRTMTRN4ZGADFRQPKJ3D/
• https://lists.fedoraproject.org/archives/list/package-announce%40lists.fedoraproject.org/message/36GUEPA5TPSC57DZTPYPBL6T7UPQ2FRH/
• https://lists.fedoraproject.org/archives/list/package-announce%40lists.fedoraproject.org/message/3L6GDN5S5QZSCFKWD3GKL2RDZQ6B4UWA/
• https://lists.fedoraproject.org/archives/list/package-announce%40lists.fedoraproject.org/message/4KDETHL5XCT6RZN2BBNOCEXRZ2W3SFU3/
• https://lists.fedoraproject.org/archives/list/package-announce%40lists.fedoraproject.org/message/5OGNAFVXSMTTT2UPH6CS3IH6L3KM42Q7/
• https://lists.fedoraproject.org/archives/list/package-announce%40lists.fedoraproject.org/message/7V7I72LSQ3IET3QJR6QPAVGJZ4CBDLN5/
• https://lists.fedoraproject.org/archives/list/package-announce%40lists.fedoraproject.org/message/AK7CJBCGERCRXYUR2EWDSSDVAQMTAZGX/
• https://lists.fedoraproject.org/archives/list/package-announce%40lists.fedoraproject.org/message/DLUJZV3HBP56ADXU6QH2V7RNYUPMVBXQ/
• https://lists.fedoraproject.org/archives/list/package-announce%40lists.fedoraproject.org/message/FY3N7H6VSDZM37B4SKM2PFFCUWU7QYWN/
• https://lists.fedoraproject.org/archives/list/package-announce%40lists.fedoraproject.org/message/HLAQRRGNSO5MYCPAXGPH2OCSHOGHSQMQ/
• https://lists.fedoraproject.org/archives/list/package-announce%40lists.fedoraproject.org/message/J5WPM42UR6XIBQNQPNQHM32X7S4LJTRX/
• https://lists.fedoraproject.org/archives/list/package-announce%40lists.fedoraproject.org/message/KBMVIQFKQDSSTHVVJWJ4QH6TW3JVB7XZ/
• https://lists.fedoraproject.org/archives/list/package-announce%40lists.fedoraproject.org/message/MH6ALXEQXIFQRQFNJ5Y2MJ5DFPIX76VN/
• https://lists.fedoraproject.org/archives/list/package-announce%40lists.fedoraproject.org/message/RN7JGC2LVHPEGSJYODFUV5FEKPBVG4D7/
• https://lists.fedoraproject.org/archives/list/package-announce%40lists.fedoraproject.org/message/SASRKYHT5ZFSVMJUQUG3UAEQRJYGJKAR/
• https://lists.fedoraproject.org/archives/list/package-announce%40lists.fedoraproject.org/message/ZKORFJTRRDJCWBTJPISKKCVMMMJBIRLG/
• https://lists.fedoraproject.org/archives/list/package-announce%40lists.fedoraproject.org/message/ZY2SLWOQR4ZURQ7UBRZ7JIX6H6F5JHJR/