CVE-2023-45142

OpenTelemetry-Go Contrib is a collection of third-party packages for OpenTelemetry-Go. A handler wrapper out of the box adds labels http.user_agent and http.method that have unbound cardinality. It leads to the servers potential memory exhaustion when many malicious requests are sent to it. HTTP header User-Agent or HTTP method for requests can be easily set by an attacker to be random and long. The library internally uses httpconv.ServerRequest that records every value for HTTP method and User-Agent. In order to be affected, a program has to use the otelhttp.NewHandler wrapper and not filter any unknown HTTP methods or User agents on the level of CDN, LB, previous middleware, etc. Version 0.44.0 fixed this issue when the values collected for attribute http.request.method were changed to be restricted to a set of well-known values and other high cardinality attributes were removed. As a workaround to stop being affected, otelhttp.WithFilter() can be used, but it requires manual careful configuration to not log certain requests entirely. For convenience and safe usage of this library, it should by default mark with the label unknown non-standard HTTP methods and User agents to show that such requests were made but do not increase cardinality. In case someone wants to stay with the current behavior, library API should allow to enable it.

Weakness

The product allocates a reusable resource or group of resources on behalf of an actor without imposing any restrictions on the size or number of resources that can be allocated, in violation of the intended security policy for that actor.

Affected Software

Extended Description

Code frequently has to work with limited resources, so programmers must be careful to ensure that resources are not consumed too quickly, or too easily. Without use of quotas, resource limits, or other protection mechanisms, it can be easy for an attacker to consume many resources by rapidly making many requests, or causing larger resources to be used than is needed. When too many resources are allocated, or if a single resource is too large, then it can prevent the code from working correctly, possibly leading to a denial of service.

Potential Mitigations

• Assume all input is malicious. Use an “accept known good” input validation strategy, i.e., use a list of acceptable inputs that strictly conform to specifications. Reject any input that does not strictly conform to specifications, or transform it into something that does.

• When performing input validation, consider all potentially relevant properties, including length, type of input, the full range of acceptable values, missing or extra inputs, syntax, consistency across related fields, and conformance to business rules. As an example of business rule logic, “boat” may be syntactically valid because it only contains alphanumeric characters, but it is not valid if the input is only expected to contain colors such as “red” or “blue.”

• Do not rely exclusively on looking for malicious or malformed inputs. This is likely to miss at least one undesirable input, especially if the code’s environment changes. This can give attackers enough room to bypass the intended validation. However, denylists can be useful for detecting potential attacks or determining which inputs are so malformed that they should be rejected outright.

• Mitigation of resource exhaustion attacks requires that the target system either:

• The first of these solutions is an issue in itself though, since it may allow attackers to prevent the use of the system by a particular valid user. If the attacker impersonates the valid user, they may be able to prevent the user from accessing the server in question.

• The second solution can be difficult to effectively institute – and even when properly done, it does not provide a full solution. It simply requires more resources on the part of the attacker.

• If the program must fail, ensure that it fails gracefully (fails closed). There may be a temptation to simply let the program fail poorly in cases such as low memory conditions, but an attacker may be able to assert control before the software has fully exited. Alternately, an uncontrolled failure could cause cascading problems with other downstream components; for example, the program could send a signal to a downstream process so the process immediately knows that a problem has occurred and has a better chance of recovery.

• Ensure that all failures in resource allocation place the system into a safe posture.

• Use resource-limiting settings provided by the operating system or environment. For example, when managing system resources in POSIX, setrlimit() can be used to set limits for certain types of resources, and getrlimit() can determine how many resources are available. However, these functions are not available on all operating systems.

• When the current levels get close to the maximum that is defined for the application (see CWE-770), then limit the allocation of further resources to privileged users; alternately, begin releasing resources for less-privileged users. While this mitigation may protect the system from attack, it will not necessarily stop attackers from adversely impacting other users.

• Ensure that the application performs the appropriate error checks and error handling in case resources become unavailable (CWE-703).

Related Attack Patterns

• https://cwe.mitre.org/data/definitions/125.html
• https://cwe.mitre.org/data/definitions/130.html
• https://cwe.mitre.org/data/definitions/147.html
• https://cwe.mitre.org/data/definitions/197.html
• https://cwe.mitre.org/data/definitions/229.html
• https://cwe.mitre.org/data/definitions/230.html
• https://cwe.mitre.org/data/definitions/231.html
• https://cwe.mitre.org/data/definitions/469.html
• https://cwe.mitre.org/data/definitions/482.html
• https://cwe.mitre.org/data/definitions/486.html
• https://cwe.mitre.org/data/definitions/487.html
• https://cwe.mitre.org/data/definitions/488.html
• https://cwe.mitre.org/data/definitions/489.html
• https://cwe.mitre.org/data/definitions/490.html
• https://cwe.mitre.org/data/definitions/491.html
• https://cwe.mitre.org/data/definitions/493.html
• https://cwe.mitre.org/data/definitions/494.html
• https://cwe.mitre.org/data/definitions/495.html
• https://cwe.mitre.org/data/definitions/496.html
• https://cwe.mitre.org/data/definitions/528.html

References

• https://github.com/advisories/GHSA-cg3q-j54f-5p7p
• https://github.com/open-telemetry/opentelemetry-go-contrib/blob/5f7e6ad5a49b45df45f61a1deb29d7f1158032df/instrumentation/net/http/otelhttp/handler.go#L63-L65
• https://github.com/open-telemetry/opentelemetry-go-contrib/pull/4277
• https://github.com/open-telemetry/opentelemetry-go-contrib/releases/tag/v1.19.0
• https://github.com/open-telemetry/opentelemetry-go-contrib/security/advisories/GHSA-5r5m-65gx-7vrh
• https://github.com/open-telemetry/opentelemetry-go-contrib/security/advisories/GHSA-rcjv-mgp8-qvmr
• https://github.com/open-telemetry/opentelemetry-go/blob/38e1b499c3da3107694ad2660b3888eee9c8b896/semconv/internal/v2/http.go#L223
• https://github.com/open-telemetry/opentelemetry-go/blob/v1.12.0/semconv/internal/v2/http.go#L159
• https://lists.fedoraproject.org/archives/list/package-announce@lists.fedoraproject.org/message/2UTRJ54INZG3OC2FTAN6AFB2RYNY2GAD/