CVE-2023-53188

In the Linux kernel, the following vulnerability has been resolved:

net: openvswitch: fix race on port output

assume the following setup on a single machine:

1. An openvswitch instance with one bridge and default flows
2. two network namespaces server and client
3. two ovs interfaces server and client on the bridge
4. for each ovs interface a veth pair with a matching name and 32 rx and tx queues
5. move the ends of the veth pairs to the respective network namespaces
6. assign ip addresses to each of the veth ends in the namespaces (needs to be the same subnet)
7. start some http server on the server network namespace
8. test if a client in the client namespace can reach the http server

when following the actions below the host has a chance of getting a cpu stuck in a infinite loop:

1. send a large amount of parallel requests to the http server (around 3000 curls should work)
2. in parallel delete the network namespace (do not delete interfaces or stop the server, just kill the namespace)

there is a low chance that this will cause the below kernel cpu stuck message. If this does not happen just retry. Below there is also the output of bpftrace for the functions mentioned in the output.

The series of events happening here is:

1. the network namespace is deleted calling unregister_netdevice_many_notify somewhere in the process
2. this sets first NETREG_UNREGISTERING on both ends of the veth and then runs synchronize_net
3. it then calls call_netdevice_notifiers with NETDEV_UNREGISTER
4. this is then handled by dp_device_event which calls ovs_netdev_detach_dev (if a vport is found, which is the case for the veth interface attached to ovs)
5. this removes the rx_handlers of the device but does not prevent packages to be sent to the device
6. dp_device_event then queues the vport deletion to work in background as a ovs_lock is needed that we do not hold in the unregistration path
7. unregister_netdevice_many_notify continues to call netdev_unregister_kobject which sets real_num_tx_queues to 0
8. port deletion continues (but details are not relevant for this issue)
9. at some future point the background task deletes the vport

If after 7. but before 9. a packet is send to the ovs vport (which is not deleted at this point in time) which forwards it to the dev_queue_xmit flow even though the device is unregistering. In skb_tx_hash (which is called in the dev_queue_xmit) path there is a while loop (if the packet has a rx_queue recorded) that is infinite if dev->real_num_tx_queues is zero.

To prevent this from happening we update do_output to handle devices without carrier the same as if the device is not found (which would be the code path after 9. is done).

Additionally we now produce a warning in skb_tx_hash if we will hit the infinite loop.

bpftrace (first word is function name):

_dev_queue_xmit server: real_num_tx_queues: 1, cpu: 2, pid: 28024, tid: 28024, skb_addr: 0xffff9edb6f207000, reg_state: 1 netdev_core_pick_tx server: addr: 0xffff9f0a46d4a000 real_num_tx_queues: 1, cpu: 2, pid: 28024, tid: 28024, skb_addr: 0xffff9edb6f207000, reg_state: 1 dp_device_event server: real_num_tx_queues: 1 cpu 9, pid: 21024, tid: 21024, event 2, reg_state: 1 synchronize_rcu_expedited: cpu 9, pid: 21024, tid: 21024 synchronize_rcu_expedited: cpu 9, pid: 21024, tid: 21024 synchronize_rcu_expedited: cpu 9, pid: 21024, tid: 21024 synchronize_rcu_expedited: cpu 9, pid: 21024, tid: 21024 dp_device_event server: real_num_tx_queues: 1 cpu 9, pid: 21024, tid: 21024, event 6, reg_state: 2 ovs_netdev_detach_dev server: real_num_tx_queues: 1 cpu 9, pid: 21024, tid: 21024, reg_state: 2 netdev_rx_handler_unregister server: real_num_tx_queues: 1, cpu: 9, pid: 21024, tid: 21024, reg_state: 2 synchronize_rcu_expedited: cpu 9, pid: 21024, tid: 21024 netdev_rx_handler_unregister ret server: real_num_tx_queues: 1, cpu: 9, pid: 21024, tid: 21024, reg_state: 2 dp —truncated—

Weakness

The product contains a concurrent code sequence that requires temporary, exclusive access to a shared resource, but a timing window exists in which the shared resource can be modified by another code sequence operating concurrently.

Affected Software

Extended Description

A race condition occurs within concurrent environments, and it is effectively a property of a code sequence. Depending on the context, a code sequence may be in the form of a function call, a small number of instructions, a series of program invocations, etc. A race condition violates these properties, which are closely related:

A race condition exists when an “interfering code sequence” can still access the shared resource, violating exclusivity. The interfering code sequence could be “trusted” or “untrusted.” A trusted interfering code sequence occurs within the product; it cannot be modified by the attacker, and it can only be invoked indirectly. An untrusted interfering code sequence can be authored directly by the attacker, and typically it is external to the vulnerable product.

Potential Mitigations

• Minimize the usage of shared resources in order to remove as much complexity as possible from the control flow and to reduce the likelihood of unexpected conditions occurring.
• Additionally, this will minimize the amount of synchronization necessary and may even help to reduce the likelihood of a denial of service where an attacker may be able to repeatedly trigger a critical section (CWE-400).

Related Attack Patterns

• https://cwe.mitre.org/data/definitions/26.html
• https://cwe.mitre.org/data/definitions/29.html

References

• https://git.kernel.org/stable/c/066b86787fa3d97b7aefb5ac0a99a22dad2d15f8
• https://git.kernel.org/stable/c/284be5db6c8d06d247ed056cfc448c4f79bbb16c
• https://git.kernel.org/stable/c/56252da41426f3d01957456f13caf46ce670ea29
• https://git.kernel.org/stable/c/5efcb301523baacd98a47553d4996e924923114d
• https://git.kernel.org/stable/c/644b3051b06ba465bc7401bfae9b14963cbc8c1c
• https://git.kernel.org/stable/c/9b0dd09c1ceb35950d2884848099fccc9ec9a123