In the Linux kernel, the following vulnerability has been resolved:
Revert wireguard: device: enable threaded NAPI
This reverts commit 933466fc50a8e4eb167acbd0d8ec96a078462e9c which is commit db9ae3b6b43c79b1ba87eea849fd65efa05b4b2e upstream.
We have had three independent production user reports in combination with Cilium utilizing WireGuard as encryption underneath that k8s Pod E/W traffic to certain peer nodes fully stalled. The situation appears as follows:
Occurs very rarely but at random times under heavy networking load.
Once the issue triggers the decryption side stops working completely for that WireGuard peer, other peers keep working fine. The stall happens also for newly initiated connections towards that particular WireGuard peer.
Only the decryption side is affected, never the encryption side.
Once it triggers, it never recovers and remains in this state, the CPU/mem on that node looks normal, no leak, busy loop or crash.
bpftrace on the affected system shows that wg_prev_queue_enqueue fails, thus the MAX_QUEUED_PACKETS (1024 skbs!) for the peers rx_queue is reached.
Also, bpftrace shows that wg_packet_rx_poll for that peer is never called again after reaching this state for that peer. For other peers wg_packet_rx_poll does get called normally.
Commit db9ae3b (wireguard: device: enable threaded NAPI) switched WireGuard to threaded NAPI by default. The default has not been changed for triggering the issue, neither did CPU hotplugging occur (i.e. 5bd8de2 (wireguard: queueing: always return valid online CPU in wg_cpumask_choose_online())).
The issue has been observed with stable kernels of v5.15 as well as v6.1. It was reported to us that v5.10 stable is working fine, and no report on v6.6 stable either (somewhat related discussion in [0] though).
In the WireGuard driver the only material difference between v5.10 stable and v5.15 stable is the switch to threaded NAPI by default.
[0] https://lore.kernel.org/netdev/CA+wXwBTT74RErDGAnj98PqS=wvdh8eM1pi4q6tTdExtjnokKqA@mail.gmail.com/
Breakdown of the problem:
| Name | Vendor | Start Version | End Version |
|---|---|---|---|
| Linux | Ubuntu | bionic | * |
| Linux | Ubuntu | esm-infra-legacy/trusty | * |
| Linux | Ubuntu | esm-infra-legacy/xenial | * |
| Linux | Ubuntu | esm-infra/bionic | * |
| Linux | Ubuntu | esm-infra/focal | * |
| Linux | Ubuntu | esm-infra/xenial | * |
| Linux | Ubuntu | focal | * |
| Linux | Ubuntu | jammy | * |
| Linux | Ubuntu | noble | * |
| Linux | Ubuntu | questing | * |
| Linux | Ubuntu | trusty | * |
| Linux | Ubuntu | trusty/esm | * |
| Linux | Ubuntu | upstream | * |
| Linux | Ubuntu | xenial | * |
| Linux-allwinner-5.19 | Ubuntu | jammy | * |
| Linux-allwinner-5.19 | Ubuntu | upstream | * |
| Linux-aws | Ubuntu | bionic | * |
| Linux-aws | Ubuntu | esm-infra-legacy/trusty | * |
| Linux-aws | Ubuntu | esm-infra-legacy/xenial | * |
| Linux-aws | Ubuntu | esm-infra/bionic | * |
| Linux-aws | Ubuntu | esm-infra/focal | * |
| Linux-aws | Ubuntu | esm-infra/xenial | * |
| Linux-aws | Ubuntu | focal | * |
| Linux-aws | Ubuntu | jammy | * |
| Linux-aws | Ubuntu | noble | * |
| Linux-aws | Ubuntu | questing | * |
| Linux-aws | Ubuntu | trusty | * |
| Linux-aws | Ubuntu | trusty/esm | * |
| Linux-aws | Ubuntu | upstream | * |
| Linux-aws | Ubuntu | xenial | * |
| Linux-aws-5.0 | Ubuntu | bionic | * |
| Linux-aws-5.0 | Ubuntu | esm-infra/bionic | * |
| Linux-aws-5.0 | Ubuntu | upstream | * |
| Linux-aws-5.11 | Ubuntu | esm-infra/focal | * |
| Linux-aws-5.11 | Ubuntu | focal | * |
| Linux-aws-5.11 | Ubuntu | upstream | * |
| Linux-aws-5.13 | Ubuntu | esm-infra/focal | * |
| Linux-aws-5.13 | Ubuntu | focal | * |
| Linux-aws-5.13 | Ubuntu | upstream | * |
| Linux-aws-5.15 | Ubuntu | esm-infra/focal | * |
| Linux-aws-5.15 | Ubuntu | focal | * |
| Linux-aws-5.15 | Ubuntu | upstream | * |
| Linux-aws-5.19 | Ubuntu | jammy | * |
| Linux-aws-5.19 | Ubuntu | upstream | * |
| Linux-aws-5.3 | Ubuntu | bionic | * |
| Linux-aws-5.3 | Ubuntu | esm-infra/bionic | * |
| Linux-aws-5.3 | Ubuntu | upstream | * |
| Linux-aws-5.4 | Ubuntu | bionic | * |
| Linux-aws-5.4 | Ubuntu | esm-infra/bionic | * |
| Linux-aws-5.4 | Ubuntu | upstream | * |
| Linux-aws-5.8 | Ubuntu | esm-infra/focal | * |
| Linux-aws-5.8 | Ubuntu | focal | * |
| Linux-aws-5.8 | Ubuntu | upstream | * |
| Linux-aws-6.14 | Ubuntu | noble | * |
| Linux-aws-6.14 | Ubuntu | upstream | * |
| Linux-aws-6.17 | Ubuntu | noble | * |
| Linux-aws-6.17 | Ubuntu | upstream | * |
| Linux-aws-6.2 | Ubuntu | jammy | * |
| Linux-aws-6.2 | Ubuntu | upstream | * |
| Linux-aws-6.5 | Ubuntu | jammy | * |
| Linux-aws-6.5 | Ubuntu | upstream | * |
| Linux-aws-6.8 | Ubuntu | jammy | * |
| Linux-aws-6.8 | Ubuntu | upstream | * |
| Linux-aws-fips | Ubuntu | fips-preview/jammy | * |
| Linux-aws-fips | Ubuntu | fips-updates/bionic | * |
| Linux-aws-fips | Ubuntu | fips-updates/focal | * |
| Linux-aws-fips | Ubuntu | fips-updates/jammy | * |
| Linux-aws-fips | Ubuntu | fips-updates/noble | * |
| Linux-aws-fips | Ubuntu | fips/bionic | * |
| Linux-aws-fips | Ubuntu | fips/focal | * |
| Linux-aws-fips | Ubuntu | trusty | * |
| Linux-aws-fips | Ubuntu | upstream | * |
| Linux-aws-fips | Ubuntu | xenial | * |
| Linux-aws-hwe | Ubuntu | esm-infra-legacy/xenial | * |
| Linux-aws-hwe | Ubuntu | esm-infra/xenial | * |
| Linux-aws-hwe | Ubuntu | upstream | * |
| Linux-aws-hwe | Ubuntu | xenial | * |
| Linux-azure | Ubuntu | bionic | * |
| Linux-azure | Ubuntu | esm-infra-legacy/trusty | * |
| Linux-azure | Ubuntu | esm-infra-legacy/xenial | * |
| Linux-azure | Ubuntu | esm-infra/bionic | * |
| Linux-azure | Ubuntu | esm-infra/focal | * |
| Linux-azure | Ubuntu | esm-infra/xenial | * |
| Linux-azure | Ubuntu | focal | * |
| Linux-azure | Ubuntu | jammy | * |
| Linux-azure | Ubuntu | noble | * |
| Linux-azure | Ubuntu | questing | * |
| Linux-azure | Ubuntu | trusty | * |
| Linux-azure | Ubuntu | trusty/esm | * |
| Linux-azure | Ubuntu | upstream | * |
| Linux-azure | Ubuntu | xenial | * |
| Linux-azure-4.15 | Ubuntu | bionic | * |
| Linux-azure-4.15 | Ubuntu | esm-infra/bionic | * |
| Linux-azure-4.15 | Ubuntu | upstream | * |
| Linux-azure-5.11 | Ubuntu | esm-infra/focal | * |
| Linux-azure-5.11 | Ubuntu | focal | * |
| Linux-azure-5.11 | Ubuntu | upstream | * |
| Linux-azure-5.13 | Ubuntu | esm-infra/focal | * |
| Linux-azure-5.13 | Ubuntu | focal | * |
| Linux-azure-5.13 | Ubuntu | upstream | * |
| Linux-azure-5.15 | Ubuntu | esm-infra/focal | * |
| Linux-azure-5.15 | Ubuntu | focal | * |
| Linux-azure-5.15 | Ubuntu | upstream | * |
| Linux-azure-5.19 | Ubuntu | jammy | * |
| Linux-azure-5.19 | Ubuntu | upstream | * |
| Linux-azure-5.3 | Ubuntu | bionic | * |
| Linux-azure-5.3 | Ubuntu | esm-infra/bionic | * |
| Linux-azure-5.3 | Ubuntu | upstream | * |
| Linux-azure-5.4 | Ubuntu | bionic | * |
| Linux-azure-5.4 | Ubuntu | esm-infra/bionic | * |
| Linux-azure-5.4 | Ubuntu | upstream | * |
| Linux-azure-5.8 | Ubuntu | esm-infra/focal | * |
| Linux-azure-5.8 | Ubuntu | focal | * |
| Linux-azure-5.8 | Ubuntu | upstream | * |
| Linux-azure-6.11 | Ubuntu | noble | * |
| Linux-azure-6.11 | Ubuntu | upstream | * |
| Linux-azure-6.14 | Ubuntu | noble | * |
| Linux-azure-6.14 | Ubuntu | upstream | * |
| Linux-azure-6.17 | Ubuntu | noble | * |
| Linux-azure-6.17 | Ubuntu | upstream | * |
| Linux-azure-6.2 | Ubuntu | jammy | * |
| Linux-azure-6.2 | Ubuntu | upstream | * |
| Linux-azure-6.5 | Ubuntu | jammy | * |
| Linux-azure-6.5 | Ubuntu | upstream | * |
| Linux-azure-6.8 | Ubuntu | jammy | * |
| Linux-azure-6.8 | Ubuntu | upstream | * |
| Linux-azure-edge | Ubuntu | bionic | * |
| Linux-azure-edge | Ubuntu | esm-infra/bionic | * |
| Linux-azure-edge | Ubuntu | upstream | * |
| Linux-azure-fde | Ubuntu | esm-infra/focal | * |
| Linux-azure-fde | Ubuntu | focal | * |
| Linux-azure-fde | Ubuntu | jammy | * |
| Linux-azure-fde | Ubuntu | noble | * |
| Linux-azure-fde | Ubuntu | questing | * |
| Linux-azure-fde | Ubuntu | upstream | * |
| Linux-azure-fde-5.15 | Ubuntu | esm-infra/focal | * |
| Linux-azure-fde-5.15 | Ubuntu | focal | * |
| Linux-azure-fde-5.15 | Ubuntu | upstream | * |
| Linux-azure-fde-5.19 | Ubuntu | jammy | * |
| Linux-azure-fde-5.19 | Ubuntu | upstream | * |
| Linux-azure-fde-6.14 | Ubuntu | noble | * |
| Linux-azure-fde-6.14 | Ubuntu | upstream | * |
| Linux-azure-fde-6.17 | Ubuntu | upstream | * |
| Linux-azure-fde-6.2 | Ubuntu | jammy | * |
| Linux-azure-fde-6.2 | Ubuntu | upstream | * |
| Linux-azure-fde-6.8 | Ubuntu | upstream | * |
| Linux-azure-fips | Ubuntu | fips-preview/jammy | * |
| Linux-azure-fips | Ubuntu | fips-updates/bionic | * |
| Linux-azure-fips | Ubuntu | fips-updates/focal | * |
| Linux-azure-fips | Ubuntu | fips-updates/jammy | * |
| Linux-azure-fips | Ubuntu | fips-updates/noble | * |
| Linux-azure-fips | Ubuntu | fips/bionic | * |
| Linux-azure-fips | Ubuntu | fips/focal | * |
| Linux-azure-fips | Ubuntu | trusty | * |
| Linux-azure-fips | Ubuntu | upstream | * |
| Linux-azure-fips | Ubuntu | xenial | * |
| Linux-azure-nvidia | Ubuntu | noble | * |
| Linux-azure-nvidia | Ubuntu | upstream | * |
| Linux-azure-nvidia-6.14 | Ubuntu | noble | * |
| Linux-azure-nvidia-6.14 | Ubuntu | upstream | * |
| Linux-bluefield | Ubuntu | bluefield/jammy | * |
| Linux-bluefield | Ubuntu | esm-infra/focal | * |
| Linux-bluefield | Ubuntu | focal | * |
| Linux-bluefield | Ubuntu | upstream | * |
| Linux-fips | Ubuntu | fips-preview/jammy | * |
| Linux-fips | Ubuntu | fips-updates/bionic | * |
| Linux-fips | Ubuntu | fips-updates/focal | * |
| Linux-fips | Ubuntu | fips-updates/jammy | * |
| Linux-fips | Ubuntu | fips-updates/noble | * |
| Linux-fips | Ubuntu | fips-updates/xenial | * |
| Linux-fips | Ubuntu | fips/bionic | * |
| Linux-fips | Ubuntu | fips/focal | * |
| Linux-fips | Ubuntu | fips/xenial | * |
| Linux-fips | Ubuntu | upstream | * |
| Linux-gcp | Ubuntu | bionic | * |
| Linux-gcp | Ubuntu | esm-infra-legacy/xenial | * |
| Linux-gcp | Ubuntu | esm-infra/bionic | * |
| Linux-gcp | Ubuntu | esm-infra/focal | * |
| Linux-gcp | Ubuntu | esm-infra/xenial | * |
| Linux-gcp | Ubuntu | focal | * |
| Linux-gcp | Ubuntu | jammy | * |
| Linux-gcp | Ubuntu | noble | * |
| Linux-gcp | Ubuntu | questing | * |
| Linux-gcp | Ubuntu | upstream | * |
| Linux-gcp | Ubuntu | xenial | * |
| Linux-gcp-4.15 | Ubuntu | bionic | * |
| Linux-gcp-4.15 | Ubuntu | esm-infra/bionic | * |
| Linux-gcp-4.15 | Ubuntu | upstream | * |
| Linux-gcp-5.11 | Ubuntu | esm-infra/focal | * |
| Linux-gcp-5.11 | Ubuntu | focal | * |
| Linux-gcp-5.11 | Ubuntu | upstream | * |
| Linux-gcp-5.13 | Ubuntu | esm-infra/focal | * |
| Linux-gcp-5.13 | Ubuntu | focal | * |
| Linux-gcp-5.13 | Ubuntu | upstream | * |
| Linux-gcp-5.15 | Ubuntu | esm-infra/focal | * |
| Linux-gcp-5.15 | Ubuntu | focal | * |
| Linux-gcp-5.15 | Ubuntu | upstream | * |
| Linux-gcp-5.19 | Ubuntu | jammy | * |
| Linux-gcp-5.19 | Ubuntu | upstream | * |
| Linux-gcp-5.3 | Ubuntu | bionic | * |
| Linux-gcp-5.3 | Ubuntu | esm-infra/bionic | * |
| Linux-gcp-5.3 | Ubuntu | upstream | * |
| Linux-gcp-5.4 | Ubuntu | bionic | * |
| Linux-gcp-5.4 | Ubuntu | esm-infra/bionic | * |
| Linux-gcp-5.4 | Ubuntu | upstream | * |
| Linux-gcp-5.8 | Ubuntu | esm-infra/focal | * |
| Linux-gcp-5.8 | Ubuntu | focal | * |
| Linux-gcp-5.8 | Ubuntu | upstream | * |
| Linux-gcp-6.11 | Ubuntu | noble | * |
| Linux-gcp-6.11 | Ubuntu | upstream | * |
| Linux-gcp-6.14 | Ubuntu | noble | * |
| Linux-gcp-6.14 | Ubuntu | upstream | * |
| Linux-gcp-6.17 | Ubuntu | noble | * |
| Linux-gcp-6.17 | Ubuntu | upstream | * |
| Linux-gcp-6.2 | Ubuntu | jammy | * |
| Linux-gcp-6.2 | Ubuntu | upstream | * |
| Linux-gcp-6.5 | Ubuntu | jammy | * |
| Linux-gcp-6.5 | Ubuntu | upstream | * |
| Linux-gcp-6.8 | Ubuntu | jammy | * |
| Linux-gcp-6.8 | Ubuntu | upstream | * |
| Linux-gcp-fips | Ubuntu | fips-preview/jammy | * |
| Linux-gcp-fips | Ubuntu | fips-updates/bionic | * |
| Linux-gcp-fips | Ubuntu | fips-updates/focal | * |
| Linux-gcp-fips | Ubuntu | fips-updates/jammy | * |
| Linux-gcp-fips | Ubuntu | fips-updates/noble | * |
| Linux-gcp-fips | Ubuntu | fips/bionic | * |
| Linux-gcp-fips | Ubuntu | fips/focal | * |
| Linux-gcp-fips | Ubuntu | trusty | * |
| Linux-gcp-fips | Ubuntu | upstream | * |
| Linux-gcp-fips | Ubuntu | xenial | * |
| Linux-gke | Ubuntu | esm-infra/focal | * |
| Linux-gke | Ubuntu | focal | * |
| Linux-gke | Ubuntu | jammy | * |
| Linux-gke | Ubuntu | noble | * |
| Linux-gke | Ubuntu | upstream | * |
| Linux-gke | Ubuntu | xenial | * |
| Linux-gke-4.15 | Ubuntu | bionic | * |
| Linux-gke-4.15 | Ubuntu | esm-infra/bionic | * |
| Linux-gke-4.15 | Ubuntu | upstream | * |
| Linux-gke-5.15 | Ubuntu | esm-infra/focal | * |
| Linux-gke-5.15 | Ubuntu | focal | * |
| Linux-gke-5.15 | Ubuntu | upstream | * |
| Linux-gke-5.4 | Ubuntu | bionic | * |
| Linux-gke-5.4 | Ubuntu | esm-infra/bionic | * |
| Linux-gke-5.4 | Ubuntu | upstream | * |
| Linux-gkeop | Ubuntu | esm-infra/focal | * |
| Linux-gkeop | Ubuntu | focal | * |
| Linux-gkeop | Ubuntu | jammy | * |
| Linux-gkeop | Ubuntu | noble | * |
| Linux-gkeop | Ubuntu | upstream | * |
| Linux-gkeop-5.15 | Ubuntu | esm-infra/focal | * |
| Linux-gkeop-5.15 | Ubuntu | focal | * |
| Linux-gkeop-5.15 | Ubuntu | upstream | * |
| Linux-gkeop-5.4 | Ubuntu | bionic | * |
| Linux-gkeop-5.4 | Ubuntu | esm-infra/bionic | * |
| Linux-gkeop-5.4 | Ubuntu | upstream | * |
| Linux-hwe | Ubuntu | bionic | * |
| Linux-hwe | Ubuntu | esm-infra-legacy/xenial | * |
| Linux-hwe | Ubuntu | esm-infra/bionic | * |
| Linux-hwe | Ubuntu | esm-infra/xenial | * |
| Linux-hwe | Ubuntu | upstream | * |
| Linux-hwe | Ubuntu | xenial | * |
| Linux-hwe-5.11 | Ubuntu | esm-infra/focal | * |
| Linux-hwe-5.11 | Ubuntu | focal | * |
| Linux-hwe-5.11 | Ubuntu | upstream | * |
| Linux-hwe-5.13 | Ubuntu | esm-infra/focal | * |
| Linux-hwe-5.13 | Ubuntu | focal | * |
| Linux-hwe-5.13 | Ubuntu | upstream | * |
| Linux-hwe-5.15 | Ubuntu | esm-infra/focal | * |
| Linux-hwe-5.15 | Ubuntu | focal | * |
| Linux-hwe-5.15 | Ubuntu | upstream | * |
| Linux-hwe-5.19 | Ubuntu | jammy | * |
| Linux-hwe-5.19 | Ubuntu | upstream | * |
| Linux-hwe-5.4 | Ubuntu | bionic | * |
| Linux-hwe-5.4 | Ubuntu | esm-infra/bionic | * |
| Linux-hwe-5.4 | Ubuntu | upstream | * |
| Linux-hwe-5.8 | Ubuntu | esm-infra/focal | * |
| Linux-hwe-5.8 | Ubuntu | focal | * |
| Linux-hwe-5.8 | Ubuntu | upstream | * |
| Linux-hwe-6.11 | Ubuntu | noble | * |
| Linux-hwe-6.11 | Ubuntu | upstream | * |
| Linux-hwe-6.14 | Ubuntu | noble | * |
| Linux-hwe-6.14 | Ubuntu | upstream | * |
| Linux-hwe-6.17 | Ubuntu | noble | * |
| Linux-hwe-6.17 | Ubuntu | upstream | * |
| Linux-hwe-6.2 | Ubuntu | jammy | * |
| Linux-hwe-6.2 | Ubuntu | upstream | * |
| Linux-hwe-6.5 | Ubuntu | jammy | * |
| Linux-hwe-6.5 | Ubuntu | upstream | * |
| Linux-hwe-6.8 | Ubuntu | jammy | * |
| Linux-hwe-6.8 | Ubuntu | upstream | * |
| Linux-hwe-7.0 | Ubuntu | upstream | * |
| Linux-hwe-edge | Ubuntu | bionic | * |
| Linux-hwe-edge | Ubuntu | esm-infra-legacy/xenial | * |
| Linux-hwe-edge | Ubuntu | esm-infra/bionic | * |
| Linux-hwe-edge | Ubuntu | esm-infra/xenial | * |
| Linux-hwe-edge | Ubuntu | upstream | * |
| Linux-hwe-edge | Ubuntu | xenial | * |
| Linux-ibm | Ubuntu | esm-infra/focal | * |
| Linux-ibm | Ubuntu | focal | * |
| Linux-ibm | Ubuntu | jammy | * |
| Linux-ibm | Ubuntu | noble | * |
| Linux-ibm | Ubuntu | upstream | * |
| Linux-ibm-5.15 | Ubuntu | esm-infra/focal | * |
| Linux-ibm-5.15 | Ubuntu | focal | * |
| Linux-ibm-5.15 | Ubuntu | upstream | * |
| Linux-ibm-5.4 | Ubuntu | bionic | * |
| Linux-ibm-5.4 | Ubuntu | esm-infra/bionic | * |
| Linux-ibm-5.4 | Ubuntu | upstream | * |
| Linux-ibm-6.8 | Ubuntu | jammy | * |
| Linux-ibm-6.8 | Ubuntu | upstream | * |
| Linux-intel-5.13 | Ubuntu | esm-infra/focal | * |
| Linux-intel-5.13 | Ubuntu | focal | * |
| Linux-intel-5.13 | Ubuntu | upstream | * |
| Linux-intel-iot-realtime | Ubuntu | jammy | * |
| Linux-intel-iot-realtime | Ubuntu | realtime/jammy | * |
| Linux-intel-iot-realtime | Ubuntu | upstream | * |
| Linux-intel-iotg | Ubuntu | jammy | * |
| Linux-intel-iotg | Ubuntu | upstream | * |
| Linux-intel-iotg-5.15 | Ubuntu | esm-infra/focal | * |
| Linux-intel-iotg-5.15 | Ubuntu | focal | * |
| Linux-intel-iotg-5.15 | Ubuntu | upstream | * |
| Linux-iot | Ubuntu | esm-infra/focal | * |
| Linux-iot | Ubuntu | focal | * |
| Linux-iot | Ubuntu | upstream | * |
| Linux-kvm | Ubuntu | bionic | * |
| Linux-kvm | Ubuntu | esm-infra-legacy/xenial | * |
| Linux-kvm | Ubuntu | esm-infra/bionic | * |
| Linux-kvm | Ubuntu | esm-infra/focal | * |
| Linux-kvm | Ubuntu | esm-infra/xenial | * |
| Linux-kvm | Ubuntu | focal | * |
| Linux-kvm | Ubuntu | jammy | * |
| Linux-kvm | Ubuntu | upstream | * |
| Linux-kvm | Ubuntu | xenial | * |
| Linux-lowlatency | Ubuntu | jammy | * |
| Linux-lowlatency | Ubuntu | noble | * |
| Linux-lowlatency | Ubuntu | upstream | * |
| Linux-lowlatency-hwe-5.15 | Ubuntu | esm-infra/focal | * |
| Linux-lowlatency-hwe-5.15 | Ubuntu | focal | * |
| Linux-lowlatency-hwe-5.15 | Ubuntu | upstream | * |
| Linux-lowlatency-hwe-5.19 | Ubuntu | jammy | * |
| Linux-lowlatency-hwe-5.19 | Ubuntu | upstream | * |
| Linux-lowlatency-hwe-6.11 | Ubuntu | noble | * |
| Linux-lowlatency-hwe-6.11 | Ubuntu | upstream | * |
| Linux-lowlatency-hwe-6.2 | Ubuntu | jammy | * |
| Linux-lowlatency-hwe-6.2 | Ubuntu | upstream | * |
| Linux-lowlatency-hwe-6.5 | Ubuntu | jammy | * |
| Linux-lowlatency-hwe-6.5 | Ubuntu | upstream | * |
| Linux-lowlatency-hwe-6.8 | Ubuntu | jammy | * |
| Linux-lowlatency-hwe-6.8 | Ubuntu | upstream | * |
| Linux-lts-xenial | Ubuntu | esm-infra-legacy/trusty | * |
| Linux-lts-xenial | Ubuntu | trusty | * |
| Linux-lts-xenial | Ubuntu | trusty/esm | * |
| Linux-lts-xenial | Ubuntu | upstream | * |
| Linux-nvidia | Ubuntu | jammy | * |
| Linux-nvidia | Ubuntu | noble | * |
| Linux-nvidia | Ubuntu | upstream | * |
| Linux-nvidia-6.11 | Ubuntu | noble | * |
| Linux-nvidia-6.11 | Ubuntu | upstream | * |
| Linux-nvidia-6.17 | Ubuntu | noble | * |
| Linux-nvidia-6.17 | Ubuntu | upstream | * |
| Linux-nvidia-6.2 | Ubuntu | jammy | * |
| Linux-nvidia-6.2 | Ubuntu | upstream | * |
| Linux-nvidia-6.5 | Ubuntu | jammy | * |
| Linux-nvidia-6.5 | Ubuntu | upstream | * |
| Linux-nvidia-6.8 | Ubuntu | jammy | * |
| Linux-nvidia-6.8 | Ubuntu | upstream | * |
| Linux-nvidia-lowlatency | Ubuntu | noble | * |
| Linux-nvidia-lowlatency | Ubuntu | upstream | * |
| Linux-nvidia-tegra | Ubuntu | jammy | * |
| Linux-nvidia-tegra | Ubuntu | noble | * |
| Linux-nvidia-tegra | Ubuntu | upstream | * |
| Linux-nvidia-tegra-5.15 | Ubuntu | esm-infra/focal | * |
| Linux-nvidia-tegra-5.15 | Ubuntu | focal | * |
| Linux-nvidia-tegra-5.15 | Ubuntu | upstream | * |
| Linux-nvidia-tegra-igx | Ubuntu | jammy | * |
| Linux-nvidia-tegra-igx | Ubuntu | upstream | * |
| Linux-oem | Ubuntu | bionic | * |
| Linux-oem | Ubuntu | esm-infra/bionic | * |
| Linux-oem | Ubuntu | upstream | * |
| Linux-oem | Ubuntu | xenial | * |
| Linux-oem-5.10 | Ubuntu | esm-infra/focal | * |
| Linux-oem-5.10 | Ubuntu | focal | * |
| Linux-oem-5.10 | Ubuntu | upstream | * |
| Linux-oem-5.13 | Ubuntu | esm-infra/focal | * |
| Linux-oem-5.13 | Ubuntu | focal | * |
| Linux-oem-5.13 | Ubuntu | upstream | * |
| Linux-oem-5.14 | Ubuntu | esm-infra/focal | * |
| Linux-oem-5.14 | Ubuntu | focal | * |
| Linux-oem-5.14 | Ubuntu | upstream | * |
| Linux-oem-5.17 | Ubuntu | jammy | * |
| Linux-oem-5.17 | Ubuntu | upstream | * |
| Linux-oem-5.6 | Ubuntu | esm-infra/focal | * |
| Linux-oem-5.6 | Ubuntu | focal | * |
| Linux-oem-5.6 | Ubuntu | upstream | * |
| Linux-oem-6.0 | Ubuntu | jammy | * |
| Linux-oem-6.0 | Ubuntu | upstream | * |
| Linux-oem-6.1 | Ubuntu | jammy | * |
| Linux-oem-6.1 | Ubuntu | upstream | * |
| Linux-oem-6.11 | Ubuntu | noble | * |
| Linux-oem-6.11 | Ubuntu | upstream | * |
| Linux-oem-6.14 | Ubuntu | noble | * |
| Linux-oem-6.14 | Ubuntu | upstream | * |
| Linux-oem-6.17 | Ubuntu | noble | * |
| Linux-oem-6.17 | Ubuntu | upstream | * |
| Linux-oem-6.5 | Ubuntu | jammy | * |
| Linux-oem-6.5 | Ubuntu | upstream | * |
| Linux-oem-6.8 | Ubuntu | noble | * |
| Linux-oem-6.8 | Ubuntu | upstream | * |
| Linux-oem-7.0 | Ubuntu | upstream | * |
| Linux-oracle | Ubuntu | bionic | * |
| Linux-oracle | Ubuntu | esm-infra-legacy/xenial | * |
| Linux-oracle | Ubuntu | esm-infra/bionic | * |
| Linux-oracle | Ubuntu | esm-infra/focal | * |
| Linux-oracle | Ubuntu | esm-infra/xenial | * |
| Linux-oracle | Ubuntu | focal | * |
| Linux-oracle | Ubuntu | jammy | * |
| Linux-oracle | Ubuntu | noble | * |
| Linux-oracle | Ubuntu | questing | * |
| Linux-oracle | Ubuntu | upstream | * |
| Linux-oracle | Ubuntu | xenial | * |
| Linux-oracle-5.0 | Ubuntu | bionic | * |
| Linux-oracle-5.0 | Ubuntu | esm-infra/bionic | * |
| Linux-oracle-5.0 | Ubuntu | upstream | * |
| Linux-oracle-5.11 | Ubuntu | esm-infra/focal | * |
| Linux-oracle-5.11 | Ubuntu | focal | * |
| Linux-oracle-5.11 | Ubuntu | upstream | * |
| Linux-oracle-5.13 | Ubuntu | esm-infra/focal | * |
| Linux-oracle-5.13 | Ubuntu | focal | * |
| Linux-oracle-5.13 | Ubuntu | upstream | * |
| Linux-oracle-5.15 | Ubuntu | esm-infra/focal | * |
| Linux-oracle-5.15 | Ubuntu | focal | * |
| Linux-oracle-5.15 | Ubuntu | upstream | * |
| Linux-oracle-5.3 | Ubuntu | bionic | * |
| Linux-oracle-5.3 | Ubuntu | esm-infra/bionic | * |
| Linux-oracle-5.3 | Ubuntu | upstream | * |
| Linux-oracle-5.4 | Ubuntu | bionic | * |
| Linux-oracle-5.4 | Ubuntu | esm-infra/bionic | * |
| Linux-oracle-5.4 | Ubuntu | upstream | * |
| Linux-oracle-5.8 | Ubuntu | esm-infra/focal | * |
| Linux-oracle-5.8 | Ubuntu | focal | * |
| Linux-oracle-5.8 | Ubuntu | upstream | * |
| Linux-oracle-6.14 | Ubuntu | noble | * |
| Linux-oracle-6.14 | Ubuntu | upstream | * |
| Linux-oracle-6.17 | Ubuntu | noble | * |
| Linux-oracle-6.17 | Ubuntu | upstream | * |
| Linux-oracle-6.5 | Ubuntu | jammy | * |
| Linux-oracle-6.5 | Ubuntu | upstream | * |
| Linux-oracle-6.8 | Ubuntu | jammy | * |
| Linux-oracle-6.8 | Ubuntu | upstream | * |
| Linux-raspi | Ubuntu | esm-infra/focal | * |
| Linux-raspi | Ubuntu | focal | * |
| Linux-raspi | Ubuntu | jammy | * |
| Linux-raspi | Ubuntu | noble | * |
| Linux-raspi | Ubuntu | questing | * |
| Linux-raspi | Ubuntu | upstream | * |
| Linux-raspi-5.4 | Ubuntu | bionic | * |
| Linux-raspi-5.4 | Ubuntu | esm-infra/bionic | * |
| Linux-raspi-5.4 | Ubuntu | upstream | * |
| Linux-raspi-realtime | Ubuntu | noble | * |
| Linux-raspi-realtime | Ubuntu | realtime/noble | * |
| Linux-raspi-realtime | Ubuntu | upstream | * |
| Linux-raspi2 | Ubuntu | bionic | * |
| Linux-raspi2 | Ubuntu | esm-infra/focal | * |
| Linux-raspi2 | Ubuntu | focal | * |
| Linux-raspi2 | Ubuntu | upstream | * |
| Linux-raspi2 | Ubuntu | xenial | * |
| Linux-realtime | Ubuntu | jammy | * |
| Linux-realtime | Ubuntu | noble | * |
| Linux-realtime | Ubuntu | questing | * |
| Linux-realtime | Ubuntu | realtime/jammy | * |
| Linux-realtime | Ubuntu | realtime/noble | * |
| Linux-realtime | Ubuntu | upstream | * |
| Linux-realtime-6.14 | Ubuntu | realtime/noble | * |
| Linux-realtime-6.14 | Ubuntu | upstream | * |
| Linux-realtime-6.17 | Ubuntu | realtime/noble | * |
| Linux-realtime-6.17 | Ubuntu | upstream | * |
| Linux-realtime-6.8 | Ubuntu | realtime/jammy | * |
| Linux-realtime-6.8 | Ubuntu | upstream | * |
| Linux-riscv | Ubuntu | esm-infra/focal | * |
| Linux-riscv | Ubuntu | focal | * |
| Linux-riscv | Ubuntu | jammy | * |
| Linux-riscv | Ubuntu | noble | * |
| Linux-riscv | Ubuntu | questing | * |
| Linux-riscv | Ubuntu | upstream | * |
| Linux-riscv-5.11 | Ubuntu | esm-infra/focal | * |
| Linux-riscv-5.11 | Ubuntu | focal | * |
| Linux-riscv-5.11 | Ubuntu | upstream | * |
| Linux-riscv-5.15 | Ubuntu | esm-infra/focal | * |
| Linux-riscv-5.15 | Ubuntu | focal | * |
| Linux-riscv-5.15 | Ubuntu | upstream | * |
| Linux-riscv-5.19 | Ubuntu | jammy | * |
| Linux-riscv-5.19 | Ubuntu | upstream | * |
| Linux-riscv-5.8 | Ubuntu | esm-infra/focal | * |
| Linux-riscv-5.8 | Ubuntu | focal | * |
| Linux-riscv-5.8 | Ubuntu | upstream | * |
| Linux-riscv-6.14 | Ubuntu | noble | * |
| Linux-riscv-6.14 | Ubuntu | upstream | * |
| Linux-riscv-6.17 | Ubuntu | noble | * |
| Linux-riscv-6.17 | Ubuntu | upstream | * |
| Linux-riscv-6.5 | Ubuntu | jammy | * |
| Linux-riscv-6.5 | Ubuntu | upstream | * |
| Linux-riscv-6.8 | Ubuntu | jammy | * |
| Linux-riscv-6.8 | Ubuntu | upstream | * |
| Linux-starfive-5.19 | Ubuntu | jammy | * |
| Linux-starfive-5.19 | Ubuntu | upstream | * |
| Linux-starfive-6.2 | Ubuntu | jammy | * |
| Linux-starfive-6.2 | Ubuntu | upstream | * |
| Linux-starfive-6.5 | Ubuntu | jammy | * |
| Linux-starfive-6.5 | Ubuntu | upstream | * |
| Linux-xilinx | Ubuntu | noble | * |
| Linux-xilinx | Ubuntu | upstream | * |
| Linux-xilinx-zynqmp | Ubuntu | esm-infra/focal | * |
| Linux-xilinx-zynqmp | Ubuntu | focal | * |
| Linux-xilinx-zynqmp | Ubuntu | jammy | * |
| Linux-xilinx-zynqmp | Ubuntu | upstream | * |