Resources requests and limits¶

In this document, we are talking about the resources values set on the virt-launcher compute container, referred to as "the container" below for simplicity.

CPU¶

Note: dedicated CPUs (and isolated emulator thread) are ignored here as they have a dedicated page.

CPU requests on the container¶

By default, the container requests (1/cpuAllocationRatio) CPU per vCPU. The number of vCPUs is sockets * cores * threads, defaults to 1.
cpuAllocationRatio defaults to 10 but can be changed in the CR.
If a CPU limit is manually set on the VM(I) and no CPU request is, the CPU requests on the container will match the CPU limits
Manually setting CPU requests on the VM(I) will override all of the above and be the CPU requests for the container

CPU limits on the container¶

By default, no CPU limit is set on the container
If auto CPU limits is enabled (see next section), then the container will have a CPU limit of 1 per vCPU
Manually setting CPU limits on the VM(I) will override all of the above and be the CPU limits for the container

Auto CPU limits¶

KubeVirt automatically set CPU limits on VM(I)s if all the following conditions are true:

The namespace where the VMI will be created has a ResourceQuota containing cpu limits.
The VMI has no manually set cpu limits.
The VMI is not requesting dedicated CPU.

Additionally, you can add the namespaceLabelSelector in the KubeVirt CR, forcing CPU limits to be set.

In both cases, the VM(I) created will have a CPU limit of 1 per vCPU.

autoCPULimitNamespaceLabelSelector configuration¶

Cluster admins can define a label selector in the KubeVirt CR.
Once that label selector is defined, if the creation namespace matches the selector, all VM(I)s created in it will have a CPU limits set.

Example:

CR:

apiVersion: kubevirt.io/v1
kind: KubeVirt
metadata:
  name: kubevirt
  namespace: kubevirt
spec:
  configuration:
    autoCPULimitNamespaceLabelSelector:
      matchLabels:
        autoCpuLimit: "true"

Namespace:

apiVersion: v1
kind: Namespace
metadata:
  labels:
    autoCpuLimit: "true"
    kubernetes.io/metadata.name: default
  name: default

Memory¶

Memory requests on the container¶

VM(I)s must specify a desired amount of memory, in either spec.domain.memory.guest or spec.domain.resources.requests.memory (ignoring hugepages, see the dedicated page). If both are set, the memory requests take precedence. A calculated amount of overhead will be added to it, forming the memory request value for the container. See Memory overhead below for what contributes to that overhead.

Memory overhead¶

Various VM resources, such as a video adapter, IOThreads, and supplementary system software, consume additional memory from the Node, beyond the requested memory intended for the guest OS consumption. In order to provide a better estimate for the scheduler, this memory overhead will be calculated and added to the requested memory.

The default memory overhead has been set to 32Mi to account for these additional system resources.

Please see how Pods with resource requests are scheduled for additional information on resource requests and limits.

Viewing memory overhead¶

Note: This feature requires the VmiMemoryOverheadReport feature gate to be enabled.

When the feature gate is enabled, the calculated overhead is exposed in the VMI status:

apiVersion: kubevirt.io/v1
kind: VirtualMachineInstance
metadata:
  name: my-vmi
status:
  memory:
    memoryOverhead: 512Mi

The status.memory.memoryOverhead field is populated when the VMI starts, and reflects the overhead calculated for the current virt-launcher pod.

Memory overhead after live migration¶

During live migration, the target node may calculate a different overhead value (for example, due to different hardware or configuration). While a migration is in progress, the target overhead is reported in status.migrationState.targetMemoryOverhead. Once the migration completes, status.memory.memoryOverhead is updated to reflect the new value:

status:
  memory:
    memoryOverhead: 520Mi
  migrationState:
    targetMemoryOverhead: 520Mi

Memory limits on the container¶

By default, no memory limit is set on the container
If auto memory limits is enabled (see next section), then the container will have a limit of 2x the requested memory.
Manually setting a memory limit on the VM(I) will set the same value on the container

Warnings¶

Memory limits have to be more than memory requests + overhead, otherwise the container will have memory requests > limits and be rejected by Kubernetes.
Memory usage bursts could lead to VM crashes when memory limits are set

Auto memory limits¶

KubeVirt automatically set memory limits on VM(I)s if all the following conditions are true:

The namespace where the VMI will be created has a ResourceQuota containing memory limits.
The VMI has no manually set memory limits.
The VMI is not requesting dedicated CPU.

If all the previous conditions are true, the memory limits will be set to a value (2x) of the memory requests. This ratio can be adjusted, per namespace, by adding the label alpha.kubevirt.io/auto-memory-limits-ratio, with the desired custom value. For example, with alpha.kubevirt.io/auto-memory-limits-ratio: "1.2", the memory limits set will be equal to (1.2x) of the memory requests.