NUMA I/O aware network queue assignments

Patent metadata

Field	Value
Publication number	US-9459904-B2
Application number	US-201414530587-A
Country	US
Kind code	B2
Filing date	Oct 31, 2014
Priority date	Sep 30, 2014
Publication date	Oct 4, 2016
Grant date	Oct 4, 2016

Abstract

Systems and methods for preferentially assigning virtual machines (VMs) on a particular NUMA node with network queues on the same NUMA node are described. A load balancer process on a host assigns multiple VMs to network queues. The assignment of the VMs to a network queues is performed with a bias toward assigning VMs using a particular NUMA node to network queues on the same NUMA node. A scheduler on the host assigns VMs to NUMA nodes. The scheduler is biased toward assigning VMs to the same NUMA node as the PNIC and/or the same NUMA node as a network queue assigned to the VM.

First claim

We claim:

1. A method of assigning a plurality of virtual machines (VMs) to a set of network queues operating on a host machine, the method comprising:
identifying a non-uniform memory access (NUMA) node associated with a VM of the plurality of VMs;
for the identified NUMA node, identifying, from the set of network queues, a network queue assigned to the NUMA node associated with the VM;
upon identifying the network queue assigned to the NUMA node associated with the VM, generating an assignment for the VM to one of the network queues from the set of network queues, said generating is biased toward assigning the VM to the network queue assigned to the NUMA node associated with the VM; and
using the generated assignment to assign the VM to the network queue assigned to the NUMA node associated with the VM, wherein the NUMA node is associated with the VM when the VM is executed by at least one processor of the NUMA node and data from the VM is stored in the memory of the NUMA node.

2. The method of claim 1 further comprising:
determining that the VM has been reassigned from a first NUMA node to a second NUMA node;
identifying a network queue assigned to the second NUMA node; and
reassigning the VM to the identified network queue assigned to the second NUMA node.

3. The method of claim 1 , wherein the generating bias comprises assigning the VM to the network queue of the NUMA node associated with the VM when a metric of the network queue of the NUMA node associated with the VM does not meet a threshold metric and assigning the VM to a network queue of a different NUMA node when the metric of the network queue of the NUMA node associated with the VM does meet the threshold metric.

4. The method of claim 3 further comprising:
after assigning the VM to the network queue of the different NUMA node, determining that the metric of the network queue of the NUMA node associated with the VM has changed from meeting the threshold metric to not meeting the threshold metric; and
responsive to said determination, reassigning the VM to the network queue of the NUMA node associated with the VM.

5. The method of claim 1 further comprising:
identifying a first plurality of network queues on the NUMA node;
identifying a set of VMs assigned to the first plurality of network queues;
determining that a metric of the first plurality of network queues, assigned to the set of VMs does not meet a threshold metric; and
generating a set of network queue assignments for the set of VMs that assigns every VM in the set of VMs with a second plurality of network queues, wherein the second plurality of network queues comprises fewer network queues than the first plurality of network queues.

6. The method of claim 1 further comprising:
identifying, a set of VMs associated with a particular network queue on the NUMA node;
determining that a metric of the set of VMs associated with the particular network queue meets a threshold metric; and
generating a set of network queue assignments for the set of VMs that assigns the set of VMs with at least two network queues.

7. The method of claim 6 , wherein the at least two network queues comprise an unassigned network queue, the NUMA node is a first NUMA node and the set of VMs comprises at least one VM on a second NUMA node, the method further comprising assigning the unassigned network queue to the second NUMA node.

8. The method of claim 1 further comprising:
identifying a set of VMs assigned to a particular network queue on the NUMA node;
identifying an additional VM, wherein the additional VM is not assigned to the NUMA node;
determining that a metric of the particular network queue would meet a threshold metric resulting from effects of the set of VMs on the metric if the set of VMs were to be assigned to the particular network queue and estimated effects on the metric by the additional VM being assigned to the particular network queue; and
based on the determining, generating a set of network queue assignments for the set of VMs and the additional VM that assigns the set of VMs and the additional VM with at least two network queues, one the of at least two network queues being the particular network queue.

9. The method of claim 1 , wherein the VM comprises a plurality of virtual central processing units (VCPUs), at least a first VCPU of the VM is assigned to a different NUMA node than at least a second VCPU of the VM, and wherein assigning the VM to one of the network queues comprises assigning the VM to a network queue assigned to a same NUMA node as at least one of the VCPUs of the VM.

10. A non-transitory machine readable medium storing a program for assigning a plurality of virtual machines (VMs) to a set of network queues operating on a host machine, the program comprising sets of instructions for:
identifying a non-uniform memory access (NUMA) node associated with a VM of the plurality of VMs;
for the identified NUMA node, identifying, from the set of network queues, a network queue assigned to the NUMA node;
upon identifying the network queue assigned to the NUMA node associated with the VM, generating an assignment for the VM to one of the network queues from the set of network queues, said generating is biased toward assigning the VM to a network queue assigned to the NUMA node associated with the VM; and
using the generated assignment to assign the VM to the network queue assigned to the NUMA node associated with the VM, wherein the NUMA node is associated with the VM when the VM is executed by at least one processor of the NUMA node and data from the VM is stored in the memory of the NUMA node.

11. The non-transitory machine readable medium of claim 10 , wherein the program further comprises sets of instructions for:
determining that the VM has been reassigned from a first NUMA node to a second NUMA node;
identifying a network queue assigned to the second NUMA node; and
reassigning the VM to the identified network queue assigned to the second NUMA node.

12. The non-transitory machine readable medium of claim 10 , wherein the generating, bias comprises assigning the VM to the network queue of the NUMA node associated with the VM when a metric of the network queue of the NUMA node associated with the VM does not meet a threshold metric and assigning the VM to a network queue of a different NUMA node when the metric of the network queue of the NUMA node associated with the VM does meet the threshold metric.

13. The machine readable medium of claim 12 , wherein the program further comprises sets of instructions for:
after assigning the VM to the network queue of the different NUMA node, determining that the metric of the network queue of the NUMA node associated with the VM has changed from meeting the threshold metric to not meeting the threshold metric; and
responsive to said determination, reassigning the VM to the network queue of the NUMA node associated with the VM.

14. The non-transitory machine readable medium of claim 10 , wherein the program further comprises sets of instructions for:
identifying a first plurality of network queues on the NUMA node;
identifying a set of VMs assigned to the first plurality of network queues;
determining that a metric of the first plurality of network queues, assigned to the set of VMs does not meet a threshold metric; and
generating a set of network queue assignments for the set of VMs that assigns every VM in the set of VMs with a second plurality of network queues, wherein the second plurality of network queues comprises fewer network queues than the first plurality of network queues.

15. The non-transitor

Assignees

• Vmware Inc

Inventors

• Mehta Rishi
• Shen Xiaochuan
• Banerjee Amitabha
• Veeraiyan Ayyappan

Classifications

• G06F2212/2542

  Non-uniform memory access [NUMA] architecture · CPC title

• G06F12/0223

  User address space allocation, e.g. contiguous or non contiguous base addressing · CPC title

• G06F12/0815

  Cache consistency protocols · CPC title

• G06F2009/4557

  Distribution of virtual machine instances; Migration and load balancing · CPC title

• G06F9/45558 Primary

  Hypervisor-specific management and integration aspects · CPC title