I/O to unpinned memory supporting memory overcommit and live migration of virtual machines

Patent metadata

Field	Value
Publication number	US-10922185-B2
Application number	US-201916279462-A
Country	US
Kind code	B2
Filing date	Feb 19, 2019
Priority date	Feb 19, 2019
Publication date	Feb 16, 2021
Grant date	Feb 16, 2021

Abstract

Systems and methods of error handling in a network interface card (NIC) are provided. For a data packet destined for a local virtual machine (VM), if the NIC cannot determine a valid translation memory address for a virtual memory address in a buffer descriptor from a receive queue of the VM, the NIC can retrieve a backup buffer descriptor from a hypervisor queue, and store the packet in a host memory location indicated by an address in the backup buffer descriptor. For a transmission request from a local VM, if the NIC cannot determine a valid translated address for a virtual memory address in the packet descriptor from a transmit queue of the VM, the NIC can send a message to a hypervisor backup queue, and generate and transmit a data packet based on data in a memory page reallocated by the hypervisor.

First claim

What is claimed is:

1. A method of error handling in a network interface card of a host device, the method comprising:
receiving, at a processor of the network interface card, a data packet addressed to a virtual machine executing on the host device;
selecting, by the processor, a receive queue corresponding to the virtual machine and residing in a memory of the network interface card;
retrieving, by the processor, a buffer descriptor from the receive queue, wherein the buffer descriptor includes a virtual memory address; and
in response to determining that the virtual memory address is not associated with a valid translated memory address associated with the virtual machine:
retrieving, by the processor, a backup buffer descriptor from a hypervisor queue residing in the network interface card memory and corresponding to a hypervisor executing on the host device; and
storing, by the processor, contents of the data packet in a host device memory location indicated by a backup memory address in the backup buffer descriptor.

2. The method of claim 1 , comprising:
receiving, at the processor, a second data packet addressed to the virtual machine;
selecting, by the processor, a second receive queue assigned to the virtual machine and residing in a memory of the network interface card;
retrieving, by the processor, a second buffer descriptor from the second receive queue, wherein the second buffer descriptor includes a second virtual address;
in response to determining that the second virtual address is associated with a valid second translated address associated with the virtual machine, storing, by the processor, the second data packet in a second host device memory location indicated by the second translated address.

3. The method of claim 1 , comprising:
allocating, by the hypervisor in response to receiving a notification from the network interface card that the address in the buffer descriptor is not associated with a valid translated address, a new page of host device memory to the virtual machine, wherein the host device memory location is in the new page.

4. The method of claim 1 , wherein the hypervisor has allocated unpinned host device memory to virtual machines.

5. The method of claim 4 , wherein the hypervisor has allocated host device memory to virtual machines that in aggregate exceeds a total host device memory available to the virtual machines.

6. The method of claim 1 , comprising:
migrating, by the hypervisor, the virtual machine to a second host device, wherein the processor receives the data packet during the migration after memory previously allocated to the virtual machine has been reallocated to a second virtual machine;
maintaining, by the processor, a list of memory locations associated with the virtual machine that change during the migration; and
transferring, by the processor, data from the memory locations maintained in the list to a second memory associated with the migrated virtual machine on the second host device.

7. A system comprising a host device executing one or more virtual machines and a hypervisor, a host device memory, and a network interface card configured to:
receive a data packet addressed to a virtual machine executing on the host device;
select a receive queue corresponding to the virtual machine and residing in a memory of the network interface card;
retrieve a buffer descriptor from the receive queue, wherein the buffer descriptor includes a virtual memory address; and
in response to determining that the virtual memory address is not associated with a valid translated memory address associated with the virtual machine:
retrieve a backup buffer descriptor from a hypervisor queue residing in the network interface card memory and corresponding to a hypervisor executing on the host device; and
store contents of the data packet in a host device memory location indicated by a backup memory address in the backup buffer descriptor.

8. The system of claim 7 , the network interface card configured to:
receive a second data packet addressed to the virtual machine;
select a second receive queue assigned to the virtual machine and residing in a memory of the network interface card;
retrieve a second buffer descriptor from the second receive queue, wherein the second buffer descriptor includes a second virtual address;
in response to determining that the second virtual address is associated with a valid second translated address associated with the virtual machine, store the second data packet in a second host device memory location indicated by the second translated address.

9. The system of claim 7 , wherein the hypervisor is configured to allocate, in response to receiving a notification from the network interface card that the address in the buffer descriptor is not associated with a valid translated address, a new page of host device memory to the virtual machine, wherein the host device memory location is in the new page.

10. The system of claim 9 , wherein the hypervisor has allocated unpinned host device memory to the virtual machines.

11. The system of claim 7 , wherein the hypervisor has allocated host device memory to the virtual machines that in aggregate exceeds a total host device memory available to the virtual machines.

12. The system of claim 7 , wherein:
the hypervisor is configured to migrate the virtual machine to a second host device, wherein the network interface card receives the data packet during the migration after memory previously allocated to the virtual machine has been reallocated to a second virtual machine; and
the network interface card is configured to:
maintain a list of memory locations associated with the virtual machine that change during the migration; and
transfer data from the memory locations maintained in the list to a second memory associated with the migrated virtual machine on the second host device.

13. A method of error handling in a network interface card of a host device, the method comprising:
receiving, at a processor of the network interface card, a transmission request from a virtual machine executing on the host device;
retrieving, by the processor, a packet descriptor from a transmit queue corresponding to the virtual machine and residing in a memory of the network interface card, wherein the packet descriptor includes a virtual memory address; and
in response to determining that the virtual memory address is not associated with a valid translation memory address associated with the virtual machine:
sending, by the processor, a message to a backup queue residing in the network interface card memory and corresponding to a hypervisor executing on the host device, the message including an identifier of the transmit queue and the virtual memory address;
receiving, by the processor, a notification from the hypervisor that a host device memory region has been reallocated to the virtual machine;
retrieving, by the processor from the transmit queue, a second packet descriptor including a second virtual memory address;
translating, by the processor, the second virtual memory address;
generating, by the processor, a data packet using data from a host device memory location indicated by the translated second virtual memory address; and
transmitting, by the processor, the data packet.

14. The method of claim 13 , comprising:
receiving, by the processor, a second transmission request from the virtual machine;
retrieving, by the processor, a third packet descriptor from a second transmit queue corresponding to the virtual machine, wherein the third packet descriptor includes a third virtual memory address;

Assignees

• Google Llc

Inventors

• Chandra Prashant
• Mclaren Ian
• Olson Jon
• Adriaens Jacob

Classifications

• G06F2009/45595

  Network integration; Enabling network access in virtual machine instances · CPC title

• G06F2009/4557

  Distribution of virtual machine instances; Migration and load balancing · CPC title

• G06F9/45558

  Hypervisor-specific management and integration aspects · CPC title

• G06F2009/45583

  Memory management, e.g. access or allocation · CPC title

• G06F12/109

  for multiple virtual address spaces, e.g. segmentation (G06F12/1036 takes precedence) · CPC title