I/O to unpinned memory supporting memory overcommit and live migration of virtual machines

Patent metadata

Field	Value
Publication number	US-11650884-B2
Application number	US-202217740565-A
Country	US
Kind code	B2
Filing date	May 10, 2022
Priority date	Feb 19, 2019
Publication date	May 16, 2023
Grant date	May 16, 2023

Abstract

Systems and methods of error handling in a network interface card (NIC) are provided. For a data packet destined for a local virtual machine (VM), if the NIC cannot determine a valid translation memory address for a virtual memory address in a buffer descriptor from a receive queue of the VM, the NIC can retrieve a backup buffer descriptor from a hypervisor queue, and store the packet in a host memory location indicated by an address in the backup buffer descriptor. For a transmission request from a local VM, if the NIC cannot determine a valid translated address for a virtual memory address in the packet descriptor from a transmit queue of the VM, the NIC can send a message to a hypervisor backup queue, and generate and transmit a data packet based on data in a memory page reallocated by the hypervisor.

First claim

What is claimed is:

1. A method of transmitting data packets in a network interface card of a host device, the method comprising:
receiving, at a processor of the network interface card, a transmission request from a virtual machine executing on the host device;
retrieving, by the processor, a packet descriptor from a transmit queue corresponding to the virtual machine and residing in a memory of the network interface card, wherein the packet descriptor includes a virtual memory address; and
in response to determining that the virtual memory address is not associated with a valid translation memory address associated with the virtual machine:
sending, by the processor, a message to a backup queue residing in the network interface card memory, the message including an identifier of the transmit queue and the virtual memory address;
retrieving, by the processor, a second packet descriptor from the transmit queue, wherein the second packet descriptor includes a second virtual memory address;
translating, by the processor, the second virtual memory address;
generating, by the processor, a data packet using data from a host device memory location indicated by the translated second virtual memory address; and
transmitting, by the processor, the data packet.

2. The method of claim 1 , comprising:
receiving, by the processor, a second transmission request from the virtual machine;
retrieving, by the processor, a third packet descriptor from a second transmit queue corresponding to the virtual machine, wherein the third packet descriptor includes a third virtual memory address;
translating, by the processor, the third virtual memory address; and
in response to determining that the third virtual memory address is associated with a valid translation memory address associated with the virtual machine:
generating, by the processor, a second data packet using data from a second host device memory location indicated by the translated third virtual memory address; and
transmitting, by the processor, the second data packet.

3. The method of claim 1 , comprising:
allocating, by a virtual machine memory manager executing on the host device, a new page of host device memory to the virtual machine;
retrieving, by the virtual machine memory manager, backup data of a host device memory location previously associated with the virtual machine; and
storing, by the processor, the backup data in the new page.

4. The method of claim 1 , wherein the virtual machine memory manager has allocated unpinned host device memory to virtual machines.

5. The method of claim 4 , wherein the virtual machine memory manager has allocated host device memory to virtual machines that in aggregate exceeds a total host device memory available to the virtual machines.

6. The method of claim 1 , comprising:
migrating, by a virtual machine memory manager executing on the host device, the virtual machine to a second host device, wherein the processor receives the data packet during the migration after memory previously allocated to the virtual machine has been reallocated to a second virtual machine;
maintaining, by the processor, a list of memory locations associated with the virtual machine that change during the migration; and
transferring, by the processor, data from the memory locations maintained in the list to a second memory associated with the migrated virtual machine on the second host device.

7. The method of claim 1 , further comprising receiving, by the processor, a notification that a host device memory region has been reallocated to the virtual machine.

8. A system comprising a host device executing one or more virtual machines and a virtual machine memory manager, a host device memory, and a network interface card configured to:
receive a transmission request from a virtual machine executing on the host device;
retrieve a packet descriptor from a transmit queue corresponding to the virtual machine and residing in a memory of the network interface card, wherein the packet descriptor includes a virtual memory address; and
in response to determining that the virtual memory address is not associated with a valid translation memory address associated with the virtual machine:
send a message to a backup queue residing in the network interface card memory, the message including an identifier of the transmit queue and the virtual memory address;
retrieve a second packet descriptor from the transmit queue, wherein the second packet descriptor includes a second virtual memory address;
translate the second virtual memory address;
generate a data packet using data from a host device memory location indicated by the translated second virtual memory address; and
transmit the data packet.

9. The system of claim 8 , wherein the network interface card is configured to:
receive a second transmission request from the virtual machine;
retrieve a third packet descriptor from a second transmit queue corresponding to the virtual machine, wherein the third packet descriptor includes a third virtual memory address;
translate the third virtual memory address; and
in response to determining that the third virtual memory address is associated with a valid translation memory address associated with the virtual machine:
generate a second data packet using data from a second host device memory location indicated by the translated third virtual memory address; and
transmit the second data packet.

10. The system of claim 8 , wherein:
the virtual machine manager is configured to:
allocate a new page of host device memory to the virtual machine; and
retrieve backup data of a host device memory location previously associated with the virtual machine; and

the network interface card is configured to store the backup data in the new page.

11. The system of claim 8 , wherein the virtual machine manager has allocated unpinned host device memory to the virtual machines.

12. The system of claim 11 , wherein the virtual machine manager has allocated host device memory to virtual machines that in aggregate exceeds a total host device memory available to the virtual machines.

13. The system of claim 8 , wherein:
the virtual machine manager is configured to migrate the virtual machine to a second host device, wherein the network interface card receives the data packet during the migration after memory previously allocated to the virtual machine has been reallocated to a second virtual machine; and
the network interface card is configured to:
maintain a list of memory locations associated with the virtual machine that change during the migration; and
transfer data from the memory locations maintained in the list to a second memory associated with the migrated virtual machine on the second host device.

14. The system of claim 8 , wherein the network interface card is configured to receive a notification from the virtual machine manager that a host device memory region has been reallocated to the virtual machine.

15. A non-transitory computer-readable medium storing instructions, that when executed by one or more processors of a network interface card of a host device, cause the one or more processors to:
receive a transmission request from a virtual machine executing on the host device;
retrieve a packet descriptor from a transmit queue corresponding to the virtual machine and residing in a memory of the network interface card, wherein the packet descriptor includes a virtual memory address; and
in response to determining that the virtual memory address is not associated with a valid translation memory address associated with the v

Assignees

• Google Llc

Inventors

• Chandra Prashant R
• Mclaren Ian
• Olson Jon
• Adriaens Jacob

Classifications

• G06F2009/45595

  Network integration; Enabling network access in virtual machine instances · CPC title

• G06F2009/4557

  Distribution of virtual machine instances; Migration and load balancing · CPC title

• G06F9/45558

  Hypervisor-specific management and integration aspects · CPC title

• G06F2009/45583

  Memory management, e.g. access or allocation · CPC title

• G06F9/544 Primary

  Buffers; Shared memory; Pipes · CPC title