Methods and apparatus for accelerating virtual machine migration

Patent metadata

Field	Value
Publication number	US-11809899-B2
Application number	US-201916584716-A
Country	US
Kind code	B2
Filing date	Sep 26, 2019
Priority date	Jun 28, 2019
Publication date	Nov 7, 2023
Grant date	Nov 7, 2023

Abstract

A server having a host processor coupled to a programmable coprocessor is provided. One or more virtual machines may run on the host processor. The coprocessor may be coupled to an auxiliary memory that stores virtual machine (VM) states. During live migration, the coprocessor may determine when to move the VM states from the auxiliary memory to a remote server node. The coprocessor may include a coherent protocol home agent and state tracking circuitry configured to track data modification at a cache line granularity. Whenever a particular cache line has been modified, only the data associated with that cache line will be moved to the remote server without having to copy over the entire page, thereby substantially reducing the amount of data that needs to be transferred during migration events.

First claim

What is claimed is:

1. An integrated circuit, comprising:
a memory controller, electrically coupled to and configured to access an external memory storing virtual machine (VM) state information of a VM, wherein the VM state information is organized into a plurality of pages each of which includes a plurality of cache lines;
processing circuitry electrically coupled to and configured to interface with an external host processor to run the VM while the VM state information is migrated to a remote server node via live migration;
a coherency protocol circuit configured to expose the external memory as an operating system (OS) managed system memory to the external host processor coupled to the integrated circuit, to service transactions issued from the external host processor, and to monitor the state of individual cache lines in the plurality of pages at a cache line granularity; and
a state tracker circuit configured to analyze one or more external host processor ownership states associated with individual cache lines of the plurality of cache lines while the VM is running on the external host processor, and to determine when it is appropriate to migrate the individual cache lines to a remote server node based on the one or more external host processor ownership states for each of the individual cache lines to optimize total migration time.

2. The integrated circuit of claim 1 , wherein the transactions received at the coherency protocol circuit are issued in accordance with a cache coherency protocol.

3. The integrated circuit of claim 1 , further comprising a coherence memory controller coupled between the coherency protocol circuit on the integrated circuit and the external memory storing the VM state information.

4. The integrated circuit of claim 1 , further comprising a direct memory access (DMA) controller coupled between the state tracker circuit on the integrated circuit and the external host processor.

5. The integrated circuit of claim 1 , wherein the coherency protocol circuit is further configured to communicate with the external host processor via a predetermined cache coherent interconnect, and wherein the state tracker circuit is further configured to communicate with the external host processor via a non-coherent interconnect.

6. The integrated circuit of claim 1 , wherein the state tracker circuit is further configured to receive coherency state information for individual cache lines from the coherency protocol circuit.

7. The integrated circuit of claim 6 , wherein the state tracker circuit keeps track of the received coherency state information using a cache line directory.

8. The integrated circuit of claim 7 , further comprising a tracking cache configured to store at least a portion of the cache line directory.

9. The integrated circuit of claim 7 , further comprising an additional memory controller configured to access an additional external memory configured to store the cache line directory.

10. The integrated circuit of claim 7 , wherein each entry in the cache line directory includes bits for representing whether the external host processor has been granted ownership of a respective cache line in the plurality of pages.

11. The integrated circuit of claim 7 , wherein each entry in the cache line directory includes bits for representing whether the external host processor has lost ownership of a respective cache line in the plurality of pages.

12. The integrated circuit of claim 7 , wherein each entry in the cache line directory includes bits for representing whether the external host processor has modified a respective cache line in the plurality of pages.

13. A method of operating an integrated circuit, the method comprising:
with a memory controller in the integrated circuit, accessing an external memory electrically coupled to the memory controller, that stores virtual machine (VM) state information, wherein the VM state information is organized into a plurality of pages each of which includes multiple cache lines;
with processing circuitry, running the VM on the integrated circuit while the VM state information is migrated to a destination server node via live migration;
with a home agent in the integrated circuit, exposing the external memory as an operating system (OS) managed system memory to an external host processor coupled to the integrated circuit; and
with a state tracker in the integrated circuit configured to determine one or more external host processor ownership states associated with individual cache lines of the multiple cache lines, receiving coherency state information from the home agent and facilitating the live migration by managing when the individual cache lines in the plurality of pages are migrated to the destination server node based on the one or more external host processor ownership states associated with each of the individual cache lines.

14. The method of claim 13 , comprising determining the one or more external host processor ownership states using the home agent to determine when the external host processor obtains ownership of a given cache line in the plurality of pages.

15. The method of claim 14 , comprising determining the one or more external host processor ownership states by using the home agent to determine when the external host processor modifies data in the given cache line.

16. The method of claim 15 , comprising determining the one or more external host processor ownership states by using the home agent to determine when the external host processor gives up ownership of the given cache line.

17. The method of claim 16 , further comprising:
in response to determining that the external host processor has modified the given cache line and given up ownership of the given cache line, moving the given cache line to the destination server node.

18. A system, comprising:
a host central processing unit (CPU) configured to host a plurality of virtual machines, the host CPU comprising processing circuitry configured to run a virtual machine (VM) of the plurality of virtual machines on the host CPU while VM state information is migrated to a remote server via live migration;
main memory coupled to the host CPU;
a programmable coprocessor coupled to the host CPU; and
auxiliary memory coupled to the programmable coprocessor, wherein the auxiliary memory is configured to store the VM state information for the plurality of virtual machines, wherein the VM state information is organized into a plurality of pages each of which includes multiple cache lines, and wherein the programmable coprocessor comprises a cache line state tracker configured to determine one or more host CPU ownership states associated with individual cache lines of the multiple cache lines and configured to determine, while the VM is running on the host CPU, when to migrate individual cache lines from the auxiliary memory over to a remote server via live migration based on the one or more host CPU ownership states associated with each of the individual cache lines.

19. The system of claim 18 , wherein the programmable coprocessor further comprises a cache coherence protocol circuit configured to expose coherency state information to the cache line state tracker.

20. The system of claim 18 , wherein the cache line state tracker is configured to implement and maintain a cache line directory, and wherein the cache line directory includes bits indicative of whether an individual cache line has been modified by the host CPU and whether the host CPU has obtained or given up ownership of that c

Assignees

• Intel Corp

Inventors

• Chitlur Nagabhushan
• Aguirre Mariano
• Chang Stephen S
• Menezes Rohan
• Werstlein Michael T
• Lo Jonathan

Classifications

• G06F9/4856 Primary

  resumption being on a different machine, e.g. task migration, virtual machine migration (G06F9/5088 takes precedence) · CPC title

• G06F9/45558 Primary

  Hypervisor-specific management and integration aspects · CPC title

• G06F12/0882

  Page mode · CPC title

• G06F12/1081

  for peripheral access to main memory, e.g. direct memory access [DMA] · CPC title

• G06F13/1668 Primary

  Details of memory controller · CPC title