Memory copy during virtual machine migration in a virtualized computing system

Patent metadata

Field	Value
Publication number	US-11995459-B2
Application number	US-202017002233-A
Country	US
Kind code	B2
Filing date	Aug 25, 2020
Priority date	Aug 25, 2020
Publication date	May 28, 2024
Grant date	May 28, 2024

Abstract

A virtual machine (VM) is migrated from a source host to a destination host in a virtualized computing system, the VM having a plurality of virtual central processing units (CPUs). The method includes copying, by VM migration software executing in the source host and the destination host, memory of the VM from the source host to the destination host by installing, at the source host, write traces spanning all of the memory and then copying the memory from the source host to the destination host over a plurality of iterations; and performing switch-over, by the VM migration software, to quiesce the VM in the source host and resume the VM in the destination host. The VM migration software installs write traces using less than all of the virtual CPUs, and using trace granularity larger than a smallest page granularity.

First claim

We claim:

1. A method of migrating a virtual machine (VM) from a source host to a destination host in a virtualized computing system, the VM having a plurality of virtual central processing units (CPUs) supported by a CPU of the source host, the method comprising:
copying, by a hypervisor executing in the source host, memory of the VM from the source host to the destination host over a plurality of iterations;
installing, by the hypervisor, write traces spanning all the memory, the write traces configured to cause the CPU to notify the hypervisor, through page fault exceptions, that guest software in the VM modified the memory; and
performing switch-over, by the hypervisor, to quiesce the VM in the source host;
wherein the hypervisor installs the write traces using less than all the plurality of virtual CPUs in parallel with at least one of the plurality of virtual CPUs executing the guest software in the VM.

2. The method of claim 1 , wherein the hypervisors installs the write traces using only one of the plurality of virtual CPUs.

3. The method of claim 1 , wherein each of the plurality of virtual CPUs is configured to support a plurality of page granularities, and wherein the hypervisor installs the write traces on large pages of the memory having granularity larger than a smallest granularity of the plurality of page granularities.

4. The method of claim 1 , wherein the step of installing the write traces comprises requesting each of the plurality of virtual CPUs to lazily flush an associated translation lookaside buffer (TLB).

5. The method of claim 1 , further comprising re-installing, by the hypervisor during each of the plurality of iterations, at least one write trace.

6. The method of claim 5 , wherein the hypervisor re-installs the at least one write trace using only one of the plurality of virtual CPUs.

7. The method of claim 1 , further comprising:
tracking, by the hypervisor, dirty pages in a tracking bitmap;
walking the tracking bitmap at the source host, by the hypervisor during the switch-over, to identify page numbers of modified pages; and
transmitting, by the hypervisor from the source host to the destination host, the page numbers.

8. The method of claim 1 , further comprising:
handling, at a hypervisor in the source host managing the VM, a trace fire in response to the guest software in the VM writing to a memory page of the memory;
wherein each of the plurality of virtual CPUs is configured to support a plurality of page granularities, wherein the hypervisor installs the write traces at a first granularity of the plurality of granularities, and wherein the hypervisor handles the trace fire at a second granularity of the plurality of granularities different than the first granularity.

9. The method of claim 1 , wherein each of the plurality of virtual CPUs is configured to support a plurality of page granularities, and wherein the method comprises:
selecting a first granularity of the plurality of page granularities dynamically during run-time;
wherein the write traces are installed at the first granularity.

10. A non-transitory computer readable medium having instructions stored thereon that when executed by a processor cause the processor to perform a method of migrating a virtual machine (VM) from a source host to a destination host in a virtualized computing system, the VM having a plurality of virtual central processing units (CPUs) supported by a CPU of the source host, the method comprising:
copying memory of the VM from the source host to the destination host over a plurality of iterations;
installing write traces spanning all the memory, the write traces configured to cause the CPU to notify a hypervisor of the source host, through page fault exceptions, that guest software in the VM modified the memory; and
performing switch-over to quiesce the VM in the source host;
wherein the step of installing the write traces is executed using less than all the plurality of virtual CPUs in parallel with at least one of the plurality of virtual CPUs executing the guest software in the VM.

11. The non-transitory computer readable medium of claim 10 , wherein the step of installing the write traces is executed using only one of the plurality of virtual CPUs.

12. The non-transitory computer readable medium of claim 10 , wherein each of the plurality of virtual CPUs is configured to support a plurality of page granularities, and wherein the write traces are installed on large pages of the memory having granularity larger than a smallest granularity of the plurality of page granularities.

13. The non-transitory computer readable medium of claim 10 , further comprising:
handling a trace fire in response to the guest software in the VM writing to a memory page of the memory;
wherein each of the plurality of virtual CPUs is configured to support a plurality of page granularities, wherein the write traces are installed at a first granularity of the plurality of granularities, and wherein the trace fire is handled at a second granularity of the plurality of granularities different than the first granularity.

14. A virtualized computing system, comprising:
a source host executing a first virtual machine (VM) migration module, the source host executing a hypervisor and a VM, the source host including a central processing unit (CPU) that supports a plurality of virtual CPUs for the VM;
a destination host executing a second VM migration module;
wherein the first VM migration module is configured to cooperate with the second VM migration module to:
copy memory of a VM executing in the source host from the source host to the destination host over a plurality of iterations;
install, at the source host, write traces spanning all the memory using less than all of the plurality of virtual CPUs in parallel with at least one of the plurality of virtual CPUs executing guest software in the VM, the write traces configured to cause the CPU to notify the hypervisor, through page fault exceptions, that the guest software in the VM modified the memory; and
perform switch-over to quiesce the VM in the source host and resume the VM in the destination host.

15. The virtualized computing system of claim 14 , wherein the first VM migration module is configured to install the write traces using only one of the plurality of virtual CPUs.

16. The virtualized computing system of claim 14 , wherein each of the plurality of virtual CPUs is configured to support a plurality of page granularities, and wherein the first VM migration module is configured to install the write traces on large pages of the memory having granularity larger than a smallest granularity of the plurality of page granularities.

17. The virtualized computing system of claim 14 , wherein the first VM migration module is configured to request each of the plurality of virtual CPUs to lazily flush an associated translation lookaside buffer (TLB) subsequent to installing the write traces.

18. The virtualized computing system of claim 14 , wherein the hypervisor in the source host is configured to handle a trace fire in response to the guest software in the VM writing to a memory page of the memory;
wherein each of the plurality of virtual CPUs is configured to support a plurality of page granularities, wherein the VM migration module is configured to install the write traces at a first granularity of the plurality of granularities, and wherein the hypervisor is configured to handle the trace fire at a second granularity of the plurality of granularities different than the first granulari

Assignees

• Vmware Inc
• VMware LLC

Inventors

• Ramanathan Arunachalam
• Zhao Yanlei
• Saxena Anurekh
• Baskakov Yury
• Sheldon Jeffrey W
• Tarasuk-Levin Gabriel
• Dunn David A
• Setty Sreekanth

Classifications

• G06F9/45558 Primary

  Hypervisor-specific management and integration aspects · CPC title

• G06F9/5077

  Logical partitioning of resources; Management or configuration of virtualized resources (specific details on emulation or internal functioning of virtual machines G06F9/455) · CPC title

• G06F12/1027

  using associative or pseudo-associative address translation means, e.g. translation look-aside buffer [TLB] · CPC title

• G06F2009/4557

  Distribution of virtual machine instances; Migration and load balancing · CPC title

• G06F2009/45583

  Memory management, e.g. access or allocation · CPC title