(RDMA: Remote Direct Memory Access)
RDMA Live Migration Specification, Version # 1
==============================================
-Wiki: http://wiki.qemu.org/Features/RDMALiveMigration
+Wiki: https://wiki.qemu.org/Features/RDMALiveMigration
* RDMA Migration Protocol Description
* Versioning and Capabilities
* QEMUFileRDMA Interface
-* Migration of pc.ram
+* Migration of VM's ram
* Error handling
* TODO
because the RDMA I/O architecture reduces the number of interrupts and
data copies by bypassing the host networking stack. In particular, a TCP-based
migration, under certain types of memory-bound workloads, may take a more
-unpredicatable amount of time to complete the migration if the amount of
+unpredictable amount of time to complete the migration if the amount of
memory tracked during each live migration iteration round cannot keep pace
with the rate of dirty memory produced by the workload.
high-performance RDMA hardware using the following command:
QEMU Monitor Command:
-$ migrate_set_capability x-rdma-pin-all on # disabled by default
+$ migrate_set_capability rdma-pin-all on # disabled by default
Performing this action will cause all 8GB to be pinned, so if that's
not what you want, then please ignore this step altogether.
Next, on the destination machine, add the following to the QEMU command line:
-qemu ..... -incoming x-rdma:host:port
+qemu ..... -incoming rdma:host:port
Finally, perform the actual migration on the source machine:
QEMU Monitor Command:
-$ migrate -d x-rdma:host:port
+$ migrate -d rdma:host:port
PERFORMANCE
===========
active use and the VM itself is completely idle using the same 40 gbps
infiniband link:
-1. x-rdma-pin-all disabled total time: approximately 7.5 seconds @ 9.5 Gbps
-2. x-rdma-pin-all enabled total time: approximately 4 seconds @ 26 Gbps
+1. rdma-pin-all disabled total time: approximately 7.5 seconds @ 9.5 Gbps
+2. rdma-pin-all enabled total time: approximately 4 seconds @ 26 Gbps
These numbers would of course scale up to whatever size virtual machine
you have to migrate using RDMA.
message is that SEND messages cause notifications
to be posted to the completion queue (CQ) on the
infiniband receiver side, whereas RDMA messages (used
-for pc.ram) do not (to behave like an actual DMA).
+for VM's ram) do not (to behave like an actual DMA).
Messages in infiniband require two things:
listed above and issue another "QEMU File" protocol command,
asking for a new SEND message to re-fill the buffer.
-Migration of pc.ram:
+Migration of VM's ram:
====================
At the beginning of the migration, (migration-rdma.c),
TODO:
=====
-1. 'migrate x-rdma:host:port' and '-incoming x-rdma' options will be
- renamed to 'rdma' after the experimental phase of this work has
- completed upstream.
-2. Currently, 'ulimit -l' mlock() limits as well as cgroups swap limits
- are not compatible with infinband memory pinning and will result in
+1. Currently, 'ulimit -l' mlock() limits as well as cgroups swap limits
+ are not compatible with infiniband memory pinning and will result in
an aborted migration (but with the source VM left unaffected).
-3. Use of the recent /proc/<pid>/pagemap would likely speed up
+2. Use of the recent /proc/<pid>/pagemap would likely speed up
the use of KSM and ballooning while using RDMA.
-4. Also, some form of balloon-device usage tracking would also
+3. Also, some form of balloon-device usage tracking would also
help alleviate some issues.
-5. Move UNREGISTER requests to a separate thread.
-6. Use LRU to provide more fine-grained direction of UNREGISTER
+4. Use LRU to provide more fine-grained direction of UNREGISTER
requests for unpinning memory in an overcommitted environment.
-7. Expose UNREGISTER support to the user by way of workload-specific
+5. Expose UNREGISTER support to the user by way of workload-specific
hints about application behavior.