X-Git-Url: https://repo.jachan.dev/qemu.git/blobdiff_plain/859389810910f232188675d2f10b15f1aad77660..e48a748fc899bbd9bfa63f08ebb83f1c31b7bb7d:/docs/rdma.txt diff --git a/docs/rdma.txt b/docs/rdma.txt index 2aca63bd72..a86e992c84 100644 --- a/docs/rdma.txt +++ b/docs/rdma.txt @@ -1,7 +1,7 @@ (RDMA: Remote Direct Memory Access) RDMA Live Migration Specification, Version # 1 ============================================== -Wiki: http://wiki.qemu-project.org/Features/RDMALiveMigration +Wiki: https://wiki.qemu.org/Features/RDMALiveMigration Github: git@github.com:hinesmr/qemu.git, 'rdma' branch Copyright (C) 2013 Michael R. Hines @@ -18,7 +18,7 @@ Contents: * RDMA Migration Protocol Description * Versioning and Capabilities * QEMUFileRDMA Interface -* Migration of pc.ram +* Migration of VM's ram * Error handling * TODO @@ -30,7 +30,7 @@ of the significantly lower latency and higher throughput over TCP/IP. This is because the RDMA I/O architecture reduces the number of interrupts and data copies by bypassing the host networking stack. In particular, a TCP-based migration, under certain types of memory-bound workloads, may take a more -unpredicatable amount of time to complete the migration if the amount of +unpredictable amount of time to complete the migration if the amount of memory tracked during each live migration iteration round cannot keep pace with the rate of dirty memory produced by the workload. @@ -66,7 +66,7 @@ bulk-phase round of the migration and can be enabled for extremely high-performance RDMA hardware using the following command: QEMU Monitor Command: -$ migrate_set_capability x-rdma-pin-all on # disabled by default +$ migrate_set_capability rdma-pin-all on # disabled by default Performing this action will cause all 8GB to be pinned, so if that's not what you want, then please ignore this step altogether. @@ -93,12 +93,12 @@ $ migrate_set_speed 40g # or whatever is the MAX of your RDMA device Next, on the destination machine, add the following to the QEMU command line: -qemu ..... -incoming x-rdma:host:port +qemu ..... -incoming rdma:host:port Finally, perform the actual migration on the source machine: QEMU Monitor Command: -$ migrate -d x-rdma:host:port +$ migrate -d rdma:host:port PERFORMANCE =========== @@ -120,8 +120,8 @@ For example, in the same 8GB RAM example with all 8GB of memory in active use and the VM itself is completely idle using the same 40 gbps infiniband link: -1. x-rdma-pin-all disabled total time: approximately 7.5 seconds @ 9.5 Gbps -2. x-rdma-pin-all enabled total time: approximately 4 seconds @ 26 Gbps +1. rdma-pin-all disabled total time: approximately 7.5 seconds @ 9.5 Gbps +2. rdma-pin-all enabled total time: approximately 4 seconds @ 26 Gbps These numbers would of course scale up to whatever size virtual machine you have to migrate using RDMA. @@ -149,7 +149,7 @@ The only difference between a SEND message and an RDMA message is that SEND messages cause notifications to be posted to the completion queue (CQ) on the infiniband receiver side, whereas RDMA messages (used -for pc.ram) do not (to behave like an actual DMA). +for VM's ram) do not (to behave like an actual DMA). Messages in infiniband require two things: @@ -355,7 +355,7 @@ If the buffer is empty, then we follow the same steps listed above and issue another "QEMU File" protocol command, asking for a new SEND message to re-fill the buffer. -Migration of pc.ram: +Migration of VM's ram: ==================== At the beginning of the migration, (migration-rdma.c), @@ -407,18 +407,14 @@ socket is broken during a non-RDMA based migration. TODO: ===== -1. 'migrate x-rdma:host:port' and '-incoming x-rdma' options will be - renamed to 'rdma' after the experimental phase of this work has - completed upstream. -2. Currently, 'ulimit -l' mlock() limits as well as cgroups swap limits - are not compatible with infinband memory pinning and will result in +1. Currently, 'ulimit -l' mlock() limits as well as cgroups swap limits + are not compatible with infiniband memory pinning and will result in an aborted migration (but with the source VM left unaffected). -3. Use of the recent /proc//pagemap would likely speed up +2. Use of the recent /proc//pagemap would likely speed up the use of KSM and ballooning while using RDMA. -4. Also, some form of balloon-device usage tracking would also +3. Also, some form of balloon-device usage tracking would also help alleviate some issues. -5. Move UNREGISTER requests to a separate thread. -6. Use LRU to provide more fine-grained direction of UNREGISTER +4. Use LRU to provide more fine-grained direction of UNREGISTER requests for unpinning memory in an overcommitted environment. -7. Expose UNREGISTER support to the user by way of workload-specific +5. Expose UNREGISTER support to the user by way of workload-specific hints about application behavior.