Max Reitz [Fri, 11 Oct 2019 15:28:02 +0000 (17:28 +0200)]
qcow2: Keep unknown extra snapshot data
The qcow2 specification says to ignore unknown extra data fields in
snapshot table entries. Currently, we discard it whenever we update the
image, which is a bit different from "ignore".
This patch makes the qcow2 driver keep all unknown extra data fields
when updating an image's snapshot table.
Max Reitz [Mon, 14 Oct 2019 15:39:28 +0000 (17:39 +0200)]
mirror: Do not dereference invalid pointers
mirror_exit_common() may be called twice (if it is called from
mirror_prepare() and fails, it will be called from mirror_abort()
again).
In such a case, many of the pointers in the MirrorBlockJob object will
already be freed. This can be seen most reliably for s->target, which
is set to NULL (and then dereferenced by blk_bs()).
Introduce an API for some shared splittable resource, like memory.
It's going to be used by backup. Backup uses both read/write io and
copy_range. copy_range may consume memory implictly, so the new API is
abstract: it doesn't allocate any real memory but only hands out
tickets.
The idea is that we have some total amount of something and callers
should wait in coroutine queue if there is not enough of the resource
at the moment.
Merge copying code into one function block_copy_do_copy, which only
calls bdrv_ io functions and don't do any synchronization (like dirty
bitmap set/reset).
Refactor block_copy() function so that it takes full decision about
size of chunk to be copied and does all the synchronization (checking
intersecting requests, set/reset dirty bitmaps).
It will help:
- introduce parallel processing of block_copy iterations: we need to
calculate chunk size, start async chunk copying and go to the next
iteration
- simplify synchronization improvement (like memory limiting in
further commit and reducing critical section (now we lock the whole
requested range, when actually we need to lock only dirty region
which we handle at the moment))
Large copy range may imply memory allocation and large io effort, so
using 2G copy range request may be bad idea. Let's limit it to 16 MiB.
It also helps the following patch to refactor copy-with-offload
fallback to copy-with-bounce-buffer.
Note, that total memory usage of backup is still not limited, it will
be fixed in further commit.
block/block-copy: allocate buffer in block_copy_with_bounce_buffer
Move bounce_buffer allocation block_copy_with_bounce_buffer. This
commit simplifies further work on implementing copying by larger chunks
(of different size) and further asynchronous handling of block_copy
iterations (with help of block/aio_task API).
Allocation works fast, a lot faster than disk io, so it's not a problem
that we now allocate/free bounce_buffer more times. And we anyway will
have to allocate several bounce_buffers for parallel execution of loop
iterations in future.
Max Reitz [Thu, 17 Oct 2019 13:31:34 +0000 (15:31 +0200)]
iotests.py: Store socket files in $SOCK_DIR
iotests.py itself does not store socket files, but machine.py and
qtest.py do. iotests.py needs to pass the respective path to them, and
they need to adhere to it.
Max Reitz [Thu, 17 Oct 2019 13:31:33 +0000 (15:31 +0200)]
iotests: Introduce $SOCK_DIR
Unix sockets generally have a maximum path length. Depending on your
$TEST_DIR, it may be exceeded and then all tests that create and use
Unix sockets there may fail.
Circumvent this by adding a new scratch directory specifically for
Unix socket files. It defaults to a temporary directory (mktemp -d)
that is completely removed after the iotests are done.
(By default, mktemp -d creates a /tmp/tmp.XXXXXXXXXX directory, which
should be short enough for our use cases.)
Use mkdir -p to create the directory (because it seems right), and do
the same for $TEST_DIR (because there is no reason for that to be
created in any different way).
block/mirror: support unaligned write in active mirror
Prior 9adc1cb49af8d do_sync_target_write had a bug: it reset aligned-up
region in the dirty bitmap, which means that we may not copy some bytes
and assume them copied, which actually leads to producing corrupted
target.
So 9adc1cb49af8d forced dirty bitmap granularity to be
request_alignment for mirror-top filter, so we are not working with
unaligned requests. However forcing large alignment obviously decreases
performance of unaligned requests.
This commit provides another solution for the problem: if unaligned
padding is already dirty, we can safely ignore it, as
1. It's dirty, it will be copied by mirror_iteration anyway
2. It's dirty, so skipping it now we don't increase dirtiness of the
bitmap and therefore don't damage "synchronicity" of the
write-blocking mirror.
If unaligned padding is not dirty, we just write it, no reason to touch
dirty bitmap if we succeed (on failure we'll set the whole region
ofcourse, but we loss "synchronicity" on failure anyway).
Note: we need to disable dirty_bitmap, otherwise we will not be able to
see in do_sync_target_write bitmap state before current operation. We
may of course check dirty bitmap before the operation in
bdrv_mirror_top_do_write and remember it, but we don't need active
dirty bitmap for write-blocking mirror anyway.
New code-path is unused until the following commit reverts 9adc1cb49af8d.
do_sync_target_write is called from bdrv_mirror_top_do_write after
write/discard operation, all inside active_write/active_write_settle
protecting us from mirror iteration. So the whole area is dirty for
sure, no reason to examine dirty bitmap.
Passing zero length to these functions leads to unpredicted results.
Zero-length set/reset may occur in active-mirror, on zero-length write
(which is unlikely, but not guaranteed to never happen).
Max Reitz [Tue, 17 Sep 2019 09:19:59 +0000 (11:19 +0200)]
iotests: Allow skipping test cases
case_notrun() does not actually skip the current test case. It just
adds a "notrun" note and then returns to the caller, who manually has to
skip the test. Generally, skipping a test case is as simple as
returning from the current function, but not always: For example, this
model does not allow skipping tests already in the setUp() function.
Thus, add a QMPTestCase.case_skip() function that invokes case_notrun()
and then self.skipTest(). To make this work, we need to filter the
information on how many test cases were skipped from the unittest
output.
Peter Maydell [Sat, 26 Oct 2019 09:13:48 +0000 (10:13 +0100)]
Merge remote-tracking branch 'remotes/stsquad/tags/pull-testing-next-251019-3' into staging
Testing updates (split from mega PR)
- various Travis dependency updates
- enable tcg debug for check-tcg
- additional Xcode build for Cirrus
- dependency tweak for gitlab
# gpg: Signature made Fri 25 Oct 2019 20:35:56 BST
# gpg: using RSA key 6685AE99E75167BCAFC8DF35FBD0DB095A9E2A44
# gpg: Good signature from "Alex Bennée (Master Work Key) <[email protected]>" [full]
# Primary key fingerprint: 6685 AE99 E751 67BC AFC8 DF35 FBD0 DB09 5A9E 2A44
* remotes/stsquad/tags/pull-testing-next-251019-3:
tests/docker: update Travis image to a more current version
tests/docker: set HOST_ARCH if we don't have ARCH
travis.yml: --enable-debug-tcg to check-tcg
gitlab-ci.yml: Use libvdeplug-dev to compile-test the VDE network backend
travis.yml: cache the clang sanitizer build
tests/vm/netbsd: Disable IPv6
tests/vm: Let subclasses disable IPv6
cirrus.yml: add latest Xcode build target
travis.yml: bump Xcode 10 to latest dot release
travis.yml: Test the release tarball
travis.yml: Fix the ccache lines
travis.yml: Use newer version of libgnutls and libpng
travis.yml: Use libsdl2 instead of libsdl1.2, and install libsdl2-image
travis.yml: Add libvdeplug-dev to compile-test net/vde.c
travis.yml: reduce scope of the --enable-debug build
Peter Maydell [Fri, 25 Oct 2019 20:57:41 +0000 (21:57 +0100)]
Merge remote-tracking branch 'remotes/stefanha/tags/block-pull-request' into staging
Pull request
# gpg: Signature made Fri 25 Oct 2019 20:18:23 BST
# gpg: using RSA key 8695A8BFD3F97CDAAC35775A9CA4ABB381AB73C8
# gpg: Good signature from "Stefan Hajnoczi <[email protected]>" [full]
# gpg: aka "Stefan Hajnoczi <[email protected]>" [full]
# Primary key fingerprint: 8695 A8BF D3F9 7CDA AC35 775A 9CA4 ABB3 81AB 73C8
* remotes/stefanha/tags/block-pull-request:
yield_until_fd_readable: make it work with any AioContect
virtio-blk: Add blk_drain() to virtio_blk_device_unrealize()
Thomas Huth [Wed, 9 Oct 2019 17:07:00 +0000 (19:07 +0200)]
travis.yml: Fix the ccache lines
The "command -v ccache && ccache ..." likely were supposed to test
the availability of ccache before running the program. But this
shell construct causes Travis to abort if ccache is not available.
Use an if-statement instead to fix this problem.
Thomas Huth [Wed, 9 Oct 2019 17:06:59 +0000 (19:06 +0200)]
travis.yml: Use newer version of libgnutls and libpng
libgnutls-dev and libpng12-dev are not available in newer versions
of Ubuntu anymore, so installing these packages fails e.g. in the
new arm64 containers on Travis. Let's use newer versions of these
packages by default instead. (The old versions still get tested in
the "gcc-9" build).
Thomas Huth [Wed, 9 Oct 2019 17:06:58 +0000 (19:06 +0200)]
travis.yml: Use libsdl2 instead of libsdl1.2, and install libsdl2-image
We've removed support for SDL 1.2 quite a while ago already, so let's
use SDL 2 now in Travis to get test coverage for SDL again.
And while we're at it, also add libsdl2-image-dev which can be used
by QEMU nowadays, too.
Peter Maydell [Fri, 25 Oct 2019 17:32:26 +0000 (18:32 +0100)]
Merge remote-tracking branch 'remotes/amarkovic/tags/mips-queue-oct-24-2019-v2' into staging
MIPS queue for October 24th, 2019 - v2
# gpg: Signature made Fri 25 Oct 2019 17:37:29 BST
# gpg: using RSA key D4972A8967F75A65
# gpg: Good signature from "Aleksandar Markovic <[email protected]>" [unknown]
# gpg: WARNING: This key is not certified with a trusted signature!
# gpg: There is no indication that the signature belongs to the owner.
# Primary key fingerprint: 8526 FBF1 5DA3 811F 4A01 DD75 D497 2A89 67F7 5A65
* remotes/amarkovic/tags/mips-queue-oct-24-2019-v2:
tests/ssh_linux_malta: Fix 64-bit target tests
tests/ssh_linux_malta: Refactor how to get image/kernel info
tests/ssh_linux_malta: Match stricter console output
tests/ssh_linux_malta: Remove duplicated test
tests/ssh_linux_malta: Run tests using a snapshot image
target/mips: Refactor handling of vector compare 'less than' (signed) instructions
target/mips: Refactor handling of vector compare 'equal' instructions
target/mips: Demacro LMI decoder
target/mips: msa: Split helpers for ASUB_<S|U>.<B|H|W|D>
target/mips: msa: Split helpers for HSUB_<S|U>.<H|W|D>
target/mips: msa: Split helpers for PCK<EV|OD>.<B|H|W|D>
target/mips: msa: Split helpers for S<LL|RA|RAR|RL|RLR>.<B|H|W|D>
target/mips: msa: Split helpers for HADD_<S|U>.<H|W|D>
target/mips: msa: Split helpers for ADD<_A|S_A|S_S|S_U|V>.<B|H|W|D>
target/mips: msa: Split helpers for ILV<EV|OD|L|R>.<B|H|W|D>
target/mips: msa: Split helpers for <MAX|MIN>_<S|U>.<B|H|W|D>
target/mips: msa: Split helpers for <MAX|MIN>_A.<B|H|W|D>
MAINTAINERS: Update mail address of Aleksandar Rikalo
target/mips: Clean up op_helper.c
target/mips: Clean up helper.c
Commit 9090d3332cdcc added tests for specific to the 32-bit
machines, which inadvertently make the 64-bit tests failing.
Now than we have this information available in the CPU_INFO
array, use it to have the 64-bit tests back.
Peter Maydell [Fri, 25 Oct 2019 13:59:53 +0000 (14:59 +0100)]
Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging
Block layer patches:
- qcow2: Fix data corruption bug that is triggered in partial cluster
allocation with default options
- qapi: add support for blkreplay driver
- doc: Describe missing generic -blockdev options
- iotests: Fix 118 when run as root
- Minor code cleanups
# gpg: Signature made Fri 25 Oct 2019 14:19:04 BST
# gpg: using RSA key 7F09B272C88F2FD6
# gpg: Good signature from "Kevin Wolf <[email protected]>" [full]
# Primary key fingerprint: DC3D EB15 9A9A F95D 3D74 56FE 7F09 B272 C88F 2FD6
* remotes/kevin/tags/for-upstream:
qcow2: Fix corruption bug in qcow2_detect_metadata_preallocation()
coroutine: Add qemu_co_mutex_assert_locked()
doc: Describe missing generic -blockdev options
block/backup: drop dead code from backup_job_create
blockdev: Use error_report() in hmp_commit()
iotests: Skip read-only cases in 118 when run as root
qapi: add support for blkreplay driver
Kevin Wolf [Thu, 24 Oct 2019 14:26:58 +0000 (16:26 +0200)]
qcow2: Fix corruption bug in qcow2_detect_metadata_preallocation()
qcow2_detect_metadata_preallocation() calls qcow2_get_refcount() which
requires s->lock to be taken to protect its accesses to the refcount
table and refcount blocks. However, nothing in this code path actually
took the lock. This could cause the same cache entry to be used by two
requests at the same time, for different tables at different offsets,
resulting in image corruption.
As it would be preferable to base the detection on consistent data (even
though it's just heuristics), let's take the lock not only around the
qcow2_get_refcount() calls, but around the whole function.
This patch takes the lock in qcow2_co_block_status() earlier and asserts
in qcow2_detect_metadata_preallocation() that we hold the lock.
Kevin Wolf [Thu, 24 Oct 2019 14:26:57 +0000 (16:26 +0200)]
coroutine: Add qemu_co_mutex_assert_locked()
Some functions require that the caller holds a certain CoMutex for them
to operate correctly. Add a function so that they can assert the lock is
really held.
* remotes/vivier2/tags/trivial-branch-pull-request:
hw/rtc/aspeed_rtc: Remove unused includes
hw/rtc/xlnx-zynqmp-rtc: Remove unused "ptimer.h" include
hw/rtc/mc146818: Include mc146818rtc_regs.h a bit less
hw: Move Aspeed RTC from hw/timer/ to hw/rtc/ subdirectory
hw: Move Exynos4210 RTC from hw/timer/ to hw/rtc/ subdirectory
hw: Move Xilinx ZynqMP RTC from hw/timer/ to hw/rtc/ subdirectory
hw: Move DS1338 device from hw/timer/ to hw/rtc/ subdirectory
hw: Move TWL92230 device from hw/timer/ to hw/rtc/ subdirectory
hw: Move sun4v hypervisor RTC from hw/timer/ to hw/rtc/ subdirectory
hw: Move M41T80 device from hw/timer/ to hw/rtc/ subdirectory
hw: Move M48T59 device from hw/timer/ to hw/rtc/ subdirectory
hw: Move MC146818 device from hw/timer/ to hw/rtc/ subdirectory
hw: Move PL031 device from hw/timer/ to hw/rtc/ subdirectory
hw/timer: Compile devices not target-dependent as common object
qemu-timer: reuse MIN macro in qemu_timeout_ns_to_ms
event_notifier: avoid dandling file descriptor in event_notifier_cleanup
util/async: avoid useless cast
pci_bridge: fix a typo in comment
qemu-options.hx: Update for reboot-timeout parameter
Signed-off-by: Peter Maydell <[email protected]>
# Conflicts:
# hw/timer/trace-events
Kevin Wolf [Tue, 15 Oct 2019 10:29:58 +0000 (12:29 +0200)]
doc: Describe missing generic -blockdev options
We added more generic options after introducing -blockdev and forgot to
update the documentation (man page and --help output) accordingly. Do
that now.
block/backup: drop dead code from backup_job_create
After commit 00e30f05de1d195, there is no more "goto error" points
after job creation, so after "error:" @job is always NULL and we don't
need roll-back job creation.
Kevin Wolf [Fri, 18 Oct 2019 11:46:42 +0000 (13:46 +0200)]
iotests: Skip read-only cases in 118 when run as root
Some tests in 118 use chmod to remove write permissions from the file
and assume that the image can indeed not be opened read-write
afterwards. This doesn't work when the test is run as root, because root
can still open the file as writable even when the permission bit isn't
set.
Introduce a @skip_if_root decorator and use it in 118 to skip the tests
in question when the script is run as root.
Pavel Dovgalyuk [Wed, 16 Oct 2019 08:40:39 +0000 (11:40 +0300)]
qapi: add support for blkreplay driver
This patch adds support for blkreplay driver to the blockdev options.
Now blkreplay can be used with -blockdev command line option
in the following format:
-blockdev driver=blkreplay,image=file-node-name,node-name=replay-node-name
This option makes possible implementation of the better command
line support for record/replay invocations.
Julia Suvorova [Fri, 18 Oct 2019 14:28:56 +0000 (16:28 +0200)]
virtio-blk: Add blk_drain() to virtio_blk_device_unrealize()
QEMU does not wait for completed I/O requests, assuming that the guest
driver will reset the device before calling unrealize(). This does not
happen on Windows, and QEMU crashes in virtio_notify(), getting the
result of a completed I/O request on hot-unplugged device.
Peter Maydell [Fri, 25 Oct 2019 12:12:16 +0000 (13:12 +0100)]
Merge remote-tracking branch 'remotes/pmaydell/tags/pull-target-arm-20191025' into staging
target-arm queue:
* raspi boards: some cleanup
* raspi: implement the bcm2835 system timer device
* raspi: implement a dummy thermal sensor
* misc devices: switch to ptimer transaction API
* cache TB flag state to improve performance of cpu_get_tb_cpu_state
* aspeed: Add an AST2600 eval board
* remotes/pmaydell/tags/pull-target-arm-20191025: (42 commits)
hw/arm/highbank: Use AddressSpace when using write_secondary_boot()
hw/arm/raspi: Use AddressSpace when using arm_boot::write_secondary_boot
hw/arm/bcm2836: Rename cpus[] as cpu[].core
hw/arm/bcm2836: Make the SoC code modular
hw/arm/bcm2835_peripherals: Use the SYS_timer
hw/timer/bcm2835: Add the BCM2835 SYS_timer
hw/arm/bcm2835_peripherals: Use the thermal sensor block
hw/misc/bcm2835_thermal: Add a dummy BCM2835 thermal sensor
hw/watchdog/milkymist-sysctl.c: Switch to transaction-based ptimer API
hw/m68k/mcf5206.c: Switch to transaction-based ptimer API
hw/timer/grlib_gptimer.c: Switch to transaction-based ptimer API
hw/timer/slavio_timer.c: Switch to transaction-based ptimer API
hw/timer/slavio_timer: Remove useless check for NULL t->timer
hw/dma/xilinx_axidma.c: Switch to transaction-based ptimer API
hw/timer/xilinx_timer.c: Switch to transaction-based ptimer API
hw/net/fsl_etsec/etsec.c: Switch to transaction-based ptimer API
target/arm: Rely on hflags correct in cpu_get_tb_cpu_state
linux-user/arm: Rebuild hflags for TARGET_WORDS_BIGENDIAN
linux-user/aarch64: Rebuild hflags for TARGET_WORDS_BIGENDIAN
target/arm: Rebuild hflags for M-profile NVIC
...
projects / qemu.git / log