Kevin Wolf [Fri, 23 Mar 2018 14:57:20 +0000 (15:57 +0100)]
block: Don't poll in parent drain callbacks
bdrv_do_drained_begin() is only safe if we have a single
BDRV_POLL_WHILE() after quiescing all affected nodes. We cannot allow
that parent callbacks introduce a nested polling loop that could cause
graph changes while we're traversing the graph.
Split off bdrv_do_drained_begin_quiesce(), which only quiesces a single
node without waiting for its requests to complete. These requests will
be waited for in the BDRV_POLL_WHILE() call down the call chain.
Kevin Wolf [Fri, 23 Mar 2018 11:40:21 +0000 (12:40 +0100)]
test-bdrv-drain: Test node deletion in subtree recursion
If bdrv_do_drained_begin() polls during its subtree recursion, the graph
can change and mess up the bs->children iteration. Test that this
doesn't happen.
Kevin Wolf [Fri, 23 Mar 2018 11:40:41 +0000 (12:40 +0100)]
block: Drain recursively with a single BDRV_POLL_WHILE()
Anything can happen inside BDRV_POLL_WHILE(), including graph
changes that may interfere with its callers (e.g. child list iteration
in recursive callers of bdrv_do_drained_begin).
Switch to a single BDRV_POLL_WHILE() call for the whole subtree at the
end of bdrv_do_drained_begin() to avoid such effects. The recursion
happens now inside the loop condition. As the graph can only change
between bdrv_drain_poll() calls, but not inside of it, doing the
recursion here is safe.
Max Reitz [Wed, 28 Feb 2018 18:04:54 +0000 (19:04 +0100)]
test-bdrv-drain: Add test for node deletion
This patch adds two bdrv-drain tests for what happens if some BDS goes
away during the drainage.
The basic idea is that you have a parent BDS with some child nodes.
Then, you drain one of the children. Because of that, the party who
actually owns the parent decides to (A) delete it, or (B) detach all its
children from it -- both while the child is still being drained.
A real-world case where this can happen is the mirror block job, which
may exit if you drain one of its children.
Kevin Wolf [Thu, 22 Mar 2018 13:35:58 +0000 (14:35 +0100)]
block: Remove bdrv_drain_recurse()
For bdrv_drain(), recursively waiting for child node requests is
pointless because we didn't quiesce their parents, so new requests could
come in anyway. Letting the function work only on a single node makes it
more consistent.
For subtree drains and drain_all, we already have the recursion in
bdrv_do_drained_begin(), so the extra recursion doesn't add anything
either.
Kevin Wolf [Thu, 22 Mar 2018 13:11:20 +0000 (14:11 +0100)]
block: Really pause block jobs on drain
We already requested that block jobs be paused in .bdrv_drained_begin,
but no guarantee was made that the job was actually inactive at the
point where bdrv_drained_begin() returned.
This introduces a new callback BdrvChildRole.bdrv_drained_poll() and
uses it to make bdrv_drain_poll() consider block jobs using the node to
be drained.
For the test case to work as expected, we have to switch from
block_job_sleep_ns() to qemu_co_sleep_ns() so that the test job is even
considered active and must be waited for when draining the node.
Kevin Wolf [Thu, 22 Mar 2018 09:57:14 +0000 (10:57 +0100)]
block: Avoid unnecessary aio_poll() in AIO_WAIT_WHILE()
Commit 91af091f923 added an additional aio_poll() to BDRV_POLL_WHILE()
in order to make sure that all pending BHs are executed on drain. This
was the wrong place to make the fix, as it is useless overhead for all
other users of the macro and unnecessarily complicates the mechanism.
This patch effectively reverts said commit (the context has changed a
bit and the code has moved to AIO_WAIT_WHILE()) and instead polls in the
loop condition for drain.
The effect is probably hard to measure in any real-world use case
because actual I/O will dominate, but if I run only the initialisation
part of 'qemu-img convert' where it calls bdrv_block_status() for the
whole image to find out how much data there is copy, this phase actually
needs only roughly half the time after this patch.
Kevin Wolf [Wed, 4 Apr 2018 11:26:16 +0000 (13:26 +0200)]
tests/test-bdrv-drain: bdrv_drain_all() works in coroutines now
Since we use bdrv_do_drained_begin/end() for bdrv_drain_all_begin/end(),
coroutine context is automatically left with a BH, preventing the
deadlocks that made bdrv_drain_all*() unsafe in coroutine context. Now
that we even removed the old polling code as dead code, it's obvious
that it's compatible now.
Enable the coroutine test cases for bdrv_drain_all().
Kevin Wolf [Thu, 14 Dec 2017 10:25:16 +0000 (11:25 +0100)]
block: Don't manually poll in bdrv_drain_all()
All involved nodes are already idle, we called bdrv_do_drain_begin() on
them.
The comment in the code suggested that this was not correct because the
completion of a request on one node could spawn a new request on a
different node (which might have been drained before, so we wouldn't
drain the new request). In reality, new requests to different nodes
aren't spawned out of nothing, but only in the context of a parent
request, and they aren't submitted to random nodes, but only to child
nodes. As long as we still poll for the completion of the parent request
(which we do), draining each root node separately is good enough.
Remove the additional polling code from bdrv_drain_all_begin() and
replace it with an assertion that all nodes are already idle after we
drained them separately.
Kevin Wolf [Thu, 14 Dec 2017 09:27:23 +0000 (10:27 +0100)]
block: Use bdrv_do_drain_begin/end in bdrv_drain_all()
bdrv_do_drain_begin/end() implement already everything that
bdrv_drain_all_begin/end() need and currently still do manually: Disable
external events, call parent drain callbacks, call block driver
callbacks.
It also does two more things:
The first is incrementing bs->quiesce_counter. bdrv_drain_all() already
stood out in the test case by behaving different from the other drain
variants. Adding this is not only safe, but in fact a bug fix.
The second is calling bdrv_drain_recurse(). We already do that later in
the same function in a loop, so basically doing an early first iteration
doesn't hurt.
Kevin Wolf [Sat, 9 Dec 2017 23:11:13 +0000 (00:11 +0100)]
test-bdrv-drain: bdrv_drain() works with cross-AioContext events
As long as nobody keeps the other I/O thread from working, there is no
reason why bdrv_drain() wouldn't work with cross-AioContext events. The
key is that the root request we're waiting for is in the AioContext
we're polling (which it always is for bdrv_drain()) so that aio_poll()
is woken up in the end.
Add a test case that shows that it works. Remove the comment in
bdrv_drain() that claims otherwise.
Peter Maydell [Fri, 15 Jun 2018 15:30:27 +0000 (16:30 +0100)]
Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging
Block layer patches:
- Fix options that work only with -drive or -blockdev, but not with
both, because of QDict type confusion
- rbd: Add options 'auth-client-required' and 'key-secret'
- Remove deprecated -drive options serial/addr/cyls/heads/secs/trans
- rbd, iscsi: Remove deprecated 'filename' option
- Fix 'qemu-img map' crash with unaligned image size
- Improve QMP documentation for jobs
# gpg: Signature made Fri 15 Jun 2018 15:20:03 BST
# gpg: using RSA key 7F09B272C88F2FD6
# gpg: Good signature from "Kevin Wolf <[email protected]>"
# Primary key fingerprint: DC3D EB15 9A9A F95D 3D74 56FE 7F09 B272 C88F 2FD6
* remotes/kevin/tags/for-upstream: (26 commits)
block: Remove dead deprecation warning code
block: Remove deprecated -drive option serial
block: Remove deprecated -drive option addr
block: Remove deprecated -drive geometry options
rbd: New parameter key-secret
rbd: New parameter auth-client-required
block: Fix -blockdev / blockdev-add for empty objects and arrays
check-block-qdict: Cover flattening of empty lists and dictionaries
check-block-qdict: Rename qdict_flatten()'s variables for clarity
block-qdict: Simplify qdict_is_list() some
block-qdict: Clean up qdict_crumple() a bit
block-qdict: Tweak qdict_flatten_qdict(), qdict_flatten_qlist()
block-qdict: Simplify qdict_flatten_qdict()
block: Make remaining uses of qobject input visitor more robust
block: Factor out qobject_input_visitor_new_flat_confused()
block: Clean up a misuse of qobject_to() in .bdrv_co_create_opts()
block: Fix -drive for certain non-string scalars
block: Fix -blockdev for certain non-string scalars
qobject: Move block-specific qdict code to block-qdict.c
block: Add block-specific QDict header
...
Peter Maydell [Fri, 15 Jun 2018 14:27:48 +0000 (15:27 +0100)]
Merge remote-tracking branch 'remotes/pmaydell/tags/pull-target-arm-20180615' into staging
target-arm and miscellaneous queue:
* fix KVM state save/restore for GICv3 priority registers for high IRQ numbers
* hw/arm/mps2-tz: Put ethernet controller behind PPC
* hw/sh/sh7750: Convert away from old_mmio
* hw/m68k/mcf5206: Convert away from old_mmio
* hw/block/pflash_cfi02: Convert away from old_mmio
* hw/watchdog/wdt_i6300esb: Convert away from old_mmio
* hw/input/pckbd: Convert away from old_mmio
* hw/char/parallel: Convert away from old_mmio
* armv7m: refactor to get rid of armv7m_init() function
* arm: Don't crash if user tries to use a Cortex-M CPU without an NVIC
* hw/core/or-irq: Support more than 16 inputs to an OR gate
* cpu-defs.h: Document CPUIOTLBEntry 'addr' field
* cputlb: Pass cpu_transaction_failed() the correct physaddr
* CODING_STYLE: Define our preferred form for multiline comments
* Add and use new stn_*_p() and ldn_*_p() memory access functions
* target/arm: More parts of the upcoming SVE support
* aspeed_scu: Implement RNG register
* m25p80: add support for two bytes WRSR for Macronix chips
* exec.c: Handle IOMMUs being in the path of TCG CPU memory accesses
* target/arm: Allow ARMv6-M Thumb2 instructions
* remotes/pmaydell/tags/pull-target-arm-20180615: (43 commits)
target/arm: Allow ARMv6-M Thumb2 instructions
exec.c: Handle IOMMUs in address_space_translate_for_iotlb()
iommu: Add IOMMU index argument to translate method
iommu: Add IOMMU index argument to notifier APIs
iommu: Add IOMMU index concept to IOMMU API
m25p80: add support for two bytes WRSR for Macronix chips
aspeed_scu: Implement RNG register
target/arm: Implement SVE Floating Point Arithmetic - Unpredicated Group
target/arm: Implement SVE Integer Wide Immediate - Unpredicated Group
target/arm: Implement FDUP/DUP
target/arm: Implement SVE Integer Compare - Scalars Group
target/arm: Implement SVE Predicate Count Group
target/arm: Implement SVE Partition Break Group
target/arm: Implement SVE Integer Compare - Immediate Group
target/arm: Implement SVE Integer Compare - Vectors Group
target/arm: Implement SVE Select Vectors Group
target/arm: Implement SVE vector splice (predicated)
target/arm: Implement SVE reverse within elements
target/arm: Implement SVE copy to vector (predicated)
target/arm: Implement SVE conditionally broadcast/extract element
...
Julia Suvorova [Fri, 15 Jun 2018 13:57:16 +0000 (14:57 +0100)]
target/arm: Allow ARMv6-M Thumb2 instructions
ARMv6-M supports 6 Thumb2 instructions. This patch checks for these
instructions and allows their execution.
Like Thumb2 cores, ARMv6-M always interprets BL instruction as 32-bit.
This patch is required for future Cortex-M0 support.
Peter Maydell [Fri, 15 Jun 2018 13:57:16 +0000 (14:57 +0100)]
exec.c: Handle IOMMUs in address_space_translate_for_iotlb()
Currently we don't support board configurations that put an IOMMU
in the path of the CPU's memory transactions, and instead just
assert() if the memory region fonud in address_space_translate_for_iotlb()
is an IOMMUMemoryRegion.
Remove this limitation by having the function handle IOMMUs.
This is mostly straightforward, but we must make sure we have
a notifier registered for every IOMMU that a transaction has
passed through, so that we can flush the TLB appropriately
when any of the IOMMUs change their mappings.
Peter Maydell [Fri, 15 Jun 2018 13:57:16 +0000 (14:57 +0100)]
iommu: Add IOMMU index argument to translate method
Add an IOMMU index argument to the translate method of
IOMMUs. Since all of our current IOMMU implementations
support only a single IOMMU index, this has no effect
on the behaviour.
Peter Maydell [Fri, 15 Jun 2018 13:57:16 +0000 (14:57 +0100)]
iommu: Add IOMMU index argument to notifier APIs
Add support for multiple IOMMU indexes to the IOMMU notifier APIs.
When initializing a notifier with iommu_notifier_init(), the caller
must pass the IOMMU index that it is interested in. When a change
happens, the IOMMU implementation must pass
memory_region_notify_iommu() the IOMMU index that has changed and
that notifiers must be called for.
IOMMUs which support only a single index don't need to change.
Callers which only really support working with IOMMUs with a single
index can use the result of passing MEMTXATTRS_UNSPECIFIED to
memory_region_iommu_attrs_to_index().
Peter Maydell [Fri, 15 Jun 2018 13:57:15 +0000 (14:57 +0100)]
iommu: Add IOMMU index concept to IOMMU API
If an IOMMU supports mappings that care about the memory
transaction attributes, then it no longer has a unique
address -> output mapping, but more than one. We can
represent these using an IOMMU index, analogous to TCG's
mmu indexes.
Cédric Le Goater [Fri, 15 Jun 2018 13:57:15 +0000 (14:57 +0100)]
m25p80: add support for two bytes WRSR for Macronix chips
On Macronix chips, two bytes can written to the WRSR. First byte will
configure the status register and the second the configuration
register. It is important to save the configuration value as it
contains the dummy cycle setting when using dual or quad IO mode.
Joel Stanley [Fri, 15 Jun 2018 13:57:15 +0000 (14:57 +0100)]
aspeed_scu: Implement RNG register
The ASPEED SoCs contain a single register that returns random data when
read. This models that register so that guests can use it.
The random number data register has a corresponding control register,
however it returns data regardless of the state of the enabled bit, so
the model follows this behaviour.
When the qcrypto call fails we exit as the guest uses the random number
device to feed it's entropy pool, which is used for cryptographic
purposes.
Rearrange the arithmetic so that we are agnostic about the total size
of the vector and the size of the element. This will allow us to index
up to the 32nd byte and with 16-byte elements.
Peter Maydell [Fri, 15 Jun 2018 13:57:14 +0000 (14:57 +0100)]
exec.c: Don't accidentally sign-extend 4-byte loads in subpage_read()
In subpage_read() we perform a load of the data into a local buffer
which we then access using ldub_p(), lduw_p(), ldl_p() or ldq_p()
depending on its size, storing the result into the uint64_t *data.
Since ldl_p() returns an 'int', this means that for the 4-byte
case we will sign-extend the data, whereas for 1 and 2 byte
reads we zero-extend it.
This ought not to matter since the caller will likely ignore values in
the high bytes of the data, but add a cast so that we're consistent.
Peter Maydell [Fri, 15 Jun 2018 13:57:14 +0000 (14:57 +0100)]
bswap: Add new stn_*_p() and ldn_*_p() memory access functions
There's a common pattern in QEMU where a function needs to perform
a data load or store of an N byte integer in a particular endianness.
At the moment this is handled by doing a switch() on the size and
calling the appropriate ld*_p or st*_p function for each size.
Provide a new family of functions ldn_*_p() and stn_*_p() which
take the size as an argument and do the switch() themselves.
Peter Maydell [Fri, 15 Jun 2018 13:57:14 +0000 (14:57 +0100)]
CODING_STYLE: Define our preferred form for multiline comments
The codebase has a bit of a mix of different multiline
comment styles. State a preference for the Linux kernel
style:
/*
* Star on the left for each line.
* Leading slash-star and trailing star-slash
* each go on a line of their own.
*/
Peter Maydell [Fri, 15 Jun 2018 13:57:14 +0000 (14:57 +0100)]
cputlb: Pass cpu_transaction_failed() the correct physaddr
The API for cpu_transaction_failed() says that it takes the physical
address for the failed transaction. However we were actually passing
it the offset within the target MemoryRegion. We don't currently
have any target CPU implementations of this hook that require the
physical address; fix this bug so we don't get confused if we ever
do add one.
Peter Maydell [Fri, 15 Jun 2018 13:57:14 +0000 (14:57 +0100)]
cpu-defs.h: Document CPUIOTLBEntry 'addr' field
The 'addr' field in the CPUIOTLBEntry struct has a rather non-obvious
use; add a comment documenting it (reverse-engineered from what
the code that sets it is doing).
Peter Maydell [Fri, 15 Jun 2018 13:57:14 +0000 (14:57 +0100)]
hw/core/or-irq: Support more than 16 inputs to an OR gate
For the IoTKit MPC support, we need to wire together the
interrupt outputs of 17 MPCs; this exceeds the current
value of MAX_OR_LINES. Increase MAX_OR_LINES to 32 (which
should be enough for anyone).
The tricky part is retaining the migration compatibility for
existing OR gates; we add a subsection which is only used
for larger OR gates, and define it such that we can freely
increase MAX_OR_LINES in future (or even move to a dynamically
allocated levels[] array without an upper size limit) without
breaking compatibility.
Peter Maydell [Fri, 15 Jun 2018 13:57:13 +0000 (14:57 +0100)]
arm: Don't crash if user tries to use a Cortex-M CPU without an NVIC
The Cortex-M CPU and its NVIC are two intimately intertwined parts of
the same hardware; it is not possible to use one without the other.
Unfortunately a lot of our board models don't do any sanity checking
on the CPU type the user asks for, so a command line like
qemu-system-arm -M versatilepb -cpu cortex-m3
will create an M3 without an NVIC, and coredump immediately.
In the other direction, trying a non-M-profile CPU in an M-profile
board won't blow up, but doesn't do anything useful either:
qemu-system-arm -M lm3s6965evb -cpu arm926
Add some checking in the NVIC and CPU realize functions that the
user isn't trying to use an NVIC without an M-profile CPU or
an M-profile CPU without an NVIC, so we can produce a helpful
error message rather than a core dump.
Peter Maydell [Fri, 15 Jun 2018 13:57:13 +0000 (14:57 +0100)]
hw/arm/armv7m: Remove unused armv7m_init() function
Remove the now-unused armv7m_init() function. This was a legacy from
before we properly QOMified ARMv7M, and it has some flaws:
* it combines work that needs to be done by an SoC object (creating
and initializing the TYPE_ARMV7M object) with work that needs to
be done by the board model (setting the system up to load the ELF
file specified with -kernel)
* TYPE_ARMV7M creation failure is fatal, but an SoC object wants to
arrange to propagate the failure outward
* it uses allocate-and-create via qdev_create() whereas the current
preferred style for SoC objects is to do creation in-place
Board and SoC models can instead do the two jobs this function
was doing themselves, in the right places and with whatever their
preferred style/error handling is.
Peter Maydell [Fri, 15 Jun 2018 13:57:13 +0000 (14:57 +0100)]
stellaris: Stop using armv7m_init()
The stellaris board is still using the legacy armv7m_init() function,
which predates conversion of the ARMv7M into a proper QOM container
object. Make the board code directly create the ARMv7M object instead.
Peter Maydell [Fri, 15 Jun 2018 13:57:13 +0000 (14:57 +0100)]
hw/char/parallel: Convert away from old_mmio
Convert the parallel device away from using the old_mmio field
of MemoryRegionOps. This change only affects the memory-mapped
variant, which is used by the MIPS Jazz boards 'magnum' and 'pica61'.
Peter Maydell [Fri, 15 Jun 2018 13:57:13 +0000 (14:57 +0100)]
hw/input/pckbd: Convert away from old_mmio
Convert the pckbd device away from using the old_mmio field
of MemoryRegionOps. This change only affects the memory-mapped
variant of the i8042, which is used by the Unicore32 'puv3'
board and the MIPS Jazz boards 'magnum' and 'pica61'.
Peter Maydell [Fri, 15 Jun 2018 13:57:13 +0000 (14:57 +0100)]
hw/arm/mps2-tz: Put ethernet controller behind PPC
The ethernet controller in the AN505 MPC FPGA image is behind
the same AHB Peripheral Protection Controller that handles
the graphics and GPIOs. (In the documentation this is clear
in the block diagram but the ethernet controller was omitted
from the table listing devices connected to the PPC.)
The ethernet sits behind AHB PPCEXP0 interface 5. We had
incorrectly claimed that this was a "gpio4", but there are
only 4 GPIOs in this image.
Shannon Zhao [Fri, 15 Jun 2018 13:57:13 +0000 (14:57 +0100)]
arm_gicv3_kvm: kvm_dist_get/put_priority: skip the registers banked by GICR_IPRIORITYR
While for_each_dist_irq_reg loop starts from GIC_INTERNAL, it forgot to
offset the date array and index. This will overlap the GICR registers
value and leave the last GIC_INTERNAL irq's registers out of update.
Balamuruhan S [Tue, 12 Jun 2018 08:50:09 +0000 (14:20 +0530)]
migration: calculate expected_downtime with ram_bytes_remaining()
expected_downtime value is not accurate with dirty_pages_rate * page_size,
using ram_bytes_remaining() would yeild it resonable.
consider to read the remaining ram just after having updated the dirty
pages count later migration_bitmap_sync_range() in migration_bitmap_sync()
and reuse the `remaining` field in ram_counters to hold ram_bytes_remaining()
for calculating expected_downtime.
migration/postcopy: Wake rate limit sleep on postcopy request
Use the 'urgent request' mechanism added in the previous patch
for entries added to the postcopy request queue for RAM. Ignore
the rate limiting while we have requests.
Rate limiting sleeps the migration thread for a while when it runs
out of bandwidth; but sometimes we want to wake up to get on with
something more urgent (like a postcopy request). Here we use
a semaphore with a timedwait instead of a simple sleep; Incrementing
the sempahore will wake it up sooner. Anything that consumes
these urgent events must decrement the sempahore.
Limit the background transfer bandwidth during the postcopy
phase to the value set on this new parameter. The default, 0,
corresponds to the existing behaviour which is unlimited bandwidth.
The migration code should be using the
RAMBLOCK_FOREACH_MIGRATABLE and qemu_ram_foreach_block_migratable
not the all-block versions; poison them so that we can't accidentally
use them.
There are still a few cases where migration code is using the macros
and functions that do all RAMBlocks rather than just the migratable
blocks; fix those up.
In file included from /tmp/qemu-test/src/migration/savevm.c:59:
/tmp/qemu-test/src/migration/qjson.h:16: error: redefinition of typedef
'QJSON'
/tmp/qemu-test/src/include/migration/vmstate.h:30: note: previous
declaration of 'QJSON' was here
make: *** [migration/savevm.o] Error 1
This happens because CentOS 6 has an old GCC 4.4.7. Even if redefining
a typedef with the same type is permitted since GCC 4.6, unless -pedantic
is passed, we don't really need to do that on purpose. Let's have a
single definition in <qemu/typedefs.h> instead.
Kevin Wolf [Wed, 13 Jun 2018 09:01:30 +0000 (11:01 +0200)]
block: Remove deprecated -drive geometry options
The -drive options cyls, heads, secs and trans were deprecated in
QEMU 2.10. It's time to remove them.
hd-geo-test tested both the old version with geometry options in -drive
and the new one with -device. Therefore the code using -drive doesn't
have to be replaced there, we just need to remove the -drive test cases.
This in turn allows some simplification of the code.
Legacy -drive supports "password-secret" parameter that isn't
available with -blockdev / blockdev-add. That's because we backed out
our first try to provide it there due to interface design doubts, in
commit 577d8c9a811, v2.9.0.
This is the second try. It brings back the parameter, except it's
named "key-secret" now.
Let's review our reasons for backing out the first try, as stated in
the commit message:
* BlockdevOptionsRbd member @password-secret isn't actually a
password, it's a key generated by Ceph.
Addressed by the rename.
* We're not sure where member @password-secret belongs (see the
previous commit).
See previous commit.
* How @password-secret interacts with settings from a configuration
file specified with @conf is undocumented.
Not actually true, the documentation for @conf says "Values in the
configuration file will be overridden by options specified via QAPI",
and we've tested this.
Parameter auth-client-required lets you configure authentication
methods. We tried to provide that in v2.9.0, but backed out due to
interface design doubts (commit 464444fcc16).
This commit is similar to what we backed out, but simpler: we use a
list of enumeration values instead of a list of objects with a member
of enumeration type.
Let's review our reasons for backing out the first try, as stated in
the commit message:
* The implementation uses deprecated rados_conf_set() key
"auth_supported". No biggie.
Fixed: we use "auth-client-required".
* The implementation makes -drive silently ignore invalid parameters
"auth" and "auth-supported.*.X" where X isn't "auth". Fixable (in
fact I'm going to fix similar bugs around parameter server), so
again no biggie.
That fix is commit 2836284db60. This commit doesn't bring the bugs
back.
* BlockdevOptionsRbd member @password-secret applies only to
authentication method cephx. Should it be a variant member of
RbdAuthMethod?
We've had time to ponder, and we decided to stick to the way Ceph
configuration works: the key configured separately, and silently
ignored if the authentication method doesn't use it.
* BlockdevOptionsRbd member @user could apply to both methods cephx
and none, but I'm not sure it's actually used with none. If it
isn't, should it be a variant member of RbdAuthMethod?
Likewise.
* The client offers a *set* of authentication methods, not a list.
Should the methods be optional members of BlockdevOptionsRbd instead
of members of list @auth-supported? The latter begs the question
what multiple entries for the same method mean. Trivial question
now that RbdAuthMethod contains nothing but @type, but less so when
RbdAuthMethod acquires other members, such the ones discussed above.
Again, we decided to stick to the way Ceph configuration works, except
we make auth-client-required a list of enumeration values instead of a
string containing keywords separated by delimiters.
* How BlockdevOptionsRbd member @auth-supported interacts with
settings from a configuration file specified with @conf is
undocumented. I suspect it's untested, too.
Not actually true, the documentation for @conf says "Values in the
configuration file will be overridden by options specified via QAPI",
and we've tested this.
block: Fix -blockdev / blockdev-add for empty objects and arrays
-blockdev and blockdev-add silently ignore empty objects and arrays in
their argument. That's because qmp_blockdev_add() converts the
argument to a flat QDict, and qdict_flatten() eats empty QDict and
QList members. For instance, we ignore an empty BlockdevOptions
member @cache. No real harm, as absent means the same as empty there.
Thus, the flaw puts an artificial restriction on the QAPI schema: we
can't have potentially empty objects and arrays within
BlockdevOptions, except when they're optional and "empty" has the same
meaning as "absent".
Our QAPI schema satisfies this restriction (I checked), but it's a
trap for the unwary, and a temptation to employ awkward workarounds
for the wary. Let's get rid of it.
Change qdict_flatten() and qdict_crumple() to treat empty dictionaries
and lists exactly like scalars.
When you mix scalar and non-scalar keys, whether you get an "already
set as scalar" or an "already set as dict" error depends on qdict
iteration order. Neither message makes much sense. Replace by
""Cannot mix scalar and non-scalar keys". This is similar to the
message we get for mixing list and non-list keys.
I find qdict_crumple()'s first loop hard to understand. Rearrange it
and add a comment.
qdict_flatten_qdict() skips copying scalars from @qdict to @target
when the two are the same. Fair enough, but it uses a non-obvious
test for "same". Replace it by the obvious one. While there, improve
comments.
There's no need to restart the loop. We don't elsewhere, e.g. in
qdict_extract_subqdict(), qdict_join() and qemu_opts_absorb_qdict().
Simplify accordingly.
block: Make remaining uses of qobject input visitor more robust
Remaining uses of qobject_input_visitor_new_keyval() in the block
subsystem:
* block_crypto_open_opts_init()
Currently doesn't visit any non-string scalars, thus safe. It's
called from
- block_crypto_open_luks()
Creates the QDict with qemu_opts_to_qdict_filtered(), which
creates only string scalars, but has a TODO asking for other types.
- qcow_open()
- qcow2_open(), qcow2_co_invalidate_cache(), qcow2_reopen_prepare()
* block_crypto_create_opts_init(), called from
- block_crypto_co_create_opts_luks()
Also creates the QDict with qemu_opts_to_qdict_filtered().
* vdi_co_create_opts()
Also creates the QDict with qemu_opts_to_qdict_filtered().
Replace these uses by qobject_input_visitor_new_flat_confused() for
robustness. This adds crumpling. Right now, that's a no-op, but if
we ever extend these things in non-flat ways, crumpling will be
needed.
block: Clean up a misuse of qobject_to() in .bdrv_co_create_opts()
The following pattern occurs in the .bdrv_co_create_opts() methods of
parallels, qcow, qcow2, qed, vhdx and vpc:
qobj = qdict_crumple_for_keyval_qiv(qdict, errp);
qobject_unref(qdict);
qdict = qobject_to(QDict, qobj);
if (qdict == NULL) {
ret = -EINVAL;
goto done;
}
v = qobject_input_visitor_new_keyval(QOBJECT(qdict));
[...]
ret = 0;
done:
qobject_unref(qdict);
[...]
return ret;
If qobject_to() fails, we return failure without setting errp. That's
wrong. As far as I can tell, it cannot fail here. Clean it up
anyway, by removing the useless conversion.
The previous commit fixed -blockdev breakage due to misuse of the
qobject input visitor's keyval flavor in bdrv_file_open(). The commit
message explain why using the plain flavor would be just as wrong; it
would break -drive. Turns out we break it in three places:
nbd_open(), sd_open() and ssh_file_open(). They are even marked
FIXME. Example breakage:
$ qemu-system-x86 -drive node-name=n1,driver=nbd,server.type=inet,server.host=localhost,server.port=1234,server.numeric=off
qemu-system-x86: -drive node-name=n1,driver=nbd,server.type=inet,server.host=localhost,server.port=1234,server.numeric=off: Invalid parameter type for 'numeric', expected: boolean
Fix it the same way: replace qdict_crumple() by
qdict_crumple_for_keyval_qiv(), and switch from plain to the keyval
flavor.
block: Fix -blockdev for certain non-string scalars
Configuration flows through the block subsystem in a rather peculiar
way. Configuration made with -drive enters it as QemuOpts.
Configuration made with -blockdev / blockdev-add enters it as QAPI
type BlockdevOptions. The block subsystem uses QDict, QemuOpts and
QAPI types internally. The precise flow is next to impossible to
explain (I tried for this commit message, but gave up after wasting
several hours). What I can explain is a flaw in the BlockDriver
interface that leads to this bug:
Here's what happens. The block layer passes configuration represented
as flat QDict (with dotted keys) to BlockDriver methods
.bdrv_file_open(). The QDict's members are typed according to the
QAPI schema.
nfs_file_open() converts it to QAPI type BlockdevOptionsNfs, with
qdict_crumple() and a qobject input visitor.
This visitor comes in two flavors. The plain flavor requires scalars
to be typed according to the QAPI schema. That's the case here. The
keyval flavor requires string scalars. That's not the case here.
nfs_file_open() uses the latter, and promptly falls apart for members
@user, @group, @tcp-syn-count, @readahead-size, @page-cache-size,
@debug.
Switching to the plain flavor would fix -blockdev, but break -drive,
because there the scalars arrive in nfs_file_open() as strings.
The proper fix would be to replace the QDict by QAPI type
BlockdevOptions in the BlockDriver interface. Sadly, that's beyond my
reach right now.
Next best would be to fix the block layer to always pass correctly
typed QDicts to the BlockDriver methods. Also beyond my reach.
What I can do is throw another hack onto the pile: have
nfs_file_open() convert all members to string, so use of the keyval
flavor actually works, by replacing qdict_crumple() by new function
qdict_crumple_for_keyval_qiv().
The pattern "pass result of qdict_crumple() to
qobject_input_visitor_new_keyval()" occurs several times more:
* qemu_rbd_open()
Same issue as nfs_file_open(), but since BlockdevOptionsRbd has only
string members, its only a latent bug. Fix it anyway.
These work, because they create the QDict with
qemu_opts_to_qdict_filtered(), which creates only string scalars.
The function sports a TODO comment asking for better typing; that's
going to be fun. Use qdict_crumple_for_keyval_qiv() to be safe.
Max Reitz [Thu, 14 Jun 2018 19:14:28 +0000 (21:14 +0200)]
block: Add block-specific QDict header
There are numerous QDict functions that have been introduced for and are
used only by the block layer. Move their declarations into an own
header file to reflect that.
While qdict_extract_subqdict() is in fact used outside of the block
layer (in util/qemu-config.c), it is still a function related very
closely to how the block layer works with nested QDicts, namely by
sometimes flattening them. Therefore, its declaration is put into this
header as well and util/qemu-config.c includes it with a comment stating
exactly which function it needs.
Eric Blake [Mon, 11 Jun 2018 21:39:27 +0000 (16:39 -0500)]
iotests: Add test 221 to catch qemu-img map regression
Although qemu-img creates aligned files (by rounding up), it
must also gracefully handle files that are not sector-aligned.
Test that the bug fixed in the previous patch does not recur.
It's a bit annoying that we can see the (implicit) hole past
the end of the file on to the next sector boundary, so if we
ever reach the point where we report a byte-accurate size rather
than our current behavior of always rounding up, this test will
probably need a slight modification.
Eric Blake [Mon, 11 Jun 2018 21:39:26 +0000 (16:39 -0500)]
qemu-img: Fix assert when mapping unaligned raw file
Commit a290f085 exposed a latent bug in qemu-img map introduced
during the conversion of block status to be byte-based. Earlier in
commit 5e344dd8, the internal interface get_block_status() switched
to take byte-based parameters, but still called a sector-based
block layer function; as such, rounding was added in the lone
caller to obey the contract. However, commit 237d78f8 changed
get_block_status() to truly be byte-based, at which point rounding
to sector boundaries can result in calling bdrv_block_status() with
'bytes == 0' (a coding error) when the boundary between data and a
hole falls mid-sector (true for the past-EOF implicit hole present
in POSIX files). Fix things by removing the rounding that is now
no longer necessary.
Peter Maydell [Fri, 15 Jun 2018 11:49:36 +0000 (12:49 +0100)]
Merge remote-tracking branch 'remotes/stsquad/tags/pull-travis-updates-140618-1' into staging
Travis updates
- show config.log when failing
- reduce time for gprof build
- reduce time for alternate trace builds
# gpg: Signature made Thu 14 Jun 2018 20:29:59 BST
# gpg: using RSA key FBD0DB095A9E2A44
# gpg: Good signature from "Alex Bennée (Master Work Key) <[email protected]>"
# Primary key fingerprint: 6685 AE99 E751 67BC AFC8 DF35 FBD0 DB09 5A9E 2A44
* remotes/stsquad/tags/pull-travis-updates-140618-1:
travis: reduce time taken for trace-backend testing
travis: reduce coverage of gprof build
travis: display config.log when configure fails
Peter Maydell [Fri, 15 Jun 2018 10:41:44 +0000 (11:41 +0100)]
Merge remote-tracking branch 'remotes/jasowang/tags/net-pull-request' into staging
# gpg: Signature made Fri 15 Jun 2018 03:47:09 BST
# gpg: using RSA key EF04965B398D6211
# gpg: Good signature from "Jason Wang (Jason Wang on RedHat) <[email protected]>"
# gpg: WARNING: This key is not certified with sufficiently trusted signatures!
# gpg: It is not certain that the signature belongs to the owner.
# Primary key fingerprint: 215D 46F4 8246 689E C77F 3562 EF04 965B 398D 6211
* remotes/jasowang/tags/net-pull-request:
vhost-user: delete net client if necessary
e1000e: Do not auto-clear ICR bits which aren't set in EIAC
net: Fix a potential segfault
tap: set vhostfd passed from qemu cli to non-blocking
projects / qemu.git / log