Greg Kurz [Wed, 26 Feb 2020 18:59:38 +0000 (19:59 +0100)]
ppc: Officially deprecate the CPU "compat" property
Server class POWER CPUs have a "compat" property, which was obsoleted
by commit 7843c0d60d and replaced by a "max-cpu-compat" property on the
pseries machine type. A hack was introduced so that passing "compat" to
-cpu would still produce the desired effect, for the sake of backward
compatibility : it strips the "compat" option from the CPU properties
and applies internally it to the pseries machine. The accessors of the
"compat" property were updated to do nothing but warn the user about the
deprecated status when doing something like:
$ qemu-system-ppc64 -global POWER9-family-powerpc64-cpu.compat=power9
qemu-system-ppc64: warning: CPU 'compat' property is deprecated and has no
effect; use max-cpu-compat machine property instead
This was merged during the QEMU 2.10 timeframe, a few weeks before we
formalized our deprecation process. As a consequence, the "compat"
property fell through the cracks and was never listed in the officialy
deprecated features.
We are now eight QEMU versions later, it is largely time to mention it
in qemu-deprecated.texi. Also, since -global XXX-powerpc64-cpu.compat=
has been emitting warnings since QEMU 2.10 and the usual way of setting
CPU properties is with -cpu, completely remove the "compat" property.
Keep the hack so that -cpu XXX,compat= stays functional some more time,
as required by our deprecation process.
The now empty powerpc_servercpu_properties[] list which was introduced
for "compat" and never had any other use is removed on the way. We can
re-add it in the future if the need for a server class POWER CPU specific
property arises again.
Greg Kurz [Mon, 24 Feb 2020 19:23:43 +0000 (20:23 +0100)]
spapr: Handle pending hot plug/unplug requests at CAS
If a hot plug or unplug request is pending at CAS, we currently trigger
a CAS reboot, which severely increases the guest boot time. This is
because SLOF doesn't handle hot plug events and we had no way to fix
the FDT that gets presented to the guest.
We can do better thanks to recent changes in QEMU and SLOF:
- we now return a full FDT to SLOF during CAS
- SLOF was fixed to correctly detect any device that was either added or
removed since boot time and to update its internal DT accordingly.
The right solution is to process all pending hot plug/unplug requests
during CAS: convert hot plugged devices to cold plugged devices and
remove the hot unplugged ones, which is exactly what spapr_drc_reset()
does. Also clear all hot plug events that are currently queued since
they're no longer relevant.
Note that SLOF cannot currently populate hot plugged PCI bridges or PHBs
at CAS. Until this limitation is lifted, SLOF will reset the machine when
this scenario occurs : this will allow the FDT to be fully processed when
SLOF is started again (ie. the same effect as the CAS reboot that would
occur anyway without this patch).
This adds vTPM support, full-FDT-rebuild-on-CAS fixes and
basic ext4 support.
The full changelog is:
Alexey Kardashevskiy (10):
disk-label: Prepare for extenting
disk-label: Support Linux GPT partition type
ext2: Prepare for extending
ext2: Rename group-desc-size
ext2: Read size of group descriptors
ext2: Read all 64bit of inode number
ext2/4: Add basic extent tree support
elf64: Add LE64 ABIv1/2 support for loading images to given address
fdt: Fix creating new nodes at H_CAS
version: update to 20200221
Greg Kurz (2):
fdt: Fix update of "interrupt-controller" node at CAS
fdt: Delete nodes of devices removed between boot and CAS
Stefan Berger (8):
slof: Implement SLOF_get_keystroke() and SLOF_reset()
slof: Make linker script variables accessible
qemu: Make print_version variable accessible
tpm: Add TPM CRQ driver implementation
tpm: Add sha256 implementation
tcgbios: Add TPM 2.0 support and firmware API
tcgbios: Implement menu to clear TPM 2 and activate its PCR banks
tcgbios: Measure the GPT table
Gerd Hoffmann [Mon, 9 Mar 2020 10:00:09 +0000 (11:00 +0100)]
stdvga+bochs-display: add dummy mmio handler
The bochs-display mmio bar has some sub-regions with the actual hardware
registers. What happens when the guest access something outside those
regions depends on the archirecture. On x86 those reads succeed (and
return 0xff I think). On risc-v qemu aborts.
This patch adds handlers for the parent region, to make the wanted
behavior explicit and to make things consistent across architectures.
v2:
- use existing unassigned_io_ops.
- also cover stdvga.
Volker Rümelin [Sun, 8 Mar 2020 19:33:19 +0000 (20:33 +0100)]
audio: change mixing engine float range to [-1.f, 1.f]
Currently the internal float range of the mixing engine is
[-.5f, .5f]. PulseAudio, SDL2 and libasound use a [-1.f, 1.f]
range. This means with float samples the audio playback volume
is 6dB too low and audio recording signals will be clipped in
most cases.
To avoid another scaling factor in the conv_natural_float_* and
clip_natural_float_* functions with FLOAT_MIXENG defined this
patch changes the mixing engine float range to [-1.f, 1.f].
Volker Rümelin [Sun, 8 Mar 2020 19:33:18 +0000 (20:33 +0100)]
audio: consistency changes
Change the clip_natural_float_from_mono() function in
audio/mixeng.c to be consistent with the clip_*_from_mono()
functions in audio/mixeng_template.h.
Peter Maydell [Fri, 13 Mar 2020 10:33:04 +0000 (10:33 +0000)]
Merge remote-tracking branch 'remotes/dgilbert/tags/pull-migration-20200313a' into staging
Migration pull 2020-03-13
zstd build fix
A new auto-converge parameter
Some COLO improvements
# gpg: Signature made Fri 13 Mar 2020 10:29:34 GMT
# gpg: using RSA key 45F5C71B4A0CB7FB977A9FA90516331EBC5BFDE7
# gpg: Good signature from "Dr. David Alan Gilbert (RH2) <[email protected]>" [full]
# Primary key fingerprint: 45F5 C71B 4A0C B7FB 977A 9FA9 0516 331E BC5B FDE7
* remotes/dgilbert/tags/pull-migration-20200313a:
migration: recognize COLO as part of activating process
ram/colo: only record bitmap of dirty pages in COLO stage
COLO: Optimize memory back-up process
migration/throttle: Add throttle-trig-thres migration parameter
configure: Improve zstd test
zhanghailiang [Mon, 24 Feb 2020 06:54:12 +0000 (14:54 +0800)]
migration: recognize COLO as part of activating process
We will migrate parts of dirty pages backgroud lively during the gap time
of two checkpoints, without this modification, it will not work
because ram_save_iterate() will check it before send RAM_SAVE_FLAG_EOS
at the end of it.
zhanghailiang [Mon, 24 Feb 2020 06:54:10 +0000 (14:54 +0800)]
COLO: Optimize memory back-up process
This patch will reduce the downtime of VM for the initial process,
Previously, we copied all these memory in preparing stage of COLO
while we need to stop VM, which is a time-consuming process.
Here we optimize it by a trick, back-up every page while in migration
process while COLO is enabled, though it affects the speed of the
migration, but it obviously reduce the downtime of back-up all SVM'S
memory in COLO preparing stage.
Currently, if the bytes_dirty_period is more than the 50% of
bytes_xfer_period, we start or increase throttling.
If we make this percentage higher, then we can tolerate higher
dirty rate during migration, which means less impact on guest.
The side effect of higher percentage is longer migration time.
We can make this parameter configurable to switch between mig-
ration time first or guest performance first.
The default value is 50 and valid range is 1 to 100.
Juan Quintela [Tue, 10 Mar 2020 11:14:31 +0000 (12:14 +0100)]
configure: Improve zstd test
There were one error on the test (missing an s for --exists).
But we really need a recent zstd (1.4.0).
Thanks to Michal Privoznik to provide the right vension.
Peter Maydell [Thu, 12 Mar 2020 17:34:34 +0000 (17:34 +0000)]
Merge remote-tracking branch 'remotes/pmaydell/tags/pull-target-arm-20200312' into staging
target-arm queue:
* Fix various bugs that might result in an assert() due to
incorrect hflags for M-profile CPUs
* Fix Aspeed SMC Controller user-mode select handling
* Report correct (with-tag) address in fault address register
when TBI is enabled
* cubieboard: make sure SOC object isn't leaked
* fsl-imx25: Wire up eSDHC controllers
* fsl-imx25: Wire up USB controllers
* New board model: orangepi-pc (OrangePi PC)
* ARM/KVM: if user doesn't select GIC version and the
host kernel can only provide GICv3, use that, rather
than defaulting to "fail because GICv2 isn't possible"
* kvm: Only do KVM_SET_VCPU_EVENTS at the last stage of sync
* remotes/pmaydell/tags/pull-target-arm-20200312: (36 commits)
target/arm: kvm: Inject events at the last stage of sync
hw/arm/virt: kvm: allow gicv3 by default if v2 cannot work
hw/arm/virt: kvm: Restructure finalize_gic_version()
target/arm/kvm: Let kvm_arm_vgic_probe() return a bitmap
hw/arm/virt: Introduce finalize_gic_version()
hw/arm/virt: Introduce VirtGICType enum type
hw/arm/virt: Document 'max' value in gic-version property description
docs: add Orange Pi PC document
tests/boot_linux_console: Test booting NetBSD via U-Boot on OrangePi PC
tests/boot_linux_console: Add a SLOW test booting Ubuntu on OrangePi PC
tests/boot_linux_console: Add a SD card test for the OrangePi PC board
tests/boot_linux_console: Add initrd test for the Orange Pi PC board
tests/boot_linux_console: Add a quick test for the OrangePi PC board
hw/arm/allwinner: add RTC device support
hw/arm/allwinner-h3: add SDRAM controller device
hw/arm/allwinner-h3: add Boot ROM support
hw/arm/allwinner-h3: add EMAC ethernet device
hw/arm/allwinner: add SD/MMC host controller
hw/arm/allwinner: add Security Identifier device
hw/arm/allwinner: add CPU Configuration module
...
Peter Maydell [Thu, 12 Mar 2020 16:51:26 +0000 (16:51 +0000)]
Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging
Block layer patches:
- Relax restrictions for blockdev-snapshot (allows libvirt to do live
storage migration with blockdev-mirror)
- luks: Delete created files when block_crypto_co_create_opts_luks fails
- Fix memleaks in qmp_object_add
# gpg: Signature made Wed 11 Mar 2020 15:38:59 GMT
# gpg: using RSA key 7F09B272C88F2FD6
# gpg: Good signature from "Kevin Wolf <[email protected]>" [full]
# Primary key fingerprint: DC3D EB15 9A9A F95D 3D74 56FE 7F09 B272 C88F 2FD6
* remotes/kevin/tags/for-upstream:
qemu-iotests: adding LUKS cleanup for non-UTF8 secret error
crypto.c: cleanup created file when block_crypto_co_create_opts_luks fails
block.c: adding bdrv_co_delete_file
block: introducing 'bdrv_co_delete_file' interface
tests/qemu-iotests: Fix socket_scm_helper build path
qapi: Add '@allow-write-only-overlay' feature for 'blockdev-snapshot'
iotests: Add iothread cases to 155
block: Fix cross-AioContext blockdev-snapshot
iotests: Test mirror with temporarily disabled target backing file
iotests: Fix run_job() with use_log=False
block: Relax restrictions for blockdev-snapshot
block: Make bdrv_get_cumulative_perm() public
qom-qmp-cmds: fix two memleaks in qmp_object_add
Beata Michalska [Thu, 12 Mar 2020 00:34:00 +0000 (00:34 +0000)]
target/arm: kvm: Inject events at the last stage of sync
KVM_SET_VCPU_EVENTS might actually lead to vcpu registers being modified.
As such this should be the last step of sync to avoid potential overwriting
of whatever changes KVM might have done.
Eric Auger [Wed, 11 Mar 2020 13:16:18 +0000 (14:16 +0100)]
hw/arm/virt: kvm: allow gicv3 by default if v2 cannot work
At the moment if the end-user does not specify the gic-version along
with KVM acceleration, v2 is set by default. However most of the
systems now have GICv3 and sometimes they do not support GICv2
compatibility.
This patch keeps the default v2 selection in all cases except
in the KVM accelerated mode when either
- the host does not support GICv2 in-kernel emulation or
- number of VCPUS exceeds 8.
Those cases did not work anyway so we do not break any compatibility.
Now we get v3 selected in such a case.
In KVM mode / in-kernel irqchip , we explictly check whether
the chosen version is supported by the host. If the end-user
explicitly sets v2/v3 and this is not supported by the host,
then the user gets an explicit error message. Note that for
old kernels where the CREATE_DEVICE ioctl doesn't exist then
we will now fail if the user specifically asked for gicv2,
where previously we (probably) would have succeeded.
In KVM mode / userspace irqchip we immediatly output an error
in case the end-user explicitly selected v3. Also we warn the
end-user about the unexpected usage of gic-version=host in
that case as only userspace GICv2 is supported.
Eric Auger [Wed, 11 Mar 2020 13:16:16 +0000 (14:16 +0100)]
target/arm/kvm: Let kvm_arm_vgic_probe() return a bitmap
Convert kvm_arm_vgic_probe() so that it returns a
bitmap of supported in-kernel emulation VGIC versions instead
of the max version: at the moment values can be v2 and v3.
This allows to expose the case where the host GICv3 also
supports GICv2 emulation. This will be useful to choose the
default version in KVM accelerated mode.
Eric Auger [Wed, 11 Mar 2020 13:16:15 +0000 (14:16 +0100)]
hw/arm/virt: Introduce finalize_gic_version()
Let's move the code which freezes which gic-version to
be applied in a dedicated function. We also now set by
default the VIRT_GIC_VERSION_NO_SET. This eventually
turns into the legacy v2 choice in the finalize() function.
Eric Auger [Wed, 11 Mar 2020 13:16:14 +0000 (14:16 +0100)]
hw/arm/virt: Introduce VirtGICType enum type
We plan to introduce yet another value for the gic version (nosel).
As we already use exotic values such as 0 and -1, let's introduce
a dedicated enum type and let vms->gic_version take this
type.
Niek Linnenbank [Wed, 11 Mar 2020 22:18:54 +0000 (23:18 +0100)]
docs: add Orange Pi PC document
The Xunlong Orange Pi PC machine is a functional ARM machine
based on the Allwinner H3 System-on-Chip. It supports mainline
Linux, U-Boot, NetBSD and is covered by acceptance tests.
This commit adds a documentation text file with a description
of the machine and instructions for the user.
Niek Linnenbank [Wed, 11 Mar 2020 22:18:48 +0000 (23:18 +0100)]
hw/arm/allwinner: add RTC device support
Allwinner System-on-Chips usually contain a Real Time Clock (RTC)
for non-volatile system date and time keeping. This commit adds a generic
Allwinner RTC device that supports the RTC devices found in Allwinner SoC
family sun4i (A10), sun7i (A20) and sun6i and newer (A31, H2+, H3, etc).
The following RTC functionality and features are implemented:
* Year-Month-Day read/write
* Hour-Minute-Second read/write
* General Purpose storage
The following boards are extended with the RTC device:
* Cubieboard (hw/arm/cubieboard.c)
* Orange Pi PC (hw/arm/orangepi.c)
Niek Linnenbank [Wed, 11 Mar 2020 22:18:47 +0000 (23:18 +0100)]
hw/arm/allwinner-h3: add SDRAM controller device
In the Allwinner H3 SoC the SDRAM controller is responsible
for interfacing with the external Synchronous Dynamic Random
Access Memory (SDRAM). Types of memory that the SDRAM controller
supports are DDR2/DDR3 and capacities of up to 2GiB. This commit
adds emulation support of the Allwinner H3 SDRAM controller.
Niek Linnenbank [Wed, 11 Mar 2020 22:18:46 +0000 (23:18 +0100)]
hw/arm/allwinner-h3: add Boot ROM support
A real Allwinner H3 SoC contains a Boot ROM which is the
first code that runs right after the SoC is powered on.
The Boot ROM is responsible for loading user code (e.g. a bootloader)
from any of the supported external devices and writing the downloaded
code to internal SRAM. After loading the SoC begins executing the code
written to SRAM.
This commits adds emulation of the Boot ROM firmware setup functionality
by loading user code from SD card in the A1 SRAM. While the A1 SRAM is
64KiB, we limit the size to 32KiB because the real H3 Boot ROM also rejects
sizes larger than 32KiB. For reference, this behaviour is documented
by the Linux Sunxi project wiki at:
Niek Linnenbank [Wed, 11 Mar 2020 22:18:45 +0000 (23:18 +0100)]
hw/arm/allwinner-h3: add EMAC ethernet device
The Allwinner Sun8i System on Chip family includes an Ethernet MAC (EMAC)
which provides 10M/100M/1000M Ethernet connectivity. This commit
adds support for the Allwinner EMAC from the Sun8i family (H2+, H3, A33, etc),
including emulation for the following functionality:
* DMA transfers
* MII interface
* Transmit CRC calculation
Niek Linnenbank [Wed, 11 Mar 2020 22:18:44 +0000 (23:18 +0100)]
hw/arm/allwinner: add SD/MMC host controller
The Allwinner System on Chip families sun4i and above contain
an integrated storage controller for Secure Digital (SD) and
Multi Media Card (MMC) interfaces. This commit adds support
for the Allwinner SD/MMC storage controller with the following
emulated features:
Niek Linnenbank [Wed, 11 Mar 2020 22:18:43 +0000 (23:18 +0100)]
hw/arm/allwinner: add Security Identifier device
The Security Identifier device found in various Allwinner System on Chip
designs gives applications a per-board unique identifier. This commit
adds support for the Allwinner Security Identifier using a 128-bit
UUID value as input.
Niek Linnenbank [Wed, 11 Mar 2020 22:18:42 +0000 (23:18 +0100)]
hw/arm/allwinner: add CPU Configuration module
Various Allwinner System on Chip designs contain multiple processors
that can be configured and reset using the generic CPU Configuration
module interface. This commit adds support for the Allwinner CPU
configuration interface which emulates the following features:
Niek Linnenbank [Wed, 11 Mar 2020 22:18:41 +0000 (23:18 +0100)]
hw/arm/allwinner-h3: add System Control module
The Allwinner H3 System on Chip has an System Control
module that provides system wide generic controls and
device information. This commit adds support for the
Allwinner H3 System Control module.
Niek Linnenbank [Wed, 11 Mar 2020 22:18:40 +0000 (23:18 +0100)]
hw/arm/allwinner-h3: add USB host controller
The Allwinner H3 System on Chip contains multiple USB 2.0 bus
connections which provide software access using the Enhanced
Host Controller Interface (EHCI) and Open Host Controller
Interface (OHCI) interfaces. This commit adds support for
both interfaces in the Allwinner H3 System on Chip.
Niek Linnenbank [Wed, 11 Mar 2020 22:18:39 +0000 (23:18 +0100)]
hw/arm/allwinner-h3: add Clock Control Unit
The Clock Control Unit is responsible for clock signal generation,
configuration and distribution in the Allwinner H3 System on Chip.
This commit adds support for the Clock Control Unit which emulates
a simple read/write register interface.
Niek Linnenbank [Wed, 11 Mar 2020 22:18:38 +0000 (23:18 +0100)]
hw/arm: add Xunlong Orange Pi PC machine
The Xunlong Orange Pi PC is an Allwinner H3 System on Chip
based embedded computer with mainline support in both U-Boot
and Linux. The board comes with a Quad Core Cortex A7 @ 1.3GHz,
1GiB RAM, 100Mbit ethernet, USB, SD/MMC, USB, HDMI and
various other I/O. This commit add support for the Xunlong
Orange Pi PC machine.
Niek Linnenbank [Wed, 11 Mar 2020 22:18:37 +0000 (23:18 +0100)]
hw/arm: add Allwinner H3 System-on-Chip
The Allwinner H3 is a System on Chip containing four ARM Cortex A7
processor cores. Features and specifications include DDR2/DDR3 memory,
SD/MMC storage cards, 10/100/1000Mbit Ethernet, USB 2.0, HDMI and
various I/O modules. This commit adds support for the Allwinner H3
System on Chip.
Igor Mammedov [Tue, 3 Mar 2020 09:12:54 +0000 (04:12 -0500)]
hw/arm/cubieboard: make sure SOC object isn't leaked
SOC object returned by object_new() is leaked in current code.
Set SOC parent explicitly to board and then unref to SOC object
to make sure that refererence returned by object_new() is taken
care of.
The SOC object will be kept alive by its parent (machine) and
will be automatically freed when MachineState is destroyed.
target/arm: Disable clean_data_tbi for system mode
We must include the tag in the FAR_ELx register when raising
an addressing exception. Which means that we should not clear
out the tag during translation.
We cannot at present comply with this for user mode, so we
retain the clean_data_tbi function for the moment, though it
no longer does what it says on the tin for system mode. This
function is to be replaced with MTE, so don't worry about the
slight misnaming.
The Aspeed SMC Controller can operate in different modes : Read, Fast
Read, Write and User modes. When the User mode is configured, it
selects automatically the SPI slave device until the CE_STOP_ACTIVE
bit is set to 1. When any other modes are configured the device is
unselected. The HW logic handles the chip select automatically when
the flash is accessed through its AHB window.
When configuring the CEx Control Register, the User mode logic to
select and unselect the slave is incorrect and data corruption can be
seen on machines using two chips, witherspoon and romulus.
Rework the handler setting the CEx Control Register to fix this issue.
Peter Maydell [Tue, 3 Mar 2020 17:49:49 +0000 (17:49 +0000)]
target/arm: Recalculate hflags correctly after writes to CONTROL
A write to the CONTROL register can change our current EL (by
writing to the nPRIV bit). That means that we can't assume
that s->current_el is still valid in trans_MSR_v7m() when
we try to rebuild the hflags.
Add a new helper rebuild_hflags_m32_newel() which, like the
existing rebuild_hflags_a32_newel(), recalculates the current
EL from scratch, and use it in trans_MSR_v7m().
This fixes an assertion about an hflags mismatch when the
guest changes privilege by writing to CONTROL.
Peter Maydell [Tue, 3 Mar 2020 17:49:48 +0000 (17:49 +0000)]
target/arm: Update hflags in trans_CPS_v7m()
For M-profile CPUs, the FAULTMASK value affects the CPU's MMU index
(it changes the NegPri bit). We update the hflags after calls
to the v7m_msr helper in trans_MSR_v7m() but forgot to do so
in trans_CPS_v7m().
Peter Maydell [Tue, 3 Mar 2020 17:49:47 +0000 (17:49 +0000)]
hw/intc/armv7m_nvic: Rebuild hflags on reset
Some of an M-profile CPU's cached hflags state depends on state that's
in our NVIC object. We already do an hflags rebuild when the NVIC
registers are written, but we also need to do this on NVIC reset,
because there's no guarantee that this will happen before the
CPU reset.
This fixes an assertion due to mismatched hflags which happens if
the CPU is reset from inside a HardFault handler.
Peter Maydell [Thu, 12 Mar 2020 15:20:52 +0000 (15:20 +0000)]
Merge remote-tracking branch 'remotes/pmaydell/tags/pull-docs-20200312' into staging
docs queue:
* Remove some no longer needed texinfo infrastructure
* Reorder the top level index docs to put most useful manuals first
* Split the Arm target-specific info into sub-pages
* Improve the Arm documentation a bit with info previously
only on the wiki page
* remotes/pmaydell/tags/pull-docs-20200312:
docs: Be consistent about capitalization of 'Arm'
docs: Move arm-cpu-features.rst into the system manual
docs/system/target-arm.rst: Add some introductory text
docs/system: Split target-arm.rst into sub-documents
Makefile: Allow for subdirectories in Sphinx manual dependencies
docs/qemu-option-trace.rst.inc: Remove redundant comment
docs/index.rst, docs/index.html.in: Reorder manuals
Makefile: Make all Sphinx documentation depend on the extensions
docs/sphinx/hxtool.py: Remove STEXI/ETEXI support
hxtool: Remove Texinfo generation support
Update comments in .hx files that mention Texinfo
Makefile: Remove redundant Texinfo related code
Peter Maydell [Mon, 9 Mar 2020 21:58:18 +0000 (21:58 +0000)]
docs: Be consistent about capitalization of 'Arm'
The company 'Arm' went through a rebranding some years back
involving a recapitalization from 'ARM' to 'Arm'. As a result
our documentation is a bit inconsistent between the two forms.
It's not worth trying to update everywhere in QEMU, but it's
easy enough to make docs/ consistent.
Note that "ARMv8" and similar architecture names, and
older CPU names like "ARM926" still retain all-caps.
Peter Maydell [Mon, 9 Mar 2020 21:58:17 +0000 (21:58 +0000)]
docs: Move arm-cpu-features.rst into the system manual
Now we have somewhere to put arm-specific rst documentation,
we can move arm-cpu-features.rst from the docs/ top level
directory into docs/system/arm/.
Peter Maydell [Mon, 9 Mar 2020 21:58:16 +0000 (21:58 +0000)]
docs/system/target-arm.rst: Add some introductory text
Now we've moved the various bits of per-board documentation into
their own files, the top level document is a little bare. Add
some introductory information, including a note that many
of the board models we support are currently undocumented.
(Most sections of this new text were originally written by me
for the wiki page https://wiki.qemu.org/Documentation/Platforms/ARM)
Peter Maydell [Mon, 9 Mar 2020 21:58:15 +0000 (21:58 +0000)]
docs/system: Split target-arm.rst into sub-documents
Currently the documentation for Arm system emulator targets is in a
single target-arm.rst. This describes only some of the boards and
often in a fairly abbreviated fashion. Restructure it so that each
board has its own documentation file in the docs/system/arm/
subdirectory.
This will hopefully encourage us to write board documentation that
describes the board in detail, rather than a few brief paragraphs
in a single long page. The table of contents should also help users
to find the board they care about faster.
Once the structure is in place we'll be able to move microvm.rst
from the top-level docs/ directory.
All the text from the old page is retained, except for the final
paragraph ("A Linux 2.6 test image is available on the QEMU web site.
More information is available in the QEMU mailing-list archive."),
which is deleted. The git history shows this was originally added
in reference to the integratorcp board (at that time the only
Arm board that was supported), and has subsequently gradually been
further and further separated from the integratorcp documentation
by the insertion of other board documentation sections. It's
extremely out of date and no longer accurate, since AFAICT there
isn't an integratorcp kernel on the website any more; so better
deleted than retained.
Peter Maydell [Mon, 9 Mar 2020 21:58:14 +0000 (21:58 +0000)]
Makefile: Allow for subdirectories in Sphinx manual dependencies
Currently we put 'docs/foo/*.rst' in the Make list of dependencies
for the Sphinx 'foo' manual, which means all the files must be
in the top level of that manual's directory. We'd like to be
able to have subdirectories inside some of the manuals, so add
'docs/foo/*/*.rst' to the dependencies too.
The Texinfo version of the tracing options documentation has now
been deleted, so we can remove the now-redundant comment at the top
of the rST version that was reminding us that the two should be
kept in sync.
Now that qemu-doc.html is no longer present, the ordering of manuals
within the top-level index page looks a bit odd. Reshuffle so that
the manuals the user is most likely to be interested in are at the
top of the list, and the reference material is at the bottom.
Similarly, we reorder the index.rst file used as the base of
the "all manuals in one" documentation for readthedocs.
The new order is:
* system
* user
* tools
* interop
* specs
* QMP reference (if present)
* Guest agent protocol reference (if present)
* devel (if present)
Peter Maydell [Fri, 6 Mar 2020 17:17:47 +0000 (17:17 +0000)]
Makefile: Make all Sphinx documentation depend on the extensions
Add the Python source files of our Sphinx extensions to the
dependencies of the Sphinx manuals, so that if we edit the
extension source code the manuals get rebuilt.
Adding this dependency unconditionally means that we'll rebuild
a manual even if it happens to not use the extension whose
source file was changed, but this is simpler and less error
prone, and it's unlikely that we'll be making frequent changes
to the extensions.
Peter Maydell [Fri, 6 Mar 2020 17:17:45 +0000 (17:17 +0000)]
hxtool: Remove Texinfo generation support
All the STEXI/ETEXI blocks and the Makfile rules that use them have now
been removed from the codebase. We can remove the code from the hxtool
script which handles the STEXI/ETEXI directives and the '-t' option.
Peter Maydell [Wed, 11 Mar 2020 17:06:40 +0000 (17:06 +0000)]
Merge remote-tracking branch 'remotes/maxreitz/tags/pull-block-2020-03-11' into staging
Block patches for the 5.0 softfreeze:
- qemu-img measure for LUKS
- Improve block-copy's performance by reducing inter-request
dependencies
- Make curl's detection of accept-ranges more robust
- Memleak fixes
- iotest fix
* remotes/maxreitz/tags/pull-block-2020-03-11:
block/block-copy: hide structure definitions
block/block-copy: reduce intersecting request lock
block/block-copy: rename start to offset in interfaces
block/block-copy: refactor interfaces to use bytes instead of end
block/block-copy: factor out find_conflicting_inflight_req
block/block-copy: use block_status
block/block-copy: specialcase first copy_range request
block/block-copy: fix progress calculation
job: refactor progress to separate object
block/qcow2-threads: fix qcow2_decompress
qemu-img: free memory before re-assign
block/qcow2: do free crypto_opts in qcow2_close()
iotests: Fix nonportable use of od --endian
block/curl: HTTP header field names are case insensitive
block/curl: HTTP header fields allow whitespace around values
iotests: add 288 luks qemu-img measure test
qemu-img: allow qemu-img measure --object without a filename
luks: implement .bdrv_measure()
luks: extract qcrypto_block_calculate_payload_offset()
Formatting '/var/tmp/pool_target/vol_1', fmt=luks size=10485760 key-secret=vol_1_encrypt0
qemu-img: /var/tmp/pool_target/vol_1: Data from secret vol_1_encrypt0 is not valid UTF-8
However, the created file '/var/tmp/pool_target/vol_1' is left behind in the
file system after the failure. This behavior can be observed when creating
the volume using Libvirt, via 'virsh vol-create', and then getting "volume
target path already exist" errors when trying to re-create the volume.
The volume file is created inside block_crypto_co_create_opts_luks(), in
block/crypto.c. If the bdrv_create_file() call is successful but any
succeeding step fails*, the existing 'fail' label does not take into
account the created file, leaving it behind.
This patch changes block_crypto_co_create_opts_luks() to delete
'filename' in case of failure. A failure in this point means that
the volume is now truncated/corrupted, so even if 'filename' was an
existing volume before calling qemu-img, it is now unusable. Deleting
the file it is not much worse than leaving it in the filesystem in
this scenario, and we don't have to deal with checking the file
pre-existence in the code.
* in our case, block_crypto_co_create_generic calls qcrypto_block_create,
which calls qcrypto_block_luks_create, and this function fails when
calling qcrypto_secret_lookup_as_utf8.
Using the new 'bdrv_co_delete_file' interface, a pure co_routine function
'bdrv_co_delete_file' inside block.c can can be used in a way similar of
the existing bdrv_create_file to to clean up a created file.
We're creating a pure co_routine because the only caller of
'bdrv_co_delete_file' will be already in co_routine context, thus there
is no need to add all the machinery to check for qemu_in_coroutine() and
create a separated co_routine to do the job.
Adding to Block Drivers the capability of being able to clean up
its created files can be useful in certain situations. For the
LUKS driver, for instance, a failure in one of its authentication
steps can leave files in the host that weren't there before.
This patch adds the 'bdrv_co_delete_file' interface to block
drivers and add it to the 'file' driver in file-posix.c. The
implementation is given by 'raw_co_delete_file'.
Kevin Wolf [Tue, 10 Mar 2020 11:38:30 +0000 (12:38 +0100)]
iotests: Add iothread cases to 155
This patch adds test cases for attaching the backing chain to a mirror
job target right before finalising the job, where the image is in a
non-mainloop AioContext (i.e. the backing chain needs to be moved to the
AioContext of the mirror target).
This requires switching the test case from virtio-blk to virtio-scsi
because virtio-blk only actually starts using the iothreads when the
guest driver initialises the device (which never happens in a test case
without a guest OS). virtio-scsi always keeps its block nodes in the
AioContext of the the requested iothread without guest interaction.
Kevin Wolf [Tue, 10 Mar 2020 11:38:29 +0000 (12:38 +0100)]
block: Fix cross-AioContext blockdev-snapshot
external_snapshot_prepare() tries to move the overlay to the AioContext
of the backing file (the snapshotted node). However, it's possible that
this doesn't work, but the backing file can instead be moved to the
overlay's AioContext (e.g. opening the backing chain for a mirror
target).
bdrv_append() already indirectly uses bdrv_attach_node(), which takes
care to move nodes to make sure they use the same AioContext and which
tries both directions.
So the problem has a simple fix: Just delete the unnecessary extra
bdrv_try_set_aio_context() call in external_snapshot_prepare() and
instead assert in bdrv_append() that both nodes were indeed moved to the
same AioContext.
Kevin Wolf [Tue, 10 Mar 2020 11:38:28 +0000 (12:38 +0100)]
iotests: Test mirror with temporarily disabled target backing file
The newly tested scenario is a common live storage migration scenario:
The target node is opened without a backing file so that the active
layer is mirrored while its backing chain can be copied in the
background.
The backing chain should be attached to the mirror target node when
finalising the job, just before switching the users of the source node
to the new copy (at which point the mirror job still has a reference to
the node). drive-mirror did this automatically, but with blockdev-mirror
this is the job of the QMP client.
This patch adds test cases for two ways to achieve the desired result,
using either x-blockdev-reopen or blockdev-snapshot.
Kevin Wolf [Tue, 10 Mar 2020 11:38:26 +0000 (12:38 +0100)]
block: Relax restrictions for blockdev-snapshot
blockdev-snapshot returned an error if the overlay was already in use,
which it defined as having any BlockBackend parent. This is in fact both
too strict (some parents can tolerate the change of visible data caused
by attaching a backing file) and too loose (some non-BlockBackend
parents may not be happy with it).
One important use case that is prevented by the too strict check is live
storage migration with blockdev-mirror. Here, the target node is
usually opened without a backing file so that the active layer is
mirrored while its backing chain can be copied in the background.
The backing chain should be attached to the mirror target node when
finalising the job, just before switching the users of the source node
to the new copy (at which point the mirror job still has a reference to
the node). drive-mirror did this automatically, but with blockdev-mirror
this is the job of the QMP client, so it needs a way to do this.
blockdev-snapshot is the obvious way, so this patch makes it work in
this scenario. The new condition is that no parent uses CONSISTENT_READ
permissions. This will ensure that the operation will still be blocked
when the node is attached to the guest device, so blockdev-snapshot
remains safe.
(For the sake of completeness, x-blockdev-reopen can be used to achieve
the same, however it is a big hammer, performs the graph change
completely unchecked and is still experimental. So even with the option
of using x-blockdev-reopen, there are reasons why blockdev-snapshot
should be able to perform this operation.)
Pan Nengyuan [Tue, 10 Mar 2020 06:46:40 +0000 (14:46 +0800)]
qom-qmp-cmds: fix two memleaks in qmp_object_add
'type/id' forgot to free in qmp_object_add, this patch fix that.
The leak stack:
Direct leak of 84 byte(s) in 6 object(s) allocated from:
#0 0x7fe2a5ebf768 in __interceptor_malloc (/lib64/libasan.so.5+0xef768)
#1 0x7fe2a5044445 in g_malloc (/lib64/libglib-2.0.so.0+0x52445)
#2 0x7fe2a505dd92 in g_strdup (/lib64/libglib-2.0.so.0+0x6bd92)
#3 0x56344954e692 in qmp_object_add /mnt/sdb/qemu-new/qemu_test/qemu/qom/qom-qmp-cmds.c:258
#4 0x563449960f5a in do_qmp_dispatch /mnt/sdb/qemu-new/qemu_test/qemu/qapi/qmp-dispatch.c:132
#5 0x563449960f5a in qmp_dispatch /mnt/sdb/qemu-new/qemu_test/qemu/qapi/qmp-dispatch.c:175
#6 0x563449498a30 in monitor_qmp_dispatch /mnt/sdb/qemu-new/qemu_test/qemu/monitor/qmp.c:145
#7 0x56344949a64f in monitor_qmp_bh_dispatcher /mnt/sdb/qemu-new/qemu_test/qemu/monitor/qmp.c:234
#8 0x563449a92a3a in aio_bh_call /mnt/sdb/qemu-new/qemu_test/qemu/util/async.c:136
Direct leak of 54 byte(s) in 6 object(s) allocated from:
#0 0x7fe2a5ebf768 in __interceptor_malloc (/lib64/libasan.so.5+0xef768)
#1 0x7fe2a5044445 in g_malloc (/lib64/libglib-2.0.so.0+0x52445)
#2 0x7fe2a505dd92 in g_strdup (/lib64/libglib-2.0.so.0+0x6bd92)
#3 0x56344954e6c4 in qmp_object_add /mnt/sdb/qemu-new/qemu_test/qemu/qom/qom-qmp-cmds.c:267
#4 0x563449960f5a in do_qmp_dispatch /mnt/sdb/qemu-new/qemu_test/qemu/qapi/qmp-dispatch.c:132
#5 0x563449960f5a in qmp_dispatch /mnt/sdb/qemu-new/qemu_test/qemu/qapi/qmp-dispatch.c:175
#6 0x563449498a30 in monitor_qmp_dispatch /mnt/sdb/qemu-new/qemu_test/qemu/monitor/qmp.c:145
#7 0x56344949a64f in monitor_qmp_bh_dispatcher /mnt/sdb/qemu-new/qemu_test/qemu/monitor/qmp.c:234
#8 0x563449a92a3a in aio_bh_call /mnt/sdb/qemu-new/qemu_test/qemu/util/async.c:136
Currently, block_copy operation lock the whole requested region. But
there is no reason to lock clusters, which are already copied, it will
disturb other parallel block_copy requests for no reason.
Let's instead do the following:
Lock only sub-region, which we are going to operate on. Then, after
copying all dirty sub-regions, we should wait for intersecting
requests block-copy, if they failed, we should retry these new dirty
clusters.
block/block-copy: refactor interfaces to use bytes instead of end
We have a lot of "chunk_end - start" invocations, let's switch to
bytes/cur_bytes scheme instead.
While being here, improve check on block_copy_do_copy parameters to not
overflow when calculating nbytes and use int64_t for bytes in
block_copy for consistency.
Use bdrv_block_status_above to chose effective chunk size and to handle
zeroes effectively.
This substitutes checking for just being allocated or not, and drops
old code path for it. Assistance by backup job is dropped too, as
caching block-status information is more difficult than just caching
is-allocated information in our dirty bitmap, and backup job is not
good place for this caching anyway.
block/block-copy: specialcase first copy_range request
In block_copy_do_copy we fallback to read+write if copy_range failed.
In this case copy_size is larger than defined for buffered IO, and
there is corresponding commit. Still, backup copies data cluster by
cluster, and most of requests are limited to one cluster anyway, so the
only source of this one bad-limited request is copy-before-write
operation.
Further patch will move backup to use block_copy directly, than for
cases where copy_range is not supported, first request will be
oversized in each backup. It's not good, let's change it now.
Fix is simple: just limit first copy_range request like buffer-based
request. If it succeed, set larger copy_range limit.
Assume we have two regions, A and B, and region B is in-flight now,
region A is not yet touched, but it is unallocated and should be
skipped.
Correspondingly, as progress we have
total = A + B
current = 0
If we reset unallocated region A and call progress_reset_callback,
it will calculate 0 bytes dirty in the bitmap and call
job_progress_set_remaining, which will set
total = current + 0 = 0 + 0 = 0
So, B bytes are actually removed from total accounting. When job
finishes we'll have
total = 0
current = B
, which doesn't sound good.
This is because we didn't considered in-flight bytes, actually when
calculating remaining, we should have set (in_flight + dirty_bytes)
as remaining, not only dirty_bytes.
To fix it, let's refactor progress calculation, moving it to block-copy
itself instead of fixing callback. And, of course, track in_flight
bytes count.
We still have to keep one callback, to maintain backup job bytes_read
calculation, but it will go on soon, when we turn the whole backup
process into one block_copy call.
On success path we return what inflate() returns instead of 0. And it
most probably works for Z_STREAM_END as it is positive, but is
definitely broken for Z_BUF_ERROR.
While being here, switch to errno return code, to be closer to
qcow2_compress API (and usual expectations).
Revert condition in if to be more positive. Drop dead initialization of
ret.
Pan Nengyuan [Thu, 27 Feb 2020 01:29:50 +0000 (09:29 +0800)]
qemu-img: free memory before re-assign
collect_image_check() is called twice in img_check(), the filename/format will be alloced without free the original memory.
It is not a big deal since the process will exit anyway, but seems like a clean code and it will remove the warning spotted by asan.
Pan Nengyuan [Thu, 27 Feb 2020 01:29:49 +0000 (09:29 +0800)]
block/qcow2: do free crypto_opts in qcow2_close()
'crypto_opts' forgot to free in qcow2_close(), this patch fix the bellow leak stack:
Direct leak of 24 byte(s) in 1 object(s) allocated from:
#0 0x7f0edd81f970 in __interceptor_calloc (/lib64/libasan.so.5+0xef970)
#1 0x7f0edc6d149d in g_malloc0 (/lib64/libglib-2.0.so.0+0x5249d)
#2 0x55d7eaede63d in qobject_input_start_struct /mnt/sdb/qemu-new/qemu_test/qemu/qapi/qobject-input-visitor.c:295
#3 0x55d7eaed78b8 in visit_start_struct /mnt/sdb/qemu-new/qemu_test/qemu/qapi/qapi-visit-core.c:49
#4 0x55d7eaf5140b in visit_type_QCryptoBlockOpenOptions qapi/qapi-visit-crypto.c:290
#5 0x55d7eae43af3 in block_crypto_open_opts_init /mnt/sdb/qemu-new/qemu_test/qemu/block/crypto.c:163
#6 0x55d7eacd2924 in qcow2_update_options_prepare /mnt/sdb/qemu-new/qemu_test/qemu/block/qcow2.c:1148
#7 0x55d7eacd33f7 in qcow2_update_options /mnt/sdb/qemu-new/qemu_test/qemu/block/qcow2.c:1232
#8 0x55d7eacd9680 in qcow2_do_open /mnt/sdb/qemu-new/qemu_test/qemu/block/qcow2.c:1512
#9 0x55d7eacdc55e in qcow2_open_entry /mnt/sdb/qemu-new/qemu_test/qemu/block/qcow2.c:1792
#10 0x55d7eacdc8fe in qcow2_open /mnt/sdb/qemu-new/qemu_test/qemu/block/qcow2.c:1819
#11 0x55d7eac3742d in bdrv_open_driver /mnt/sdb/qemu-new/qemu_test/qemu/block.c:1317
#12 0x55d7eac3e990 in bdrv_open_common /mnt/sdb/qemu-new/qemu_test/qemu/block.c:1575
#13 0x55d7eac4442c in bdrv_open_inherit /mnt/sdb/qemu-new/qemu_test/qemu/block.c:3126
#14 0x55d7eac45c3f in bdrv_open /mnt/sdb/qemu-new/qemu_test/qemu/block.c:3219
#15 0x55d7ead8e8a4 in blk_new_open /mnt/sdb/qemu-new/qemu_test/qemu/block/block-backend.c:397
#16 0x55d7eacde74c in qcow2_co_create /mnt/sdb/qemu-new/qemu_test/qemu/block/qcow2.c:3534
#17 0x55d7eacdfa6d in qcow2_co_create_opts /mnt/sdb/qemu-new/qemu_test/qemu/block/qcow2.c:3668
#18 0x55d7eac1c678 in bdrv_create_co_entry /mnt/sdb/qemu-new/qemu_test/qemu/block.c:485
#19 0x55d7eb0024d2 in coroutine_trampoline /mnt/sdb/qemu-new/qemu_test/qemu/util/coroutine-ucontext.c:115
projects / qemu.git / log