Max Reitz [Fri, 10 Jul 2020 16:32:52 +0000 (18:32 +0200)]
iotests: Set LC_ALL=C for sort
Otherwise the result is basically unpredictable.
(Note that the precise environment variable to control sorting order is
LC_COLLATE, but LC_ALL overrides LC_COLLATE, and we do not want the
sorting order to be messed up if LC_ALL is set in the environment.)
Compiling all virtio-gpu objects into a single module isn't a good plan
because the individual objects have different CONFIG_* dependencies.
Leads to module load failures on s390x due to vga support being
disabled, which in turn breaks '-device virtio-gpu-device' (flagged by
travis ci).
So back to the drawing board for modular virtio-gpu ...
Max Filippov [Thu, 9 Jul 2020 14:13:26 +0000 (15:13 +0100)]
tests/docker: update toolchain set in debian-xtensa-cross
Switch to the prebuilt xtensa toolchains release 2020.07.
Drop csp toolchain as the csp core is not a part of QEMU.
Add de233_fpu and dsp3400 toolchains to enable DFPU and FPU2000 tests.
Alex Bennée [Thu, 9 Jul 2020 14:13:25 +0000 (15:13 +0100)]
tests/docker: fall back more gracefully when pull fails
I only spotted this in the small window between my testing with my
registry while waiting for the gitlab PR to go in. As we pre-pull the
registry image we know if that fails there isn't any point attempting
to use the cache. Fall back to the way we used to do it at that point.
In 06c4cc3660b3, we split the multiplication in two parts to avoid
a clang warning. But because double still rounds to 53 bits, this
does not provide additional precision beyond multiplication by
nextafter(0x1p64, 0), the largest representable value smaller
than 2**64.
However, since we have eliminated 1.0, mutiplying by 2**64 produces
a better distribution of input values to the output values.
Since the seed must be non-zero, subtracting 1 means puts the
rate in 0..UINT64_MAX-1, which allows the 0 and UINT64_MAX
thresholds to corrspond to 0% (never) and 100% (always).
Thomas Huth [Mon, 8 Jun 2020 11:40:49 +0000 (13:40 +0200)]
travis.yml: Test also the other targets on s390x
s390x is our only big endian host in our CI, so building and testing QEMU
there is quite valuable. Thus let's also test the other targets with
additional jobs (also using different sets of pre-installed libraries to
get a better coverage of the things that we test).
Alex Bennée [Wed, 1 Jul 2020 13:56:50 +0000 (14:56 +0100)]
containers.yml: build with docker.py tooling
Instead of building the docker files directly use the same docker.py
scripting as we do for building locally. This should help ensure we
use the exact same steps and allow us to cache properly when building
locally.
To get this working you have to have a fairly recent docker binary
otherwise you will see the error message:
=> ERROR importing cache manifest from registry.gitlab....
So far docker 19.03.12 works (from the docker apt repos) but 18.09.1,
build 4c52b90 which is packaged in Debian Buster fails.
tests: improve performance of device-introspect-test
Total execution time with "-m slow" and x86_64 QEMU, drops from 3
minutes 15 seconds, down to 54 seconds.
Individual tests drop from 17-20 seconds, down to 3-4 seconds.
The cost of this change is that any QOM bugs resulting in the test
failure will not be directly associated with the device that caused
the failure. The test case is not frequently identifying such bugs
though, and the cause is likely easily visible in the patch series
that causes the failure. So overall the shorter running time is
considered the more important factor.
Alex Bennée [Wed, 1 Jul 2020 13:56:45 +0000 (14:56 +0100)]
linux-user/elfload: use MAP_FIXED_NOREPLACE in pgb_reserved_va
Given we assert the requested address matches what we asked we should
also make that clear in the mmap flags. Otherwise we see failures in
the GitLab environment for some currently unknown but allowable
reason. We use MAP_FIXED_NOREPLACE if we can so we don't just clobber
an existing mapping. Also include the strerror string for a bit more
info on failure.
Alex Bennée [Wed, 1 Jul 2020 13:56:44 +0000 (14:56 +0100)]
tests/docker: add a linux-user testing focused image
We happily use all the cross images for both cross-building QEMU as
well as building the linux-user tests. However calling docker from
within docker seems not to work. As we can build in Debian anyway why
not include an image that has all the compilers available for
non-docker invocation.
Alex Bennée [Wed, 1 Jul 2020 13:56:43 +0000 (14:56 +0100)]
tests/tcg: add more default compilers to configure.sh
We were missing a bunch of compilers which we could use if they were
locally installed. The defaults are based on Debian as they seem to be
the best distro for well distributed cross-build compilers.
Alex Bennée [Wed, 1 Jul 2020 13:56:42 +0000 (14:56 +0100)]
gitlab: add acceptance testing to system builds
As part of migrating things from Travis to GitLab add the acceptance
tests. To do this:
- rename system1 to system-ubuntu-main
- rename system2 to system-fedora-misc
- split into build/check/acceptance
- remove -j from check stages
- use artifacts to save build stage
- add post acceptance template and use
Alex Bennée [Wed, 1 Jul 2020 13:56:40 +0000 (14:56 +0100)]
tests/acceptance: skip multicore mips_malta tests on GitLab
For some reason these tests fail all the time on GitLab. I can
re-create the hang around 3% of the time locally but it doesn't seem
to be MTTCG related. For now skipIf on GITLAB_CI.
Alex Bennée [Wed, 1 Jul 2020 13:56:36 +0000 (14:56 +0100)]
tests/docker: add --registry support to tooling
This allows us to point the tools towards a registry from which they
can grab pre-built layers instead of doing everything from scratch
each time. To enable this we need to be using the DOCKER_BUILDKIT
engine.
Alex Bennée [Wed, 1 Jul 2020 13:56:35 +0000 (14:56 +0100)]
gitlab: build containers with buildkit and metadata
According to the documentation to be able to use --cache-from for
remote registries you need to enable both buildkit and inline the
metadata. We want to do this to support pulling from gitlab when users
build their local docker images.
gitlab: convert jobs to use custom built containers
Now that we're building standard container images from
dockerfiles in tests/docker/dockerfiles, we can convert
the build jobs to use them. The key benefit of this is
that a contributor can now more easily replicate the CI
environment on their local machine. The container images
are cached too, so we are not spending time waiting for
the apt-get/dnf package installs to complete.
We have a number of container images in tests/docker/dockerfiles
that are intended to provide well defined environments for doing
test builds. We want our CI system to use these containers too.
This introduces builds of all of them as the first stage in the
CI, so that the built containers are available for later build
jobs. The containers are setup to use the GitLab container
registry as the cache, so we only pay the penalty of the full
build when the dockerfiles change. The main qemu-project/qemu
repo is used as a second cache, so that users forking QEMU will
see a fast turnaround time on their CI jobs.
gitlab: introduce explicit "container" and "build" stages
If no stage is listed, jobs get put in an implicit "test" stage.
Some jobs which create container images to be used by later stages
are currently listed as in a "build" stages.
Alex Bennée [Wed, 1 Jul 2020 13:56:29 +0000 (14:56 +0100)]
tests/docker: change tag naming scheme of our images
We've been misusing the tag naming scheme for some time by overloading
the post : section with the image type. Really it should be saved for
the revision of that particular build. Move the details to the other
side so we have:
qemu/image-name
with the implied :latest version added by the tooling.
Alex Bennée [Wed, 1 Jul 2020 13:56:27 +0000 (14:56 +0100)]
tests/vm: allow us to take advantage of MTTCG
We currently limit TCG guests to -smp 1 but now we have added some
aarch64 guests we can do better when running on x86_64 hardware.
Raise the limit for TCG guests when it is safe to do so.
Alex Bennée [Wed, 1 Jul 2020 13:56:26 +0000 (14:56 +0100)]
tests/vm: switch from optsparse to argparse
optparse has been deprecated since version 3.2 and argparse is the
blessed replacement. Take the opportunity to enhance our help output
showing defaults when called.
Robert Foley [Wed, 1 Jul 2020 13:56:25 +0000 (14:56 +0100)]
tests/vm: Add workaround to consume console
This adds support to basevm.py so that we always
drain the console chars. This makes use of
support added in an earlier commit that allows
QEMUMachine to use the ConsoleSocket.
This is a workaround we found was needed since
there is a known issue where QEMU will hang waiting
for console characters to be consumed.
We also added the option of logging the console to a file.
LOG_CONSOLE=1 will now log the output to a file.
Robert Foley [Wed, 1 Jul 2020 13:56:24 +0000 (14:56 +0100)]
python/qemu: Add ConsoleSocket for optional use in QEMUMachine
We add the ConsoleSocket object, which has a socket interface
and which will consume all arriving characters on the
socket, placing them into an in memory buffer.
This will also provide those chars via recv() as
would a regular socket.
ConsoleSocket also has the option of dumping
the console bytes to a log file.
We also give QEMUMachine the option of using ConsoleSocket
to drain and to use for logging console to a file.
By default QEMUMachine does not use ConsoleSocket.
This is added in preparation for use by basevm.py in a later commit.
This is a workaround we found was needed for basevm.py since
there is a known issue where QEMU will hang waiting
for console characters to be consumed.
Robert Foley [Wed, 1 Jul 2020 13:56:23 +0000 (14:56 +0100)]
tests/vm: change scripts to use self._config
This change converts existing scripts to using for example self.ROOT_PASS,
to self._config['root_pass'].
We made similar changes for GUEST_USER, and GUEST_PASS.
This allows us also to remove the change in basevm.py,
which adds __getattr__ for backwards compatibility.
Robert Foley [Wed, 1 Jul 2020 13:56:21 +0000 (14:56 +0100)]
tests/vm: Added a new script for ubuntu.aarch64.
ubuntu.aarch64 provides a script to create an Ubuntu 18.04 VM.
Another new file is also added aarch64vm.py, which is a module with
common methods used by aarch64 VMs, such as how to create the
flash images.
Robert Foley [Wed, 1 Jul 2020 13:56:20 +0000 (14:56 +0100)]
tests/vm: Add common Ubuntu python module
Add a common Ubuntu python module and make use of
it with the ubuntu.i386 script.
This is preparation for adding an Ubuntu script
ubuntu.aarch64. Splitting out the common
logic such as build_image() will reduce duplication.
Robert Foley [Wed, 1 Jul 2020 13:56:19 +0000 (14:56 +0100)]
tests/vm: Added configuration file support
Changes to tests/vm/basevm.py to allow accepting a configuration file
as a parameter. Allows for specifying VM options such as
cpu, machine, memory, and arbitrary qemu arguments for specifying options
such as NUMA configuration.
Also added an example conf_example_aarch64.yml and conf_example_x86.yml.
Robert Foley [Wed, 1 Jul 2020 13:56:18 +0000 (14:56 +0100)]
tests/vm: Add configuration to basevm.py
Added use of a configuration to tests/vm/basevm.py.
The configuration provides parameters used to configure a VM.
This allows for providing alternate configurations to the VM being
created/launched. cpu, machine, memory, and NUMA configuration are all
examples of configuration which we might want to vary on the VM being created
or launched.
This will for example allow for creating an aarch64 vm.
Robert Foley [Wed, 1 Jul 2020 13:56:17 +0000 (14:56 +0100)]
tests/vm: pass args through to BaseVM's __init__
Adding the args parameter to BaseVM's __init__.
We will shortly need to pass more parameters to the class
so let's just pass args rather than growing the parameter list.
Robert Foley [Wed, 1 Jul 2020 13:56:16 +0000 (14:56 +0100)]
util/coroutine: Cleanup start_switch_fiber_ for TSAN.
This is a cleanup patch to follow-up the patch which introduced TSAN.
This patch makes separate start_switch_fiber_ functions for TSAN and ASAN.
This does two things:
1. Unrelated ASAN and TSAN code is separate and each function only
has arguments that are actually needed.
2. The co->tsan_caller_fiber and co->tsan_co_fiber fields are only
access from within #ifdef CONFIG_TSAN.
* remotes/armbru/tags/pull-error-2020-07-07-v2: (53 commits)
xen: Use ERRP_GUARD()
nbd: Use ERRP_GUARD()
virtio-9p: Use ERRP_GUARD()
fw_cfg: Use ERRP_GUARD()
pflash: Use ERRP_GUARD()
sd: Use ERRP_GUARD()
scripts: Coccinelle script to use ERRP_GUARD()
error: New macro ERRP_GUARD()
hmp: Ignore Error objects where the return value suffices
qdev: Ignore Error objects where the return value suffices
qemu-img: Ignore Error objects where the return value suffices
error: Avoid error_propagate() after migrate_add_blocker()
qapi: Purge error_propagate() from QAPI core
qapi: Smooth visitor error checking in generated code
qapi: Smooth another visitor error checking pattern
block/parallels: Simplify parallels_open() after previous commit
error: Reduce unnecessary error propagation
error: Eliminate error_propagate() manually
error: Eliminate error_propagate() with Coccinelle, part 2
error: Eliminate error_propagate() with Coccinelle, part 1
...
If we want to check error after errp-function call, we need to
introduce local_err and then propagate it to errp. Instead, use
the ERRP_GUARD() macro, benefits are:
1. No need of explicit error_propagate call
2. No need of explicit local_err variable: use errp directly
3. ERRP_GUARD() leaves errp as is if it's not NULL or
&error_fatal, this means that we don't break error_abort
(we'll abort on error_set, not on error_propagate)
If we want to add some info to errp (by error_prepend() or
error_append_hint()), we must use the ERRP_GUARD() macro.
Otherwise, this info will not be added when errp == &error_fatal
(the program will exit prior to the error_append_hint() or
error_prepend() call). No such cases are being fixed here.
If we want to check error after errp-function call, we need to
introduce local_err and then propagate it to errp. Instead, use
the ERRP_GUARD() macro, benefits are:
1. No need of explicit error_propagate call
2. No need of explicit local_err variable: use errp directly
3. ERRP_GUARD() leaves errp as is if it's not NULL or
&error_fatal, this means that we don't break error_abort
(we'll abort on error_set, not on error_propagate)
If we want to add some info to errp (by error_prepend() or
error_append_hint()), we must use the ERRP_GUARD() macro.
Otherwise, this info will not be added when errp == &error_fatal
(the program will exit prior to the error_append_hint() or
error_prepend() call). Fix several such cases, e.g. in nbd_read().
If we want to check error after errp-function call, we need to
introduce local_err and then propagate it to errp. Instead, use
the ERRP_GUARD() macro, benefits are:
1. No need of explicit error_propagate call
2. No need of explicit local_err variable: use errp directly
3. ERRP_GUARD() leaves errp as is if it's not NULL or
&error_fatal, this means that we don't break error_abort
(we'll abort on error_set, not on error_propagate)
If we want to add some info to errp (by error_prepend() or
error_append_hint()), we must use the ERRP_GUARD() macro.
Otherwise, this info will not be added when errp == &error_fatal
(the program will exit prior to the error_append_hint() or
error_prepend() call). Fix such a case in
v9fs_device_realize_common().
If we want to check error after errp-function call, we need to
introduce local_err and then propagate it to errp. Instead, use
the ERRP_GUARD() macro, benefits are:
1. No need of explicit error_propagate call
2. No need of explicit local_err variable: use errp directly
3. ERRP_GUARD() leaves errp as is if it's not NULL or
&error_fatal, this means that we don't break error_abort
(we'll abort on error_set, not on error_propagate)
If we want to add some info to errp (by error_prepend() or
error_append_hint()), we must use the ERRP_GUARD() macro.
Otherwise, this info will not be added when errp == &error_fatal
(the program will exit prior to the error_append_hint() or
error_prepend() call). No such cases are being fixed here.
Reported-by: Kevin Wolf <[email protected]> Reported-by: Greg Kurz <[email protected]> Signed-off-by: Vladimir Sementsov-Ogievskiy <[email protected]> Reviewed-by: Philippe Mathieu-Daudé <[email protected]>
[Commit message tweaked] Signed-off-by: Markus Armbruster <[email protected]>
Message-Id: <20200707165037.1026246[email protected]>
[ERRP_AUTO_PROPAGATE() renamed to ERRP_GUARD(), and
auto-propagated-errp.cocci to errp-guard.cocci. Commit message
tweaked again. Coccinelle script rerun for commit 3203148917
"hw/nvram/fw_cfg: Add the FW_CFG_DATA_GENERATOR interface"]
If we want to check error after errp-function call, we need to
introduce local_err and then propagate it to errp. Instead, use
the ERRP_GUARD() macro, benefits are:
1. No need of explicit error_propagate call
2. No need of explicit local_err variable: use errp directly
3. ERRP_GUARD() leaves errp as is if it's not NULL or
&error_fatal, this means that we don't break error_abort
(we'll abort on error_set, not on error_propagate)
If we want to add some info to errp (by error_prepend() or
error_append_hint()), we must use the ERRP_GUARD() macro.
Otherwise, this info will not be added when errp == &error_fatal
(the program will exit prior to the error_append_hint() or
error_prepend() call). No such cases are being fixed here.
If we want to check error after errp-function call, we need to
introduce local_err and then propagate it to errp. Instead, use
the ERRP_GUARD() macro, benefits are:
1. No need of explicit error_propagate call
2. No need of explicit local_err variable: use errp directly
3. ERRP_GUARD() leaves errp as is if it's not NULL or
&error_fatal, this means that we don't break error_abort
(we'll abort on error_set, not on error_propagate)
If we want to add some info to errp (by error_prepend() or
error_append_hint()), we must use the ERRP_GUARD() macro.
Otherwise, this info will not be added when errp == &error_fatal
(the program will exit prior to the error_append_hint() or
error_prepend() call). No such cases are being fixed here.
Introduce a new ERRP_GUARD() macro, to be used at start of functions
with an errp OUT parameter.
It has three goals:
1. Fix issue with error_fatal and error_prepend/error_append_hint: the
user can't see this additional information, because exit() happens in
error_setg earlier than information is added. [Reported by Greg Kurz]
2. Fix issue with error_abort and error_propagate: when we wrap
error_abort by local_err+error_propagate, the resulting coredump will
refer to error_propagate and not to the place where error happened.
(the macro itself doesn't fix the issue, but it allows us to [3.] drop
the local_err+error_propagate pattern, which will definitely fix the
issue) [Reported by Kevin Wolf]
3. Drop local_err+error_propagate pattern, which is used to workaround
void functions with errp parameter, when caller wants to know resulting
status. (Note: actually these functions could be merely updated to
return int error code).
To achieve these goals, later patches will add invocations
of this macro at the start of functions with either use
error_prepend/error_append_hint (solving 1) or which use
local_err+error_propagate to check errors, switching those
functions to use *errp instead (solving 2 and 3).
Signed-off-by: Vladimir Sementsov-Ogievskiy <[email protected]> Reviewed-by: Paul Durrant <[email protected]> Reviewed-by: Greg Kurz <[email protected]> Reviewed-by: Eric Blake <[email protected]>
[Merge comments properly with recent commit "error: Document Error API
usage rules", and edit for clarity. Put ERRP_AUTO_PROPAGATE() before
its helpers, and touch up style. Tweak commit message.] Signed-off-by: Markus Armbruster <[email protected]>
Message-Id: <20200707165037.1026246[email protected]>
[Rename ERRP_AUTO_PROPAGATE() to ERRP_GUARD(), tweak commit message
again]
hmp: Ignore Error objects where the return value suffices
qdev_print_props() receives and throws away Error objects just to
check for object_property_get_str() and object_property_print()
failure. Unnecessary, both return suitable values, so use those
instead.
When all we do with an Error we receive into a local variable is
propagating to somewhere else, we can just as well receive it there
right away, even when we need to keep error_propagate() for other
error paths.
When all we do with an Error we receive into a local variable is
propagating to somewhere else, we can just as well receive it there
right away. The previous two commits did that for sufficiently simple
cases with Coccinelle. Do it for several more manually.
error: Eliminate error_propagate() with Coccinelle, part 2
When all we do with an Error we receive into a local variable is
propagating to somewhere else, we can just as well receive it there
right away. The previous commit did that with a Coccinelle script I
consider fairly trustworthy. This commit uses the same script with
the matching of return taken out, i.e. we convert
if (!foo(..., &err)) {
...
error_propagate(errp, err);
...
}
to
if (!foo(..., errp)) {
...
...
}
This is unsound: @err could still be read between afterwards. I don't
know how to express "no read of @err without an intervening write" in
Coccinelle. Instead, I manually double-checked for uses of @err.
Suboptimal line breaks tweaked manually. qdev_realize() simplified
further to placate scripts/checkpatch.pl.
error: Eliminate error_propagate() with Coccinelle, part 1
When all we do with an Error we receive into a local variable is
propagating to somewhere else, we can just as well receive it there
right away. Convert
@rule1 forall@
identifier fun, err, errp, lbl;
expression list args, args2;
binary operator op;
constant c1, c2;
symbol false;
@@
if (
(
- fun(args, &err, args2)
+ fun(args, errp, args2)
|
- !fun(args, &err, args2)
+ !fun(args, errp, args2)
|
- fun(args, &err, args2) op c1
+ fun(args, errp, args2) op c1
)
)
{
... when != err
when != lbl:
when strict
- error_propagate(errp, err);
... when != err
(
return;
|
return c2;
|
return false;
)
}
@rule2 forall@
identifier fun, err, errp, lbl;
expression list args, args2;
expression var;
binary operator op;
constant c1, c2;
symbol false;
@@
- var = fun(args, &err, args2);
+ var = fun(args, errp, args2);
... when != err
if (
(
var
|
!var
|
var op c1
)
)
{
... when != err
when != lbl:
when strict
- error_propagate(errp, err);
... when != err
(
return;
|
return c2;
|
return false;
|
return var;
)
}
@depends on rule1 || rule2@
identifier err;
@@
- Error *err = NULL;
... when != err
Not exactly elegant, I'm afraid.
The "when != lbl:" is necessary to avoid transforming
if (fun(args, &err)) {
goto out
}
...
out:
error_propagate(errp, err);
even though other paths to label out still need the error_propagate().
For an actual example, see sclp_realize().
Without the "when strict", Coccinelle transforms vfio_msix_setup(),
incorrectly. I don't know what exactly "when strict" does, only that
it helps here.
The match of return is narrower than what I want, but I can't figure
out how to express "return where the operand doesn't use @err". For
an example where it's too narrow, see vfio_intx_enable().
Silently fails to convert hw/arm/armsse.c, because Coccinelle gets
confused by ARMSSE being used both as typedef and function-like macro
there. Converted manually.
Line breaks tidied up manually. One nested declaration of @local_err
deleted manually. Preexisting unwanted blank line dropped in
hw/riscv/sifive_e.c.
Fails to convert hw/arm/armsse.c, because Coccinelle gets confused by
ARMSSE being used both as typedef and function-like macro there.
Convert manually.
Chokes on hw/arm/musicpal.c's lcd_refresh() with the unhelpful error
message "no position information". Convert that one manually.
Fails to convert hw/arm/armsse.c, because Coccinelle gets confused by
ARMSSE being used both as typedef and function-like macro there.
Convert manually.
Fails to convert hw/rx/rx-gdbsim.c, because Coccinelle gets confused
by RXCPU being used both as typedef and function-like macro there.
Convert manually. The other files using RXCPU that way don't need
conversion.
s390x/pci: Fix harmless mistake in zpci's property fid's setter
s390_pci_set_fid() sets zpci->fid_defined to true even when
visit_type_uint32() failed. Reproducer: "-device zpci,fid=junk".
Harmless in practice, because qdev_device_add() then fails, throwing
away @zpci. Fix it anyway.
qapi: Use returned bool to check for failure, manual part
The previous commit used Coccinelle to convert from checking the Error
object to checking the return value. Convert a few more manually.
Also tweak control flow in places to conform to the conventional "if
error bail out" pattern.
block: Avoid error accumulation in bdrv_img_create()
When creating an image fails because the format doesn't support option
"backing_file" or "backing_fmt", bdrv_img_create() first has
qemu_opt_set() put a generic error into @local_err, then puts the real
error into @errp with error_setg(), and then propagates the former to
the latter, which throws away the generic error. A bit complicated,
but works.
Now that qemu_opt_set() returns a useful value, we can simply ignore
the generic error instead.
qemu-option: Replace opt_set() by cleaner opt_validate()
opt_set() frees its argument @value on failure. Slightly unclean;
functions ideally do nothing on failure.
To tidy this up, move opt_create() from opt_set() into its callers,
along with the cleanup. Rename opt_set() to opt_validate(), noting
its similarity to qemu_opts_validate(). Drop redundant parameter
@opts; use opt->opts instead.
projects / qemu.git / log