Like all locally-built programs, dlfilters may need to be re-built if
shared libraries they use change. Also there may be unexpected results
if the dfilter uses different versions of the shared libraries that perf
uses.
John Garry [Thu, 29 Jul 2021 13:56:23 +0000 (21:56 +0800)]
perf test: Add more pmu-events uncore aliases
Add more events to cover the scenarios fixed and also inadvertently
broken by commit c47a5599eda324ba ("perf tools: Fix pattern matching for
same substring in different PMU type")
John Garry [Thu, 29 Jul 2021 13:56:22 +0000 (21:56 +0800)]
perf test: Re-add pmu-event uncore PMU alias test
Add support to match aliases for uncore PMUs.
Since we cannot rely on the PMUs being present on the host system, use
fake PMUs.
The following conditions in the test are ensures:
- Expected count of aliases created
- All aliases can be matched to an expected alias in
perf_pmu_test_pmu.aliases
This will catch the condition fixed in commit c47a5599eda3 ("perf tools:
Fix pattern matching for same substring in different PMU type"), where
excess events were created for a PMU. It will also fix the scenario
inadvertently broken there, where no aliases were created for aliases
with multiple tokens.
John Garry [Thu, 29 Jul 2021 13:56:20 +0000 (21:56 +0800)]
perf test: Test pmu-events core aliases separately
The current method to test uncore event aliasing is limited, as it
relies on the uncore PMU being present in the host system to test.
As such, breakages of uncore PMU aliases goes unnoticed. To make this
more robust, a new method of testing uncore PMUs with fake PMUs will be
used in future. This will be separate to testing core PMU aliases.
So make the current test function core PMU only. Uncore PMU alias
support will be re-added later.
perf bench: Add benchmark for evlist open/close operations
This new benchmark finds the total time that is taken to open, mmap,
enable, disable, munmap, close an evlist (time taken for new,
create_maps, config, delete is not counted in).
The evlist can be configured as in perf-record using the
-a,-C,-e,-u,--per-thread,-t,-p options.
The events can be duplicated in the evlist to quickly test performance
with many events using the -n options.
Furthermore, also the number of iterations used to calculate the
statistics is customizable.
Examples:
- Open one dummy event system-wide:
$ sudo ./perf bench internals evlist-open-close
Number of cpus: 4
Number of threads: 1
Number of events: 1 (4 fds)
Number of iterations: 100
Average open-close took: 613.870 usec (+- 32.852 usec)
- Open the group '{cs,cycles}' on CPU 0
$ sudo ./perf bench internals evlist-open-close -e '{cs,cycles}' -C 0
Number of cpus: 1
Number of threads: 1
Number of events: 2 (2 fds)
Number of iterations: 100
Average open-close took: 8503.220 usec (+- 252.652 usec)
- Open 10 'cycles' events for user 0, calculate average over 100 runs
$ sudo ./perf bench internals evlist-open-close -e cycles -n 10 -u 0 -i 100
Number of cpus: 4
Number of threads: 328
Number of events: 10 (13120 fds)
Number of iterations: 100
Average open-close took: 180043.140 usec (+- 2295.889 usec)
Committer notes:
Replaced a deprecated bzero() call with designated initialized zeroing.
Added some missing evlist allocation checks, one noted by Riccardo on
the mailing list.
Since the function auxtrace_mmap__read_snapshot_head() is exactly same
with auxtrace_mmap__read_head(), whether the session is in snapshot mode
or not, it's unified to use function auxtrace_mmap__read_head() for
reading AUX buffer head.
And the function auxtrace_mmap__read_snapshot_head() is unused so this
patch removes it.
Leo Yan [Mon, 9 Aug 2021 11:14:04 +0000 (19:14 +0800)]
perf auxtrace: Drop legacy __sync functions
The main purpose for using __sync built-in functions is to support
compat mode for 32-bit perf with 64-bit kernel. But using these
built-in functions might cause potential issues.
__sync functions originally support Intel Itanium processoer [1] but it
cannot promise to support all 32-bit archs. Now these functions have
become the legacy functions.
Considering __sync functions cannot really fix the 64-bit value
atomicity on 32-bit archs, thus this patch drops __sync functions.
Stephen Brennan [Fri, 6 Aug 2021 20:45:01 +0000 (13:45 -0700)]
perf script python: Fix unintended underline
The text ranging from "subsystem__event_name" to "raw_syscalls__sys_enter()"
is interpreted by asciidoc as a pair of unconstrained text formatting markers.
The result is that the manual page displayed this text as underlined,
and the HTML pages displayed this text as italicized. Escape the first
double-underscore to prevent this.
James Clark [Thu, 5 Aug 2021 13:03:54 +0000 (14:03 +0100)]
perf cs-etm: Add warnings for missing DSOs
Currently decode will silently fail if no binary data is available for
the decode. This is made worse if only partial data is available because
the decode will appear to work, but any trace from that missing DSO will
silently not be generated.
Add a UI popup once if there is any data missing, and then warn in the
bottom left for each individual DSO that's missing.
Davidlohr Bueso [Mon, 9 Aug 2021 04:33:01 +0000 (21:33 -0700)]
perf bench futex, requeue: Add --pi parameter
This extends the program to measure WAIT_REQUEUE_PI+CMP_REQUEUE_PI
pairs, which are the underlying machinery behind priority-inheritance
aware condition variables. The defaults are the same as with the regular
non-pi version, requeueing one task at a time, with the exception that
PI will always wakeup the first waiter.
Davidlohr Bueso [Mon, 9 Aug 2021 04:32:58 +0000 (21:32 -0700)]
perf bench futex: Add --mlockall parameter
This adds, across all futex benchmarks, the -m/--mlockall option
which is a common operation for realtime workloads by not incurring
in page faults in paths that want determinism. As such, threads
started after a call to mlockall(2) will generate page faults
immediately since the new stack is immediately forced to memory,
due to the MCL_FUTURE flag.
Davidlohr Bueso [Mon, 9 Aug 2021 04:32:55 +0000 (21:32 -0700)]
perf bench futex: Group test parameters cleanup
Do this across all futex-bench tests such that all program parameters
neatly share a common structure, which is nicer than how we have them
now. No changes in program behavior are expected.
James Clark [Thu, 29 Jul 2021 15:58:04 +0000 (16:58 +0100)]
perf cs-etm: Improve Coresight zero timestamp warning
Only show the warning if the user hasn't already set timeless mode and
improve the text because there was ambiguity around the meaning of '...'
Change the warning to a UI warning instead of printing straight to
stderr because this corrupts the UI when perf report TUI is used. The UI
warning function also handles printing to stderr when in perf script
mode.
James Clark [Thu, 29 Jul 2021 15:58:03 +0000 (16:58 +0100)]
perf tools: Add flag for tracking warnings of missing DSOs
Auxtrace support may need DSOs for decoding (for example Arm Coresight).
If one of these is missing it would make sense to warn once for each one
that's missing, but not flood the output with every address as there
could be thousands of lookups.
This flag will allow tracking whether a warning was shown for each DSO.
James Clark [Thu, 29 Jul 2021 15:58:02 +0000 (16:58 +0100)]
perf annotate: Add disassembly warnings for annotate --stdio
Currently 'perf annotate --stdio' (and --stdio2) will exit without
printing anything if there are disassembly errors. Apply the same error
handler that's used for TUI and GTK modes. This makes comparing
disassembly across the different modes more consistent.
Setting annotate_warned to true on errors was removed in
commit ee51d851392e ("perf annotate: Introduce strerror for handling
symbol__disassemble() errors") which means when 'perf annotate
--skip-missing' is used warnings are shown multiple times for the same
DSO.
Setting this again restores the original behavior of only one warning
each.
Namhyung Kim [Mon, 19 Jul 2021 22:31:53 +0000 (15:31 -0700)]
perf tools: Add pipe_test.sh to verify pipe operations
It builds a test program and use it to verify pipe behavior with perf
record, inject and report.
$ perf test pipe -v
80: perf pipe recording and injection test :
--- start ---
test child forked, pid 1109301
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.000 MB - ] 11093151109315 -1 |test.file.MGNff
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.000 MB - ]
99.99% test.file.MGNff test.file.MGNffM [.] noploop
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.000 MB - ]
99.99% test.file.MGNff test.file.MGNffM [.] noploop
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.153 MB /tmp/perf.data.dmsnlx (3995 samples) ]
99.99% test.file.MGNff test.file.MGNffM [.] noploop
test child finished with 0
---- end ----
perf pipe recording and injection test: Ok
Namhyung Kim [Mon, 19 Jul 2021 22:31:52 +0000 (15:31 -0700)]
perf inject: Fix output from a file to a pipe
When the input is a regular file but the output is a pipe, it should
write a pipe header. But just repiping would write a portion of the
existing header which is different in 'size' value. So we need to
prevent it and write a new pipe header along with other information
like event attributes and features.
This can handle something like this:
# perf record -a -B sleep 1
# perf inject -b -i perf.data | perf report -i -
Factor out perf_event__synthesize_for_pipe() to be shared between perf
record and inject.
Namhyung Kim [Mon, 19 Jul 2021 22:31:51 +0000 (15:31 -0700)]
perf inject: Fix output from a pipe to a file
Sometimes it needs to save the perf inject data to a file for debugging.
But normally it assumes the same format for input and output, so the end
result cannot be used due to a broken format.
# perf record -a -o - sleep 1 | perf inject -b -o my.data
# perf report -i my.data --stdio
0x208 [0]: failed to process type: 0 [Invalid argument]
Error:
failed to process sample
# To display the perf.data header info, please use --header/--header-only options.
#
In this case, it thought the data has a regular file header since the
output is not a pipe. But actually it doesn't have one and has a pipe
file header. At the end of the session, it tries to rewrite the regular
file header with updated features and it overwrites the data just
follows the pipe header.
Fix it by checking either the input and the output is a pipe.
Namhyung Kim [Mon, 19 Jul 2021 22:31:50 +0000 (15:31 -0700)]
perf tools: Pass a fd to perf_file_header__read_pipe()
Currently it unconditionally writes to stdout for repipe. But perf
inject can direct its output to a regular file. Then it needs to
write the header to the file as well.
Namhyung Kim [Mon, 19 Jul 2021 22:31:49 +0000 (15:31 -0700)]
perf tools: Remove repipe argument from perf_session__new()
The repipe argument is only used by perf inject and the all others
passes 'false'. Let's remove it from the function signature and add
__perf_session__new() to be called from perf inject directly.
This is a preparation of the change the pipe input/output.
Eirik Fuller [Sat, 26 Jun 2021 02:38:25 +0000 (22:38 -0400)]
perf test: Handle fd gaps in test__dso_data_reopen
https://github.com/beaker-project/restraint/issues/215 describes a file
descriptor leak which revealed the test failure described here.
The 'DSO data reopen' perf test assumes that RLIMIT_NOFILE limits the
number of open file descriptors, but it actually limits newly opened
file descriptors. When the file descriptor limit is reduced, file
descriptors already open remain open regardless of the new limit. This
test failure does not occur if open file descriptors are contiguous,
beginning at zero.
The following command triggers this perf test failure.
perf test 'DSO data reopen' 3>/dev/null 8>/dev/null
This patch determines the file descriptor limit by opening four files
and then closing them. The limit is set to the fourth file descriptor,
leaving only the first three available because any newly opened file
descriptor must be less than the limit.
Li Huafei [Mon, 26 Jul 2021 12:38:53 +0000 (20:38 +0800)]
perf env: Normalize aarch64.* and arm64.* to arm64 in normalize_arch()
On my aarch64 big endian machine, the perf annotate does not work.
# perf annotate
Percent | Source code & Disassembly of [kernel.kallsyms] for cycles (253 samples, percent: local period)
--------------------------------------------------------------------------------------------------------------
Percent | Source code & Disassembly of [kernel.kallsyms] for cycles (1 samples, percent: local period)
------------------------------------------------------------------------------------------------------------
Percent | Source code & Disassembly of [kernel.kallsyms] for cycles (47 samples, percent: local period)
-------------------------------------------------------------------------------------------------------------
...
This is because the arch_find() function uses the normalized architecture
name provided by normalize_arch(), and my machine's architecture name
aarch64_be is not normalized to arm64. Like other architectures such as
arm and powerpc, we can fuzzy match the architecture names associated with
aarch64.* and normalize them.
It seems that there is also arm64_be architecture name, which we also
normalize to arm64.
Ian Rogers [Thu, 13 May 2021 06:04:41 +0000 (23:04 -0700)]
perf beauty: Reuse the generic arch errno switch
Previously the code would see if, for example,
tools/perf/arch/arm/include/uapi/asm/errno.h exists and if not generate
a "generic" switch statement using the asm-generic/errno.h.
This creates multiple identical "generic" switch statements before the
default generic switch statement for an unknown architecture.
By simplifying the archlist to be only for architectures that are not
"generic" the amount of generated code can be reduced from 14 down to 6
functions.
Remove the special case of x86, instead reverse the architecture names
so that it comes first.
Committer testing:
$ tools/perf/trace/beauty/arch_errno_names.sh gcc tools > before
Apply this patch and:
$ tools/perf/trace/beauty/arch_errno_names.sh gcc tools > after
14 arches down to 6, that are the ones with an explicit errno.h file:
Ian Rogers [Thu, 15 Jul 2021 01:33:39 +0000 (18:33 -0700)]
perf doc: Remove references to user-manual
Perf doesn't have a user-manual.txt, but git does and this explains why
there are references here. Having these references breaks 'make info' as
user-manual.info can't be created given the missing dependency. Remove
all references to user-manual so that 'make info' can succeed.
Ian Rogers [Thu, 15 Jul 2021 01:33:37 +0000 (18:33 -0700)]
perf doc: Fix perfman.info build
Before this change 'make perfman.info' fails as cat-texi.perl is
missing. It also fails as the makeinfo output isn't written into the
appropriate file. Add cat-texi.perl from git. Add missing output file
flag for makeinfo.
James Clark [Wed, 21 Jul 2021 15:02:02 +0000 (16:02 +0100)]
perf cs-etm: Pass unformatted flag to decoder
The TRBE (Trace Buffer Extension) feature allows a separate trace buffer
for each trace source, therefore the trace wouldn't need to be
formatted. The driver was introduced in commit 3fbf7f011f24
("coresight: sink: Add TRBE driver").
The formatted/unformatted mode is encoded in one of the flags of the
AUX record. The first AUX record encountered for each event is used to
determine the mode, and this will persist for the remaining trace that
is either decoded or dumped.
James Clark [Wed, 21 Jul 2021 15:02:01 +0000 (16:02 +0100)]
perf cs-etm: Use existing decoder instead of resetting it
When dumping trace, the decoder is continually deleted and recreated to
decode each buffer. To support both formatted and unformatted trace in
a later commit, the decoder will be configured in advance.
This commit removes the deletion of the decoder and allows the
formatted/unformatted setting to persist.
James Clark [Wed, 21 Jul 2021 15:02:00 +0000 (16:02 +0100)]
perf cs-etm: Suppress printing when resetting decoder
The decoder is quite noisy when being reset. In a future commit,
dump-raw-trace will use a code path that resets the decoder rather than
creating a new one, so printing has to be suppressed to not flood the
output.
James Clark [Wed, 21 Jul 2021 15:01:59 +0000 (16:01 +0100)]
perf cs-etm: Only setup queues when they are modified
Continually creating queues in cs_etm__process_event() is unnecessary.
They only need to be created when a buffer for a new CPU or thread is
encountered. This can be in two places, when building the queues in
advance in cs_etm__process_auxtrace_info(), or in
cs_etm__process_auxtrace_event() when data_queued is false and the
index wasn't available (pipe mode).
This change will allow the 'formatted' decoder setting to applied when
iterating over aux records in a later commit.
James Clark [Wed, 21 Jul 2021 15:01:58 +0000 (16:01 +0100)]
perf cs-etm: Split setup and timestamp search functions
This refactoring has some benefits:
* Decoding is done to find the timestamp. If we want to print errors
when maps aren't available, then doing it from cs_etm__setup_queue()
may cause warnings to be printed.
* The cs_etm__setup_queue() flow is shared between timed and timeless
modes, so it needs to be guarded by an if statement which can now
be removed.
* Allows moving the setup queues function earlier.
* If data was piped in, then not all queues would be filled so it
wouldn't have worked properly anyway. Now it waits for flush so
data in all queues will be available.
The motivation for this is to decouple setup functions with ones that
involve decoding. That way we can move the setup function earlier when
the formatted/unformatted trace information is available.
James Clark [Wed, 21 Jul 2021 15:01:57 +0000 (16:01 +0100)]
perf cs-etm: Refactor initialisation of kernel start address
The kernel start address is already cached in the machine struct once it
is initialised, so storing it in the cs_etm struct is unnecessary.
It also depends on kernel maps being available to be initialised.
Therefore cs_etm__setup_queues() isn't an appropriate place to call it
because it could be called before processing starts. It would be better
to initialise it at the point when it is needed, then we can be sure
that all the necessary maps are available. Also by calling
machine__kernel_start() multiple times it can be initialised at some
point, even if it failed to initialise previously due to missing maps.
In a later commit cs_etm__setup_queues() will be moved which is the
motivation for this change.
Linus Torvalds [Sun, 1 Aug 2021 19:25:30 +0000 (12:25 -0700)]
Merge tag 'perf-tools-fixes-for-v5.14-2021-08-01' of git://git.kernel.org/pub/scm/linux/kernel/git/acme/linux
Pull perf tools fixes from Arnaldo Carvalho de Melo:
- Revert "perf map: Fix dso->nsinfo refcounting", this makes 'perf top'
abort, uncovering a design flaw on how namespace information is kept.
The fix for that is more than we can do right now, leave it for the
next merge window.
- Split --dump-raw-trace by AUX records for ARM's CoreSight, fixing up
the decoding of some records.
- Fix PMU alias matching.
Thanks to James Clark and John Garry for these fixes.
* tag 'perf-tools-fixes-for-v5.14-2021-08-01' of git://git.kernel.org/pub/scm/linux/kernel/git/acme/linux:
Revert "perf map: Fix dso->nsinfo refcounting"
perf pmu: Fix alias matching
perf cs-etm: Split --dump-raw-trace by AUX records
Linus Torvalds [Sun, 1 Aug 2021 19:18:44 +0000 (12:18 -0700)]
Merge tag 'powerpc-5.14-4' of git://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux
Pull powerpc fixes from Michael Ellerman:
- Don't use r30 in VDSO code, to avoid breaking existing Go lang
programs.
- Change an export symbol to allow non-GPL modules to use spinlocks
again.
Thanks to Paul Menzel, and Srikar Dronamraju.
* tag 'powerpc-5.14-4' of git://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux:
powerpc/vdso: Don't use r30 to avoid breaking Go lang
powerpc/pseries: Fix regression while building external modules
Linus Torvalds [Sun, 1 Aug 2021 19:07:23 +0000 (12:07 -0700)]
Merge tag 'xfs-5.14-fixes-2' of git://git.kernel.org/pub/scm/fs/xfs/xfs-linux
Pull xfs fixes from Darrick Wong:
"This contains a bunch of bug fixes in XFS.
Dave and I have been busy the last couple of weeks to find and fix as
many log recovery bugs as we can find; here are the results so far. Go
fstests -g recoveryloop! ;)
- Fix a number of coordination bugs relating to cache flushes for
metadata writeback, cache flushes for multi-buffer log writes, and
FUA writes for single-buffer log writes
- Fix a bug with incorrect replay of attr3 blocks
- Fix unnecessary stalls when flushing logs to disk
- Fix spoofing problems when recovering realtime bitmap blocks"
* tag 'xfs-5.14-fixes-2' of git://git.kernel.org/pub/scm/fs/xfs/xfs-linux:
xfs: prevent spoofing of rtbitmap blocks when recovering buffers
xfs: limit iclog tail updates
xfs: need to see iclog flags in tracing
xfs: Enforce attr3 buffer recovery order
xfs: logging the on disk inode LSN can make it go backwards
xfs: avoid unnecessary waits in xfs_log_force_lsn()
xfs: log forces imply data device cache flushes
xfs: factor out forced iclog flushes
xfs: fix ordering violation between cache flushes and tail updates
xfs: fold __xlog_state_release_iclog into xlog_state_release_iclog
xfs: external logs need to flush data device
xfs: flush data dev on external log write
Merge tag '5.14-rc3-smb3-fixes' of git://git.samba.org/sfrench/cifs-2.6
Pull cifs fixes from Steve French:
"Three cifs/smb3 fixes, including two for stable, and a fix for an
fallocate problem noticed by Clang"
* tag '5.14-rc3-smb3-fixes' of git://git.samba.org/sfrench/cifs-2.6:
cifs: add missing parsing of backupuid
smb3: rc uninitialized in one fallocate path
SMB3: fix readpage for large swap cache
- mlx5e: RX, avoid possible data corruption w/ relaxed ordering and
LRO
- phy: re-add check for PHY_BRCM_DIS_TXCRXC_NOENRGY on the BCM54811
PHY
- sctp: fix return value check in __sctp_rcv_asconf_lookup
Previous releases - always broken:
- bpf:
- more spectre corner case fixes, introduce a BPF nospec
instruction for mitigating Spectre v4
- fix OOB read when printing XDP link fdinfo
- sockmap: fix cleanup related races
- mac80211: fix enabling 4-address mode on a sta vif after assoc
- can:
- raw: raw_setsockopt(): fix raw_rcv panic for sock UAF
- j1939: j1939_session_deactivate(): clarify lifetime of session
object, avoid UAF
- fix number of identical memory leaks in USB drivers
- tipc:
- do not blindly write skb_shinfo frags when doing decryption
- fix sleeping in tipc accept routine"
* tag 'net-5.14-rc4' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net: (91 commits)
gve: Update MAINTAINERS list
can: esd_usb2: fix memory leak
can: ems_usb: fix memory leak
can: usb_8dev: fix memory leak
can: mcba_usb_start(): add missing urb->transfer_dma initialization
can: hi311x: fix a signedness bug in hi3110_cmd()
MAINTAINERS: add Yasushi SHOJI as reviewer for the Microchip CAN BUS Analyzer Tool driver
bpf: Fix leakage due to insufficient speculative store bypass mitigation
bpf: Introduce BPF nospec instruction for mitigating Spectre v4
sis900: Fix missing pci_disable_device() in probe and remove
net: let flow have same hash in two directions
nfc: nfcsim: fix use after free during module unload
tulip: windbond-840: Fix missing pci_disable_device() in probe and remove
sctp: fix return value check in __sctp_rcv_asconf_lookup
nfc: s3fwrn5: fix undefined parameter values in dev_err()
net/mlx5: Fix mlx5_vport_tbl_attr chain from u16 to u32
net/mlx5e: Fix nullptr in mlx5e_hairpin_get_mdev()
net/mlx5: Unload device upon firmware fatal error
net/mlx5e: Fix page allocation failure for ptp-RQ over SF
net/mlx5e: Fix page allocation failure for trap-RQ over SF
...
Merge tag 'acpi-5.14-rc4' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm
Pull ACPI fixes from Rafael Wysocki:
"These revert a recent IRQ resources handling modification that turned
out to be problematic, fix suspend-to-idle handling on AMD platforms
to take upcoming systems into account properly and fix the retrieval
of the DPTF attributes of the PCH FIVR.
Specifics:
- Revert recent change of the ACPI IRQ resources handling that
attempted to improve the ACPI IRQ override selection logic, but
introduced serious regressions on some systems (Hui Wang).
- Fix up quirks for AMD platforms in the suspend-to-idle support code
so as to take upcoming systems using uPEP HID AMDI007 into account
as appropriate (Mario Limonciello).
- Fix the code retrieving DPTF attributes of the PCH FIVR so that it
agrees on the return data type with the ACPI control method
evaluated for this purpose (Srinivas Pandruvada)"
* tag 'acpi-5.14-rc4' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm:
ACPI: DPTF: Fix reading of attributes
Revert "ACPI: resources: Add checks for ACPI IRQ override"
ACPI: PM: Add support for upcoming AMD uPEP HID AMDI007
Since commit 1b6b26ae7053 ("pipe: fix and clarify pipe write wakeup
logic") we have sanitized the pipe write logic, and would only try to
wake up readers if they needed it.
In particular, if the pipe already had data in it before the write,
there was no point in trying to wake up a reader, since any existing
readers must have been aware of the pre-existing data already. Doing
extraneous wakeups will only cause potential thundering herd problems.
However, it turns out that some Android libraries have misused the EPOLL
interface, and expected "edge triggered" be to "any new write will
trigger it". Even if there was no edge in sight.
Quoting Sandeep Patil:
"The commit 1b6b26ae7053 ('pipe: fix and clarify pipe write wakeup
logic') changed pipe write logic to wakeup readers only if the pipe
was empty at the time of write. However, there are libraries that
relied upon the older behavior for notification scheme similar to
what's described in [1]
One such library 'realm-core'[2] is used by numerous Android
applications. The library uses a similar notification mechanism as GNU
Make but it never drains the pipe until it is full. When Android moved
to v5.10 kernel, all applications using this library stopped working.
The library has since been fixed[3] but it will be a while before all
applications incorporate the updated library"
Our regression rule for the kernel is that if applications break from
new behavior, it's a regression, even if it was because the application
did something patently wrong. Also note the original report [4] by
Michal Kerrisk about a test for this epoll behavior - but at that point
we didn't know of any actual broken use case.
So add the extraneous wakeup, to approximate the old behavior.
[ I say "approximate", because the exact old behavior was to do a wakeup
not for each write(), but for each pipe buffer chunk that was filled
in. The behavior introduced by this change is not that - this is just
"every write will cause a wakeup, whether necessary or not", which
seems to be sufficient for the broken library use. ]
It's worth noting that this adds the extraneous wakeup only for the
write side, while the read side still considers the "edge" to be purely
about reading enough from the pipe to allow further writes.
See commit f467a6a66419 ("pipe: fix and clarify pipe read wakeup logic")
for the pipe read case, which remains that "only wake up if the pipe was
full, and we read something from it".
This makes 'perf top' abort in some cases, and the right fix will
involve surgery that is too much to do at this stage, so revert for now
and fix it in the next merge window.
Merge tag 'block-5.14-2021-07-30' of git://git.kernel.dk/linux-block
Pull block fixes from Jens Axboe:
- gendisk freeing fix (Christoph)
- blk-iocost wake ordering fix (Tejun)
- tag allocation error handling fix (John)
- loop locking fix. While this isn't the prettiest fix in the world,
nobody has any good alternatives for 5.14. Something to likely
revisit for 5.15. (Tetsuo)
* tag 'block-5.14-2021-07-30' of git://git.kernel.dk/linux-block:
block: delay freeing the gendisk
blk-iocost: fix operation ordering in iocg_wake_fn()
blk-mq-sched: Fix blk_mq_sched_alloc_tags() error handling
loop: reintroduce global lock for safe loop_validate_file() traversal
Merge tag 'libata-5.14-2021-07-30' of git://git.kernel.dk/linux-block
Pull libata fixlets from Jens Axboe:
- A fix for PIO highmem (Christoph)
- Kill HAVE_IDE as it's now unused (Lukas)
* tag 'libata-5.14-2021-07-30' of git://git.kernel.dk/linux-block:
arch: Kconfig: clean up obsolete use of HAVE_IDE
libata: fix ata_pio_sector for CONFIG_HIGHMEM
Merge tag 'for-5.14-rc3-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux
Pull btrfs fixes from David Sterba:
- fix -Warray-bounds warning, to help external patchset to make it
default treewide
- fix writeable device accounting (syzbot report)
- fix fsync and log replay after a rename and inode eviction
- fix potentially lost error code when submitting multiple bios for
compressed range
* tag 'for-5.14-rc3-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux:
btrfs: calculate number of eb pages properly in csum_tree_block
btrfs: fix rw device counting in __btrfs_free_extra_devids
btrfs: fix lost inode on log replay after mix of fsync, rename and inode eviction
btrfs: mark compressed range uptodate only if all bio succeed
Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/hid/hid
Pull HID fixes from Jiri Kosina:
- resume timing fix for intel-ish driver (Ye Xiang)
- fix for using incorrect MMIO register in amd_sfh driver (Dylan
MacKenzie)
- Cintiq 24HDT / 27QHDT regression fix and touch processing fix for
Wacom driver (Jason Gerecke)
- device removal bugfix for ft260 driver (Michael Zaidman)
- other small assorted fixes
* 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/hid/hid:
HID: ft260: fix device removal due to USB disconnect
HID: wacom: Skip processing of touches with negative slot values
HID: wacom: Re-enable touch by default for Cintiq 24HDT / 27QHDT
HID: Kconfig: Fix spelling mistake "Uninterruptable" -> "Uninterruptible"
HID: apple: Add support for Keychron K1 wireless keyboard
HID: fix typo in Kconfig
HID: ft260: fix format type warning in ft260_word_show()
HID: amd_sfh: Use correct MMIO register for DMA address
HID: asus: Remove check for same LED brightness on set
HID: intel-ish-hid: use async resume function
Subsystems affected by this patch series: lib, ocfs2, and mm (slub,
migration, and memcg)"
* emailed patches from Andrew Morton <[email protected]>:
mm/memcg: fix NULL pointer dereference in memcg_slab_free_hook()
slub: fix unreclaimable slab stat for bulk free
mm/migrate: fix NR_ISOLATED corruption on 64-bit
mm: memcontrol: fix blocking rstat function called from atomic cgroup1 thresholding code
ocfs2: issue zeroout to EOF blocks
ocfs2: fix zero out valid data
lib/test_string.c: move string selftest in the Runtime Testing menu
Jakub Kicinski [Fri, 30 Jul 2021 17:29:52 +0000 (19:29 +0200)]
Merge tag 'linux-can-fixes-for-5.14-20210730' of git://git.kernel.org/pub/scm/linux/kernel/git/mkl/linux-can
Marc Kleine-Budde says:
====================
pull-request: can 2021-07-30
The first patch is by me and adds Yasushi SHOJI as a reviewer for the
Microchip CAN BUS Analyzer Tool driver.
Dan Carpenter's patch fixes a signedness bug in the hi311x driver.
Pavel Skripkin provides 4 patches, the first targets the mcba_usb
driver by adding the missing urb->transfer_dma initialization, which
was broken in a previous commit. The last 3 patches fix a memory leak
in the usb_8dev, ems_usb and esd_usb2 driver.
* tag 'linux-can-fixes-for-5.14-20210730' of git://git.kernel.org/pub/scm/linux/kernel/git/mkl/linux-can:
can: esd_usb2: fix memory leak
can: ems_usb: fix memory leak
can: usb_8dev: fix memory leak
can: mcba_usb_start(): add missing urb->transfer_dma initialization
can: hi311x: fix a signedness bug in hi3110_cmd()
MAINTAINERS: add Yasushi SHOJI as reviewer for the Microchip CAN BUS Analyzer Tool driver
====================
When kmalloc_node() a large memory, page is allocated, not slab, so when
freeing memory via kfree_rcu(), this large memory should not be used by
memcg_slab_free_hook(), because memcg_slab_free_hook() is is used for
slab.
Using page_objcgs_check() instead of page_objcgs() in
memcg_slab_free_hook() to fix this bug.
SLUB uses page allocator for higher order allocations and update
unreclaimable slab stat for such allocations. At the moment, the bulk
free for SLUB does not share code with normal free code path for these
type of allocations and have missed the stat update. So, fix the stat
update by common code. The user visible impact of the bug is the
potential of inconsistent unreclaimable slab stat visible through
meminfo and vmstat.
Similar to commit 2da9f6305f30 ("mm/vmscan: fix NR_ISOLATED_FILE
corruption on 64-bit") avoid using unsigned int for nr_pages. With
unsigned int type the large unsigned int converts to a large positive
signed long.
Symptoms include CMA allocations hanging forever due to
alloc_contig_range->...->isolate_migratepages_block waiting forever in
"while (unlikely(too_many_isolated(pgdat)))".
Johannes Weiner [Thu, 29 Jul 2021 21:53:44 +0000 (14:53 -0700)]
mm: memcontrol: fix blocking rstat function called from atomic cgroup1 thresholding code
Dan Carpenter reports:
The patch 2d146aa3aa84: "mm: memcontrol: switch to rstat" from Apr
29, 2021, leads to the following static checker warning:
kernel/cgroup/rstat.c:200 cgroup_rstat_flush()
warn: sleeping in atomic context
mm/memcontrol.c
3572 static unsigned long mem_cgroup_usage(struct mem_cgroup *memcg, bool swap)
3573 {
3574 unsigned long val;
3575
3576 if (mem_cgroup_is_root(memcg)) {
3577 cgroup_rstat_flush(memcg->css.cgroup);
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
This is from static analysis and potentially a false positive. The
problem is that mem_cgroup_usage() is called from __mem_cgroup_threshold()
which holds an rcu_read_lock(). And the cgroup_rstat_flush() function
can sleep.
3578 val = memcg_page_state(memcg, NR_FILE_PAGES) +
3579 memcg_page_state(memcg, NR_ANON_MAPPED);
3580 if (swap)
3581 val += memcg_page_state(memcg, MEMCG_SWAP);
3582 } else {
3583 if (!swap)
3584 val = page_counter_read(&memcg->memory);
3585 else
3586 val = page_counter_read(&memcg->memsw);
3587 }
3588 return val;
3589 }
__mem_cgroup_threshold() indeed holds the rcu lock. In addition, the
thresholding code is invoked during stat changes, and those contexts
have irqs disabled as well. If the lock breaking occurs inside the
flush function, it will result in a sleep from an atomic context.
Use the irqsafe flushing variant in mem_cgroup_usage() to fix this.
Junxiao Bi [Thu, 29 Jul 2021 21:53:41 +0000 (14:53 -0700)]
ocfs2: issue zeroout to EOF blocks
For punch holes in EOF blocks, fallocate used buffer write to zero the
EOF blocks in last cluster. But since ->writepage will ignore EOF
pages, those zeros will not be flushed.
This "looks" ok as commit 6bba4471f0cc ("ocfs2: fix data corruption by
fallocate") will zero the EOF blocks when extend the file size, but it
isn't. The problem happened on those EOF pages, before writeback, those
pages had DIRTY flag set and all buffer_head in them also had DIRTY flag
set, when writeback run by write_cache_pages(), DIRTY flag on the page
was cleared, but DIRTY flag on the buffer_head not.
When next write happened to those EOF pages, since buffer_head already
had DIRTY flag set, it would not mark page DIRTY again. That made
writeback ignore them forever. That will cause data corruption. Even
directio write can't work because it will fail when trying to drop pages
caches before direct io, as it found the buffer_head for those pages
still had DIRTY flag set, then it will fall back to buffer io mode.
To make a summary of the issue, as writeback ingores EOF pages, once any
EOF page is generated, any write to it will only go to the page cache,
it will never be flushed to disk even file size extends and that page is
not EOF page any more. The fix is to avoid zero EOF blocks with buffer
write.
The following code snippet from qemu-img could trigger the corruption.
Junxiao Bi [Thu, 29 Jul 2021 21:53:38 +0000 (14:53 -0700)]
ocfs2: fix zero out valid data
If append-dio feature is enabled, direct-io write and fallocate could
run in parallel to extend file size, fallocate used "orig_isize" to
record i_size before taking "ip_alloc_sem", when
ocfs2_zeroout_partial_cluster() zeroout EOF blocks, i_size maybe already
extended by ocfs2_dio_end_io_write(), that will cause valid data zeroed
out.
lib/test_string.c: move string selftest in the Runtime Testing menu
STRING_SELFTEST is presented in the "Library routines" menu. Move it in
Kernel hacking > Kernel Testing and Coverage > Runtime Testing together
with other similar tests found in lib/
--- Runtime Testing
<*> Test functions located in the hexdump module at runtime
<*> Test string functions (NEW)
<*> Test functions located in the string_helpers module at runtime
<*> Test strscpy*() family of functions at runtime
<*> Test kstrto*() family of functions at runtime
<*> Test printf() family of functions at runtime
<*> Test scanf() family of functions at runtime
The arch-specific Kconfig files use HAVE_IDE to indicate if IDE is
supported.
As IDE support and the HAVE_IDE config vanishes with commit b7fb14d3ac63
("ide: remove the legacy ide driver"), there is no need to mention
HAVE_IDE in all those arch-specific Kconfig files.
The issue was identified with ./scripts/checkkconfigsymbols.py.
Pavel Skripkin [Tue, 27 Jul 2021 17:00:46 +0000 (20:00 +0300)]
can: esd_usb2: fix memory leak
In esd_usb2_setup_rx_urbs() MAX_RX_URBS coherent buffers are allocated
and there is nothing, that frees them:
1) In callback function the urb is resubmitted and that's all
2) In disconnect function urbs are simply killed, but URB_FREE_BUFFER
is not set (see esd_usb2_setup_rx_urbs) and this flag cannot be used
with coherent buffers.
So, all allocated buffers should be freed with usb_free_coherent()
explicitly.
Side note: This code looks like a copy-paste of other can drivers. The
same patch was applied to mcba_usb driver and it works nice with real
hardware. There is no change in functionality, only clean-up code for
coherent buffers.
Pavel Skripkin [Tue, 27 Jul 2021 17:00:33 +0000 (20:00 +0300)]
can: ems_usb: fix memory leak
In ems_usb_start() MAX_RX_URBS coherent buffers are allocated and
there is nothing, that frees them:
1) In callback function the urb is resubmitted and that's all
2) In disconnect function urbs are simply killed, but URB_FREE_BUFFER
is not set (see ems_usb_start) and this flag cannot be used with
coherent buffers.
So, all allocated buffers should be freed with usb_free_coherent()
explicitly.
Side note: This code looks like a copy-paste of other can drivers. The
same patch was applied to mcba_usb driver and it works nice with real
hardware. There is no change in functionality, only clean-up code for
coherent buffers.
projects / linux.git / log