doc.2021.01.06a: Documentation updates.
fixes.2021.01.04b: Miscellaneous fixes.
kfree_rcu.2021.01.04a: kfree_rcu() updates.
mmdumpobj.2021.01.22a: Dump allocation point for memory blocks.
nocb.2021.01.06a: RCU callback offload updates and cblist segment lengths.
rt.2021.01.04a: Real-time updates.
stall.2021.01.06a: RCU CPU stall warning updates.
torture.2021.01.12a: Torture-test updates and polling SRCU grace-period API.
tortureall.2021.01.06a: Torture-test script updates.
RCU's grace-period guarantee allows updaters to wait for the completion
of all pre-existing RCU read-side critical sections. An RCU read-side
--------critical section begins with the marker ``rcu_read_lock()`` and ends
--------with the marker ``rcu_read_unlock()``. These markers may be nested, and
++++++++critical section begins with the marker rcu_read_lock() and ends
++++++++with the marker rcu_read_unlock(). These markers may be nested, and
RCU treats a nested set as one big RCU read-side critical section.
--------Production-quality implementations of ``rcu_read_lock()`` and
--------``rcu_read_unlock()`` are extremely lightweight, and in fact have
++++++++Production-quality implementations of rcu_read_lock() and
++++++++rcu_read_unlock() are extremely lightweight, and in fact have
exactly zero overhead in Linux kernels built for production use with
--------``CONFIG_PREEMPT=n``.
++++++++``CONFIG_PREEMPTION=n``.
This guarantee allows ordering to be enforced with extremely low
overhead to readers, for example:
15 WRITE_ONCE(y, 1);
16 }
--------Because the ``synchronize_rcu()`` on line 14 waits for all pre-existing
--------readers, any instance of ``thread0()`` that loads a value of zero from
--------``x`` must complete before ``thread1()`` stores to ``y``, so that
++++++++Because the synchronize_rcu() on line 14 waits for all pre-existing
++++++++readers, any instance of thread0() that loads a value of zero from
++++++++``x`` must complete before thread1() stores to ``y``, so that
instance must also load a value of zero from ``y``. Similarly, any
--------instance of ``thread0()`` that loads a value of one from ``y`` must have
--------started after the ``synchronize_rcu()`` started, and must therefore also
++++++++instance of thread0() that loads a value of one from ``y`` must have
++++++++started after the synchronize_rcu() started, and must therefore also
load a value of one from ``x``. Therefore, the outcome:
::
+-----------------------------------------------------------------------+
| Wait a minute! You said that updaters can make useful forward |
| progress concurrently with readers, but pre-existing readers will |
--------| block ``synchronize_rcu()``!!! |
++++++++| block synchronize_rcu()!!! |
| Just who are you trying to fool??? |
+-----------------------------------------------------------------------+
| **Answer**: |
+-----------------------------------------------------------------------+
| First, if updaters do not wish to be blocked by readers, they can use |
--------| ``call_rcu()`` or ``kfree_rcu()``, which will be discussed later. |
--------| Second, even when using ``synchronize_rcu()``, the other update-side |
++++++++| call_rcu() or kfree_rcu(), which will be discussed later. |
++++++++| Second, even when using synchronize_rcu(), the other update-side |
| code does run concurrently with readers, whether pre-existing or not. |
+-----------------------------------------------------------------------+
29 WRITE_ONCE(state, STATE_NORMAL);
30 }
--------The RCU read-side critical section in ``do_something_dlm()`` works with
--------the ``synchronize_rcu()`` in ``start_recovery()`` to guarantee that
--------``do_something()`` never runs concurrently with ``recovery()``, but with
--------little or no synchronization overhead in ``do_something_dlm()``.
++++++++The RCU read-side critical section in do_something_dlm() works with
++++++++the synchronize_rcu() in start_recovery() to guarantee that
++++++++do_something() never runs concurrently with recovery(), but with
++++++++little or no synchronization overhead in do_something_dlm().
+-----------------------------------------------------------------------+
| **Quick Quiz**: |
+-----------------------------------------------------------------------+
--------| Why is the ``synchronize_rcu()`` on line 28 needed? |
++++++++| Why is the synchronize_rcu() on line 28 needed? |
+-----------------------------------------------------------------------+
| **Answer**: |
+-----------------------------------------------------------------------+
| Without that extra grace period, memory reordering could result in |
--------| ``do_something_dlm()`` executing ``do_something()`` concurrently with |
--------| the last bits of ``recovery()``. |
++++++++| do_something_dlm() executing do_something() concurrently with |
++++++++| the last bits of recovery(). |
+-----------------------------------------------------------------------+
In order to avoid fatal problems such as deadlocks, an RCU read-side
--------critical section must not contain calls to ``synchronize_rcu()``.
++++++++critical section must not contain calls to synchronize_rcu().
Similarly, an RCU read-side critical section must not contain anything
that waits, directly or indirectly, on completion of an invocation of
--------``synchronize_rcu()``.
++++++++synchronize_rcu().
Although RCU's grace-period guarantee is useful in and of itself, with
`quite a few use cases <https://lwn.net/Articles/573497/>`__, it would
be good to be able to use RCU to coordinate read-side access to linked
data structures. For this, the grace-period guarantee is not sufficient,
--------as can be seen in function ``add_gp_buggy()`` below. We will look at the
++++++++as can be seen in function add_gp_buggy() below. We will look at the
reader's code later, but in the meantime, just think of the reader as
locklessly picking up the ``gp`` pointer, and, if the value loaded is
non-\ ``NULL``, locklessly accessing the ``->a`` and ``->b`` fields.
RCU's publish-subscribe guarantee allows data to be inserted into a
linked data structure without disrupting RCU readers. The updater uses
--------``rcu_assign_pointer()`` to insert the new data, and readers use
--------``rcu_dereference()`` to access data, whether new or old. The following
++++++++rcu_assign_pointer() to insert the new data, and readers use
++++++++rcu_dereference() to access data, whether new or old. The following
shows an example of insertion:
::
15 return true;
16 }
--------The ``rcu_assign_pointer()`` on line 13 is conceptually equivalent to a
++++++++The rcu_assign_pointer() on line 13 is conceptually equivalent to a
simple assignment statement, but also guarantees that its assignment
will happen after the two assignments in lines 11 and 12, similar to the
C11 ``memory_order_release`` store operation. It also prevents any
+-----------------------------------------------------------------------+
| **Quick Quiz**: |
+-----------------------------------------------------------------------+
--------| But ``rcu_assign_pointer()`` does nothing to prevent the two |
++++++++| But rcu_assign_pointer() does nothing to prevent the two |
| assignments to ``p->a`` and ``p->b`` from being reordered. Can't that |
| also cause problems? |
+-----------------------------------------------------------------------+
It is tempting to assume that the reader need not do anything special to
control its accesses to the RCU-protected data, as shown in
--------``do_something_gp_buggy()`` below:
++++++++do_something_gp_buggy() below:
::
12 }
However, this temptation must be resisted because there are a
--------surprisingly large number of ways that the compiler (to say nothing of
--------`DEC Alpha CPUs <https://h71000.www7.hp.com/wizard/wiz_2637.html>`__)
--------can trip this code up. For but one example, if the compiler were short
--------of registers, it might choose to refetch from ``gp`` rather than keeping
--------a separate copy in ``p`` as follows:
++++++++surprisingly large number of ways that the compiler (or weak ordering
++++++++CPUs like the DEC Alpha) can trip this code up. For but one example, if
++++++++the compiler were short of registers, it might choose to refetch from
++++++++``gp`` rather than keeping a separate copy in ``p`` as follows:
::
the current structure with a new one, the fetches of ``gp->a`` and
``gp->b`` might well come from two different structures, which could
cause serious confusion. To prevent this (and much else besides),
--------``do_something_gp()`` uses ``rcu_dereference()`` to fetch from ``gp``:
++++++++do_something_gp() uses rcu_dereference() to fetch from ``gp``:
::
11 return false;
12 }
--------The ``rcu_dereference()`` uses volatile casts and (for DEC Alpha) memory
++++++++The rcu_dereference() uses volatile casts and (for DEC Alpha) memory
barriers in the Linux kernel. Should a `high-quality implementation of
C11 ``memory_order_consume``
[PDF] <http://www.rdrop.com/users/paulmck/RCU/consume.2015.07.13a.pdf>`__
--------ever appear, then ``rcu_dereference()`` could be implemented as a
++++++++ever appear, then rcu_dereference() could be implemented as a
``memory_order_consume`` load. Regardless of the exact implementation, a
--------pointer fetched by ``rcu_dereference()`` may not be used outside of the
++++++++pointer fetched by rcu_dereference() may not be used outside of the
outermost RCU read-side critical section containing that
--------``rcu_dereference()``, unless protection of the corresponding data
++++++++rcu_dereference(), unless protection of the corresponding data
element has been passed from RCU to some other synchronization
mechanism, most commonly locking or `reference
counting <https://www.kernel.org/doc/Documentation/RCU/rcuref.txt>`__.
--------In short, updaters use ``rcu_assign_pointer()`` and readers use
--------``rcu_dereference()``, and these two RCU API elements work together to
++++++++In short, updaters use rcu_assign_pointer() and readers use
++++++++rcu_dereference(), and these two RCU API elements work together to
ensure that readers have a consistent view of newly added data elements.
Of course, it is also necessary to remove elements from RCU-protected
the newly removed data element).
#. At this point, only the updater has a reference to the newly removed
data element, so it can safely reclaim the data element, for example,
-------- by passing it to ``kfree()``.
++++++++ by passing it to kfree().
--------This process is implemented by ``remove_gp_synchronous()``:
++++++++This process is implemented by remove_gp_synchronous():
::
This function is straightforward, with line 13 waiting for a grace
period before line 14 frees the old data element. This waiting ensures
--------that readers will reach line 7 of ``do_something_gp()`` before the data
--------element referenced by ``p`` is freed. The ``rcu_access_pointer()`` on
--------line 6 is similar to ``rcu_dereference()``, except that:
++++++++that readers will reach line 7 of do_something_gp() before the data
++++++++element referenced by ``p`` is freed. The rcu_access_pointer() on
++++++++line 6 is similar to rcu_dereference(), except that:
--------#. The value returned by ``rcu_access_pointer()`` cannot be
++++++++#. The value returned by rcu_access_pointer() cannot be
dereferenced. If you want to access the value pointed to as well as
-------- the pointer itself, use ``rcu_dereference()`` instead of
-------- ``rcu_access_pointer()``.
--------#. The call to ``rcu_access_pointer()`` need not be protected. In
-------- contrast, ``rcu_dereference()`` must either be within an RCU
++++++++ the pointer itself, use rcu_dereference() instead of
++++++++ rcu_access_pointer().
++++++++#. The call to rcu_access_pointer() need not be protected. In
++++++++ contrast, rcu_dereference() must either be within an RCU
read-side critical section or in a code segment where the pointer
cannot change, for example, in code protected by the corresponding
update-side lock.
+-----------------------------------------------------------------------+
| **Quick Quiz**: |
+-----------------------------------------------------------------------+
--------| Without the ``rcu_dereference()`` or the ``rcu_access_pointer()``, |
++++++++| Without the rcu_dereference() or the rcu_access_pointer(), |
| what destructive optimizations might the compiler make use of? |
+-----------------------------------------------------------------------+
| **Answer**: |
+-----------------------------------------------------------------------+
--------| Let's start with what happens to ``do_something_gp()`` if it fails to |
--------| use ``rcu_dereference()``. It could reuse a value formerly fetched |
++++++++| Let's start with what happens to do_something_gp() if it fails to |
++++++++| use rcu_dereference(). It could reuse a value formerly fetched |
| from this same pointer. It could also fetch the pointer from ``gp`` |
| in a byte-at-a-time manner, resulting in *load tearing*, in turn |
| resulting a bytewise mash-up of two distinct pointer values. It might |
| update has changed the pointer to match the wrong guess. Too bad |
| about any dereferences that returned pre-initialization garbage in |
| the meantime! |
--------| For ``remove_gp_synchronous()``, as long as all modifications to |
++++++++| For remove_gp_synchronous(), as long as all modifications to |
| ``gp`` are carried out while holding ``gp_lock``, the above |
| optimizations are harmless. However, ``sparse`` will complain if you |
| define ``gp`` with ``__rcu`` and then access it without using either |
--------| ``rcu_access_pointer()`` or ``rcu_dereference()``. |
++++++++| rcu_access_pointer() or rcu_dereference(). |
+-----------------------------------------------------------------------+
In short, RCU's publish-subscribe guarantee is provided by the
--------combination of ``rcu_assign_pointer()`` and ``rcu_dereference()``. This
++++++++combination of rcu_assign_pointer() and rcu_dereference(). This
guarantee allows data elements to be safely added to RCU-protected
linked data structures without disrupting RCU readers. This guarantee
can be used in combination with the grace-period guarantee to also allow
This guarantee was only partially premeditated. DYNIX/ptx used an
explicit memory barrier for publication, but had nothing resembling
--------``rcu_dereference()`` for subscription, nor did it have anything
++++++++rcu_dereference() for subscription, nor did it have anything
resembling the dependency-ordering barrier that was later subsumed
--------into ``rcu_dereference()`` and later still into ``READ_ONCE()``. The
++++++++into rcu_dereference() and later still into READ_ONCE(). The
need for these operations made itself known quite suddenly at a
late-1990s meeting with the DEC Alpha architects, back in the days when
DEC was still a free-standing company. It took the Alpha architects a
and C++ standards committees have provided much education on tricks and
traps from the compiler. In short, compilers were much less tricky in
the early 1990s, but in 2015, don't even think about omitting
--------``rcu_dereference()``!
++++++++rcu_dereference()!
Memory-Barrier Guarantees
~~~~~~~~~~~~~~~~~~~~~~~~~
systems with more than one CPU:
#. Each CPU that has an RCU read-side critical section that begins
-------- before ``synchronize_rcu()`` starts is guaranteed to execute a full
++++++++ before synchronize_rcu() starts is guaranteed to execute a full
memory barrier between the time that the RCU read-side critical
-------- section ends and the time that ``synchronize_rcu()`` returns. Without
++++++++ section ends and the time that synchronize_rcu() returns. Without
this guarantee, a pre-existing RCU read-side critical section might
hold a reference to the newly removed ``struct foo`` after the
-------- ``kfree()`` on line 14 of ``remove_gp_synchronous()``.
++++++++ kfree() on line 14 of remove_gp_synchronous().
#. Each CPU that has an RCU read-side critical section that ends after
-------- ``synchronize_rcu()`` returns is guaranteed to execute a full memory
-------- barrier between the time that ``synchronize_rcu()`` begins and the
++++++++ synchronize_rcu() returns is guaranteed to execute a full memory
++++++++ barrier between the time that synchronize_rcu() begins and the
time that the RCU read-side critical section begins. Without this
guarantee, a later RCU read-side critical section running after the
-------- ``kfree()`` on line 14 of ``remove_gp_synchronous()`` might later run
-------- ``do_something_gp()`` and find the newly deleted ``struct foo``.
--------#. If the task invoking ``synchronize_rcu()`` remains on a given CPU,
++++++++ kfree() on line 14 of remove_gp_synchronous() might later run
++++++++ do_something_gp() and find the newly deleted ``struct foo``.
++++++++#. If the task invoking synchronize_rcu() remains on a given CPU,
then that CPU is guaranteed to execute a full memory barrier sometime
-------- during the execution of ``synchronize_rcu()``. This guarantee ensures
-------- that the ``kfree()`` on line 14 of ``remove_gp_synchronous()`` really
++++++++ during the execution of synchronize_rcu(). This guarantee ensures
++++++++ that the kfree() on line 14 of remove_gp_synchronous() really
does execute after the removal on line 11.
--------#. If the task invoking ``synchronize_rcu()`` migrates among a group of
++++++++#. If the task invoking synchronize_rcu() migrates among a group of
CPUs during that invocation, then each of the CPUs in that group is
guaranteed to execute a full memory barrier sometime during the
-------- execution of ``synchronize_rcu()``. This guarantee also ensures that
-------- the ``kfree()`` on line 14 of ``remove_gp_synchronous()`` really does
++++++++ execution of synchronize_rcu(). This guarantee also ensures that
++++++++ the kfree() on line 14 of remove_gp_synchronous() really does
execute after the removal on line 11, but also in the case where the
-------- thread executing the ``synchronize_rcu()`` migrates in the meantime.
++++++++ thread executing the synchronize_rcu() migrates in the meantime.
+-----------------------------------------------------------------------+
| **Quick Quiz**: |
| Given that multiple CPUs can start RCU read-side critical sections at |
| any time without any ordering whatsoever, how can RCU possibly tell |
| whether or not a given RCU read-side critical section starts before a |
--------| given instance of ``synchronize_rcu()``? |
++++++++| given instance of synchronize_rcu()? |
+-----------------------------------------------------------------------+
| **Answer**: |
+-----------------------------------------------------------------------+
| If RCU cannot tell whether or not a given RCU read-side critical |
--------| section starts before a given instance of ``synchronize_rcu()``, then |
++++++++| section starts before a given instance of synchronize_rcu(), then |
| it must assume that the RCU read-side critical section started first. |
--------| In other words, a given instance of ``synchronize_rcu()`` can avoid |
++++++++| In other words, a given instance of synchronize_rcu() can avoid |
| waiting on a given RCU read-side critical section only if it can |
--------| prove that ``synchronize_rcu()`` started first. |
--------| A related question is “When ``rcu_read_lock()`` doesn't generate any |
++++++++| prove that synchronize_rcu() started first. |
++++++++| A related question is “When rcu_read_lock() doesn't generate any |
| code, why does it matter how it relates to a grace period?” The |
--------| answer is that it is not the relationship of ``rcu_read_lock()`` |
++++++++| answer is that it is not the relationship of rcu_read_lock() |
| itself that is important, but rather the relationship of the code |
| within the enclosed RCU read-side critical section to the code |
| preceding and following the grace period. If we take this viewpoint, |
| Yes, they really are required. To see why the first guarantee is |
| required, consider the following sequence of events: |
| |
--------| #. CPU 1: ``rcu_read_lock()`` |
++++++++| #. CPU 1: rcu_read_lock() |
| #. CPU 1: ``q = rcu_dereference(gp); /* Very likely to return p. */`` |
| #. CPU 0: ``list_del_rcu(p);`` |
--------| #. CPU 0: ``synchronize_rcu()`` starts. |
++++++++| #. CPU 0: synchronize_rcu() starts. |
| #. CPU 1: ``do_something_with(q->a);`` |
| ``/* No smp_mb(), so might happen after kfree(). */`` |
--------| #. CPU 1: ``rcu_read_unlock()`` |
--------| #. CPU 0: ``synchronize_rcu()`` returns. |
++++++++| #. CPU 1: rcu_read_unlock() |
++++++++| #. CPU 0: synchronize_rcu() returns. |
| #. CPU 0: ``kfree(p);`` |
| |
| Therefore, there absolutely must be a full memory barrier between the |
| is roughly similar: |
| |
| #. CPU 0: ``list_del_rcu(p);`` |
--------| #. CPU 0: ``synchronize_rcu()`` starts. |
--------| #. CPU 1: ``rcu_read_lock()`` |
++++++++| #. CPU 0: synchronize_rcu() starts. |
++++++++| #. CPU 1: rcu_read_lock() |
| #. CPU 1: ``q = rcu_dereference(gp);`` |
| ``/* Might return p if no memory barrier. */`` |
--------| #. CPU 0: ``synchronize_rcu()`` returns. |
++++++++| #. CPU 0: synchronize_rcu() returns. |
| #. CPU 0: ``kfree(p);`` |
| #. CPU 1: ``do_something_with(q->a); /* Boom!!! */`` |
--------| #. CPU 1: ``rcu_read_unlock()`` |
++++++++| #. CPU 1: rcu_read_unlock() |
| |
| And similarly, without a memory barrier between the beginning of the |
| grace period and the beginning of the RCU read-side critical section, |
+-----------------------------------------------------------------------+
| **Quick Quiz**: |
+-----------------------------------------------------------------------+
--------| You claim that ``rcu_read_lock()`` and ``rcu_read_unlock()`` generate |
++++++++| You claim that rcu_read_lock() and rcu_read_unlock() generate |
| absolutely no code in some kernel builds. This means that the |
| compiler might arbitrarily rearrange consecutive RCU read-side |
| critical sections. Given such rearrangement, if a given RCU read-side |
+-----------------------------------------------------------------------+
| **Answer**: |
+-----------------------------------------------------------------------+
--------| In cases where ``rcu_read_lock()`` and ``rcu_read_unlock()`` generate |
++++++++| In cases where rcu_read_lock() and rcu_read_unlock() generate |
| absolutely no code, RCU infers quiescent states only at special |
| locations, for example, within the scheduler. Because calls to |
--------| ``schedule()`` had better prevent calling-code accesses to shared |
--------| variables from being rearranged across the call to ``schedule()``, if |
++++++++| schedule() had better prevent calling-code accesses to shared |
++++++++| variables from being rearranged across the call to schedule(), if |
| RCU detects the end of a given RCU read-side critical section, it |
| will necessarily detect the end of all prior RCU read-side critical |
| sections, no matter how aggressively the compiler scrambles the code. |
then might acquire the update-side spinlock in order to update that
element, all while remaining in that RCU read-side critical section. Of
course, it is necessary to exit the RCU read-side critical section
--------before invoking ``synchronize_rcu()``, however, this inconvenience can
--------be avoided through use of the ``call_rcu()`` and ``kfree_rcu()`` API
++++++++before invoking synchronize_rcu(), however, this inconvenience can
++++++++be avoided through use of the call_rcu() and kfree_rcu() API
members described later in this document.
+-----------------------------------------------------------------------+
Readers Impose Minimal Ordering
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
--------Reader-side markers such as ``rcu_read_lock()`` and
--------``rcu_read_unlock()`` provide absolutely no ordering guarantees except
++++++++Reader-side markers such as rcu_read_lock() and
++++++++rcu_read_unlock() provide absolutely no ordering guarantees except
through their interaction with the grace-period APIs such as
--------``synchronize_rcu()``. To see this, consider the following pair of
++++++++synchronize_rcu(). To see this, consider the following pair of
threads:
::
18 rcu_read_unlock();
19 }
--------After ``thread0()`` and ``thread1()`` execute concurrently, it is quite
++++++++After thread0() and thread1() execute concurrently, it is quite
possible to have
::
(r1 == 1 && r2 == 0)
(that is, ``y`` appears to have been assigned before ``x``), which would
--------not be possible if ``rcu_read_lock()`` and ``rcu_read_unlock()`` had
++++++++not be possible if rcu_read_lock() and rcu_read_unlock() had
much in the way of ordering properties. But they do not, so the CPU is
within its rights to do significant reordering. This is by design: Any
significant ordering constraints would slow down these fast-path APIs.
+-----------------------------------------------------------------------+
| **Answer**: |
+-----------------------------------------------------------------------+
--------| No, the volatile casts in ``READ_ONCE()`` and ``WRITE_ONCE()`` |
++++++++| No, the volatile casts in READ_ONCE() and WRITE_ONCE() |
| prevent the compiler from reordering in this particular case. |
+-----------------------------------------------------------------------+
Readers Do Not Exclude Updaters
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
--------Neither ``rcu_read_lock()`` nor ``rcu_read_unlock()`` exclude updates.
++++++++Neither rcu_read_lock() nor rcu_read_unlock() exclude updates.
All they do is to prevent grace periods from ending. The following
example illustrates this:
18 spin_unlock(&my_lock);
19 }
--------If the ``thread0()`` function's ``rcu_read_lock()`` excluded the
--------``thread1()`` function's update, the ``WARN_ON()`` could never fire. But
--------the fact is that ``rcu_read_lock()`` does not exclude much of anything
--------aside from subsequent grace periods, of which ``thread1()`` has none, so
--------the ``WARN_ON()`` can and does fire.
++++++++If the thread0() function's rcu_read_lock() excluded the
++++++++thread1() function's update, the WARN_ON() could never fire. But
++++++++the fact is that rcu_read_lock() does not exclude much of anything
++++++++aside from subsequent grace periods, of which thread1() has none, so
++++++++the WARN_ON() can and does fire.
Updaters Only Wait For Old Readers
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
--------It might be tempting to assume that after ``synchronize_rcu()``
++++++++It might be tempting to assume that after synchronize_rcu()
completes, there are no readers executing. This temptation must be
avoided because new readers can start immediately after
--------``synchronize_rcu()`` starts, and ``synchronize_rcu()`` is under no
++++++++synchronize_rcu() starts, and synchronize_rcu() is under no
obligation to wait for these new readers.
+-----------------------------------------------------------------------+
+-----------------------------------------------------------------------+
| **Answer**: |
+-----------------------------------------------------------------------+
--------| For no time at all. Even if ``synchronize_rcu()`` were to wait until |
++++++++| For no time at all. Even if synchronize_rcu() were to wait until |
| all readers had completed, a new reader might start immediately after |
--------| ``synchronize_rcu()`` completed. Therefore, the code following |
--------| ``synchronize_rcu()`` can *never* rely on there being no readers. |
++++++++| synchronize_rcu() completed. Therefore, the code following |
++++++++| synchronize_rcu() can *never* rely on there being no readers. |
+-----------------------------------------------------------------------+
Grace Periods Don't Partition Read-Side Critical Sections
28 rcu_read_unlock();
29 }
--------Here, if ``(r1 == 1)``, then ``thread0()``'s write to ``b`` must happen
--------before the end of ``thread1()``'s grace period. If in addition
--------``(r4 == 1)``, then ``thread3()``'s read from ``b`` must happen after
--------the beginning of ``thread2()``'s grace period. If it is also the case
--------that ``(r2 == 1)``, then the end of ``thread1()``'s grace period must
--------precede the beginning of ``thread2()``'s grace period. This mean that
++++++++Here, if ``(r1 == 1)``, then thread0()'s write to ``b`` must happen
++++++++before the end of thread1()'s grace period. If in addition
++++++++``(r4 == 1)``, then thread3()'s read from ``b`` must happen after
++++++++the beginning of thread2()'s grace period. If it is also the case
++++++++that ``(r2 == 1)``, then the end of thread1()'s grace period must
++++++++precede the beginning of thread2()'s grace period. This mean that
the two RCU read-side critical sections cannot overlap, guaranteeing
that ``(r3 == 1)``. As a result, the outcome:
b. Wait-free read-side primitives for real-time use.
This focus on read-mostly situations means that RCU must interoperate
--------with other synchronization primitives. For example, the ``add_gp()`` and
--------``remove_gp_synchronous()`` examples discussed earlier use RCU to
++++++++with other synchronization primitives. For example, the add_gp() and
++++++++remove_gp_synchronous() examples discussed earlier use RCU to
protect readers and locking to coordinate updaters. However, the need
extends much farther, requiring that a variety of synchronization
primitives be legal within RCU read-side critical sections, including
| sections. |
| Note that it *is* legal for a normal RCU read-side critical section |
| to conditionally acquire a sleeping locks (as in |
--------| ``mutex_trylock()``), but only as long as it does not loop |
++++++++| mutex_trylock()), but only as long as it does not loop |
| indefinitely attempting to conditionally acquire that sleeping locks. |
--------| The key point is that things like ``mutex_trylock()`` either return |
++++++++| The key point is that things like mutex_trylock() either return |
| with the mutex held, or return an error indication if the mutex was |
--------| not immediately available. Either way, ``mutex_trylock()`` returns |
++++++++| not immediately available. Either way, mutex_trylock() returns |
| immediately without sleeping. |
+-----------------------------------------------------------------------+
costs have plummeted. However, as I learned from Matt Mackall's
`bloatwatch <http://elinux.org/Linux_Tiny-FAQ>`__ efforts, memory
footprint is critically important on single-CPU systems with
--------non-preemptible (``CONFIG_PREEMPT=n``) kernels, and thus `tiny
++++++++non-preemptible (``CONFIG_PREEMPTION=n``) kernels, and thus `tiny
was born. Josh Triplett has since taken over the small-memory banner
with his `Linux kernel tinification <https://tiny.wiki.kernel.org/>`__
project, which resulted in `SRCU <#Sleepable%20RCU>`__ becoming optional
The remaining performance requirements are, for the most part,
unsurprising. For example, in keeping with RCU's read-side
--------specialization, ``rcu_dereference()`` should have negligible overhead
++++++++specialization, rcu_dereference() should have negligible overhead
(for example, suppression of a few minor compiler optimizations).
--------Similarly, in non-preemptible environments, ``rcu_read_lock()`` and
--------``rcu_read_unlock()`` should have exactly zero overhead.
++++++++Similarly, in non-preemptible environments, rcu_read_lock() and
++++++++rcu_read_unlock() should have exactly zero overhead.
In preemptible environments, in the case where the RCU read-side
critical section was not preempted (as will be the case for the
--------highest-priority real-time process), ``rcu_read_lock()`` and
--------``rcu_read_unlock()`` should have minimal overhead. In particular, they
++++++++highest-priority real-time process), rcu_read_lock() and
++++++++rcu_read_unlock() should have minimal overhead. In particular, they
should not contain atomic read-modify-write operations, memory-barrier
instructions, preemption disabling, interrupt disabling, or backwards
branches. However, in the case where the RCU read-side critical section
--------was preempted, ``rcu_read_unlock()`` may acquire spinlocks and disable
++++++++was preempted, rcu_read_unlock() may acquire spinlocks and disable
interrupts. This is why it is better to nest an RCU read-side critical
section within a preempt-disable region than vice versa, at least in
cases where that critical section is short enough to avoid unduly
degrading real-time latencies.
--------The ``synchronize_rcu()`` grace-period-wait primitive is optimized for
++++++++The synchronize_rcu() grace-period-wait primitive is optimized for
throughput. It may therefore incur several milliseconds of latency in
addition to the duration of the longest RCU read-side critical section.
On the other hand, multiple concurrent invocations of
--------``synchronize_rcu()`` are required to use batching optimizations so that
++++++++synchronize_rcu() are required to use batching optimizations so that
they can be satisfied by a single underlying grace-period-wait
operation. For example, in the Linux kernel, it is not unusual for a
single grace-period-wait operation to serve more than `1,000 separate
invocations <https://www.usenix.org/conference/2004-usenix-annual-technical-conference/making-rcu-safe-deep-sub-millisecond-response>`__
--------of ``synchronize_rcu()``, thus amortizing the per-invocation overhead
++++++++of synchronize_rcu(), thus amortizing the per-invocation overhead
down to nearly zero. However, the grace-period optimization is also
required to avoid measurable degradation of real-time scheduling and
interrupt latencies.
--------In some cases, the multi-millisecond ``synchronize_rcu()`` latencies are
--------unacceptable. In these cases, ``synchronize_rcu_expedited()`` may be
++++++++In some cases, the multi-millisecond synchronize_rcu() latencies are
++++++++unacceptable. In these cases, synchronize_rcu_expedited() may be
used instead, reducing the grace-period latency down to a few tens of
microseconds on small systems, at least in cases where the RCU read-side
critical sections are short. There are currently no special latency
--------requirements for ``synchronize_rcu_expedited()`` on large systems, but,
++++++++requirements for synchronize_rcu_expedited() on large systems, but,
consistent with the empirical nature of the RCU specification, that is
subject to change. However, there most definitely are scalability
--------requirements: A storm of ``synchronize_rcu_expedited()`` invocations on
++++++++requirements: A storm of synchronize_rcu_expedited() invocations on
4096 CPUs should at least make reasonable forward progress. In return
--------for its shorter latencies, ``synchronize_rcu_expedited()`` is permitted
++++++++for its shorter latencies, synchronize_rcu_expedited() is permitted
to impose modest degradation of real-time latency on non-idle online
CPUs. Here, “modest” means roughly the same latency degradation as a
scheduling-clock interrupt.
There are a number of situations where even
--------``synchronize_rcu_expedited()``'s reduced grace-period latency is
--------unacceptable. In these situations, the asynchronous ``call_rcu()`` can
--------be used in place of ``synchronize_rcu()`` as follows:
++++++++synchronize_rcu_expedited()'s reduced grace-period latency is
++++++++unacceptable. In these situations, the asynchronous call_rcu() can
++++++++be used in place of synchronize_rcu() as follows:
::
28 }
A definition of ``struct foo`` is finally needed, and appears on
--------lines 1-5. The function ``remove_gp_cb()`` is passed to ``call_rcu()``
++++++++lines 1-5. The function remove_gp_cb() is passed to call_rcu()
on line 25, and will be invoked after the end of a subsequent grace
--------period. This gets the same effect as ``remove_gp_synchronous()``, but
++++++++period. This gets the same effect as remove_gp_synchronous(), but
without forcing the updater to wait for a grace period to elapse. The
--------``call_rcu()`` function may be used in a number of situations where
--------neither ``synchronize_rcu()`` nor ``synchronize_rcu_expedited()`` would
--------be legal, including within preempt-disable code, ``local_bh_disable()``
++++++++call_rcu() function may be used in a number of situations where
++++++++neither synchronize_rcu() nor synchronize_rcu_expedited() would
++++++++be legal, including within preempt-disable code, local_bh_disable()
code, interrupt-disable code, and interrupt handlers. However, even
--------``call_rcu()`` is illegal within NMI handlers and from idle and offline
--------CPUs. The callback function (``remove_gp_cb()`` in this case) will be
++++++++call_rcu() is illegal within NMI handlers and from idle and offline
++++++++CPUs. The callback function (remove_gp_cb() in this case) will be
executed within softirq (software interrupt) environment within the
Linux kernel, either within a real softirq handler or under the
--------protection of ``local_bh_disable()``. In both the Linux kernel and in
++++++++protection of local_bh_disable(). In both the Linux kernel and in
userspace, it is bad practice to write an RCU callback function that
takes too long. Long-running operations should be relegated to separate
threads or (in the Linux kernel) workqueues.
+-----------------------------------------------------------------------+
| **Quick Quiz**: |
+-----------------------------------------------------------------------+
--------| Why does line 19 use ``rcu_access_pointer()``? After all, |
--------| ``call_rcu()`` on line 25 stores into the structure, which would |
++++++++| Why does line 19 use rcu_access_pointer()? After all, |
++++++++| call_rcu() on line 25 stores into the structure, which would |
| interact badly with concurrent insertions. Doesn't this mean that |
--------| ``rcu_dereference()`` is required? |
++++++++| rcu_dereference() is required? |
+-----------------------------------------------------------------------+
| **Answer**: |
+-----------------------------------------------------------------------+
| Presumably the ``->gp_lock`` acquired on line 18 excludes any |
--------| changes, including any insertions that ``rcu_dereference()`` would |
++++++++| changes, including any insertions that rcu_dereference() would |
| protect against. Therefore, any insertions will be delayed until |
| after ``->gp_lock`` is released on line 25, which in turn means that |
--------| ``rcu_access_pointer()`` suffices. |
++++++++| rcu_access_pointer() suffices. |
+-----------------------------------------------------------------------+
--------However, all that ``remove_gp_cb()`` is doing is invoking ``kfree()`` on
++++++++However, all that remove_gp_cb() is doing is invoking kfree() on
the data element. This is a common idiom, and is supported by
--------``kfree_rcu()``, which allows “fire and forget” operation as shown
++++++++kfree_rcu(), which allows “fire and forget” operation as shown
below:
::
20 return true;
21 }
--------Note that ``remove_gp_faf()`` simply invokes ``kfree_rcu()`` and
++++++++Note that remove_gp_faf() simply invokes kfree_rcu() and
proceeds, without any need to pay any further attention to the
--------subsequent grace period and ``kfree()``. It is permissible to invoke
--------``kfree_rcu()`` from the same environments as for ``call_rcu()``.
--------Interestingly enough, DYNIX/ptx had the equivalents of ``call_rcu()``
--------and ``kfree_rcu()``, but not ``synchronize_rcu()``. This was due to the
++++++++subsequent grace period and kfree(). It is permissible to invoke
++++++++kfree_rcu() from the same environments as for call_rcu().
++++++++Interestingly enough, DYNIX/ptx had the equivalents of call_rcu()
++++++++and kfree_rcu(), but not synchronize_rcu(). This was due to the
fact that RCU was not heavily used within DYNIX/ptx, so the very few
--------places that needed something like ``synchronize_rcu()`` simply
++++++++places that needed something like synchronize_rcu() simply
open-coded it.
+-----------------------------------------------------------------------+
| **Quick Quiz**: |
+-----------------------------------------------------------------------+
--------| Earlier it was claimed that ``call_rcu()`` and ``kfree_rcu()`` |
++++++++| Earlier it was claimed that call_rcu() and kfree_rcu() |
| allowed updaters to avoid being blocked by readers. But how can that |
| be correct, given that the invocation of the callback and the freeing |
| of the memory (respectively) must still wait for a grace period to |
| definition would say that updates in garbage-collected languages |
| cannot complete until the next time the garbage collector runs, which |
| does not seem at all reasonable. The key point is that in most cases, |
--------| an updater using either ``call_rcu()`` or ``kfree_rcu()`` can proceed |
--------| to the next update as soon as it has invoked ``call_rcu()`` or |
--------| ``kfree_rcu()``, without having to wait for a subsequent grace |
++++++++| an updater using either call_rcu() or kfree_rcu() can proceed |
++++++++| to the next update as soon as it has invoked call_rcu() or |
++++++++| kfree_rcu(), without having to wait for a subsequent grace |
| period. |
+-----------------------------------------------------------------------+
But what if the updater must wait for the completion of code to be
executed after the end of the grace period, but has other tasks that can
be carried out in the meantime? The polling-style
--------``get_state_synchronize_rcu()`` and ``cond_synchronize_rcu()`` functions
++++++++get_state_synchronize_rcu() and cond_synchronize_rcu() functions
may be used for this purpose, as shown below:
::
18 return true;
19 }
--------On line 14, ``get_state_synchronize_rcu()`` obtains a “cookie” from RCU,
++++++++On line 14, get_state_synchronize_rcu() obtains a “cookie” from RCU,
then line 15 carries out other tasks, and finally, line 16 returns
immediately if a grace period has elapsed in the meantime, but otherwise
waits as required. The need for ``get_state_synchronize_rcu`` and
--------``cond_synchronize_rcu()`` has appeared quite recently, so it is too
++++++++cond_synchronize_rcu() has appeared quite recently, so it is too
early to tell whether they will stand the test of time.
RCU thus provides a range of tools to allow updaters to strike the
definition prevent later grace periods from ever completing. For a more
involved example, consider a 64-CPU system built with
``CONFIG_RCU_NOCB_CPU=y`` and booted with ``rcu_nocbs=1-63``, where
--------CPUs 1 through 63 spin in tight loops that invoke ``call_rcu()``. Even
--------if these tight loops also contain calls to ``cond_resched()`` (thus
++++++++CPUs 1 through 63 spin in tight loops that invoke call_rcu(). Even
++++++++if these tight loops also contain calls to cond_resched() (thus
allowing grace periods to complete), CPU 0 simply will not be able to
invoke callbacks as fast as the other 63 CPUs can register them, at
least not until the system runs out of memory. In both of these
periods:
#. If a grace period fails to complete within 100 milliseconds, RCU
-------- causes future invocations of ``cond_resched()`` on the holdout CPUs
++++++++ causes future invocations of cond_resched() on the holdout CPUs
to provide an RCU quiescent state. RCU also causes those CPUs'
-------- ``need_resched()`` invocations to return ``true``, but only after the
++++++++ need_resched() invocations to return ``true``, but only after the
corresponding CPU's next scheduling-clock.
#. CPUs mentioned in the ``nohz_full`` kernel boot parameter can run
indefinitely in the kernel without scheduling-clock interrupts, which
-------- defeats the above ``need_resched()`` strategem. RCU will therefore
-------- invoke ``resched_cpu()`` on any ``nohz_full`` CPUs still holding out
++++++++ defeats the above need_resched() strategem. RCU will therefore
++++++++ invoke resched_cpu() on any ``nohz_full`` CPUs still holding out
after 109 milliseconds.
#. In kernels built with ``CONFIG_RCU_BOOST=y``, if a given task that
has been preempted within an RCU read-side critical section is
holding out for more than 500 milliseconds, RCU will resort to
priority boosting.
#. If a CPU is still holding out 10 seconds into the grace period, RCU
-------- will invoke ``resched_cpu()`` on it regardless of its ``nohz_full``
++++++++ will invoke resched_cpu() on it regardless of its ``nohz_full``
state.
The above values are defaults for systems running with ``HZ=1000``. They
are provided only for RCU, not for `SRCU <#Sleepable%20RCU>`__ or `Tasks
RCU <#Tasks%20RCU>`__.
--------RCU takes the following steps in ``call_rcu()`` to encourage timely
++++++++RCU takes the following steps in call_rcu() to encourage timely
invocation of callbacks when any given non-\ ``rcu_nocbs`` CPU has
10,000 callbacks, or has 10,000 more callbacks than it had the last time
encouragement was provided:
RCU <#Tasks%20RCU>`__. Even for RCU, callback-invocation forward
progress for ``rcu_nocbs`` CPUs is much less well-developed, in part
because workloads benefiting from ``rcu_nocbs`` CPUs tend to invoke
--------``call_rcu()`` relatively infrequently. If workloads emerge that need
--------both ``rcu_nocbs`` CPUs and high ``call_rcu()`` invocation rates, then
++++++++call_rcu() relatively infrequently. If workloads emerge that need
++++++++both ``rcu_nocbs`` CPUs and high call_rcu() invocation rates, then
additional forward-progress work will be required.
Composability
real-world implementations of composable constructs, there are
limitations.
--------Implementations of RCU for which ``rcu_read_lock()`` and
--------``rcu_read_unlock()`` generate no code, such as Linux-kernel RCU when
--------``CONFIG_PREEMPT=n``, can be nested arbitrarily deeply. After all, there
++++++++Implementations of RCU for which rcu_read_lock() and
++++++++rcu_read_unlock() generate no code, such as Linux-kernel RCU when
++++++++``CONFIG_PREEMPTION=n``, can be nested arbitrarily deeply. After all, there
is no overhead. Except that if all these instances of
--------``rcu_read_lock()`` and ``rcu_read_unlock()`` are visible to the
++++++++rcu_read_lock() and rcu_read_unlock() are visible to the
compiler, compilation will eventually fail due to exhausting memory,
mass storage, or user patience, whichever comes first. If the nesting is
not visible to the compiler, as is the case with mutually recursive
the fact remains that RCU must handle such workloads gracefully. This
requirement is another factor driving batching of grace periods, but it
is also the driving force behind the checks for large numbers of queued
--------RCU callbacks in the ``call_rcu()`` code path. Finally, high update
++++++++RCU callbacks in the call_rcu() code path. Finally, high update
rates should not delay RCU read-side critical sections, although some
small read-side delays can occur when using
--------``synchronize_rcu_expedited()``, courtesy of this function's use of
--------``smp_call_function_single()``.
++++++++synchronize_rcu_expedited(), courtesy of this function's use of
++++++++smp_call_function_single().
Although all three of these corner cases were understood in the early
1990s, a simple user-level test consisting of ``close(open(path))`` in a
Between Murphy's Law and “To err is human”, it is necessary to guard
against mishaps and misuse:
--------#. It is all too easy to forget to use ``rcu_read_lock()`` everywhere
++++++++#. It is all too easy to forget to use rcu_read_lock() everywhere
that it is needed, so kernels built with ``CONFIG_PROVE_RCU=y`` will
-------- splat if ``rcu_dereference()`` is used outside of an RCU read-side
++++++++ splat if rcu_dereference() is used outside of an RCU read-side
critical section. Update-side code can use
-------- ``rcu_dereference_protected()``, which takes a `lockdep
++++++++ rcu_dereference_protected(), which takes a `lockdep
expression <https://lwn.net/Articles/371986/>`__ to indicate what is
providing the protection. If the indicated protection is not
provided, a lockdep splat is emitted.
Code shared between readers and updaters can use
-------- ``rcu_dereference_check()``, which also takes a lockdep expression,
-------- and emits a lockdep splat if neither ``rcu_read_lock()`` nor the
++++++++ rcu_dereference_check(), which also takes a lockdep expression,
++++++++ and emits a lockdep splat if neither rcu_read_lock() nor the
indicated protection is in place. In addition,
-------- ``rcu_dereference_raw()`` is used in those (hopefully rare) cases
++++++++ rcu_dereference_raw() is used in those (hopefully rare) cases
where the required protection cannot be easily described. Finally,
-------- ``rcu_read_lock_held()`` is provided to allow a function to verify
++++++++ rcu_read_lock_held() is provided to allow a function to verify
that it has been invoked within an RCU read-side critical section. I
was made aware of this set of requirements shortly after Thomas
Gleixner audited a number of RCU uses.
#. A given function might wish to check for RCU-related preconditions
upon entry, before using any other RCU API. The
-------- ``rcu_lockdep_assert()`` does this job, asserting the expression in
++++++++ rcu_lockdep_assert() does this job, asserting the expression in
kernels having lockdep enabled and doing nothing otherwise.
--------#. It is also easy to forget to use ``rcu_assign_pointer()`` and
-------- ``rcu_dereference()``, perhaps (incorrectly) substituting a simple
++++++++#. It is also easy to forget to use rcu_assign_pointer() and
++++++++ rcu_dereference(), perhaps (incorrectly) substituting a simple
assignment. To catch this sort of error, a given RCU-protected
pointer may be tagged with ``__rcu``, after which sparse will
complain about simple-assignment accesses to that pointer. Arnd
Bergmann made me aware of this requirement, and also supplied the
needed `patch series <https://lwn.net/Articles/376011/>`__.
#. Kernels built with ``CONFIG_DEBUG_OBJECTS_RCU_HEAD=y`` will splat if
-------- a data element is passed to ``call_rcu()`` twice in a row, without a
++++++++ a data element is passed to call_rcu() twice in a row, without a
grace period in between. (This error is similar to a double free.)
The corresponding ``rcu_head`` structures that are dynamically
allocated are automatically tracked, but ``rcu_head`` structures
allocated on the stack must be initialized with
-------- ``init_rcu_head_on_stack()`` and cleaned up with
-------- ``destroy_rcu_head_on_stack()``. Similarly, statically allocated
++++++++ init_rcu_head_on_stack() and cleaned up with
++++++++ destroy_rcu_head_on_stack(). Similarly, statically allocated
non-stack ``rcu_head`` structures must be initialized with
-------- ``init_rcu_head()`` and cleaned up with ``destroy_rcu_head()``.
++++++++ init_rcu_head() and cleaned up with destroy_rcu_head().
Mathieu Desnoyers made me aware of this requirement, and also
supplied the needed
-------- `patch <https://lkml.kernel.org/g/20100319013024.GA28456@Krystal>`__.
++++++++ `patch <https://lore.kernel.org/r/20100319013024.GA28456@Krystal>`__.
#. An infinite loop in an RCU read-side critical section will eventually
trigger an RCU CPU stall warning splat, with the duration of
“eventually” being controlled by the ``RCU_CPU_STALL_TIMEOUT``
``rcupdate.rcu_cpu_stall_suppress`` to suppress the splats. This
kernel parameter may also be set via ``sysfs``. Furthermore, RCU CPU
stall warnings are counter-productive during sysrq dumps and during
-------- panics. RCU therefore supplies the ``rcu_sysrq_start()`` and
-------- ``rcu_sysrq_end()`` API members to be called before and after long
-------- sysrq dumps. RCU also supplies the ``rcu_panic()`` notifier that is
++++++++ panics. RCU therefore supplies the rcu_sysrq_start() and
++++++++ rcu_sysrq_end() API members to be called before and after long
++++++++ sysrq dumps. RCU also supplies the rcu_panic() notifier that is
automatically invoked at the beginning of a panic to suppress further
RCU CPU stall warnings.
synchronization mechanism, for example, reference counting.
#. In kernels built with ``CONFIG_RCU_TRACE=y``, RCU-related information
is provided via event tracing.
--------#. Open-coded use of ``rcu_assign_pointer()`` and ``rcu_dereference()``
++++++++#. Open-coded use of rcu_assign_pointer() and rcu_dereference()
to create typical linked data structures can be surprisingly
error-prone. Therefore, RCU-protected `linked
lists <https://lwn.net/Articles/609973/#RCU%20List%20APIs>`__ and,
other special-purpose RCU-protected data structures are available in
the Linux kernel and the userspace RCU library.
#. Some linked structures are created at compile time, but still require
-------- ``__rcu`` checking. The ``RCU_POINTER_INITIALIZER()`` macro serves
++++++++ ``__rcu`` checking. The RCU_POINTER_INITIALIZER() macro serves
this purpose.
--------#. It is not necessary to use ``rcu_assign_pointer()`` when creating
++++++++#. It is not necessary to use rcu_assign_pointer() when creating
linked structures that are to be published via a single external
-------- pointer. The ``RCU_INIT_POINTER()`` macro is provided for this task
-------- and also for assigning ``NULL`` pointers at runtime.
++++++++ pointer. The RCU_INIT_POINTER() macro is provided for this task.
This not a hard-and-fast list: RCU's diagnostic capabilities will
continue to be guided by the number and type of usage bugs found in
This all should be quite obvious, but the fact remains that Linus
Torvalds recently had to
--------`remind <https://lkml.kernel.org/g/CA+55aFy4wcCwaL4okTs8wXhGZ5h-ibecy_Meg9C4MNQrUnwMcg@mail.gmail.com>`__
++++++++`remind <https://lore.kernel.org/r/CA+55aFy4wcCwaL4okTs8wXhGZ5h-ibecy_Meg9C4MNQrUnwMcg@mail.gmail.com>`__
me of this requirement.
Firmware Interface
~~~~~~~~~~
The Linux kernel's boot sequence is an interesting process, and RCU is
--------used early, even before ``rcu_init()`` is invoked. In fact, a number of
++++++++used early, even before rcu_init() is invoked. In fact, a number of
RCU's primitives can be used as soon as the initial task's
``task_struct`` is available and the boot CPU's per-CPU variables are
--------set up. The read-side primitives (``rcu_read_lock()``,
--------``rcu_read_unlock()``, ``rcu_dereference()``, and
--------``rcu_access_pointer()``) will operate normally very early on, as will
--------``rcu_assign_pointer()``.
++++++++set up. The read-side primitives (rcu_read_lock(),
++++++++rcu_read_unlock(), rcu_dereference(), and
++++++++rcu_access_pointer()) will operate normally very early on, as will
++++++++rcu_assign_pointer().
--------Although ``call_rcu()`` may be invoked at any time during boot,
++++++++Although call_rcu() may be invoked at any time during boot,
callbacks are not guaranteed to be invoked until after all of RCU's
--------kthreads have been spawned, which occurs at ``early_initcall()`` time.
++++++++kthreads have been spawned, which occurs at early_initcall() time.
This delay in callback invocation is due to the fact that RCU does not
invoke callbacks until it is fully initialized, and this full
initialization cannot occur until after the scheduler has initialized
panacea because there would be severe restrictions on what operations
those callbacks could invoke.
--------Perhaps surprisingly, ``synchronize_rcu()`` and
--------``synchronize_rcu_expedited()``, will operate normally during very early
++++++++Perhaps surprisingly, synchronize_rcu() and
++++++++synchronize_rcu_expedited(), will operate normally during very early
boot, the reason being that there is only one CPU and preemption is
--------disabled. This means that the call ``synchronize_rcu()`` (or friends)
++++++++disabled. This means that the call synchronize_rcu() (or friends)
itself is a quiescent state and thus a grace period, so the early-boot
implementation can be a no-op.
However, once the scheduler has spawned its first kthread, this early
--------boot trick fails for ``synchronize_rcu()`` (as well as for
--------``synchronize_rcu_expedited()``) in ``CONFIG_PREEMPT=y`` kernels. The
++++++++boot trick fails for synchronize_rcu() (as well as for
++++++++synchronize_rcu_expedited()) in ``CONFIG_PREEMPTION=y`` kernels. The
reason is that an RCU read-side critical section might be preempted,
--------which means that a subsequent ``synchronize_rcu()`` really does have to
++++++++which means that a subsequent synchronize_rcu() really does have to
wait for something, as opposed to simply returning immediately.
--------Unfortunately, ``synchronize_rcu()`` can't do this until all of its
++++++++Unfortunately, synchronize_rcu() can't do this until all of its
kthreads are spawned, which doesn't happen until some time during
--------``early_initcalls()`` time. But this is no excuse: RCU is nevertheless
++++++++early_initcalls() time. But this is no excuse: RCU is nevertheless
required to correctly handle synchronous grace periods during this time
period. Once all of its kthreads are up and running, RCU starts running
normally.
The Linux kernel has interrupts, and RCU read-side critical sections are
legal within interrupt handlers and within interrupt-disabled regions of
--------code, as are invocations of ``call_rcu()``.
++++++++code, as are invocations of call_rcu().
Some Linux-kernel architectures can enter an interrupt handler from
non-idle process context, and then just never leave it, instead
The Linux kernel has non-maskable interrupts (NMIs), and RCU read-side
critical sections are legal within NMI handlers. Thankfully, RCU
--------update-side primitives, including ``call_rcu()``, are prohibited within
++++++++update-side primitives, including call_rcu(), are prohibited within
NMI handlers.
The name notwithstanding, some Linux-kernel architectures can have
nested NMIs, which RCU must handle correctly. Andy Lutomirski `surprised
--------me <https://lkml.kernel.org/r/CALCETrXLq1y7e_dKFPgou-FKHB6Pu-r8+t-6Ds+8=va7anBWDA@mail.gmail.com>`__
++++++++me <https://lore.kernel.org/r/CALCETrXLq1y7e_dKFPgou-FKHB6Pu-r8+t-6Ds+8=va7anBWDA@mail.gmail.com>`__
with this requirement; he also kindly surprised me with `an
--------algorithm <https://lkml.kernel.org/r/CALCETrXSY9JpW3uE6H8WYk81sg56qasA2aqmjMPsq5dOtzso=g@mail.gmail.com>`__
++++++++algorithm <https://lore.kernel.org/r/CALCETrXSY9JpW3uE6H8WYk81sg56qasA2aqmjMPsq5dOtzso=g@mail.gmail.com>`__
that meets this requirement.
Furthermore, NMI handlers can be interrupted by what appear to RCU to be
normal interrupts. One way that this can happen is for code that
--------directly invokes ``rcu_irq_enter()`` and ``rcu_irq_exit()`` to be called
++++++++directly invokes rcu_irq_enter() and rcu_irq_exit() to be called
from an NMI handler. This astonishing fact of life prompted the current
--------code structure, which has ``rcu_irq_enter()`` invoking
--------``rcu_nmi_enter()`` and ``rcu_irq_exit()`` invoking ``rcu_nmi_exit()``.
++++++++code structure, which has rcu_irq_enter() invoking
++++++++rcu_nmi_enter() and rcu_irq_exit() invoking rcu_nmi_exit().
And yes, I also learned of this requirement the hard way.
Loadable Modules
unloaded. After a given module has been unloaded, any attempt to call
one of its functions results in a segmentation fault. The module-unload
functions must therefore cancel any delayed calls to loadable-module
--------functions, for example, any outstanding ``mod_timer()`` must be dealt
--------with via ``del_timer_sync()`` or similar.
++++++++functions, for example, any outstanding mod_timer() must be dealt
++++++++with via del_timer_sync() or similar.
Unfortunately, there is no way to cancel an RCU callback; once you
--------invoke ``call_rcu()``, the callback function is eventually going to be
++++++++invoke call_rcu(), the callback function is eventually going to be
invoked, unless the system goes down first. Because it is normally
considered socially irresponsible to crash the system in response to a
module unload request, we need some other way to deal with in-flight RCU
callbacks.
--------RCU therefore provides ``rcu_barrier()``, which waits until all
++++++++RCU therefore provides rcu_barrier(), which waits until all
in-flight RCU callbacks have been invoked. If a module uses
--------``call_rcu()``, its exit function should therefore prevent any future
--------invocation of ``call_rcu()``, then invoke ``rcu_barrier()``. In theory,
--------the underlying module-unload code could invoke ``rcu_barrier()``
++++++++call_rcu(), its exit function should therefore prevent any future
++++++++invocation of call_rcu(), then invoke rcu_barrier(). In theory,
++++++++the underlying module-unload code could invoke rcu_barrier()
unconditionally, but in practice this would incur unacceptable
latencies.
Nikita Danilov noted this requirement for an analogous
filesystem-unmount situation, and Dipankar Sarma incorporated
--------``rcu_barrier()`` into RCU. The need for ``rcu_barrier()`` for module
++++++++rcu_barrier() into RCU. The need for rcu_barrier() for module
unloading became apparent later.
.. important::
-------- The ``rcu_barrier()`` function is not, repeat,
++++++++ The rcu_barrier() function is not, repeat,
*not*, obligated to wait for a grace period. It is instead only required
to wait for RCU callbacks that have already been posted. Therefore, if
there are no RCU callbacks posted anywhere in the system,
-------- ``rcu_barrier()`` is within its rights to return immediately. Even if
-------- there are callbacks posted, ``rcu_barrier()`` does not necessarily need
++++++++ rcu_barrier() is within its rights to return immediately. Even if
++++++++ there are callbacks posted, rcu_barrier() does not necessarily need
to wait for a grace period.
+-----------------------------------------------------------------------+
| **Quick Quiz**: |
+-----------------------------------------------------------------------+
| Wait a minute! Each RCU callbacks must wait for a grace period to |
--------| complete, and ``rcu_barrier()`` must wait for each pre-existing |
--------| callback to be invoked. Doesn't ``rcu_barrier()`` therefore need to |
++++++++| complete, and rcu_barrier() must wait for each pre-existing |
++++++++| callback to be invoked. Doesn't rcu_barrier() therefore need to |
| wait for a full grace period if there is even one callback posted |
| anywhere in the system? |
+-----------------------------------------------------------------------+
| Absolutely not!!! |
| Yes, each RCU callbacks must wait for a grace period to complete, but |
| it might well be partly (or even completely) finished waiting by the |
--------| time ``rcu_barrier()`` is invoked. In that case, ``rcu_barrier()`` |
++++++++| time rcu_barrier() is invoked. In that case, rcu_barrier() |
| need only wait for the remaining portion of the grace period to |
| elapse. So even if there are quite a few callbacks posted, |
--------| ``rcu_barrier()`` might well return quite quickly. |
++++++++| rcu_barrier() might well return quite quickly. |
| |
| So if you need to wait for a grace period as well as for all |
| pre-existing callbacks, you will need to invoke both |
--------| ``synchronize_rcu()`` and ``rcu_barrier()``. If latency is a concern, |
++++++++| synchronize_rcu() and rcu_barrier(). If latency is a concern, |
| you can always use workqueues to invoke them concurrently. |
+-----------------------------------------------------------------------+
to allow the various kernel subsystems (including RCU) to respond
appropriately to a given CPU-hotplug operation. Most RCU operations may
be invoked from CPU-hotplug notifiers, including even synchronous
- grace-period operations such as ``synchronize_rcu()`` and
- ``synchronize_rcu_expedited()``.
-
- However, all-callback-wait operations such as ``rcu_barrier()`` are also
- not supported, due to the fact that there are phases of CPU-hotplug
- operations where the outgoing CPU's callbacks will not be invoked until
- after the CPU-hotplug operation ends, which could also result in
- deadlock. Furthermore, ``rcu_barrier()`` blocks CPU-hotplug operations
- during its execution, which results in another type of deadlock when
- invoked from a CPU-hotplug notifier.
-------grace-period operations such as (``synchronize_rcu()`` and
-------``synchronize_rcu_expedited()``). However, these synchronous operations
++++++++grace-period operations such as (synchronize_rcu() and
++++++++synchronize_rcu_expedited()). However, these synchronous operations
+ do block and therefore cannot be invoked from notifiers that execute via
-------``stop_machine()``, specifically those between the ``CPUHP_AP_OFFLINE``
++++++++stop_machine(), specifically those between the ``CPUHP_AP_OFFLINE``
+ and ``CPUHP_AP_ONLINE`` states.
+
-------In addition, all-callback-wait operations such as ``rcu_barrier()`` may
++++++++In addition, all-callback-wait operations such as rcu_barrier() may
+ not be invoked from any CPU-hotplug notifier. This restriction is due
+ to the fact that there are phases of CPU-hotplug operations where the
+ outgoing CPU's callbacks will not be invoked until after the CPU-hotplug
+ operation ends, which could also result in deadlock. Furthermore,
-------``rcu_barrier()`` blocks CPU-hotplug operations during its execution,
++++++++rcu_barrier() blocks CPU-hotplug operations during its execution,
+ which results in another type of deadlock when invoked from a CPU-hotplug
+ notifier.
+
+ Finally, RCU must avoid deadlocks due to interaction between hotplug,
+ timers and grace period processing. It does so by maintaining its own set
+ of books that duplicate the centrally maintained ``cpu_online_mask``,
+ and also by reporting quiescent states explicitly when a CPU goes
+ offline. This explicit reporting of quiescent states avoids any need
+ for the force-quiescent-state loop (FQS) to report quiescent states for
+ offline CPUs. However, as a debugging measure, the FQS loop does splat
+ if offline CPUs block an RCU grace period for too long.
+
+ An offline CPU's quiescent state will be reported either:
+
-------1. As the CPU goes offline using RCU's hotplug notifier (``rcu_report_dead()``).
-------2. When grace period initialization (``rcu_gp_init()``) detects a
++++++++1. As the CPU goes offline using RCU's hotplug notifier (rcu_report_dead()).
++++++++2. When grace period initialization (rcu_gp_init()) detects a
+ race either with CPU offlining or with a task unblocking on a leaf
+ ``rcu_node`` structure whose CPUs are all offline.
+
-------The CPU-online path (``rcu_cpu_starting()``) should never need to report
++++++++The CPU-online path (rcu_cpu_starting()) should never need to report
+ a quiescent state for an offline CPU. However, as a debugging measure,
+ it does emit a warning if a quiescent state was not already reported
+ for that CPU.
+
+ During the checking/modification of RCU's hotplug bookkeeping, the
+ corresponding CPU's leaf node lock is held. This avoids race conditions
+ between RCU's hotplug notifier hooks, the grace period initialization
+ code, and the FQS loop, all of which refer to or modify this bookkeeping.
Scheduler and RCU
~~~~~~~~~~~~~~~~~
There is no longer any prohibition against holding any of
scheduler's runqueue or priority-inheritance spinlocks across an
--------``rcu_read_unlock()``, even if interrupts and preemption were enabled
++++++++rcu_read_unlock(), even if interrupts and preemption were enabled
somewhere within the corresponding RCU read-side critical section.
--------Therefore, it is now perfectly legal to execute ``rcu_read_lock()``
++++++++Therefore, it is now perfectly legal to execute rcu_read_lock()
with preemption enabled, acquire one of the scheduler locks, and hold
--------that lock across the matching ``rcu_read_unlock()``.
++++++++that lock across the matching rcu_read_unlock().
Similarly, the RCU flavor consolidation has removed the need for negative
nesting. The fact that interrupt-disabled regions of code act as RCU
~~~~~~~~~~~~~~~
It is possible to use tracing on RCU code, but tracing itself uses RCU.
--------For this reason, ``rcu_dereference_raw_check()`` is provided for use
++++++++For this reason, rcu_dereference_raw_check() is provided for use
by tracing, which avoids the destructive recursion that could otherwise
ensue. This API is also used by virtualization in some architectures,
where RCU readers execute in environments in which tracing cannot be
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The kernel needs to access user-space memory, for example, to access data
--------referenced by system-call parameters. The ``get_user()`` macro does this job.
++++++++referenced by system-call parameters. The get_user() macro does this job.
However, user-space memory might well be paged out, which means that
--------``get_user()`` might well page-fault and thus block while waiting for the
++++++++get_user() might well page-fault and thus block while waiting for the
resulting I/O to complete. It would be a very bad thing for the compiler to
--------reorder a ``get_user()`` invocation into an RCU read-side critical section.
++++++++reorder a get_user() invocation into an RCU read-side critical section.
For example, suppose that the source code looked like this:
5 rcu_read_unlock();
6 do_something_with(v, user_v);
--------If the compiler did make this transformation in a ``CONFIG_PREEMPT=n`` kernel
--------build, and if ``get_user()`` did page fault, the result would be a quiescent
++++++++If the compiler did make this transformation in a ``CONFIG_PREEMPTION=n`` kernel
++++++++build, and if get_user() did page fault, the result would be a quiescent
state in the middle of an RCU read-side critical section. This misplaced
quiescent state could result in line 4 being a use-after-free access,
which could be bad for your kernel's actuarial statistics. Similar examples
--------can be constructed with the call to ``get_user()`` preceding the
--------``rcu_read_lock()``.
++++++++can be constructed with the call to get_user() preceding the
++++++++rcu_read_lock().
--------Unfortunately, ``get_user()`` doesn't have any particular ordering properties,
++++++++Unfortunately, get_user() doesn't have any particular ordering properties,
and in some architectures the underlying ``asm`` isn't even marked
``volatile``. And even if it was marked ``volatile``, the above access to
``p->value`` is not volatile, so the compiler would not have any reason to keep
those two accesses in order.
--------Therefore, the Linux-kernel definitions of ``rcu_read_lock()`` and
--------``rcu_read_unlock()`` must act as compiler barriers, at least for outermost
--------instances of ``rcu_read_lock()`` and ``rcu_read_unlock()`` within a nested set
++++++++Therefore, the Linux-kernel definitions of rcu_read_lock() and
++++++++rcu_read_unlock() must act as compiler barriers, at least for outermost
++++++++instances of rcu_read_lock() and rcu_read_unlock() within a nested set
of RCU read-side critical sections.
Energy Efficiency
Because RCU avoids interrupting idle CPUs, it is illegal to execute an
RCU read-side critical section on an idle CPU. (Kernels built with
--------``CONFIG_PROVE_RCU=y`` will splat if you try it.) The ``RCU_NONIDLE()``
++++++++``CONFIG_PROVE_RCU=y`` will splat if you try it.) The RCU_NONIDLE()
macro and ``_rcuidle`` event tracing is provided to work around this
--------restriction. In addition, ``rcu_is_watching()`` may be used to test
++++++++restriction. In addition, rcu_is_watching() may be used to test
whether or not it is currently legal to run RCU read-side critical
sections on this CPU. I learned of the need for diagnostics on the one
--------hand and ``RCU_NONIDLE()`` on the other while inspecting idle-loop code.
++++++++hand and RCU_NONIDLE() on the other while inspecting idle-loop code.
Steven Rostedt supplied ``_rcuidle`` event tracing, which is used quite
heavily in the idle loop. However, there are some restrictions on the
--------code placed within ``RCU_NONIDLE()``:
++++++++code placed within RCU_NONIDLE():
#. Blocking is prohibited. In practice, this is not a serious
restriction given that idle tasks are prohibited from blocking to
begin with.
--------#. Although nesting ``RCU_NONIDLE()`` is permitted, they cannot nest
++++++++#. Although nesting RCU_NONIDLE() is permitted, they cannot nest
indefinitely deeply. However, given that they can be nested on the
order of a million deep, even on 32-bit systems, this should not be a
serious restriction. This nesting limit would probably be reached
long after the compiler OOMed or the stack overflowed.
--------#. Any code path that enters ``RCU_NONIDLE()`` must sequence out of that
-------- same ``RCU_NONIDLE()``. For example, the following is grossly
++++++++#. Any code path that enters RCU_NONIDLE() must sequence out of that
++++++++ same RCU_NONIDLE(). For example, the following is grossly
illegal:
::
It is just as illegal to transfer control into the middle of
-------- ``RCU_NONIDLE()``'s argument. Yes, in theory, you could transfer in
++++++++ RCU_NONIDLE()'s argument. Yes, in theory, you could transfer in
as long as you also transferred out, but in practice you could also
expect to get sharply worded review comments.
sections, and RCU believes this CPU to be idle, no problem. This
sort of thing is used by some architectures for light-weight
exception handlers, which can then avoid the overhead of
-------- ``rcu_irq_enter()`` and ``rcu_irq_exit()`` at exception entry and
++++++++ rcu_irq_enter() and rcu_irq_exit() at exception entry and
exit, respectively. Some go further and avoid the entireties of
-------- ``irq_enter()`` and ``irq_exit()``.
++++++++ irq_enter() and irq_exit().
Just make very sure you are running some of your tests with
``CONFIG_PROVE_RCU=y``, just in case one of your code paths was in
fact joking about not doing RCU read-side critical sections.
| **Quick Quiz**: |
+-----------------------------------------------------------------------+
| But what if my driver has a hardware interrupt handler that can run |
--------| for many seconds? I cannot invoke ``schedule()`` from an hardware |
++++++++| for many seconds? I cannot invoke schedule() from an hardware |
| interrupt handler, after all! |
+-----------------------------------------------------------------------+
| **Answer**: |
Although small-memory non-realtime systems can simply use Tiny RCU, code
size is only one aspect of memory efficiency. Another aspect is the size
--------of the ``rcu_head`` structure used by ``call_rcu()`` and
--------``kfree_rcu()``. Although this structure contains nothing more than a
++++++++of the ``rcu_head`` structure used by call_rcu() and
++++++++kfree_rcu(). Although this structure contains nothing more than a
pair of pointers, it does appear in many RCU-protected data structures,
including some that are size critical. The ``page`` structure is a case
in point, as evidenced by the many occurrences of the ``union`` keyword
singly linked lists to track the ``rcu_head`` structures that are
waiting for a grace period to elapse. It is also the reason why
``rcu_head`` structures do not contain debug information, such as fields
--------tracking the file and line of the ``call_rcu()`` or ``kfree_rcu()`` that
++++++++tracking the file and line of the call_rcu() or kfree_rcu() that
posted them. Although this information might appear in debug-only kernel
builds at some point, in the meantime, the ``->func`` field will often
provide the needed debug information.
``rcu_head`` field shares storage with a great many other structures
that are used at various points in the corresponding page's lifetime. In
order to correctly resolve certain `race
--------conditions <https://lkml.kernel.org/g/1439976106-137226-1-git-send-email-kirill.shutemov@linux.intel.com>`__,
++++++++conditions <https://lore.kernel.org/r/1439976106-137226-1-git-send-email-kirill.shutemov@linux.intel.com>`__,
the Linux kernel's memory-management subsystem needs a particular bit to
remain zero during all phases of grace-period processing, and that bit
happens to map to the bottom bit of the ``rcu_head`` structure's
--------``->next`` field. RCU makes this guarantee as long as ``call_rcu()`` is
--------used to post the callback, as opposed to ``kfree_rcu()`` or some future
--------“lazy” variant of ``call_rcu()`` that might one day be created for
++++++++``->next`` field. RCU makes this guarantee as long as call_rcu() is
++++++++used to post the callback, as opposed to kfree_rcu() or some future
++++++++“lazy” variant of call_rcu() that might one day be created for
energy-efficiency purposes.
That said, there are limits. RCU requires that the ``rcu_head``
structure be aligned to a two-byte boundary, and passing a misaligned
--------``rcu_head`` structure to one of the ``call_rcu()`` family of functions
++++++++``rcu_head`` structure to one of the call_rcu() family of functions
will result in a splat. It is therefore necessary to exercise caution
when packing structures containing fields of type ``rcu_head``. Why not
a four-byte or even eight-byte alignment requirement? Because the m68k
networking, security, virtualization, and scheduling code paths. RCU
must therefore use efficient implementations, especially in its
read-side primitives. To that end, it would be good if preemptible RCU's
--------implementation of ``rcu_read_lock()`` could be inlined, however, doing
++++++++implementation of rcu_read_lock() could be inlined, however, doing
this requires resolving ``#include`` issues with the ``task_struct``
structure.
continuously invoking any combination of RCU's runtime primitives with
minimal per-operation overhead. In fact, in many cases, increasing load
must *decrease* the per-operation overhead, witness the batching
--------optimizations for ``synchronize_rcu()``, ``call_rcu()``,
--------``synchronize_rcu_expedited()``, and ``rcu_barrier()``. As a general
++++++++optimizations for synchronize_rcu(), call_rcu(),
++++++++synchronize_rcu_expedited(), and rcu_barrier(). As a general
rule, RCU must cheerfully accept whatever the rest of the Linux kernel
decides to throw at it.
The Linux kernel is used for real-time workloads, especially in
conjunction with the `-rt
--------patchset <https://rt.wiki.kernel.org/index.php/Main_Page>`__. The
++++++++patchset <https://wiki.linuxfoundation.org/realtime/>`__. The
real-time-latency response requirements are such that the traditional
approach of disabling preemption across RCU read-side critical sections
--------is inappropriate. Kernels built with ``CONFIG_PREEMPT=y`` therefore use
++++++++is inappropriate. Kernels built with ``CONFIG_PREEMPTION=y`` therefore use
an RCU implementation that allows RCU read-side critical sections to be
preempted. This requirement made its presence known after users made it
clear that an earlier `real-time
patch <https://lwn.net/Articles/107930/>`__ did not meet their needs, in
conjunction with some `RCU
encountered by a very early version of the -rt patchset.
In addition, RCU must make do with a sub-100-microsecond real-time
RCU must avoid degrading real-time response for CPU-bound threads,
whether executing in usermode (which is one use case for
``CONFIG_NO_HZ_FULL=y``) or in the kernel. That said, CPU-bound loops in
--------the kernel must execute ``cond_resched()`` at least once per few tens of
++++++++the kernel must execute cond_resched() at least once per few tens of
milliseconds in order to avoid receiving an IPI from RCU.
Finally, RCU's status as a synchronization primitive means that any RCU
condition and a system hang.
The solution was the creation of RCU-bh, which does
--------``local_bh_disable()`` across its read-side critical sections, and which
++++++++local_bh_disable() across its read-side critical sections, and which
uses the transition from one type of softirq processing to another as a
quiescent state in addition to context switch, idle, user mode, and
offline. This means that RCU-bh grace periods can complete even when
algorithms based on RCU-bh to withstand network-based denial-of-service
attacks.
--------Because ``rcu_read_lock_bh()`` and ``rcu_read_unlock_bh()`` disable and
++++++++Because rcu_read_lock_bh() and rcu_read_unlock_bh() disable and
re-enable softirq handlers, any attempt to start a softirq handlers
during the RCU-bh read-side critical section will be deferred. In this
--------case, ``rcu_read_unlock_bh()`` will invoke softirq processing, which can
++++++++case, rcu_read_unlock_bh() will invoke softirq processing, which can
take considerable time. One can of course argue that this softirq
overhead should be associated with the code following the RCU-bh
--------read-side critical section rather than ``rcu_read_unlock_bh()``, but the
++++++++read-side critical section rather than rcu_read_unlock_bh(), but the
fact is that most profiling tools cannot be expected to make this sort
of fine distinction. For example, suppose that a three-millisecond-long
RCU-bh read-side critical section executes during a time of heavy
networking load. There will very likely be an attempt to invoke at least
one softirq handler during that three milliseconds, but any such
invocation will be delayed until the time of the
--------``rcu_read_unlock_bh()``. This can of course make it appear at first
--------glance as if ``rcu_read_unlock_bh()`` was executing very slowly.
++++++++rcu_read_unlock_bh(). This can of course make it appear at first
++++++++glance as if rcu_read_unlock_bh() was executing very slowly.
The `RCU-bh
API <https://lwn.net/Articles/609973/#RCU%20Per-Flavor%20API%20Table>`__
--------includes ``rcu_read_lock_bh()``, ``rcu_read_unlock_bh()``,
--------``rcu_dereference_bh()``, ``rcu_dereference_bh_check()``,
--------``synchronize_rcu_bh()``, ``synchronize_rcu_bh_expedited()``,
--------``call_rcu_bh()``, ``rcu_barrier_bh()``, and
--------``rcu_read_lock_bh_held()``. However, the update-side APIs are now
--------simple wrappers for other RCU flavors, namely RCU-sched in
--------CONFIG_PREEMPT=n kernels and RCU-preempt otherwise.
++++++++includes rcu_read_lock_bh(), rcu_read_unlock_bh(), rcu_dereference_bh(),
++++++++rcu_dereference_bh_check(), and rcu_read_lock_bh_held(). However, the
++++++++old RCU-bh update-side APIs are now gone, replaced by synchronize_rcu(),
++++++++synchronize_rcu_expedited(), call_rcu(), and rcu_barrier(). In addition,
++++++++anything that disables bottom halves also marks an RCU-bh read-side
++++++++critical section, including local_bh_disable() and local_bh_enable(),
++++++++local_irq_save() and local_irq_restore(), and so on.
Sched Flavor (Historical)
~~~~~~~~~~~~~~~~~~~~~~~~~
RCU read-side critical section can be a quiescent state. Therefore,
*RCU-sched* was created, which follows “classic” RCU in that an
RCU-sched grace period waits for pre-existing interrupt and NMI
--------handlers. In kernels built with ``CONFIG_PREEMPT=n``, the RCU and
++++++++handlers. In kernels built with ``CONFIG_PREEMPTION=n``, the RCU and
RCU-sched APIs have identical implementations, while kernels built with
--------``CONFIG_PREEMPT=y`` provide a separate implementation for each.
++++++++``CONFIG_PREEMPTION=y`` provide a separate implementation for each.
--------Note well that in ``CONFIG_PREEMPT=y`` kernels,
--------``rcu_read_lock_sched()`` and ``rcu_read_unlock_sched()`` disable and
++++++++Note well that in ``CONFIG_PREEMPTION=y`` kernels,
++++++++rcu_read_lock_sched() and rcu_read_unlock_sched() disable and
re-enable preemption, respectively. This means that if there was a
preemption attempt during the RCU-sched read-side critical section,
--------``rcu_read_unlock_sched()`` will enter the scheduler, with all the
--------latency and overhead entailed. Just as with ``rcu_read_unlock_bh()``,
--------this can make it look as if ``rcu_read_unlock_sched()`` was executing
++++++++rcu_read_unlock_sched() will enter the scheduler, with all the
++++++++latency and overhead entailed. Just as with rcu_read_unlock_bh(),
++++++++this can make it look as if rcu_read_unlock_sched() was executing
very slowly. However, the highest-priority task won't be preempted, so
--------that task will enjoy low-overhead ``rcu_read_unlock_sched()``
++++++++that task will enjoy low-overhead rcu_read_unlock_sched()
invocations.
The `RCU-sched
API <https://lwn.net/Articles/609973/#RCU%20Per-Flavor%20API%20Table>`__
--------includes ``rcu_read_lock_sched()``, ``rcu_read_unlock_sched()``,
--------``rcu_read_lock_sched_notrace()``, ``rcu_read_unlock_sched_notrace()``,
--------``rcu_dereference_sched()``, ``rcu_dereference_sched_check()``,
--------``synchronize_sched()``, ``synchronize_rcu_sched_expedited()``,
--------``call_rcu_sched()``, ``rcu_barrier_sched()``, and
--------``rcu_read_lock_sched_held()``. However, anything that disables
--------preemption also marks an RCU-sched read-side critical section, including
--------``preempt_disable()`` and ``preempt_enable()``, ``local_irq_save()`` and
--------``local_irq_restore()``, and so on.
++++++++includes rcu_read_lock_sched(), rcu_read_unlock_sched(),
++++++++rcu_read_lock_sched_notrace(), rcu_read_unlock_sched_notrace(),
++++++++rcu_dereference_sched(), rcu_dereference_sched_check(), and
++++++++rcu_read_lock_sched_held(). However, the old RCU-sched update-side APIs
++++++++are now gone, replaced by synchronize_rcu(), synchronize_rcu_expedited(),
++++++++call_rcu(), and rcu_barrier(). In addition, anything that disables
++++++++preemption also marks an RCU-sched read-side critical section,
++++++++including preempt_disable() and preempt_enable(), local_irq_save()
++++++++and local_irq_restore(), and so on.
Sleepable RCU
~~~~~~~~~~~~~
structure. The key benefit of these domains is that a slow SRCU reader
in one domain does not delay an SRCU grace period in some other domain.
That said, one consequence of these domains is that read-side code must
--------pass a “cookie” from ``srcu_read_lock()`` to ``srcu_read_unlock()``, for
++++++++pass a “cookie” from srcu_read_lock() to srcu_read_unlock(), for
example, as follows:
::
6 srcu_read_unlock(&ss, idx);
However, if line 5 acquired a mutex that was held across a
--------``synchronize_srcu()`` for domain ``ss``, deadlock would still be
++++++++synchronize_srcu() for domain ``ss``, deadlock would still be
possible. Furthermore, if line 5 acquired a mutex that was held across a
--------``synchronize_srcu()`` for some other domain ``ss1``, and if an
++++++++synchronize_srcu() for some other domain ``ss1``, and if an
``ss1``-domain SRCU read-side critical section acquired another mutex
--------that was held across as ``ss``-domain ``synchronize_srcu()``, deadlock
++++++++that was held across as ``ss``-domain synchronize_srcu(), deadlock
would again be possible. Such a deadlock cycle could extend across an
arbitrarily large number of different SRCU domains. Again, with great
power comes great responsibility.
Unlike the other RCU flavors, SRCU read-side critical sections can run
on idle and even offline CPUs. This ability requires that
--------``srcu_read_lock()`` and ``srcu_read_unlock()`` contain memory barriers,
++++++++srcu_read_lock() and srcu_read_unlock() contain memory barriers,
which means that SRCU readers will run a bit slower than would RCU
--------readers. It also motivates the ``smp_mb__after_srcu_read_unlock()`` API,
--------which, in combination with ``srcu_read_unlock()``, guarantees a full
++++++++readers. It also motivates the smp_mb__after_srcu_read_unlock() API,
++++++++which, in combination with srcu_read_unlock(), guarantees a full
memory barrier.
--------Also unlike other RCU flavors, ``synchronize_srcu()`` may **not** be
++++++++Also unlike other RCU flavors, synchronize_srcu() may **not** be
invoked from CPU-hotplug notifiers, due to the fact that SRCU grace
periods make use of timers and the possibility of timers being
temporarily “stranded” on the outgoing CPU. This stranding of timers
an SRCU grace period, that grace period is waiting on a timer, and that
timer is stranded on the outgoing CPU, then the notifier will never be
awakened, in other words, deadlock has occurred. This same situation of
--------course also prohibits ``srcu_barrier()`` from being invoked from
++++++++course also prohibits srcu_barrier() from being invoked from
CPU-hotplug notifiers.
SRCU also differs from other RCU flavors in that SRCU's expedited and
current implementation, not necessarily of future implementations.) In
addition, if SRCU has been idle for longer than the interval specified
by the ``srcutree.exp_holdoff`` kernel boot parameter (25 microseconds
--------by default), and if a ``synchronize_srcu()`` invocation ends this idle
++++++++by default), and if a synchronize_srcu() invocation ends this idle
period, that invocation will be automatically expedited.
As of v4.12, SRCU's callbacks are maintained per-CPU, eliminating a
locking bottleneck present in prior kernel versions. Although this will
--------allow users to put much heavier stress on ``call_srcu()``, it is
++++++++allow users to put much heavier stress on call_srcu(), it is
important to note that SRCU does not yet take any special steps to deal
with callback flooding. So if you are posting (say) 10,000 SRCU
callbacks per second per CPU, you are probably totally OK, but if you
The `SRCU
API <https://lwn.net/Articles/609973/#RCU%20Per-Flavor%20API%20Table>`__
--------includes ``srcu_read_lock()``, ``srcu_read_unlock()``,
--------``srcu_dereference()``, ``srcu_dereference_check()``,
--------``synchronize_srcu()``, ``synchronize_srcu_expedited()``,
--------``call_srcu()``, ``srcu_barrier()``, and ``srcu_read_lock_held()``. It
--------also includes ``DEFINE_SRCU()``, ``DEFINE_STATIC_SRCU()``, and
--------``init_srcu_struct()`` APIs for defining and initializing
++++++++includes srcu_read_lock(), srcu_read_unlock(),
++++++++srcu_dereference(), srcu_dereference_check(),
++++++++synchronize_srcu(), synchronize_srcu_expedited(),
++++++++call_srcu(), srcu_barrier(), and srcu_read_lock_held(). It
++++++++also includes DEFINE_SRCU(), DEFINE_STATIC_SRCU(), and
++++++++init_srcu_struct() APIs for defining and initializing
``srcu_struct`` structures.
+++++++ +More recently, the SRCU API has added polling interfaces:
+++++++ +
+++++++ +#. start_poll_synchronize_srcu() returns a cookie identifying
+++++++ + the completion of a future SRCU grace period and ensures
+++++++ + that this grace period will be started.
+++++++ +#. poll_state_synchronize_srcu() returns ``true`` iff the
+++++++ + specified cookie corresponds to an already-completed
+++++++ + SRCU grace period.
+++++++ +#. get_state_synchronize_srcu() returns a cookie just like
+++++++ + start_poll_synchronize_srcu() does, but differs in that
+++++++ + it does nothing to ensure that any future SRCU grace period
+++++++ + will be started.
+++++++ +
+++++++ +These functions are used to avoid unnecessary SRCU grace periods in
+++++++ +certain types of buffer-cache algorithms having multi-stage age-out
+++++++ +mechanisms. The idea is that by the time the block has aged completely
+++++++ +from the cache, an SRCU grace period will be very likely to have elapsed.
+++++++ +
Tasks RCU
~~~~~~~~~
able to free old trampolines, which sounds like a job for some form of
RCU. However, because it is necessary to be able to install a trace
anywhere in the code, it is not possible to use read-side markers such
--------as ``rcu_read_lock()`` and ``rcu_read_unlock()``. In addition, it does
++++++++as rcu_read_lock() and rcu_read_unlock(). In addition, it does
not work to have these markers in the trampoline itself, because there
--------would need to be instructions following ``rcu_read_unlock()``. Although
--------``synchronize_rcu()`` would guarantee that execution reached the
--------``rcu_read_unlock()``, it would not be able to guarantee that execution
++++++++would need to be instructions following rcu_read_unlock(). Although
++++++++synchronize_rcu() would guarantee that execution reached the
++++++++rcu_read_unlock(), it would not be able to guarantee that execution
had completely left the trampoline. Worse yet, in some situations
the trampoline's protection must extend a few instructions *prior* to
execution reaching the trampoline. For example, these few instructions
The solution, in the form of `Tasks
RCU <https://lwn.net/Articles/607117/>`__, is to have implicit read-side
critical sections that are delimited by voluntary context switches, that
--------is, calls to ``schedule()``, ``cond_resched()``, and
--------``synchronize_rcu_tasks()``. In addition, transitions to and from
++++++++is, calls to schedule(), cond_resched(), and
++++++++synchronize_rcu_tasks(). In addition, transitions to and from
userspace execution also delimit tasks-RCU read-side critical sections.
The tasks-RCU API is quite compact, consisting only of
--------``call_rcu_tasks()``, ``synchronize_rcu_tasks()``, and
--------``rcu_barrier_tasks()``. In ``CONFIG_PREEMPT=n`` kernels, trampolines
--------cannot be preempted, so these APIs map to ``call_rcu()``,
--------``synchronize_rcu()``, and ``rcu_barrier()``, respectively. In
--------``CONFIG_PREEMPT=y`` kernels, trampolines can be preempted, and these
++++++++call_rcu_tasks(), synchronize_rcu_tasks(), and
++++++++rcu_barrier_tasks(). In ``CONFIG_PREEMPTION=n`` kernels, trampolines
++++++++cannot be preempted, so these APIs map to call_rcu(),
++++++++synchronize_rcu(), and rcu_barrier(), respectively. In
++++++++``CONFIG_PREEMPTION=y`` kernels, trampolines can be preempted, and these
three APIs are therefore implemented by separate functions that check
for voluntary context switches.
latency.
RCU disables CPU hotplug in a few places, perhaps most notably in the
--------``rcu_barrier()`` operations. If there is a strong reason to use
--------``rcu_barrier()`` in CPU-hotplug notifiers, it will be necessary to
++++++++rcu_barrier() operations. If there is a strong reason to use
++++++++rcu_barrier() in CPU-hotplug notifiers, it will be necessary to
avoid disabling CPU hotplug. This would introduce some complexity, so
there had better be a *very* good reason.
nodes nor does it align the CPU groups with hardware features such as
sockets or cores. Such spreading and alignment is currently believed to
be unnecessary because the hotpath read-side primitives do not access
--------the combining tree, nor does ``call_rcu()`` in the common case. If you
++++++++the combining tree, nor does call_rcu() in the common case. If you
believe that your architecture needs such spreading and alignment, then
your architecture should also benefit from the
``rcutree.rcu_fanout_leaf`` boot parameter, which can be set to the
extreme loads. It might also be necessary to be able to relate CPU
utilization by RCU's kthreads and softirq handlers to the code that
instigated this CPU utilization. For example, RCU callback overhead
--------might be charged back to the originating ``call_rcu()`` instance, though
++++++++might be charged back to the originating call_rcu() instance, though
probably not in production kernels.
Additional work may be required to provide reasonable forward-progress
- A CPU looping with bottom halves disabled.
--------- For !CONFIG_PREEMPT kernels, a CPU looping anywhere in the kernel
++++++++- For !CONFIG_PREEMPTION kernels, a CPU looping anywhere in the kernel
without invoking schedule(). If the looping in the kernel is
really expected and desirable behavior, you might need to add
some calls to cond_resched().
result in the ``rcu_.*kthread starved for`` console-log message,
which will include additional debugging information.
--------- A CPU-bound real-time task in a CONFIG_PREEMPT kernel, which might
++++++++- A CPU-bound real-time task in a CONFIG_PREEMPTION kernel, which might
happen to preempt a low-priority task in the middle of an RCU
read-side critical section. This is especially damaging if
that low-priority task is not permitted to run on any other CPU,
buggy timer hardware through bugs in the interrupt or exception
path (whether hardware, firmware, or software) through bugs
in Linux's timer subsystem through bugs in the scheduler, and,
------ -- yes, even including bugs in RCU itself.
++++++ ++ yes, even including bugs in RCU itself. It can also result in
++++++ ++ the ``rcu_.*timer wakeup didn't happen for`` console-log message,
++++++ ++ which will include additional debugging information.
- A bug in the RCU implementation.
task_struct ->state field, and the "cpu" indicates that the grace-period
kthread last ran on CPU 5.
++++++ ++If the relevant grace-period kthread does not wake from FQS wait in a
++++++ ++reasonable time, then the following additional line is printed::
++++++ ++
++++++ ++ kthread timer wakeup didn't happen for 23804 jiffies! g7076 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402
++++++ ++
++++++ ++The "23804" indicates that kthread's timer expired more than 23 thousand
++++++ ++jiffies ago. The rest of the line has meaning similar to the kthread
++++++ ++starvation case.
++++++ ++
++++++ ++Additionally, the following line is printed::
++++++ ++
++++++ ++ Possible timer handling issue on cpu=4 timer-softirq=11142
++++++ ++
++++++ ++Here "cpu" indicates that the grace-period kthread last ran on CPU 4,
++++++ ++where it queued the fqs timer. The number following the "timer-softirq"
++++++ ++is the current ``TIMER_SOFTIRQ`` count on cpu 4. If this value does not
++++++ ++change on successive RCU CPU stall warnings, there is further reason to
++++++ ++suspect a timer problem.
++++++ ++
Multiple Warnings From One Stall
================================
Note that using this option lowers the security
provided by tboot because it makes the system
vulnerable to DMA attacks.
- nobounce [Default off]
- Disable bounce buffer for untrusted devices such as
- the Thunderbolt devices. This will treat the untrusted
- devices as the trusted ones, hence might expose security
- risks of DMA attacks.
intel_idle.max_cstate= [KNL,HW,ACPI,X86]
0 disables intel_idle and fall back on acpi_idle.
for all guests.
Default is 1 (enabled) if in 64-bit or 32-bit PAE mode.
+ kvm-arm.mode=
+ [KVM,ARM] Select one of KVM/arm64's modes of operation.
+
+ protected: nVHE-based mode with support for guests whose
+ state is kept private from the host.
+ Not valid if the kernel is running in EL2.
+
+ Defaults to VHE/nVHE based on hardware support and
+ the value of CONFIG_ARM64_VHE.
+
kvm-arm.vgic_v3_group0_trap=
[KVM,ARM] Trap guest accesses to GICv3 group-0
system registers
option description.
memmap=nn[KMG]@ss[KMG]
- [KNL] Force usage of a specific region of memory.
+ [KNL, X86, MIPS, XTENSA] Force usage of a specific region of memory.
Region of memory to be used is from ss to ss+nn.
If @ss[KMG] is omitted, it is equivalent to mem=nn[KMG],
which limits max address to nn[KMG].
mtdset= [ARM]
ARM/S3C2412 JIVE boot control
- See arch/arm/mach-s3c2412/mach-jive.c
+ See arch/arm/mach-s3c/mach-jive.c
mtouchusb.raw_coordinates=
[HW] Make the MicroTouch USB driver use raw coordinates
nosep [BUGS=X86-32] Disables x86 SYSENTER/SYSEXIT support.
+ nosgx [X86-64,SGX] Disables Intel SGX kernel support.
+
nosmp [SMP] Tells an SMP kernel to act as a UP kernel,
and disable the IO APIC. legacy for "maxcpus=0".
value, meaning that RCU_SOFTIRQ is used by default.
Specify rcutree.use_softirq=0 to use rcuc kthreads.
+++++ +++ But note that CONFIG_PREEMPT_RT=y kernels disable
+++++ +++ this kernel boot parameter, forcibly setting it
+++++ +++ to zero.
+++++ +++
rcutree.rcu_fanout_exact= [KNL]
Disable autobalancing of the rcu_node combining
tree. This is used by rcutorture, and might
Set wakeup interval for idle CPUs that have
RCU callbacks (RCU_FAST_NO_HZ=y).
-------- rcutree.rcu_idle_lazy_gp_delay= [KNL]
-------- Set wakeup interval for idle CPUs that have
-------- only "lazy" RCU callbacks (RCU_FAST_NO_HZ=y).
-------- Lazy RCU callbacks are those which RCU can
-------- prove do nothing more than free memory.
--------
rcutree.rcu_kick_kthreads= [KNL]
Cause the grace-period kthread to get an extra
wake_up() if it sleeps three times longer than
stress RCU, they don't participate in the actual
test, hence the "fake".
++++ ++++ rcutorture.nocbs_nthreads= [KNL]
++++ ++++ Set number of RCU callback-offload togglers.
++++ ++++ Zero (the default) disables toggling.
++++ ++++
++++ ++++ rcutorture.nocbs_toggle= [KNL]
++++ ++++ Set the delay in milliseconds between successive
++++ ++++ callback-offload toggling attempts.
++++ ++++
rcutorture.nreaders= [KNL]
Set number of RCU readers. The value -1 selects
N-1, where N is the number of CPUs. A value
only normal grace-period primitives. No effect
on CONFIG_TINY_RCU kernels.
+++++ +++ But note that CONFIG_PREEMPT_RT=y kernels enables
+++++ +++ this kernel boot parameter, forcibly setting
+++++ +++ it to the value one, that is, converting any
+++++ +++ post-boot attempt at an expedited RCU grace
+++++ +++ period to instead use normal non-expedited
+++++ +++ grace-period processing.
+++++ +++
rcupdate.rcu_task_ipi_delay= [KNL]
Set time in jiffies during which RCU tasks will
avoid sending IPIs, starting with the beginning
refscale.verbose= [KNL]
Enable additional printk() statements.
+++++++ + refscale.verbose_batched= [KNL]
+++++++ + Batch the additional printk() statements. If zero
+++++++ + (the default) or negative, print everything. Otherwise,
+++++++ + print every Nth verbose statement, where N is the value
+++++++ + specified.
+++++++ +
relax_domain_level=
[KNL, SMP] Set scheduler's default relax_domain_level.
See Documentation/admin-guide/cgroup-v1/cpusets.rst.
are running concurrently, especially on systems
with rotating-rust storage.
+++++++ + torture.verbose_sleep_frequency= [KNL]
+++++++ + Specifies how many verbose printk()s should be
+++++++ + emitted between each sleep. The default of zero
+++++++ + disables verbose-printk() sleeping.
+++++++ +
+++++++ + torture.verbose_sleep_duration= [KNL]
+++++++ + Duration of each verbose-printk() sleep in jiffies.
+++++++ +
tp720= [HW,PS2]
tpm_suspend_pcr=[HW,TPM]
device);
j = NO_REPORT_LUNS (don't use report luns
command, uas only);
+ k = NO_SAME (do not use WRITE_SAME, uas only)
l = NOT_LOCKABLE (don't try to lock and
unlock ejectable media, not on uas);
m = MAX_SECTORS_64 (don't transfer more
#define ULONG_CMP_GE(a, b) (ULONG_MAX / 2 >= (a) - (b))
#define ULONG_CMP_LT(a, b) (ULONG_MAX / 2 < (a) - (b))
#define ulong2long(a) (*(long *)(&(a)))
+++++++ +#define USHORT_CMP_GE(a, b) (USHRT_MAX / 2 >= (unsigned short)((a) - (b)))
+++++++ +#define USHORT_CMP_LT(a, b) (USHRT_MAX / 2 < (unsigned short)((a) - (b)))
/* Exported common interfaces */
void call_rcu(struct rcu_head *head, rcu_callback_t func);
void rcu_report_dead(unsigned int cpu);
void rcutree_migrate_callbacks(int cpu);
+ +++++++#ifdef CONFIG_TASKS_RCU_GENERIC
+ +++++++void rcu_init_tasks_generic(void);
+ +++++++#else
+ +++++++static inline void rcu_init_tasks_generic(void) { }
+ +++++++#endif
+ +++++++
#ifdef CONFIG_RCU_STALL_COMMON
void rcu_sysrq_start(void);
void rcu_sysrq_end(void);
#ifdef CONFIG_RCU_NOCB_CPU
void rcu_init_nohz(void);
++++ ++++int rcu_nocb_cpu_offload(int cpu);
++++ ++++int rcu_nocb_cpu_deoffload(int cpu);
#else /* #ifdef CONFIG_RCU_NOCB_CPU */
static inline void rcu_init_nohz(void) { }
++++ ++++static inline int rcu_nocb_cpu_offload(int cpu) { return -EINVAL; }
++++ ++++static inline int rcu_nocb_cpu_deoffload(int cpu) { return 0; }
#endif /* #else #ifdef CONFIG_RCU_NOCB_CPU */
/**
static inline bool rcu_lockdep_current_cpu_online(void) { return true; }
#endif /* #else #if defined(CONFIG_HOTPLUG_CPU) && defined(CONFIG_PROVE_RCU) */
+ extern struct lockdep_map rcu_lock_map;
+ extern struct lockdep_map rcu_bh_lock_map;
+ extern struct lockdep_map rcu_sched_lock_map;
+ extern struct lockdep_map rcu_callback_map;
+
#ifdef CONFIG_DEBUG_LOCK_ALLOC
static inline void rcu_lock_acquire(struct lockdep_map *map)
lock_release(map, _THIS_IP_);
}
- extern struct lockdep_map rcu_lock_map;
- extern struct lockdep_map rcu_bh_lock_map;
- extern struct lockdep_map rcu_sched_lock_map;
- extern struct lockdep_map rcu_callback_map;
int debug_lockdep_rcu_enabled(void);
int rcu_read_lock_held(void);
int rcu_read_lock_bh_held(void);
#else /* #ifdef CONFIG_PROVE_RCU */
- #define RCU_LOCKDEP_WARN(c, s) do { } while (0)
+ #define RCU_LOCKDEP_WARN(c, s) do { } while (0 && (c))
#define rcu_sleep_check() do { } while (0)
#endif /* #else #ifdef CONFIG_PROVE_RCU */
*/
#define __is_kvfree_rcu_offset(offset) ((offset) < 4096)
-- ------/*
-- ------ * Helper macro for kfree_rcu() to prevent argument-expansion eyestrain.
-- ------ */
-- ------#define __kvfree_rcu(head, offset) \
-- ------ do { \
-- ------ BUILD_BUG_ON(!__is_kvfree_rcu_offset(offset)); \
-- ------ kvfree_call_rcu(head, (rcu_callback_t)(unsigned long)(offset)); \
-- ------ } while (0)
-- ------
/**
* kfree_rcu() - kfree an object after a grace period.
-- ------ * @ptr: pointer to kfree
-- ------ * @rhf: the name of the struct rcu_head within the type of @ptr.
++ ++++++ * @ptr: pointer to kfree for both single- and double-argument invocations.
++ ++++++ * @rhf: the name of the struct rcu_head within the type of @ptr,
++ ++++++ * but only for double-argument invocations.
*
* Many rcu callbacks functions just call kfree() on the base structure.
* These functions are trivial, but their size adds up, and furthermore
* Because the functions are not allowed in the low-order 4096 bytes of
* kernel virtual memory, offsets up to 4095 bytes can be accommodated.
* If the offset is larger than 4095 bytes, a compile-time error will
-- ------ * be generated in __kvfree_rcu(). If this error is triggered, you can
++ ++++++ * be generated in kvfree_rcu_arg_2(). If this error is triggered, you can
* either fall back to use of call_rcu() or rearrange the structure to
* position the rcu_head structure into the first 4096 bytes.
*
* The BUILD_BUG_ON check must not involve any function calls, hence the
* checks are done in macros here.
*/
-- ------#define kfree_rcu(ptr, rhf) \
-- ------do { \
-- ------ typeof (ptr) ___p = (ptr); \
-- ------ \
-- ------ if (___p) \
-- ------ __kvfree_rcu(&((___p)->rhf), offsetof(typeof(*(ptr)), rhf)); \
-- ------} while (0)
++ ++++++#define kfree_rcu kvfree_rcu
/**
* kvfree_rcu() - kvfree an object after a grace period.
kvfree_rcu_arg_2, kvfree_rcu_arg_1)(__VA_ARGS__)
#define KVFREE_GET_MACRO(_1, _2, NAME, ...) NAME
-- ------#define kvfree_rcu_arg_2(ptr, rhf) kfree_rcu(ptr, rhf)
++ ++++++#define kvfree_rcu_arg_2(ptr, rhf) \
++ ++++++do { \
++ ++++++ typeof (ptr) ___p = (ptr); \
++ ++++++ \
++ ++++++ if (___p) { \
++ ++++++ BUILD_BUG_ON(!__is_kvfree_rcu_offset(offsetof(typeof(*(ptr)), rhf))); \
++ ++++++ kvfree_call_rcu(&((___p)->rhf), (rcu_callback_t)(unsigned long) \
++ ++++++ (offsetof(typeof(*(ptr)), rhf))); \
++ ++++++ } \
++ ++++++} while (0)
++ ++++++
#define kvfree_rcu_arg_1(ptr) \
do { \
typeof(ptr) ___p = (ptr); \
#include <linux/rmap.h>
#include <linux/mempolicy.h>
#include <linux/key.h>
- #include <linux/buffer_head.h>
#include <linux/page_ext.h>
#include <linux/debug_locks.h>
#include <linux/debugobjects.h>
* bigger than MAX_ORDER unless SPARSEMEM.
*/
page_ext_init_flatmem();
- init_debug_pagealloc();
+ init_mem_debugging_and_hardening();
report_meminit();
mem_init();
+ /* page_owner must be initialized after buddy is ready */
+ page_ext_init_flatmem_late();
kmem_cache_init();
kmemleak_init();
pgtable_init();
fork_init();
proc_caches_init();
uts_ns_init();
- buffer_init();
key_init();
security_init();
dbg_late_init();
struct file *file = filp_open("/dev/console", O_RDWR, 0);
if (IS_ERR(file)) {
- pr_err("Warning: unable to open an initial console.\n");
- return;
+ pr_err("Warning: unable to open an initial console. Fallback to ttynull.\n");
+ register_ttynull_console();
+
+ file = filp_open("/dev/console", O_RDWR, 0);
+ if (IS_ERR(file)) {
+ pr_err("Warning: Failed to add ttynull console. No stdin, stdout, and stderr for the init process!\n");
+ return;
+ }
}
init_dup(file);
init_dup(file);
init_mm_internals();
+ +++++++ rcu_init_tasks_generic();
do_pre_smp_initcalls();
lockup_detector_init();
#include <linux/moduleparam.h>
#include <linux/delay.h>
#include <linux/slab.h>
- -------#include <linux/percpu-rwsem.h>
#include <linux/torture.h>
+ #include <linux/reboot.h>
MODULE_LICENSE("GPL");
static bool lock_is_write_held;
static bool lock_is_read_held;
+ static unsigned long last_lock_release;
struct lock_stress_stats {
long n_lock_fail;
*/
struct lock_torture_ops {
void (*init)(void);
+ void (*exit)(void);
int (*writelock)(void);
void (*write_delay)(struct torture_random_state *trsp);
void (*task_boost)(struct torture_random_state *trsp);
int nrealwriters_stress;
int nrealreaders_stress;
bool debug_lock;
+ bool init_called;
atomic_t n_lock_torture_errors;
struct lock_torture_ops *cur_ops;
struct lock_stress_stats *lwsa; /* writer statistics */
struct lock_stress_stats *lrsa; /* reader statistics */
};
- static struct lock_torture_cxt cxt = { 0, 0, false,
+ static struct lock_torture_cxt cxt = { 0, 0, false, false,
ATOMIC_INIT(0),
NULL, NULL};
/*
BUG_ON(percpu_init_rwsem(&pcpu_rwsem));
}
+ static void torture_percpu_rwsem_exit(void)
+ {
+ percpu_free_rwsem(&pcpu_rwsem);
+ }
+
static int torture_percpu_rwsem_down_write(void) __acquires(pcpu_rwsem)
{
percpu_down_write(&pcpu_rwsem);
static struct lock_torture_ops percpu_rwsem_lock_ops = {
.init = torture_percpu_rwsem_init,
+ .exit = torture_percpu_rwsem_exit,
.writelock = torture_percpu_rwsem_down_write,
.write_delay = torture_rwsem_write_delay,
.task_boost = torture_boost_dummy,
lwsp->n_lock_acquired++;
cxt.cur_ops->write_delay(&rand);
lock_is_write_held = false;
+ WRITE_ONCE(last_lock_release, jiffies);
cxt.cur_ops->writeunlock();
stutter_wait("lock_torture_writer");
/*
* Indicates early cleanup, meaning that the test has not run,
- * such as when passing bogus args when loading the module. As
- * such, only perform the underlying torture-specific cleanups,
- * and avoid anything related to locktorture.
+ * such as when passing bogus args when loading the module.
+ * However cxt->cur_ops.init() may have been invoked, so beside
+ * perform the underlying torture-specific cleanups, cur_ops.exit()
+ * will be invoked if needed.
*/
if (!cxt.lwsa && !cxt.lrsa)
goto end;
cxt.lrsa = NULL;
end:
+ if (cxt.init_called) {
+ if (cxt.cur_ops->exit)
+ cxt.cur_ops->exit();
+ cxt.init_called = false;
+ }
torture_cleanup_end();
}
goto unwind;
}
- if (nwriters_stress == 0 && nreaders_stress == 0) {
+ if (nwriters_stress == 0 &&
+ (!cxt.cur_ops->readlock || nreaders_stress == 0)) {
pr_alert("lock-torture: must run at least one locking thread\n");
firsterr = -EINVAL;
goto unwind;
}
- if (cxt.cur_ops->init)
+ if (cxt.cur_ops->init) {
cxt.cur_ops->init();
+ cxt.init_called = true;
+ }
if (nwriters_stress >= 0)
cxt.nrealwriters_stress = nwriters_stress;
unwind:
torture_init_end();
lock_torture_cleanup();
+ if (shutdown_secs) {
+ WARN_ON(!IS_MODULE(CONFIG_LOCK_TORTURE_TEST));
+ kernel_power_off();
+ }
return firsterr;
}
config TASKS_TRACE_RCU
def_bool 0
+ +++++++ select IRQ_WORK
help
This option enables a task-based RCU implementation that uses
explicit rcu_read_lock_trace() read-side markers, and allows
config RCU_BOOST
bool "Enable RCU priority boosting"
----- --- depends on RT_MUTEXES && PREEMPT_RCU && RCU_EXPERT
----- --- default n
+++++ +++ depends on (RT_MUTEXES && PREEMPT_RCU && RCU_EXPERT) || PREEMPT_RT
+++++ +++ default y if PREEMPT_RT
help
This option boosts the priority of preempted RCU readers that
block the current preemptible RCU grace period for too long.
Use this option to reduce OS jitter for aggressive HPC or
real-time workloads. It can also be used to offload RCU
callback invocation to energy-efficient CPUs in battery-powered
- asymmetric multiprocessors.
+ asymmetric multiprocessors. The price of this reduced jitter
+ is that the overhead of call_rcu() increases and that some
+ workloads will incur significant increases in context-switch
+ rates.
This option offloads callback invocation from the set of CPUs
specified at boot time by the rcu_nocbs parameter. For each
such CPU, a kthread ("rcuox/N") will be created to invoke
callbacks, where the "N" is the CPU being offloaded, and where
- the "p" for RCU-preempt (PREEMPTION kernels) and "s" for RCU-sched
- (!PREEMPTION kernels). Nothing prevents this kthread from running
- on the specified CPUs, but (1) the kthreads may be preempted
- between each callback, and (2) affinity or cgroups can be used
- to force the kthreads to run on whatever set of CPUs is desired.
-
- Say Y here if you want to help to debug reduced OS jitter.
+ the "x" is "p" for RCU-preempt (PREEMPTION kernels) and "s" for
+ RCU-sched (!PREEMPTION kernels). Nothing prevents this kthread
+ from running on the specified CPUs, but (1) the kthreads may be
+ preempted between each callback, and (2) affinity or cgroups can
+ be used to force the kthreads to run on whatever set of CPUs is
+ desired.
+
+ Say Y here if you need reduced OS jitter, despite added overhead.
Say N here if you are unsure.
config TASKS_TRACE_RCU_READ_MB
smp_mb__after_unlock_lock(); \
} while (0)
- -------#define raw_spin_unlock_rcu_node(p) raw_spin_unlock(&ACCESS_PRIVATE(p, lock))
+ +++++++#define raw_spin_unlock_rcu_node(p) \
+ +++++++do { \
+ +++++++ lockdep_assert_irqs_disabled(); \
+ +++++++ raw_spin_unlock(&ACCESS_PRIVATE(p, lock)); \
+ +++++++} while (0)
#define raw_spin_lock_irq_rcu_node(p) \
do { \
} while (0)
#define raw_spin_unlock_irq_rcu_node(p) \
- ------- raw_spin_unlock_irq(&ACCESS_PRIVATE(p, lock))
+ +++++++do { \
+ +++++++ lockdep_assert_irqs_disabled(); \
+ +++++++ raw_spin_unlock_irq(&ACCESS_PRIVATE(p, lock)); \
+ +++++++} while (0)
#define raw_spin_lock_irqsave_rcu_node(p, flags) \
do { \
} while (0)
#define raw_spin_unlock_irqrestore_rcu_node(p, flags) \
- ------- raw_spin_unlock_irqrestore(&ACCESS_PRIVATE(p, lock), flags)
+ +++++++do { \
+ +++++++ lockdep_assert_irqs_disabled(); \
+ +++++++ raw_spin_unlock_irqrestore(&ACCESS_PRIVATE(p, lock), flags); \
+ +++++++} while (0)
#define raw_spin_trylock_rcu_node(p) \
({ \
static inline void rcu_bind_current_to_nocb(void) { }
#endif
+ #if !defined(CONFIG_TINY_RCU) && defined(CONFIG_TASKS_RCU)
+ void show_rcu_tasks_classic_gp_kthread(void);
+ #else
+ static inline void show_rcu_tasks_classic_gp_kthread(void) {}
+ #endif
+ #if !defined(CONFIG_TINY_RCU) && defined(CONFIG_TASKS_RUDE_RCU)
+ void show_rcu_tasks_rude_gp_kthread(void);
+ #else
+ static inline void show_rcu_tasks_rude_gp_kthread(void) {}
+ #endif
+ #if !defined(CONFIG_TINY_RCU) && defined(CONFIG_TASKS_TRACE_RCU)
+ void show_rcu_tasks_trace_gp_kthread(void);
+ #else
+ static inline void show_rcu_tasks_trace_gp_kthread(void) {}
+ #endif
+
#endif /* __LINUX_RCU_H */
torture_param(bool, gp_exp, false, "Use expedited GP wait primitives");
torture_param(bool, gp_normal, false,
"Use normal (non-expedited) GP wait primitives");
+++++++ +torture_param(bool, gp_poll, false, "Use polling GP wait primitives");
torture_param(bool, gp_sync, false, "Use synchronous GP wait primitives");
torture_param(int, irqreader, 1, "Allow RCU readers from irq handlers");
torture_param(int, leakpointer, 0, "Leak pointer dereferences from readers");
torture_param(int, onoff_holdoff, 0, "Time after boot before CPU hotplugs (s)");
torture_param(int, onoff_interval, 0,
"Time between CPU hotplugs (jiffies), 0=disable");
++++ ++++torture_param(int, nocbs_nthreads, 0, "Number of NOCB toggle threads, 0 to disable");
++++ ++++torture_param(int, nocbs_toggle, 1000, "Time between toggling nocb state (ms)");
torture_param(int, read_exit_delay, 13,
"Delay between read-then-exit episodes (s)");
torture_param(int, read_exit_burst, 16,
module_param(torture_type, charp, 0444);
MODULE_PARM_DESC(torture_type, "Type of RCU to torture (rcu, srcu, ...)");
++++ ++++static int nrealnocbers;
static int nrealreaders;
static struct task_struct *writer_task;
static struct task_struct **fakewriter_tasks;
static struct task_struct **reader_tasks;
++++ ++++static struct task_struct **nocb_tasks;
static struct task_struct *stats_task;
static struct task_struct *fqs_task;
static struct task_struct *boost_tasks[NR_CPUS];
#define RCU_TORTURE_PIPE_LEN 10
+++++++ +// Mailbox-like structure to check RCU global memory ordering.
+++++++ +struct rcu_torture_reader_check {
+++++++ + unsigned long rtc_myloops;
+++++++ + int rtc_chkrdr;
+++++++ + unsigned long rtc_chkloops;
+++++++ + int rtc_ready;
+++++++ + struct rcu_torture_reader_check *rtc_assigner;
+++++++ +} ____cacheline_internodealigned_in_smp;
+++++++ +
+++++++ +// Update-side data structure used to check RCU readers.
struct rcu_torture {
struct rcu_head rtort_rcu;
int rtort_pipe_count;
struct list_head rtort_free;
int rtort_mbtest;
+++++++ + struct rcu_torture_reader_check *rtort_chkp;
};
static LIST_HEAD(rcu_torture_freelist);
static DEFINE_PER_CPU(long [RCU_TORTURE_PIPE_LEN + 1], rcu_torture_count);
static DEFINE_PER_CPU(long [RCU_TORTURE_PIPE_LEN + 1], rcu_torture_batch);
static atomic_t rcu_torture_wcount[RCU_TORTURE_PIPE_LEN + 1];
+++++++ +static struct rcu_torture_reader_check *rcu_torture_reader_mbchk;
static atomic_t n_rcu_torture_alloc;
static atomic_t n_rcu_torture_alloc_fail;
static atomic_t n_rcu_torture_free;
static atomic_t n_rcu_torture_mberror;
+++++++ +static atomic_t n_rcu_torture_mbchk_fail;
+++++++ +static atomic_t n_rcu_torture_mbchk_tries;
static atomic_t n_rcu_torture_error;
static long n_rcu_torture_barrier_error;
static long n_rcu_torture_boost_ktrerror;
static struct list_head rcu_torture_removed;
static unsigned long shutdown_jiffies;
static unsigned long start_gp_seq;
++++ ++++static atomic_long_t n_nocb_offload;
++++ ++++static atomic_long_t n_nocb_deoffload;
static int rcu_torture_writer_state;
#define RTWS_FIXED_DELAY 0
#define RTWS_EXP_SYNC 4
#define RTWS_COND_GET 5
#define RTWS_COND_SYNC 6
------- -#define RTWS_SYNC 7
------- -#define RTWS_STUTTER 8
------- -#define RTWS_STOPPING 9
+++++++ +#define RTWS_POLL_GET 7
+++++++ +#define RTWS_POLL_WAIT 8
+++++++ +#define RTWS_SYNC 9
+++++++ +#define RTWS_STUTTER 10
+++++++ +#define RTWS_STOPPING 11
static const char * const rcu_torture_writer_state_names[] = {
"RTWS_FIXED_DELAY",
"RTWS_DELAY",
"RTWS_EXP_SYNC",
"RTWS_COND_GET",
"RTWS_COND_SYNC",
+++++++ + "RTWS_POLL_GET",
+++++++ + "RTWS_POLL_WAIT",
"RTWS_SYNC",
"RTWS_STUTTER",
"RTWS_STOPPING",
void (*deferred_free)(struct rcu_torture *p);
void (*sync)(void);
void (*exp_sync)(void);
------- - unsigned long (*get_state)(void);
+++++++ + unsigned long (*get_gp_state)(void);
+++++++ + unsigned long (*start_gp_poll)(void);
+++++++ + bool (*poll_gp_state)(unsigned long oldstate);
void (*cond_sync)(unsigned long oldstate);
call_rcu_func_t call;
void (*cb_barrier)(void);
void (*fqs)(void);
void (*stats)(void);
+ void (*gp_kthread_dbg)(void);
int (*stall_dur)(void);
int irq_capable;
int can_boost;
rcu_torture_pipe_update_one(struct rcu_torture *rp)
{
int i;
+++++++ + struct rcu_torture_reader_check *rtrcp = READ_ONCE(rp->rtort_chkp);
+++++++ + if (rtrcp) {
+++++++ + WRITE_ONCE(rp->rtort_chkp, NULL);
+++++++ + smp_store_release(&rtrcp->rtc_ready, 1); // Pair with smp_load_acquire().
+++++++ + }
i = READ_ONCE(rp->rtort_pipe_count);
if (i > RCU_TORTURE_PIPE_LEN)
i = RCU_TORTURE_PIPE_LEN;
.deferred_free = rcu_torture_deferred_free,
.sync = synchronize_rcu,
.exp_sync = synchronize_rcu_expedited,
------- - .get_state = get_state_synchronize_rcu,
+++++++ + .get_gp_state = get_state_synchronize_rcu,
.cond_sync = cond_synchronize_rcu,
.call = call_rcu,
.cb_barrier = rcu_barrier,
.fqs = rcu_force_quiescent_state,
.stats = NULL,
+ .gp_kthread_dbg = show_rcu_gp_kthreads,
.stall_dur = rcu_jiffies_till_stall_check,
.irq_capable = 1,
.can_boost = rcu_can_boost(),
synchronize_srcu(srcu_ctlp);
}
+++++++ +static unsigned long srcu_torture_get_gp_state(void)
+++++++ +{
+++++++ + return get_state_synchronize_srcu(srcu_ctlp);
+++++++ +}
+++++++ +
+++++++ +static unsigned long srcu_torture_start_gp_poll(void)
+++++++ +{
+++++++ + return start_poll_synchronize_srcu(srcu_ctlp);
+++++++ +}
+++++++ +
+++++++ +static bool srcu_torture_poll_gp_state(unsigned long oldstate)
+++++++ +{
+++++++ + return poll_state_synchronize_srcu(srcu_ctlp, oldstate);
+++++++ +}
+++++++ +
static void srcu_torture_call(struct rcu_head *head,
rcu_callback_t func)
{
.deferred_free = srcu_torture_deferred_free,
.sync = srcu_torture_synchronize,
.exp_sync = srcu_torture_synchronize_expedited,
+++++++ + .get_gp_state = srcu_torture_get_gp_state,
+++++++ + .start_gp_poll = srcu_torture_start_gp_poll,
+++++++ + .poll_gp_state = srcu_torture_poll_gp_state,
.call = srcu_torture_call,
.cb_barrier = srcu_torture_barrier,
.stats = srcu_torture_stats,
.exp_sync = synchronize_rcu_mult_test,
.call = call_rcu_tasks,
.cb_barrier = rcu_barrier_tasks,
+ .gp_kthread_dbg = show_rcu_tasks_classic_gp_kthread,
.fqs = NULL,
.stats = NULL,
.irq_capable = 1,
.exp_sync = synchronize_rcu_tasks_rude,
.call = call_rcu_tasks_rude,
.cb_barrier = rcu_barrier_tasks_rude,
+ .gp_kthread_dbg = show_rcu_tasks_rude_gp_kthread,
.fqs = NULL,
.stats = NULL,
.irq_capable = 1,
.exp_sync = synchronize_rcu_tasks_trace,
.call = call_rcu_tasks_trace,
.cb_barrier = rcu_barrier_tasks_trace,
+ .gp_kthread_dbg = show_rcu_tasks_trace_gp_kthread,
.fqs = NULL,
.stats = NULL,
.irq_capable = 1,
oldstarttime = boost_starttime;
while (time_before(jiffies, oldstarttime)) {
schedule_timeout_interruptible(oldstarttime - jiffies);
- stutter_wait("rcu_torture_boost");
+ if (stutter_wait("rcu_torture_boost"))
+ sched_set_fifo_low(current);
if (torture_must_stop())
goto checkwait;
}
jiffies);
call_rcu_time = jiffies;
}
- stutter_wait("rcu_torture_boost");
+ if (stutter_wait("rcu_torture_boost"))
+ sched_set_fifo_low(current);
if (torture_must_stop())
goto checkwait;
}
}
/* Go do the stutter. */
- checkwait: stutter_wait("rcu_torture_boost");
+ checkwait: if (stutter_wait("rcu_torture_boost"))
+ sched_set_fifo_low(current);
} while (!torture_must_stop());
/* Clean up and exit. */
{
unsigned long fqs_resume_time;
int fqs_burst_remaining;
+ int oldnice = task_nice(current);
VERBOSE_TOROUT_STRING("rcu_torture_fqs task started");
do {
udelay(fqs_holdoff);
fqs_burst_remaining -= fqs_holdoff;
}
- stutter_wait("rcu_torture_fqs");
+ if (stutter_wait("rcu_torture_fqs"))
+ sched_set_normal(current, oldnice);
} while (!torture_must_stop());
torture_kthread_stopping("rcu_torture_fqs");
return 0;
}
+++++++ +// Used by writers to randomly choose from the available grace-period
+++++++ +// primitives. The only purpose of the initialization is to size the array.
+++++++ +static int synctype[] = { RTWS_DEF_FREE, RTWS_EXP_SYNC, RTWS_COND_GET, RTWS_POLL_GET, RTWS_SYNC };
+++++++ +static int nsynctypes;
+++++++ +
/*
------- - * RCU torture writer kthread. Repeatedly substitutes a new structure
------- - * for that pointed to by rcu_torture_current, freeing the old structure
------- - * after a series of grace periods (the "pipeline").
+++++++ + * Determine which grace-period primitives are available.
*/
------- -static int
------- -rcu_torture_writer(void *arg)
+++++++ +static void rcu_torture_write_types(void)
{
------- - bool can_expedite = !rcu_gp_is_expedited() && !rcu_gp_is_normal();
------- - int expediting = 0;
------- - unsigned long gp_snap;
bool gp_cond1 = gp_cond, gp_exp1 = gp_exp, gp_normal1 = gp_normal;
------- - bool gp_sync1 = gp_sync;
------- - int i;
- ----- - int oldnice = task_nice(current);
------- - struct rcu_torture *rp;
------- - struct rcu_torture *old_rp;
------- - static DEFINE_TORTURE_RANDOM(rand);
- ----- - bool stutter_waited;
------- - int synctype[] = { RTWS_DEF_FREE, RTWS_EXP_SYNC,
------- - RTWS_COND_GET, RTWS_SYNC };
------- - int nsynctypes = 0;
------- -
------- - VERBOSE_TOROUT_STRING("rcu_torture_writer task started");
------- - if (!can_expedite)
------- - pr_alert("%s" TORTURE_FLAG
------- - " GP expediting controlled from boot/sysfs for %s.\n",
------- - torture_type, cur_ops->name);
+++++++ + bool gp_poll1 = gp_poll, gp_sync1 = gp_sync;
/* Initialize synctype[] array. If none set, take default. */
------- - if (!gp_cond1 && !gp_exp1 && !gp_normal1 && !gp_sync1)
------- - gp_cond1 = gp_exp1 = gp_normal1 = gp_sync1 = true;
------- - if (gp_cond1 && cur_ops->get_state && cur_ops->cond_sync) {
+++++++ + if (!gp_cond1 && !gp_exp1 && !gp_normal1 && !gp_poll1 && !gp_sync1)
+++++++ + gp_cond1 = gp_exp1 = gp_normal1 = gp_poll1 = gp_sync1 = true;
+++++++ + if (gp_cond1 && cur_ops->get_gp_state && cur_ops->cond_sync) {
synctype[nsynctypes++] = RTWS_COND_GET;
pr_info("%s: Testing conditional GPs.\n", __func__);
------- - } else if (gp_cond && (!cur_ops->get_state || !cur_ops->cond_sync)) {
+++++++ + } else if (gp_cond && (!cur_ops->get_gp_state || !cur_ops->cond_sync)) {
pr_alert("%s: gp_cond without primitives.\n", __func__);
}
if (gp_exp1 && cur_ops->exp_sync) {
} else if (gp_normal && !cur_ops->deferred_free) {
pr_alert("%s: gp_normal without primitives.\n", __func__);
}
+++++++ + if (gp_poll1 && cur_ops->start_gp_poll && cur_ops->poll_gp_state) {
+++++++ + synctype[nsynctypes++] = RTWS_POLL_GET;
+++++++ + pr_info("%s: Testing polling GPs.\n", __func__);
+++++++ + } else if (gp_poll && (!cur_ops->start_gp_poll || !cur_ops->poll_gp_state)) {
+++++++ + pr_alert("%s: gp_poll without primitives.\n", __func__);
+++++++ + }
if (gp_sync1 && cur_ops->sync) {
synctype[nsynctypes++] = RTWS_SYNC;
pr_info("%s: Testing normal GPs.\n", __func__);
} else if (gp_sync && !cur_ops->sync) {
pr_alert("%s: gp_sync without primitives.\n", __func__);
}
+++++++ +}
+++++++ +
+++++++ +/*
+++++++ + * RCU torture writer kthread. Repeatedly substitutes a new structure
+++++++ + * for that pointed to by rcu_torture_current, freeing the old structure
+++++++ + * after a series of grace periods (the "pipeline").
+++++++ + */
+++++++ +static int
+++++++ +rcu_torture_writer(void *arg)
+++++++ +{
+++++++ + bool boot_ended;
+++++++ + bool can_expedite = !rcu_gp_is_expedited() && !rcu_gp_is_normal();
+++++++ + unsigned long cookie;
+++++++ + int expediting = 0;
+++++++ + unsigned long gp_snap;
+++++++ + int i;
+++++++ + int idx;
+++++++ + int oldnice = task_nice(current);
+++++++ + struct rcu_torture *rp;
+++++++ + struct rcu_torture *old_rp;
+++++++ + static DEFINE_TORTURE_RANDOM(rand);
+++++++ + bool stutter_waited;
+++++++ +
+++++++ + VERBOSE_TOROUT_STRING("rcu_torture_writer task started");
+++++++ + if (!can_expedite)
+++++++ + pr_alert("%s" TORTURE_FLAG
+++++++ + " GP expediting controlled from boot/sysfs for %s.\n",
+++++++ + torture_type, cur_ops->name);
if (WARN_ONCE(nsynctypes == 0,
"rcu_torture_writer: No update-side primitives.\n")) {
/*
do {
rcu_torture_writer_state = RTWS_FIXED_DELAY;
------- - schedule_timeout_uninterruptible(1);
+++++++ + torture_hrtimeout_us(500, 1000, &rand);
rp = rcu_torture_alloc();
if (rp == NULL)
continue;
atomic_inc(&rcu_torture_wcount[i]);
WRITE_ONCE(old_rp->rtort_pipe_count,
old_rp->rtort_pipe_count + 1);
+++++++ + if (cur_ops->get_gp_state && cur_ops->poll_gp_state) {
+++++++ + idx = cur_ops->readlock();
+++++++ + cookie = cur_ops->get_gp_state();
+++++++ + WARN_ONCE(rcu_torture_writer_state != RTWS_DEF_FREE &&
+++++++ + cur_ops->poll_gp_state(cookie),
+++++++ + "%s: Cookie check 1 failed %s(%d) %lu->%lu\n",
+++++++ + __func__,
+++++++ + rcu_torture_writer_state_getname(),
+++++++ + rcu_torture_writer_state,
+++++++ + cookie, cur_ops->get_gp_state());
+++++++ + cur_ops->readunlock(idx);
+++++++ + }
switch (synctype[torture_random(&rand) % nsynctypes]) {
case RTWS_DEF_FREE:
rcu_torture_writer_state = RTWS_DEF_FREE;
break;
case RTWS_COND_GET:
rcu_torture_writer_state = RTWS_COND_GET;
------- - gp_snap = cur_ops->get_state();
------- - i = torture_random(&rand) % 16;
------- - if (i != 0)
------- - schedule_timeout_interruptible(i);
------- - udelay(torture_random(&rand) % 1000);
+++++++ + gp_snap = cur_ops->get_gp_state();
+++++++ + torture_hrtimeout_jiffies(torture_random(&rand) % 16, &rand);
rcu_torture_writer_state = RTWS_COND_SYNC;
cur_ops->cond_sync(gp_snap);
rcu_torture_pipe_update(old_rp);
break;
+++++++ + case RTWS_POLL_GET:
+++++++ + rcu_torture_writer_state = RTWS_POLL_GET;
+++++++ + gp_snap = cur_ops->start_gp_poll();
+++++++ + rcu_torture_writer_state = RTWS_POLL_WAIT;
+++++++ + while (!cur_ops->poll_gp_state(gp_snap))
+++++++ + torture_hrtimeout_jiffies(torture_random(&rand) % 16,
+++++++ + &rand);
+++++++ + rcu_torture_pipe_update(old_rp);
+++++++ + break;
case RTWS_SYNC:
rcu_torture_writer_state = RTWS_SYNC;
cur_ops->sync();
WARN_ON_ONCE(1);
break;
}
+++++++ + if (cur_ops->get_gp_state && cur_ops->poll_gp_state)
+++++++ + WARN_ONCE(rcu_torture_writer_state != RTWS_DEF_FREE &&
+++++++ + !cur_ops->poll_gp_state(cookie),
+++++++ + "%s: Cookie check 2 failed %s(%d) %lu->%lu\n",
+++++++ + __func__,
+++++++ + rcu_torture_writer_state_getname(),
+++++++ + rcu_torture_writer_state,
+++++++ + cookie, cur_ops->get_gp_state());
}
WRITE_ONCE(rcu_torture_current_version,
rcu_torture_current_version + 1);
!rcu_gp_is_normal();
}
rcu_torture_writer_state = RTWS_STUTTER;
- if (stutter_wait("rcu_torture_writer") &&
+++++++ + boot_ended = rcu_inkernel_boot_has_ended();
+ stutter_waited = stutter_wait("rcu_torture_writer");
+ if (stutter_waited &&
!READ_ONCE(rcu_fwd_cb_nodelay) &&
!cur_ops->slow_gps &&
!torture_must_stop() &&
------- - rcu_inkernel_boot_has_ended())
+++++++ + boot_ended)
for (i = 0; i < ARRAY_SIZE(rcu_tortures); i++)
if (list_empty(&rcu_tortures[i].rtort_free) &&
rcu_access_pointer(rcu_torture_current) !=
rcu_ftrace_dump(DUMP_ALL);
WARN(1, "%s: rtort_pipe_count: %d\n", __func__, rcu_tortures[i].rtort_pipe_count);
}
+ if (stutter_waited)
+ sched_set_normal(current, oldnice);
} while (!torture_must_stop());
rcu_torture_current = NULL; // Let stats task know that we are done.
/* Reset expediting back to unexpedited. */
static int
rcu_torture_fakewriter(void *arg)
{
+++++++ + unsigned long gp_snap;
DEFINE_TORTURE_RANDOM(rand);
VERBOSE_TOROUT_STRING("rcu_torture_fakewriter task started");
set_user_nice(current, MAX_NICE);
do {
------- - schedule_timeout_uninterruptible(1 + torture_random(&rand)%10);
------- - udelay(torture_random(&rand) & 0x3ff);
+++++++ + torture_hrtimeout_jiffies(torture_random(&rand) % 10, &rand);
if (cur_ops->cb_barrier != NULL &&
torture_random(&rand) % (nfakewriters * 8) == 0) {
cur_ops->cb_barrier();
------- - } else if (gp_normal == gp_exp) {
------- - if (cur_ops->sync && torture_random(&rand) & 0x80)
------- - cur_ops->sync();
------- - else if (cur_ops->exp_sync)
+++++++ + } else {
+++++++ + switch (synctype[torture_random(&rand) % nsynctypes]) {
+++++++ + case RTWS_DEF_FREE:
+++++++ + break;
+++++++ + case RTWS_EXP_SYNC:
cur_ops->exp_sync();
------- - } else if (gp_normal && cur_ops->sync) {
------- - cur_ops->sync();
------- - } else if (cur_ops->exp_sync) {
------- - cur_ops->exp_sync();
+++++++ + break;
+++++++ + case RTWS_COND_GET:
+++++++ + gp_snap = cur_ops->get_gp_state();
+++++++ + torture_hrtimeout_jiffies(torture_random(&rand) % 16, &rand);
+++++++ + cur_ops->cond_sync(gp_snap);
+++++++ + break;
+++++++ + case RTWS_POLL_GET:
+++++++ + gp_snap = cur_ops->start_gp_poll();
+++++++ + while (!cur_ops->poll_gp_state(gp_snap)) {
+++++++ + torture_hrtimeout_jiffies(torture_random(&rand) % 16,
+++++++ + &rand);
+++++++ + }
+++++++ + break;
+++++++ + case RTWS_SYNC:
+++++++ + cur_ops->sync();
+++++++ + break;
+++++++ + default:
+++++++ + WARN_ON_ONCE(1);
+++++++ + break;
+++++++ + }
}
stutter_wait("rcu_torture_fakewriter");
} while (!torture_must_stop());
kfree(rhp);
}
+++++++ +// Set up and carry out testing of RCU's global memory ordering
+++++++ +static void rcu_torture_reader_do_mbchk(long myid, struct rcu_torture *rtp,
+++++++ + struct torture_random_state *trsp)
+++++++ +{
+++++++ + unsigned long loops;
+++++++ + int noc = torture_num_online_cpus();
+++++++ + int rdrchked;
+++++++ + int rdrchker;
+++++++ + struct rcu_torture_reader_check *rtrcp; // Me.
+++++++ + struct rcu_torture_reader_check *rtrcp_assigner; // Assigned us to do checking.
+++++++ + struct rcu_torture_reader_check *rtrcp_chked; // Reader being checked.
+++++++ + struct rcu_torture_reader_check *rtrcp_chker; // Reader doing checking when not me.
+++++++ +
+++++++ + if (myid < 0)
+++++++ + return; // Don't try this from timer handlers.
+++++++ +
+++++++ + // Increment my counter.
+++++++ + rtrcp = &rcu_torture_reader_mbchk[myid];
+++++++ + WRITE_ONCE(rtrcp->rtc_myloops, rtrcp->rtc_myloops + 1);
+++++++ +
+++++++ + // Attempt to assign someone else some checking work.
+++++++ + rdrchked = torture_random(trsp) % nrealreaders;
+++++++ + rtrcp_chked = &rcu_torture_reader_mbchk[rdrchked];
+++++++ + rdrchker = torture_random(trsp) % nrealreaders;
+++++++ + rtrcp_chker = &rcu_torture_reader_mbchk[rdrchker];
+++++++ + if (rdrchked != myid && rdrchked != rdrchker && noc >= rdrchked && noc >= rdrchker &&
+++++++ + smp_load_acquire(&rtrcp->rtc_chkrdr) < 0 && // Pairs with smp_store_release below.
+++++++ + !READ_ONCE(rtp->rtort_chkp) &&
+++++++ + !smp_load_acquire(&rtrcp_chker->rtc_assigner)) { // Pairs with smp_store_release below.
+++++++ + rtrcp->rtc_chkloops = READ_ONCE(rtrcp_chked->rtc_myloops);
+++++++ + WARN_ON_ONCE(rtrcp->rtc_chkrdr >= 0);
+++++++ + rtrcp->rtc_chkrdr = rdrchked;
+++++++ + WARN_ON_ONCE(rtrcp->rtc_ready); // This gets set after the grace period ends.
+++++++ + if (cmpxchg_relaxed(&rtrcp_chker->rtc_assigner, NULL, rtrcp) ||
+++++++ + cmpxchg_relaxed(&rtp->rtort_chkp, NULL, rtrcp))
+++++++ + (void)cmpxchg_relaxed(&rtrcp_chker->rtc_assigner, rtrcp, NULL); // Back out.
+++++++ + }
+++++++ +
+++++++ + // If assigned some completed work, do it!
+++++++ + rtrcp_assigner = READ_ONCE(rtrcp->rtc_assigner);
+++++++ + if (!rtrcp_assigner || !smp_load_acquire(&rtrcp_assigner->rtc_ready))
+++++++ + return; // No work or work not yet ready.
+++++++ + rdrchked = rtrcp_assigner->rtc_chkrdr;
+++++++ + if (WARN_ON_ONCE(rdrchked < 0))
+++++++ + return;
+++++++ + rtrcp_chked = &rcu_torture_reader_mbchk[rdrchked];
+++++++ + loops = READ_ONCE(rtrcp_chked->rtc_myloops);
+++++++ + atomic_inc(&n_rcu_torture_mbchk_tries);
+++++++ + if (ULONG_CMP_LT(loops, rtrcp_assigner->rtc_chkloops))
+++++++ + atomic_inc(&n_rcu_torture_mbchk_fail);
+++++++ + rtrcp_assigner->rtc_chkloops = loops + ULONG_MAX / 2;
+++++++ + rtrcp_assigner->rtc_ready = 0;
+++++++ + smp_store_release(&rtrcp->rtc_assigner, NULL); // Someone else can assign us work.
+++++++ + smp_store_release(&rtrcp_assigner->rtc_chkrdr, -1); // Assigner can again assign.
+++++++ +}
+++++++ +
/*
* Do one extension of an RCU read-side critical section using the
* current reader state in readstate (set to zero for initial entry
* no data to read. Can be invoked both from process context and
* from a timer handler.
*/
------- -static bool rcu_torture_one_read(struct torture_random_state *trsp)
+++++++ +static bool rcu_torture_one_read(struct torture_random_state *trsp, long myid)
{
+++++++ + unsigned long cookie;
int i;
unsigned long started;
unsigned long completed;
WARN_ON_ONCE(!rcu_is_watching());
newstate = rcutorture_extend_mask(readstate, trsp);
rcutorture_one_extend(&readstate, newstate, trsp, rtrsp++);
+++++++ + if (cur_ops->get_gp_state && cur_ops->poll_gp_state)
+++++++ + cookie = cur_ops->get_gp_state();
started = cur_ops->get_gp_seq();
ts = rcu_trace_clock_local();
p = rcu_dereference_check(rcu_torture_current,
}
if (p->rtort_mbtest == 0)
atomic_inc(&n_rcu_torture_mberror);
+++++++ + rcu_torture_reader_do_mbchk(myid, p, trsp);
rtrsp = rcutorture_loop_extend(&readstate, trsp, rtrsp);
preempt_disable();
pipe_count = READ_ONCE(p->rtort_pipe_count);
}
__this_cpu_inc(rcu_torture_batch[completed]);
preempt_enable();
+++++++ + if (cur_ops->get_gp_state && cur_ops->poll_gp_state)
+++++++ + WARN_ONCE(cur_ops->poll_gp_state(cookie),
+++++++ + "%s: Cookie check 3 failed %s(%d) %lu->%lu\n",
+++++++ + __func__,
+++++++ + rcu_torture_writer_state_getname(),
+++++++ + rcu_torture_writer_state,
+++++++ + cookie, cur_ops->get_gp_state());
rcutorture_one_extend(&readstate, 0, trsp, rtrsp);
WARN_ON_ONCE(readstate & RCUTORTURE_RDR_MASK);
// This next splat is expected behavior if leakpointer, especially
static void rcu_torture_timer(struct timer_list *unused)
{
atomic_long_inc(&n_rcu_torture_timers);
------- - (void)rcu_torture_one_read(this_cpu_ptr(&rcu_torture_timer_rand));
+++++++ + (void)rcu_torture_one_read(this_cpu_ptr(&rcu_torture_timer_rand), -1);
/* Test call_rcu() invocation from interrupt handler. */
if (cur_ops->call) {
if (!timer_pending(&t))
mod_timer(&t, jiffies + 1);
}
------- - if (!rcu_torture_one_read(&rand) && !torture_must_stop())
+++++++ + if (!rcu_torture_one_read(&rand, myid) && !torture_must_stop())
schedule_timeout_interruptible(HZ);
if (time_after(jiffies, lastsleep) && !torture_must_stop()) {
------- - schedule_timeout_interruptible(1);
+++++++ + torture_hrtimeout_us(500, 1000, &rand);
lastsleep = jiffies + 10;
}
------- - while (num_online_cpus() < mynumonline && !torture_must_stop())
+++++++ + while (torture_num_online_cpus() < mynumonline && !torture_must_stop())
schedule_timeout_interruptible(HZ / 5);
stutter_wait("rcu_torture_reader");
} while (!torture_must_stop());
return 0;
}
++++ ++++/*
++++ ++++ * Randomly Toggle CPUs' callback-offload state. This uses hrtimers to
++++ ++++ * increase race probabilities and fuzzes the interval between toggling.
++++ ++++ */
++++ ++++static int rcu_nocb_toggle(void *arg)
++++ ++++{
++++ ++++ int cpu;
++++ ++++ int maxcpu = -1;
++++ ++++ int oldnice = task_nice(current);
++++ ++++ long r;
++++ ++++ DEFINE_TORTURE_RANDOM(rand);
++++ ++++ ktime_t toggle_delay;
++++ ++++ unsigned long toggle_fuzz;
++++ ++++ ktime_t toggle_interval = ms_to_ktime(nocbs_toggle);
++++ ++++
++++ ++++ VERBOSE_TOROUT_STRING("rcu_nocb_toggle task started");
++++ ++++ while (!rcu_inkernel_boot_has_ended())
++++ ++++ schedule_timeout_interruptible(HZ / 10);
++++ ++++ for_each_online_cpu(cpu)
++++ ++++ maxcpu = cpu;
++++ ++++ WARN_ON(maxcpu < 0);
++++ ++++ if (toggle_interval > ULONG_MAX)
++++ ++++ toggle_fuzz = ULONG_MAX >> 3;
++++ ++++ else
++++ ++++ toggle_fuzz = toggle_interval >> 3;
++++ ++++ if (toggle_fuzz <= 0)
++++ ++++ toggle_fuzz = NSEC_PER_USEC;
++++ ++++ do {
++++ ++++ r = torture_random(&rand);
++++ ++++ cpu = (r >> 4) % (maxcpu + 1);
++++ ++++ if (r & 0x1) {
++++ ++++ rcu_nocb_cpu_offload(cpu);
++++ ++++ atomic_long_inc(&n_nocb_offload);
++++ ++++ } else {
++++ ++++ rcu_nocb_cpu_deoffload(cpu);
++++ ++++ atomic_long_inc(&n_nocb_deoffload);
++++ ++++ }
++++ ++++ toggle_delay = torture_random(&rand) % toggle_fuzz + toggle_interval;
++++ ++++ set_current_state(TASK_INTERRUPTIBLE);
++++ ++++ schedule_hrtimeout(&toggle_delay, HRTIMER_MODE_REL);
++++ ++++ if (stutter_wait("rcu_nocb_toggle"))
++++ ++++ sched_set_normal(current, oldnice);
++++ ++++ } while (!torture_must_stop());
++++ ++++ torture_kthread_stopping("rcu_nocb_toggle");
++++ ++++ return 0;
++++ ++++}
++++ ++++
/*
* Print torture statistics. Caller must ensure that there is only
* one call to this function at a given time!!! This is normally
atomic_read(&n_rcu_torture_alloc),
atomic_read(&n_rcu_torture_alloc_fail),
atomic_read(&n_rcu_torture_free));
------- - pr_cont("rtmbe: %d rtbe: %ld rtbke: %ld rtbre: %ld ",
+++++++ + pr_cont("rtmbe: %d rtmbkf: %d/%d rtbe: %ld rtbke: %ld rtbre: %ld ",
atomic_read(&n_rcu_torture_mberror),
+++++++ + atomic_read(&n_rcu_torture_mbchk_fail), atomic_read(&n_rcu_torture_mbchk_tries),
n_rcu_torture_barrier_error,
n_rcu_torture_boost_ktrerror,
n_rcu_torture_boost_rterror);
data_race(n_barrier_successes),
data_race(n_barrier_attempts),
data_race(n_rcu_torture_barrier_error));
---- ---- pr_cont("read-exits: %ld\n", data_race(n_read_exits));
++++ ++++ pr_cont("read-exits: %ld ", data_race(n_read_exits)); // Statistic.
++++ ++++ pr_cont("nocb-toggles: %ld:%ld\n",
++++ ++++ atomic_long_read(&n_nocb_offload), atomic_long_read(&n_nocb_deoffload));
pr_alert("%s%s ", torture_type, TORTURE_FLAG);
if (atomic_read(&n_rcu_torture_mberror) ||
+++++++ + atomic_read(&n_rcu_torture_mbchk_fail) ||
n_rcu_torture_barrier_error || n_rcu_torture_boost_ktrerror ||
n_rcu_torture_boost_rterror || n_rcu_torture_boost_failure ||
i > 1) {
pr_cont("%s", "!!! ");
atomic_inc(&n_rcu_torture_error);
WARN_ON_ONCE(atomic_read(&n_rcu_torture_mberror));
+++++++ + WARN_ON_ONCE(atomic_read(&n_rcu_torture_mbchk_fail));
WARN_ON_ONCE(n_rcu_torture_barrier_error); // rcu_barrier()
WARN_ON_ONCE(n_rcu_torture_boost_ktrerror); // no boost kthread
WARN_ON_ONCE(n_rcu_torture_boost_rterror); // can't set RT prio
sched_show_task(wtp);
splatted = true;
}
- show_rcu_gp_kthreads();
+ if (cur_ops->gp_kthread_dbg)
+ cur_ops->gp_kthread_dbg();
rcu_ftrace_dump(DUMP_ALL);
}
rtcv_snap = rcu_torture_current_version;
"stall_cpu_block=%d "
"n_barrier_cbs=%d "
"onoff_interval=%d onoff_holdoff=%d "
---- ---- "read_exit_delay=%d read_exit_burst=%d\n",
++++ ++++ "read_exit_delay=%d read_exit_burst=%d "
++++ ++++ "nocbs_nthreads=%d nocbs_toggle=%d\n",
torture_type, tag, nrealreaders, nfakewriters,
stat_interval, verbose, test_no_idle_hz, shuffle_interval,
stutter, irqreader, fqs_duration, fqs_holdoff, fqs_stutter,
stall_cpu_block,
n_barrier_cbs,
onoff_interval, onoff_holdoff,
---- ---- read_exit_delay, read_exit_burst);
++++ ++++ read_exit_delay, read_exit_burst,
++++ ++++ nocbs_nthreads, nocbs_toggle);
}
static int rcutorture_booster_cleanup(unsigned int cpu)
unsigned long stopat;
static DEFINE_TORTURE_RANDOM(trs);
- if (cur_ops->call && cur_ops->sync && cur_ops->cb_barrier) {
+ if (!cur_ops->sync)
+ return; // Cannot do need_resched() forward progress testing without ->sync.
+ if (cur_ops->call && cur_ops->cb_barrier) {
init_rcu_head_on_stack(&fcs.rh);
selfpropcb = true;
}
/* Carry out grace-period forward-progress testing. */
static int rcu_torture_fwd_prog(void *args)
{
+ int oldnice = task_nice(current);
struct rcu_fwd *rfp = args;
int tested = 0;
int tested_tries = 0;
rcu_torture_fwd_prog_cr(rfp);
/* Avoid slow periods, better to test when busy. */
- stutter_wait("rcu_torture_fwd_prog");
+ if (stutter_wait("rcu_torture_fwd_prog"))
+ sched_set_normal(current, oldnice);
} while (!torture_must_stop());
/* Short runs might not contain a valid forward-progress attempt. */
WARN_ON(!tested && tested_tries >= 5);
if (!fwd_progress)
return 0; /* Not requested, so don't do it. */
- if (!cur_ops->stall_dur || cur_ops->stall_dur() <= 0 ||
- cur_ops == &rcu_busted_ops) {
+ if ((!cur_ops->sync && !cur_ops->call) ||
+ !cur_ops->stall_dur || cur_ops->stall_dur() <= 0 || cur_ops == &rcu_busted_ops) {
VERBOSE_TOROUT_STRING("rcu_torture_fwd_prog_init: Disabled, unsupported by RCU flavor under test");
return 0;
}
// Minimize time between reading and exiting.
while (!kthread_should_stop())
schedule_timeout_uninterruptible(1);
------- - (void)rcu_torture_one_read(trsp);
+++++++ + (void)rcu_torture_one_read(trsp, -1);
return 0;
}
return;
}
- show_rcu_gp_kthreads();
+ if (cur_ops->gp_kthread_dbg)
+ cur_ops->gp_kthread_dbg();
rcu_torture_read_exit_cleanup();
rcu_torture_barrier_cleanup();
rcu_torture_fwd_prog_cleanup();
torture_stop_kthread(rcu_torture_stall, stall_task);
torture_stop_kthread(rcu_torture_writer, writer_task);
++++ ++++ if (nocb_tasks) {
++++ ++++ for (i = 0; i < nrealnocbers; i++)
++++ ++++ torture_stop_kthread(rcu_nocb_toggle, nocb_tasks[i]);
++++ ++++ kfree(nocb_tasks);
++++ ++++ nocb_tasks = NULL;
++++ ++++ }
++++ ++++
if (reader_tasks) {
for (i = 0; i < nrealreaders; i++)
torture_stop_kthread(rcu_torture_reader,
reader_tasks[i]);
kfree(reader_tasks);
+ reader_tasks = NULL;
}
+++++++ + kfree(rcu_torture_reader_mbchk);
+++++++ + rcu_torture_reader_mbchk = NULL;
if (fakewriter_tasks) {
- for (i = 0; i < nfakewriters; i++) {
+ for (i = 0; i < nfakewriters; i++)
torture_stop_kthread(rcu_torture_fakewriter,
fakewriter_tasks[i]);
- }
kfree(fakewriter_tasks);
fakewriter_tasks = NULL;
}
#ifdef CONFIG_DEBUG_OBJECTS_RCU_HEAD
struct rcu_head rh1;
struct rcu_head rh2;
+++++++ + struct rcu_head *rhp = kmalloc(sizeof(*rhp), GFP_KERNEL);
init_rcu_head_on_stack(&rh1);
init_rcu_head_on_stack(&rh2);
local_irq_disable(); /* Make it harder to start a new grace period. */
call_rcu(&rh2, rcu_torture_leak_cb);
call_rcu(&rh2, rcu_torture_err_cb); /* Duplicate callback. */
+++++++ + if (rhp) {
+++++++ + call_rcu(rhp, rcu_torture_leak_cb);
+++++++ + call_rcu(rhp, rcu_torture_err_cb); /* Another duplicate callback. */
+++++++ + }
local_irq_enable();
rcu_read_unlock();
preempt_enable();
for (i = 0; i < ARRAY_SIZE(torture_ops); i++)
pr_cont(" %s", torture_ops[i]->name);
pr_cont("\n");
- WARN_ON(!IS_MODULE(CONFIG_RCU_TORTURE_TEST));
firsterr = -EINVAL;
cur_ops = NULL;
goto unwind;
atomic_set(&n_rcu_torture_alloc_fail, 0);
atomic_set(&n_rcu_torture_free, 0);
atomic_set(&n_rcu_torture_mberror, 0);
+++++++ + atomic_set(&n_rcu_torture_mbchk_fail, 0);
+++++++ + atomic_set(&n_rcu_torture_mbchk_tries, 0);
atomic_set(&n_rcu_torture_error, 0);
n_rcu_torture_barrier_error = 0;
n_rcu_torture_boost_ktrerror = 0;
/* Start up the kthreads. */
+++++++ + rcu_torture_write_types();
firsterr = torture_create_kthread(rcu_torture_writer, NULL,
writer_task);
if (firsterr)
}
reader_tasks = kcalloc(nrealreaders, sizeof(reader_tasks[0]),
GFP_KERNEL);
------- - if (reader_tasks == NULL) {
+++++++ + rcu_torture_reader_mbchk = kcalloc(nrealreaders, sizeof(*rcu_torture_reader_mbchk),
+++++++ + GFP_KERNEL);
+++++++ + if (!reader_tasks || !rcu_torture_reader_mbchk) {
VERBOSE_TOROUT_ERRSTRING("out of memory");
firsterr = -ENOMEM;
goto unwind;
}
for (i = 0; i < nrealreaders; i++) {
+++++++ + rcu_torture_reader_mbchk[i].rtc_chkrdr = -1;
firsterr = torture_create_kthread(rcu_torture_reader, (void *)i,
reader_tasks[i]);
if (firsterr)
goto unwind;
}
++++ ++++ nrealnocbers = nocbs_nthreads;
++++ ++++ if (WARN_ON(nrealnocbers < 0))
++++ ++++ nrealnocbers = 1;
++++ ++++ if (WARN_ON(nocbs_toggle < 0))
++++ ++++ nocbs_toggle = HZ;
++++ ++++ if (nrealnocbers > 0) {
++++ ++++ nocb_tasks = kcalloc(nrealnocbers, sizeof(nocb_tasks[0]), GFP_KERNEL);
++++ ++++ if (nocb_tasks == NULL) {
++++ ++++ VERBOSE_TOROUT_ERRSTRING("out of memory");
++++ ++++ firsterr = -ENOMEM;
++++ ++++ goto unwind;
++++ ++++ }
++++ ++++ } else {
++++ ++++ nocb_tasks = NULL;
++++ ++++ }
++++ ++++ for (i = 0; i < nrealnocbers; i++) {
++++ ++++ firsterr = torture_create_kthread(rcu_nocb_toggle, NULL, nocb_tasks[i]);
++++ ++++ if (firsterr)
++++ ++++ goto unwind;
++++ ++++ }
if (stat_interval > 0) {
firsterr = torture_create_kthread(rcu_torture_stats, NULL,
stats_task);
unwind:
torture_init_end();
rcu_torture_cleanup();
+ if (shutdown_secs) {
+ WARN_ON(!IS_MODULE(CONFIG_RCU_TORTURE_TEST));
+ kernel_power_off();
+ }
return firsterr;
}
INIT_DELAYED_WORK(&ssp->work, process_srcu);
if (!is_static)
ssp->sda = alloc_percpu(struct srcu_data);
+ if (!ssp->sda)
+ return -ENOMEM;
init_srcu_struct_nodes(ssp, is_static);
ssp->srcu_gp_seq_needed_exp = 0;
ssp->srcu_last_gp_end = ktime_get_mono_fast_ns();
smp_store_release(&ssp->srcu_gp_seq_needed, 0); /* Init done. */
- return ssp->sda ? 0 : -ENOMEM;
+ return 0;
}
#ifdef CONFIG_DEBUG_LOCK_ALLOC
{
}
+++++++ +/*
+++++++ + * Start an SRCU grace period, and also queue the callback if non-NULL.
+++++++ + */
+++++++ +static unsigned long srcu_gp_start_if_needed(struct srcu_struct *ssp,
+++++++ + struct rcu_head *rhp, bool do_norm)
+++++++ +{
+++++++ + unsigned long flags;
+++++++ + int idx;
+++++++ + bool needexp = false;
+++++++ + bool needgp = false;
+++++++ + unsigned long s;
+++++++ + struct srcu_data *sdp;
+++++++ +
+++++++ + check_init_srcu_struct(ssp);
+++++++ + idx = srcu_read_lock(ssp);
+++++++ + sdp = raw_cpu_ptr(ssp->sda);
+++++++ + spin_lock_irqsave_rcu_node(sdp, flags);
+++++++ + if (rhp)
+++++++ + rcu_segcblist_enqueue(&sdp->srcu_cblist, rhp);
+++++++ + rcu_segcblist_advance(&sdp->srcu_cblist,
+++++++ + rcu_seq_current(&ssp->srcu_gp_seq));
+++++++ + s = rcu_seq_snap(&ssp->srcu_gp_seq);
+++++++ + (void)rcu_segcblist_accelerate(&sdp->srcu_cblist, s);
+++++++ + if (ULONG_CMP_LT(sdp->srcu_gp_seq_needed, s)) {
+++++++ + sdp->srcu_gp_seq_needed = s;
+++++++ + needgp = true;
+++++++ + }
+++++++ + if (!do_norm && ULONG_CMP_LT(sdp->srcu_gp_seq_needed_exp, s)) {
+++++++ + sdp->srcu_gp_seq_needed_exp = s;
+++++++ + needexp = true;
+++++++ + }
+++++++ + spin_unlock_irqrestore_rcu_node(sdp, flags);
+++++++ + if (needgp)
+++++++ + srcu_funnel_gp_start(ssp, sdp, s, do_norm);
+++++++ + else if (needexp)
+++++++ + srcu_funnel_exp_start(ssp, sdp->mynode, s);
+++++++ + srcu_read_unlock(ssp, idx);
+++++++ + return s;
+++++++ +}
+++++++ +
/*
* Enqueue an SRCU callback on the srcu_data structure associated with
* the current CPU and the specified srcu_struct structure, initiating
static void __call_srcu(struct srcu_struct *ssp, struct rcu_head *rhp,
rcu_callback_t func, bool do_norm)
{
------- - unsigned long flags;
------- - int idx;
------- - bool needexp = false;
------- - bool needgp = false;
------- - unsigned long s;
------- - struct srcu_data *sdp;
------- -
------- - check_init_srcu_struct(ssp);
if (debug_rcu_head_queue(rhp)) {
/* Probable double call_srcu(), so leak the callback. */
WRITE_ONCE(rhp->func, srcu_leak_callback);
return;
}
rhp->func = func;
------- - idx = srcu_read_lock(ssp);
------- - sdp = raw_cpu_ptr(ssp->sda);
------- - spin_lock_irqsave_rcu_node(sdp, flags);
------- - rcu_segcblist_enqueue(&sdp->srcu_cblist, rhp);
------- - rcu_segcblist_advance(&sdp->srcu_cblist,
------- - rcu_seq_current(&ssp->srcu_gp_seq));
------- - s = rcu_seq_snap(&ssp->srcu_gp_seq);
------- - (void)rcu_segcblist_accelerate(&sdp->srcu_cblist, s);
------- - if (ULONG_CMP_LT(sdp->srcu_gp_seq_needed, s)) {
------- - sdp->srcu_gp_seq_needed = s;
------- - needgp = true;
------- - }
------- - if (!do_norm && ULONG_CMP_LT(sdp->srcu_gp_seq_needed_exp, s)) {
------- - sdp->srcu_gp_seq_needed_exp = s;
------- - needexp = true;
------- - }
------- - spin_unlock_irqrestore_rcu_node(sdp, flags);
------- - if (needgp)
------- - srcu_funnel_gp_start(ssp, sdp, s, do_norm);
------- - else if (needexp)
------- - srcu_funnel_exp_start(ssp, sdp->mynode, s);
------- - srcu_read_unlock(ssp, idx);
+++++++ + (void)srcu_gp_start_if_needed(ssp, rhp, do_norm);
}
/**
{
struct rcu_synchronize rcu;
- RCU_LOCKDEP_WARN(lock_is_held(&ssp->dep_map) ||
+ RCU_LOCKDEP_WARN(lockdep_is_held(ssp) ||
lock_is_held(&rcu_bh_lock_map) ||
lock_is_held(&rcu_lock_map) ||
lock_is_held(&rcu_sched_lock_map),
}
EXPORT_SYMBOL_GPL(synchronize_srcu);
+++++++ +/**
+++++++ + * get_state_synchronize_srcu - Provide an end-of-grace-period cookie
+++++++ + * @ssp: srcu_struct to provide cookie for.
+++++++ + *
+++++++ + * This function returns a cookie that can be passed to
+++++++ + * poll_state_synchronize_srcu(), which will return true if a full grace
+++++++ + * period has elapsed in the meantime. It is the caller's responsibility
+++++++ + * to make sure that grace period happens, for example, by invoking
+++++++ + * call_srcu() after return from get_state_synchronize_srcu().
+++++++ + */
+++++++ +unsigned long get_state_synchronize_srcu(struct srcu_struct *ssp)
+++++++ +{
+++++++ + // Any prior manipulation of SRCU-protected data must happen
+++++++ + // before the load from ->srcu_gp_seq.
+++++++ + smp_mb();
+++++++ + return rcu_seq_snap(&ssp->srcu_gp_seq);
+++++++ +}
+++++++ +EXPORT_SYMBOL_GPL(get_state_synchronize_srcu);
+++++++ +
+++++++ +/**
+++++++ + * start_poll_synchronize_srcu - Provide cookie and start grace period
+++++++ + * @ssp: srcu_struct to provide cookie for.
+++++++ + *
+++++++ + * This function returns a cookie that can be passed to
+++++++ + * poll_state_synchronize_srcu(), which will return true if a full grace
+++++++ + * period has elapsed in the meantime. Unlike get_state_synchronize_srcu(),
+++++++ + * this function also ensures that any needed SRCU grace period will be
+++++++ + * started. This convenience does come at a cost in terms of CPU overhead.
+++++++ + */
+++++++ +unsigned long start_poll_synchronize_srcu(struct srcu_struct *ssp)
+++++++ +{
+++++++ + return srcu_gp_start_if_needed(ssp, NULL, true);
+++++++ +}
+++++++ +EXPORT_SYMBOL_GPL(start_poll_synchronize_srcu);
+++++++ +
+++++++ +/**
+++++++ + * poll_state_synchronize_srcu - Has cookie's grace period ended?
+++++++ + * @ssp: srcu_struct to provide cookie for.
+++++++ + * @cookie: Return value from get_state_synchronize_srcu() or start_poll_synchronize_srcu().
+++++++ + *
+++++++ + * This function takes the cookie that was returned from either
+++++++ + * get_state_synchronize_srcu() or start_poll_synchronize_srcu(), and
+++++++ + * returns @true if an SRCU grace period elapsed since the time that the
+++++++ + * cookie was created.
+++++++ + *
+++++++ + * Because cookies are finite in size, wrapping/overflow is possible.
+++++++ + * This is more pronounced on 32-bit systems where cookies are 32 bits,
+++++++ + * where in theory wrapping could happen in about 14 hours assuming
+++++++ + * 25-microsecond expedited SRCU grace periods. However, a more likely
+++++++ + * overflow lower bound is on the order of 24 days in the case of
+++++++ + * one-millisecond SRCU grace periods. Of course, wrapping in a 64-bit
+++++++ + * system requires geologic timespans, as in more than seven million years
+++++++ + * even for expedited SRCU grace periods.
+++++++ + *
+++++++ + * Wrapping/overflow is much more of an issue for CONFIG_SMP=n systems
+++++++ + * that also have CONFIG_PREEMPTION=n, which selects Tiny SRCU. This uses
+++++++ + * a 16-bit cookie, which rcutorture routinely wraps in a matter of a
+++++++ + * few minutes. If this proves to be a problem, this counter will be
+++++++ + * expanded to the same size as for Tree SRCU.
+++++++ + */
+++++++ +bool poll_state_synchronize_srcu(struct srcu_struct *ssp, unsigned long cookie)
+++++++ +{
+++++++ + if (!rcu_seq_done(&ssp->srcu_gp_seq, cookie))
+++++++ + return false;
+++++++ + // Ensure that the end of the SRCU grace period happens before
+++++++ + // any subsequent code that the caller might execute.
+++++++ + smp_mb(); // ^^^
+++++++ + return true;
+++++++ +}
+++++++ +EXPORT_SYMBOL_GPL(poll_state_synchronize_srcu);
+++++++ +
/*
* Callback function for srcu_barrier() use.
*/
*/
static void srcu_invoke_callbacks(struct work_struct *work)
{
++++ ++++ long len;
bool more;
struct rcu_cblist ready_cbs;
struct rcu_head *rhp;
/* We are on the job! Extract and invoke ready callbacks. */
sdp->srcu_cblist_invoking = true;
rcu_segcblist_extract_done_cbs(&sdp->srcu_cblist, &ready_cbs);
++++ ++++ len = ready_cbs.len;
spin_unlock_irq_rcu_node(sdp);
rhp = rcu_cblist_dequeue(&ready_cbs);
for (; rhp != NULL; rhp = rcu_cblist_dequeue(&ready_cbs)) {
rhp->func(rhp);
local_bh_enable();
}
++++ ++++ WARN_ON_ONCE(ready_cbs.len);
/*
* Update counts, accelerate new callbacks, and if needed,
* schedule another round of callback invocation.
*/
spin_lock_irq_rcu_node(sdp);
---- ---- rcu_segcblist_insert_count(&sdp->srcu_cblist, &ready_cbs);
++++ ++++ rcu_segcblist_add_len(&sdp->srcu_cblist, -len);
(void)rcu_segcblist_accelerate(&sdp->srcu_cblist,
rcu_seq_snap(&ssp->srcu_gp_seq));
sdp->srcu_cblist_invoking = false;
}
}
- -------/* Spawn RCU-tasks grace-period kthread, e.g., at core_initcall() time. */
+ +++++++/* Spawn RCU-tasks grace-period kthread. */
static void __init rcu_spawn_tasks_kthread_generic(struct rcu_tasks *rtp)
{
struct task_struct *t;
".C"[!!data_race(rtp->cbs_head)],
s);
}
- #endif /* #ifndef CONFIG_TINY_RCU */
+ #endif // #ifndef CONFIG_TINY_RCU
static void exit_tasks_rcu_finish_trace(struct task_struct *t);
// Start off with initial wait and slowly back off to 1 HZ wait.
fract = rtp->init_fract;
- if (fract > HZ)
- fract = HZ;
- for (;;) {
+ while (!list_empty(&holdouts)) {
bool firstreport;
bool needreport;
int rtst;
- if (list_empty(&holdouts))
- break;
-
/* Slowly back off waiting for holdouts */
set_tasks_gp_state(rtp, RTGS_WAIT_SCAN_HOLDOUTS);
- schedule_timeout_idle(HZ/fract);
+ schedule_timeout_idle(fract);
- if (fract > 1)
- fract--;
+ if (fract < HZ)
+ fract++;
rtst = READ_ONCE(rcu_task_stall_timeout);
needreport = rtst > 0 && time_after(jiffies, lastreport + rtst);
static int __init rcu_spawn_tasks_kthread(void)
{
rcu_tasks.gp_sleep = HZ / 10;
- rcu_tasks.init_fract = 10;
+ rcu_tasks.init_fract = HZ / 10;
rcu_tasks.pregp_func = rcu_tasks_pregp_step;
rcu_tasks.pertask_func = rcu_tasks_pertask;
rcu_tasks.postscan_func = rcu_tasks_postscan;
rcu_spawn_tasks_kthread_generic(&rcu_tasks);
return 0;
}
- -------core_initcall(rcu_spawn_tasks_kthread);
- #ifndef CONFIG_TINY_RCU
- static void show_rcu_tasks_classic_gp_kthread(void)
+ #if !defined(CONFIG_TINY_RCU)
+ void show_rcu_tasks_classic_gp_kthread(void)
{
show_rcu_tasks_generic_gp_kthread(&rcu_tasks, "");
}
- #endif /* #ifndef CONFIG_TINY_RCU */
+ EXPORT_SYMBOL_GPL(show_rcu_tasks_classic_gp_kthread);
+ #endif // !defined(CONFIG_TINY_RCU)
/* Do the srcu_read_lock() for the above synchronize_srcu(). */
void exit_tasks_rcu_start(void) __acquires(&tasks_rcu_exit_srcu)
}
#else /* #ifdef CONFIG_TASKS_RCU */
- static inline void show_rcu_tasks_classic_gp_kthread(void) { }
void exit_tasks_rcu_start(void) { }
void exit_tasks_rcu_finish(void) { exit_tasks_rcu_finish_trace(current); }
#endif /* #else #ifdef CONFIG_TASKS_RCU */
rcu_spawn_tasks_kthread_generic(&rcu_tasks_rude);
return 0;
}
- -------core_initcall(rcu_spawn_tasks_rude_kthread);
- #ifndef CONFIG_TINY_RCU
- static void show_rcu_tasks_rude_gp_kthread(void)
+ #if !defined(CONFIG_TINY_RCU)
+ void show_rcu_tasks_rude_gp_kthread(void)
{
show_rcu_tasks_generic_gp_kthread(&rcu_tasks_rude, "");
}
- #endif /* #ifndef CONFIG_TINY_RCU */
-
- #else /* #ifdef CONFIG_TASKS_RUDE_RCU */
- static void show_rcu_tasks_rude_gp_kthread(void) {}
- #endif /* #else #ifdef CONFIG_TASKS_RUDE_RCU */
+ EXPORT_SYMBOL_GPL(show_rcu_tasks_rude_gp_kthread);
+ #endif // !defined(CONFIG_TINY_RCU)
+ #endif /* #ifdef CONFIG_TASKS_RUDE_RCU */
////////////////////////////////////////////////////////////////////////
//
static void rcu_tasks_trace_pertask(struct task_struct *t,
struct list_head *hop)
{
+ +++++++ // During early boot when there is only the one boot CPU, there
+ +++++++ // is no idle task for the other CPUs. Just return.
+ +++++++ if (unlikely(t == NULL))
+ +++++++ return;
+ +++++++
WRITE_ONCE(t->trc_reader_special.b.need_qs, false);
WRITE_ONCE(t->trc_reader_checked, false);
t->trc_ipi_to_cpu = -1;
{
if (IS_ENABLED(CONFIG_TASKS_TRACE_RCU_READ_MB)) {
rcu_tasks_trace.gp_sleep = HZ / 10;
- rcu_tasks_trace.init_fract = 10;
+ rcu_tasks_trace.init_fract = HZ / 10;
} else {
rcu_tasks_trace.gp_sleep = HZ / 200;
if (rcu_tasks_trace.gp_sleep <= 0)
rcu_tasks_trace.gp_sleep = 1;
- rcu_tasks_trace.init_fract = HZ / 5;
+ rcu_tasks_trace.init_fract = HZ / 200;
if (rcu_tasks_trace.init_fract <= 0)
rcu_tasks_trace.init_fract = 1;
}
rcu_spawn_tasks_kthread_generic(&rcu_tasks_trace);
return 0;
}
- -------core_initcall(rcu_spawn_tasks_trace_kthread);
- #ifndef CONFIG_TINY_RCU
- static void show_rcu_tasks_trace_gp_kthread(void)
+ #if !defined(CONFIG_TINY_RCU)
+ void show_rcu_tasks_trace_gp_kthread(void)
{
char buf[64];
data_race(n_heavy_reader_attempts));
show_rcu_tasks_generic_gp_kthread(&rcu_tasks_trace, buf);
}
- #endif /* #ifndef CONFIG_TINY_RCU */
+ EXPORT_SYMBOL_GPL(show_rcu_tasks_trace_gp_kthread);
+ #endif // !defined(CONFIG_TINY_RCU)
#else /* #ifdef CONFIG_TASKS_TRACE_RCU */
static void exit_tasks_rcu_finish_trace(struct task_struct *t) { }
- static inline void show_rcu_tasks_trace_gp_kthread(void) {}
#endif /* #else #ifdef CONFIG_TASKS_TRACE_RCU */
#ifndef CONFIG_TINY_RCU
}
#endif /* #ifndef CONFIG_TINY_RCU */
+ +++++++#ifdef CONFIG_PROVE_RCU
+ +++++++struct rcu_tasks_test_desc {
+ +++++++ struct rcu_head rh;
+ +++++++ const char *name;
+ +++++++ bool notrun;
+ +++++++};
+ +++++++
+ +++++++static struct rcu_tasks_test_desc tests[] = {
+ +++++++ {
+ +++++++ .name = "call_rcu_tasks()",
+ +++++++ /* If not defined, the test is skipped. */
+ +++++++ .notrun = !IS_ENABLED(CONFIG_TASKS_RCU),
+ +++++++ },
+ +++++++ {
+ +++++++ .name = "call_rcu_tasks_rude()",
+ +++++++ /* If not defined, the test is skipped. */
+ +++++++ .notrun = !IS_ENABLED(CONFIG_TASKS_RUDE_RCU),
+ +++++++ },
+ +++++++ {
+ +++++++ .name = "call_rcu_tasks_trace()",
+ +++++++ /* If not defined, the test is skipped. */
+ +++++++ .notrun = !IS_ENABLED(CONFIG_TASKS_TRACE_RCU)
+ +++++++ }
+ +++++++};
+ +++++++
+ +++++++static void test_rcu_tasks_callback(struct rcu_head *rhp)
+ +++++++{
+ +++++++ struct rcu_tasks_test_desc *rttd =
+ +++++++ container_of(rhp, struct rcu_tasks_test_desc, rh);
+ +++++++
+ +++++++ pr_info("Callback from %s invoked.\n", rttd->name);
+ +++++++
+ +++++++ rttd->notrun = true;
+ +++++++}
+ +++++++
+ +++++++static void rcu_tasks_initiate_self_tests(void)
+ +++++++{
+ +++++++ pr_info("Running RCU-tasks wait API self tests\n");
+ +++++++#ifdef CONFIG_TASKS_RCU
+ +++++++ synchronize_rcu_tasks();
+ +++++++ call_rcu_tasks(&tests[0].rh, test_rcu_tasks_callback);
+ +++++++#endif
+ +++++++
+ +++++++#ifdef CONFIG_TASKS_RUDE_RCU
+ +++++++ synchronize_rcu_tasks_rude();
+ +++++++ call_rcu_tasks_rude(&tests[1].rh, test_rcu_tasks_callback);
+ +++++++#endif
+ +++++++
+ +++++++#ifdef CONFIG_TASKS_TRACE_RCU
+ +++++++ synchronize_rcu_tasks_trace();
+ +++++++ call_rcu_tasks_trace(&tests[2].rh, test_rcu_tasks_callback);
+ +++++++#endif
+ +++++++}
+ +++++++
+ +++++++static int rcu_tasks_verify_self_tests(void)
+ +++++++{
+ +++++++ int ret = 0;
+ +++++++ int i;
+ +++++++
+ +++++++ for (i = 0; i < ARRAY_SIZE(tests); i++) {
+ +++++++ if (!tests[i].notrun) { // still hanging.
+ +++++++ pr_err("%s has been failed.\n", tests[i].name);
+ +++++++ ret = -1;
+ +++++++ }
+ +++++++ }
+ +++++++
+ +++++++ if (ret)
+ +++++++ WARN_ON(1);
+ +++++++
+ +++++++ return ret;
+ +++++++}
+ +++++++late_initcall(rcu_tasks_verify_self_tests);
+ +++++++#else /* #ifdef CONFIG_PROVE_RCU */
+ +++++++static void rcu_tasks_initiate_self_tests(void) { }
+ +++++++#endif /* #else #ifdef CONFIG_PROVE_RCU */
+ +++++++
+ +++++++void __init rcu_init_tasks_generic(void)
+ +++++++{
+ +++++++#ifdef CONFIG_TASKS_RCU
+ +++++++ rcu_spawn_tasks_kthread();
+ +++++++#endif
+ +++++++
+ +++++++#ifdef CONFIG_TASKS_RUDE_RCU
+ +++++++ rcu_spawn_tasks_rude_kthread();
+ +++++++#endif
+ +++++++
+ +++++++#ifdef CONFIG_TASKS_TRACE_RCU
+ +++++++ rcu_spawn_tasks_trace_kthread();
+ +++++++#endif
+ +++++++
+ +++++++ // Run the self-tests.
+ +++++++ rcu_tasks_initiate_self_tests();
+ +++++++}
+ +++++++
#else /* #ifdef CONFIG_TASKS_RCU_GENERIC */
static inline void rcu_tasks_bootup_oddness(void) {}
void show_rcu_tasks_gp_kthreads(void) {}
.dynticks_nesting = 1,
.dynticks_nmi_nesting = DYNTICK_IRQ_NONIDLE,
.dynticks = ATOMIC_INIT(RCU_DYNTICK_CTRL_CTR),
++++ ++++#ifdef CONFIG_RCU_NOCB_CPU
++++ ++++ .cblist.flags = SEGCBLIST_SOFTIRQ_ONLY,
++++ ++++#endif
};
static struct rcu_state rcu_state = {
.level = { &rcu_state.node[0] },
static bool dump_tree;
module_param(dump_tree, bool, 0444);
/* By default, use RCU_SOFTIRQ instead of rcuc kthreads. */
----- ---static bool use_softirq = true;
+++++ +++static bool use_softirq = !IS_ENABLED(CONFIG_PREEMPT_RT);
+++++ +++#ifndef CONFIG_PREEMPT_RT
module_param(use_softirq, bool, 0444);
+++++ +++#endif
/* Control rcu_node-tree auto-balancing at boot time. */
static bool rcu_fanout_exact;
module_param(rcu_fanout_exact, bool, 0444);
* per-CPU. Object size is equal to one page. This value
* can be changed at boot time.
*/
- static int rcu_min_cached_objs = 2;
+ static int rcu_min_cached_objs = 5;
module_param(rcu_min_cached_objs, int, 0444);
/* Retrieve RCU kthreads priority for rcutorture */
return !(snap & RCU_DYNTICK_CTRL_CTR);
}
+ /* Return true if the specified CPU is currently idle from an RCU viewpoint. */
+ bool rcu_is_idle_cpu(int cpu)
+ {
+ struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu);
+
+ return rcu_dynticks_in_eqs(rcu_dynticks_snap(rdp));
+ }
+
/*
* Return true if the CPU corresponding to the specified rcu_data
* structure has spent some time in an extended quiescent state since
return ret;
}
- static struct kernel_param_ops first_fqs_jiffies_ops = {
+ static const struct kernel_param_ops first_fqs_jiffies_ops = {
.set = param_set_first_fqs_jiffies,
.get = param_get_ulong,
};
- static struct kernel_param_ops next_fqs_jiffies_ops = {
+ static const struct kernel_param_ops next_fqs_jiffies_ops = {
.set = param_set_next_fqs_jiffies,
.get = param_get_ulong,
};
{
struct rcu_data *rdp = this_cpu_ptr(&rcu_data);
- // Enabling the tick is unsafe in NMI handlers.
- if (WARN_ON_ONCE(in_nmi()))
+ // If we're here from NMI there's nothing to do.
+ if (in_nmi())
return;
RCU_LOCKDEP_WARN(rcu_dynticks_curr_cpu_in_eqs(),
* CPU can safely enter RCU read-side critical sections. In other words,
* if the current CPU is not in its idle loop or is in an interrupt or
* NMI handler, return true.
+ *
+ * Make notrace because it can be called by the internal functions of
+ * ftrace, and making this notrace removes unnecessary recursion calls.
*/
- bool rcu_is_watching(void)
+ notrace bool rcu_is_watching(void)
{
bool ret;
preempt_disable_notrace();
rdp = this_cpu_ptr(&rcu_data);
rnp = rdp->mynode;
- if (rdp->grpmask & rcu_rnp_online_cpus(rnp))
+ if (rdp->grpmask & rcu_rnp_online_cpus(rnp) || READ_ONCE(rnp->ofl_seq) & 0x1)
ret = true;
preempt_enable_notrace();
return ret;
if (IS_ENABLED(CONFIG_IRQ_WORK) &&
!rdp->rcu_iw_pending && rdp->rcu_iw_gp_seq != rnp->gp_seq &&
(rnp->ffmask & rdp->grpmask)) {
- init_irq_work(&rdp->rcu_iw, rcu_iw_handler);
- atomic_set(&rdp->rcu_iw.flags, IRQ_WORK_HARD_IRQ);
rdp->rcu_iw_pending = true;
rdp->rcu_iw_gp_seq = rnp->gp_seq;
irq_work_queue_on(&rdp->rcu_iw, rdp->cpu);
if (!rcu_segcblist_pend_cbs(&rdp->cblist))
return false;
++++ ++++ trace_rcu_segcb_stats(&rdp->cblist, TPS("SegCbPreAcc"));
++++ ++++
/*
* Callbacks are often registered with incomplete grace-period
* information. Something about the fact that getting exact
else
trace_rcu_grace_period(rcu_state.name, gp_seq_req, TPS("AccReadyCB"));
++++ ++++ trace_rcu_segcb_stats(&rdp->cblist, TPS("SegCbPostAcc"));
++++ ++++
return ret;
}
{
bool ret = false;
bool need_qs;
- const bool offloaded = IS_ENABLED(CONFIG_RCU_NOCB_CPU) &&
- rcu_segcblist_is_offloaded(&rdp->cblist);
+ const bool offloaded = rcu_segcblist_is_offloaded(&rdp->cblist);
raw_lockdep_assert_held_rcu_node(rnp);
*/
static bool rcu_gp_init(void)
{
+ unsigned long firstseq;
unsigned long flags;
unsigned long oldmask;
unsigned long mask;
* go offline later. Please also refer to "Hotplug CPU" section
* of RCU's Requirements documentation.
*/
------ -- rcu_state.gp_state = RCU_GP_ONOFF;
++++++ ++ WRITE_ONCE(rcu_state.gp_state, RCU_GP_ONOFF);
rcu_for_each_leaf_node(rnp) {
+ smp_mb(); // Pair with barriers used when updating ->ofl_seq to odd values.
+ firstseq = READ_ONCE(rnp->ofl_seq);
+ if (firstseq & 0x1)
+ while (firstseq == READ_ONCE(rnp->ofl_seq))
+ schedule_timeout_idle(1); // Can't wake unless RCU is watching.
+ smp_mb(); // Pair with barriers used when updating ->ofl_seq to even values.
raw_spin_lock(&rcu_state.ofl_lock);
raw_spin_lock_irq_rcu_node(rnp);
if (rnp->qsmaskinit == rnp->qsmaskinitnext &&
* The grace period cannot complete until the initialization
* process finishes, because this kthread handles both.
*/
------ -- rcu_state.gp_state = RCU_GP_INIT;
++++++ ++ WRITE_ONCE(rcu_state.gp_state, RCU_GP_INIT);
rcu_for_each_node_breadth_first(rnp) {
rcu_gp_slow(gp_init_delay);
raw_spin_lock_irqsave_rcu_node(rnp, flags);
ret = 0;
for (;;) {
if (!ret) {
------ -- rcu_state.jiffies_force_qs = jiffies + j;
++++++ ++ WRITE_ONCE(rcu_state.jiffies_force_qs, jiffies + j);
++++++ ++ /*
++++++ ++ * jiffies_force_qs before RCU_GP_WAIT_FQS state
++++++ ++ * update; required for stall checks.
++++++ ++ */
++++++ ++ smp_wmb();
WRITE_ONCE(rcu_state.jiffies_kick_kthreads,
jiffies + (j ? 3 * j : 2));
}
trace_rcu_grace_period(rcu_state.name, rcu_state.gp_seq,
TPS("fqswait"));
------ -- rcu_state.gp_state = RCU_GP_WAIT_FQS;
++++++ ++ WRITE_ONCE(rcu_state.gp_state, RCU_GP_WAIT_FQS);
ret = swait_event_idle_timeout_exclusive(
rcu_state.gp_wq, rcu_gp_fqs_check_wake(&gf), j);
rcu_gp_torture_wait();
------ -- rcu_state.gp_state = RCU_GP_DOING_FQS;
++++++ ++ WRITE_ONCE(rcu_state.gp_state, RCU_GP_DOING_FQS);
/* Locking provides needed memory barriers. */
/* If grace period done, leave loop. */
if (!READ_ONCE(rnp->qsmask) &&
trace_rcu_grace_period(rcu_state.name, rcu_state.gp_seq, TPS("end"));
rcu_seq_end(&rcu_state.gp_seq);
ASSERT_EXCLUSIVE_WRITER(rcu_state.gp_seq);
------ -- rcu_state.gp_state = RCU_GP_IDLE;
++++++ ++ WRITE_ONCE(rcu_state.gp_state, RCU_GP_IDLE);
/* Check for GP requests since above loop. */
rdp = this_cpu_ptr(&rcu_data);
if (!needgp && ULONG_CMP_LT(rnp->gp_seq, rnp->gp_seq_needed)) {
needgp = true;
}
/* Advance CBs to reduce false positives below. */
- offloaded = IS_ENABLED(CONFIG_RCU_NOCB_CPU) &&
- rcu_segcblist_is_offloaded(&rdp->cblist);
+ offloaded = rcu_segcblist_is_offloaded(&rdp->cblist);
if ((offloaded || !rcu_accelerate_cbs(rnp, rdp)) && needgp) {
WRITE_ONCE(rcu_state.gp_flags, RCU_GP_FLAG_INIT);
WRITE_ONCE(rcu_state.gp_req_activity, jiffies);
for (;;) {
trace_rcu_grace_period(rcu_state.name, rcu_state.gp_seq,
TPS("reqwait"));
------ -- rcu_state.gp_state = RCU_GP_WAIT_GPS;
++++++ ++ WRITE_ONCE(rcu_state.gp_state, RCU_GP_WAIT_GPS);
swait_event_idle_exclusive(rcu_state.gp_wq,
READ_ONCE(rcu_state.gp_flags) &
RCU_GP_FLAG_INIT);
rcu_gp_torture_wait();
------ -- rcu_state.gp_state = RCU_GP_DONE_GPS;
++++++ ++ WRITE_ONCE(rcu_state.gp_state, RCU_GP_DONE_GPS);
/* Locking provides needed memory barrier. */
if (rcu_gp_init())
break;
rcu_gp_fqs_loop();
/* Handle grace-period end. */
------ -- rcu_state.gp_state = RCU_GP_CLEANUP;
++++++ ++ WRITE_ONCE(rcu_state.gp_state, RCU_GP_CLEANUP);
rcu_gp_cleanup();
------ -- rcu_state.gp_state = RCU_GP_CLEANED;
++++++ ++ WRITE_ONCE(rcu_state.gp_state, RCU_GP_CLEANED);
}
}
unsigned long flags;
unsigned long mask;
bool needwake = false;
- const bool offloaded = IS_ENABLED(CONFIG_RCU_NOCB_CPU) &&
- rcu_segcblist_is_offloaded(&rdp->cblist);
+ const bool offloaded = rcu_segcblist_is_offloaded(&rdp->cblist);
struct rcu_node *rnp;
WARN_ON_ONCE(rdp->cpu != smp_processor_id());
if (!IS_ENABLED(CONFIG_HOTPLUG_CPU))
return 0;
+ WRITE_ONCE(rcu_state.n_online_cpus, rcu_state.n_online_cpus - 1);
/* Adjust any no-longer-needed kthreads. */
rcu_boost_kthread_setaffinity(rnp, -1);
/* Do any needed no-CB deferred wakeups from this CPU. */
static void rcu_do_batch(struct rcu_data *rdp)
{
int div;
++++ ++++ bool __maybe_unused empty;
unsigned long flags;
- const bool offloaded = IS_ENABLED(CONFIG_RCU_NOCB_CPU) &&
- rcu_segcblist_is_offloaded(&rdp->cblist);
+ const bool offloaded = rcu_segcblist_is_offloaded(&rdp->cblist);
struct rcu_head *rhp;
struct rcu_cblist rcl = RCU_CBLIST_INITIALIZER(rcl);
---- ---- long bl, count;
++++ ++++ long bl, count = 0;
long pending, tlimit = 0;
/* If no callbacks are ready, just return. */
rcu_segcblist_extract_done_cbs(&rdp->cblist, &rcl);
if (offloaded)
rdp->qlen_last_fqs_check = rcu_segcblist_n_cbs(&rdp->cblist);
++++ ++++
++++ ++++ trace_rcu_segcb_stats(&rdp->cblist, TPS("SegCbDequeued"));
rcu_nocb_unlock_irqrestore(rdp, flags);
/* Invoke callbacks. */
tick_dep_set_task(current, TICK_DEP_BIT_RCU);
rhp = rcu_cblist_dequeue(&rcl);
++++ ++++
for (; rhp; rhp = rcu_cblist_dequeue(&rcl)) {
rcu_callback_t f;
++++ ++++ count++;
debug_rcu_head_unqueue(rhp);
rcu_lock_acquire(&rcu_callback_map);
/*
* Stop only if limit reached and CPU has something to do.
---- ---- * Note: The rcl structure counts down from zero.
*/
---- ---- if (-rcl.len >= bl && !offloaded &&
++++ ++++ if (count >= bl && !offloaded &&
(need_resched() ||
(!is_idle_task(current) && !rcu_is_callbacks_kthread())))
break;
if (unlikely(tlimit)) {
/* only call local_clock() every 32 callbacks */
---- ---- if (likely((-rcl.len & 31) || local_clock() < tlimit))
++++ ++++ if (likely((count & 31) || local_clock() < tlimit))
continue;
/* Exceeded the time limit, so leave. */
break;
}
---- ---- if (offloaded) {
---- ---- WARN_ON_ONCE(in_serving_softirq());
++++ ++++ if (!in_serving_softirq()) {
local_bh_enable();
lockdep_assert_irqs_enabled();
cond_resched_tasks_rcu_qs();
local_irq_save(flags);
rcu_nocb_lock(rdp);
---- ---- count = -rcl.len;
rdp->n_cbs_invoked += count;
trace_rcu_batch_end(rcu_state.name, count, !!rcl.head, need_resched(),
is_idle_task(current), rcu_is_callbacks_kthread());
/* Update counts and requeue any remaining callbacks. */
rcu_segcblist_insert_done_cbs(&rdp->cblist, &rcl);
---- ---- smp_mb(); /* List handling before counting for rcu_barrier(). */
---- ---- rcu_segcblist_insert_count(&rdp->cblist, &rcl);
++++ ++++ rcu_segcblist_add_len(&rdp->cblist, -count);
/* Reinstate batch limit if we have worked down the excess. */
count = rcu_segcblist_n_cbs(&rdp->cblist);
* The following usually indicates a double call_rcu(). To track
* this down, try building with CONFIG_DEBUG_OBJECTS_RCU_HEAD=y.
*/
---- ---- WARN_ON_ONCE(count == 0 && !rcu_segcblist_empty(&rdp->cblist));
++++ ++++ empty = rcu_segcblist_empty(&rdp->cblist);
++++ ++++ WARN_ON_ONCE(count == 0 && !empty);
WARN_ON_ONCE(!IS_ENABLED(CONFIG_RCU_NOCB_CPU) &&
---- ---- count != 0 && rcu_segcblist_empty(&rdp->cblist));
++++ ++++ count != 0 && empty);
++++ ++++ WARN_ON_ONCE(count == 0 && rcu_segcblist_n_segment_cbs(&rdp->cblist) != 0);
++++ ++++ WARN_ON_ONCE(!empty && rcu_segcblist_n_segment_cbs(&rdp->cblist) == 0);
rcu_nocb_unlock_irqrestore(rdp, flags);
void rcu_sched_clock_irq(int user)
{
trace_rcu_utilization(TPS("Start scheduler-tick"));
+ +++++++ lockdep_assert_irqs_disabled();
raw_cpu_inc(rcu_data.ticks_this_gp);
/* The load-acquire pairs with the store-release setting to true. */
if (smp_load_acquire(this_cpu_ptr(&rcu_data.rcu_urgent_qs))) {
rcu_flavor_sched_clock_irq(user);
if (rcu_pending(user))
invoke_rcu_core();
+ +++++++ lockdep_assert_irqs_disabled();
trace_rcu_utilization(TPS("End scheduler-tick"));
}
unsigned long flags;
struct rcu_data *rdp = raw_cpu_ptr(&rcu_data);
struct rcu_node *rnp = rdp->mynode;
- -- ---- const bool offloaded = rcu_segcblist_is_offloaded(&rdp->cblist);
- const bool offloaded = IS_ENABLED(CONFIG_RCU_NOCB_CPU) &&
- rcu_segcblist_is_offloaded(&rdp->cblist);
++++ ++++ const bool do_batch = !rcu_segcblist_completely_offloaded(&rdp->cblist);
if (cpu_is_offline(smp_processor_id()))
return;
/* No grace period and unregistered callbacks? */
if (!rcu_gp_in_progress() &&
---- ---- rcu_segcblist_is_enabled(&rdp->cblist) && !offloaded) {
---- ---- local_irq_save(flags);
++++ ++++ rcu_segcblist_is_enabled(&rdp->cblist) && do_batch) {
++++ ++++ rcu_nocb_lock_irqsave(rdp, flags);
if (!rcu_segcblist_restempty(&rdp->cblist, RCU_NEXT_READY_TAIL))
rcu_accelerate_cbs_unlocked(rnp, rdp);
---- ---- local_irq_restore(flags);
++++ ++++ rcu_nocb_unlock_irqrestore(rdp, flags);
}
rcu_check_gp_start_stall(rnp, rdp, rcu_jiffies_till_stall_check());
/* If there are callbacks ready, invoke them. */
---- ---- if (!offloaded && rcu_segcblist_ready_cbs(&rdp->cblist) &&
++++ ++++ if (do_batch && rcu_segcblist_ready_cbs(&rdp->cblist) &&
likely(READ_ONCE(rcu_scheduler_fully_active)))
rcu_do_batch(rdp);
static void
__call_rcu(struct rcu_head *head, rcu_callback_t func)
{
+++ +++++ static atomic_t doublefrees;
unsigned long flags;
struct rcu_data *rdp;
bool was_alldone;
* Use rcu:rcu_callback trace event to find the previous
* time callback was passed to __call_rcu().
*/
--- ----- WARN_ONCE(1, "__call_rcu(): Double-freed CB %p->%pS()!!!\n",
--- ----- head, head->func);
+++ +++++ if (atomic_inc_return(&doublefrees) < 4) {
+++ +++++ pr_err("%s(): Double-freed CB %p->%pS()!!! ", __func__, head, head->func);
+++ +++++ mem_dump_obj(head);
+++ +++++ }
WRITE_ONCE(head->func, rcu_leak_callback);
return;
}
trace_rcu_callback(rcu_state.name, head,
rcu_segcblist_n_cbs(&rdp->cblist));
++++ ++++ trace_rcu_segcb_stats(&rdp->cblist, TPS("SegCBQueued"));
++++ ++++
/* Go handle any RCU core processing required. */
- if (IS_ENABLED(CONFIG_RCU_NOCB_CPU) &&
- unlikely(rcu_segcblist_is_offloaded(&rdp->cblist))) {
+ if (unlikely(rcu_segcblist_is_offloaded(&rdp->cblist))) {
__call_rcu_nocb_wake(rdp, was_alldone, flags); /* unlocks */
} else {
__call_rcu_core(rdp, head, flags);
* In order to save some per-cpu space the list is singular.
* Even though it is lockless an access has to be protected by the
* per-cpu lock.
+ * @page_cache_work: A work to refill the cache when it is empty
+ * @work_in_progress: Indicates that page_cache_work is running
+ * @hrtimer: A hrtimer for scheduling a page_cache_work
* @nr_bkv_objs: number of allocated objects at @bkvcache.
*
* This is a per-CPU structure. The reason that it is not included in
bool monitor_todo;
bool initialized;
int count;
+
+ struct work_struct page_cache_work;
+ atomic_t work_in_progress;
+ struct hrtimer hrtimer;
+
struct llist_head bkvcache;
int nr_bkv_objs;
};
}
rcu_lock_release(&rcu_callback_map);
- krcp = krc_this_cpu_lock(&flags);
+ raw_spin_lock_irqsave(&krcp->lock, flags);
if (put_cached_bnode(krcp, bkvhead[i]))
bkvhead[i] = NULL;
- krc_this_cpu_unlock(krcp, flags);
+ raw_spin_unlock_irqrestore(&krcp->lock, flags);
if (bkvhead[i])
free_page((unsigned long) bkvhead[i]);
raw_spin_unlock_irqrestore(&krcp->lock, flags);
}
+ static enum hrtimer_restart
+ schedule_page_work_fn(struct hrtimer *t)
+ {
+ struct kfree_rcu_cpu *krcp =
+ container_of(t, struct kfree_rcu_cpu, hrtimer);
+
+ queue_work(system_highpri_wq, &krcp->page_cache_work);
+ return HRTIMER_NORESTART;
+ }
+
+ static void fill_page_cache_func(struct work_struct *work)
+ {
+ struct kvfree_rcu_bulk_data *bnode;
+ struct kfree_rcu_cpu *krcp =
+ container_of(work, struct kfree_rcu_cpu,
+ page_cache_work);
+ unsigned long flags;
+ bool pushed;
+ int i;
+
+ for (i = 0; i < rcu_min_cached_objs; i++) {
+ bnode = (struct kvfree_rcu_bulk_data *)
+ __get_free_page(GFP_KERNEL | __GFP_NOWARN);
+
+ if (bnode) {
+ raw_spin_lock_irqsave(&krcp->lock, flags);
+ pushed = put_cached_bnode(krcp, bnode);
+ raw_spin_unlock_irqrestore(&krcp->lock, flags);
+
+ if (!pushed) {
+ free_page((unsigned long) bnode);
+ break;
+ }
+ }
+ }
+
+ atomic_set(&krcp->work_in_progress, 0);
+ }
+
+ static void
+ run_page_cache_worker(struct kfree_rcu_cpu *krcp)
+ {
+ if (rcu_scheduler_active == RCU_SCHEDULER_RUNNING &&
+ !atomic_xchg(&krcp->work_in_progress, 1)) {
+ hrtimer_init(&krcp->hrtimer, CLOCK_MONOTONIC,
+ HRTIMER_MODE_REL);
+ krcp->hrtimer.function = schedule_page_work_fn;
+ hrtimer_start(&krcp->hrtimer, 0, HRTIMER_MODE_REL);
+ }
+ }
+
static inline bool
kvfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp, void *ptr)
{
if (!krcp->bkvhead[idx] ||
krcp->bkvhead[idx]->nr_records == KVFREE_BULK_MAX_ENTR) {
bnode = get_cached_bnode(krcp);
- if (!bnode) {
- /*
- * To keep this path working on raw non-preemptible
- * sections, prevent the optional entry into the
- * allocator as it uses sleeping locks. In fact, even
- * if the caller of kfree_rcu() is preemptible, this
- * path still is not, as krcp->lock is a raw spinlock.
- * With additional page pre-allocation in the works,
- * hitting this return is going to be much less likely.
- */
- if (IS_ENABLED(CONFIG_PREEMPT_RT))
- return false;
-
- /*
- * NOTE: For one argument of kvfree_rcu() we can
- * drop the lock and get the page in sleepable
- * context. That would allow to maintain an array
- * for the CONFIG_PREEMPT_RT as well if no cached
- * pages are available.
- */
- bnode = (struct kvfree_rcu_bulk_data *)
- __get_free_page(GFP_NOWAIT | __GFP_NOWARN);
- }
-
/* Switch to emergency path. */
- if (unlikely(!bnode))
+ if (!bnode)
return false;
/* Initialize the new block. */
goto unlock_return;
}
- /*
- * Under high memory pressure GFP_NOWAIT can fail,
- * in that case the emergency path is maintained.
- */
++ ++++++ kasan_record_aux_stack(ptr);
success = kvfree_call_rcu_add_ptr_to_bulk(krcp, ptr);
if (!success) {
+ run_page_cache_worker(krcp);
+
if (head == NULL)
// Inline if kvfree_rcu(one_arg) call.
goto unlock_return;
* During early boot, any blocking grace-period wait automatically
* implies a grace period. Later on, this is never the case for PREEMPTION.
*
- * Howevr, because a context switch is a grace period for !PREEMPTION, any
+ * However, because a context switch is a grace period for !PREEMPTION, any
* blocking grace-period wait automatically implies a grace period if
* there is only one CPU online at any point time during execution of
* either synchronize_rcu() or synchronize_rcu_expedited(). It is OK to
return rcu_scheduler_active == RCU_SCHEDULER_INACTIVE;
might_sleep(); /* Check for RCU read-side critical section. */
preempt_disable();
- ret = num_online_cpus() <= 1;
+ /*
+ * If the rcu_state.n_online_cpus counter is equal to one,
+ * there is only one CPU, and that CPU sees all prior accesses
+ * made by any CPU that was online at the time of its access.
+ * Furthermore, if this counter is equal to one, its value cannot
+ * change until after the preempt_enable() below.
+ *
+ * Furthermore, if rcu_state.n_online_cpus is equal to one here,
+ * all later CPUs (both this one and any that come online later
+ * on) are guaranteed to see all accesses prior to this point
+ * in the code, without the need for additional memory barriers.
+ * Those memory barriers are provided by CPU-hotplug code.
+ */
+ ret = READ_ONCE(rcu_state.n_online_cpus) <= 1;
preempt_enable();
return ret;
}
lock_is_held(&rcu_sched_lock_map),
"Illegal synchronize_rcu() in RCU read-side critical section");
if (rcu_blocking_is_gp())
- return;
+ return; // Context allows vacuous grace periods.
if (rcu_gp_is_expedited())
synchronize_rcu_expedited();
else
struct rcu_data *rdp = this_cpu_ptr(&rcu_data);
struct rcu_node *rnp = rdp->mynode;
+ +++++++ lockdep_assert_irqs_disabled();
+ +++++++
/* Check for CPU stalls, if enabled. */
check_cpu_stall(rdp);
return 1;
/* Does this CPU have callbacks ready to invoke? */
- if (rcu_segcblist_ready_cbs(&rdp->cblist))
+ if (!rcu_segcblist_is_offloaded(&rdp->cblist) &&
+ rcu_segcblist_ready_cbs(&rdp->cblist))
return 1;
/* Has RCU gone idle with this CPU needing another grace period? */
if (!gp_in_progress && rcu_segcblist_is_enabled(&rdp->cblist) &&
- (!IS_ENABLED(CONFIG_RCU_NOCB_CPU) ||
- !rcu_segcblist_is_offloaded(&rdp->cblist)) &&
+ !rcu_segcblist_is_offloaded(&rdp->cblist) &&
!rcu_segcblist_restempty(&rdp->cblist, RCU_NEXT_READY_TAIL))
return 1;
rdp->qlen_last_fqs_check = 0;
rdp->n_force_qs_snap = rcu_state.n_force_qs;
rdp->blimit = blimit;
---- ---- if (rcu_segcblist_empty(&rdp->cblist) && /* No early-boot CBs? */
---- ---- !rcu_segcblist_is_offloaded(&rdp->cblist))
---- ---- rcu_segcblist_init(&rdp->cblist); /* Re-enable callbacks. */
rdp->dynticks_nesting = 1; /* CPU not up, no tearing. */
rcu_dynticks_eqs_online();
raw_spin_unlock_rcu_node(rnp); /* irqs remain disabled. */
++++ ++++ /*
++++ ++++ * Lock in case the CB/GP kthreads are still around handling
++++ ++++ * old callbacks (longer term we should flush all callbacks
++++ ++++ * before completing CPU offline)
++++ ++++ */
++++ ++++ rcu_nocb_lock(rdp);
++++ ++++ if (rcu_segcblist_empty(&rdp->cblist)) /* No early-boot CBs? */
++++ ++++ rcu_segcblist_init(&rdp->cblist); /* Re-enable callbacks. */
++++ ++++ rcu_nocb_unlock(rdp);
/*
* Add CPU to leaf rcu_node pending-online bitmask. Any needed
rdp->cpu_no_qs.b.norm = true;
rdp->core_needs_qs = false;
rdp->rcu_iw_pending = false;
+ rdp->rcu_iw = IRQ_WORK_INIT_HARD(rcu_iw_handler);
rdp->rcu_iw_gp_seq = rdp->gp_seq - 1;
trace_rcu_grace_period(rcu_state.name, rdp->gp_seq, TPS("cpuonl"));
raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
rcu_prepare_kthreads(cpu);
rcu_spawn_cpu_nocb_kthread(cpu);
+ WRITE_ONCE(rcu_state.n_online_cpus, rcu_state.n_online_cpus + 1);
return 0;
}
rnp = rdp->mynode;
mask = rdp->grpmask;
+ WRITE_ONCE(rnp->ofl_seq, rnp->ofl_seq + 1);
+ WARN_ON_ONCE(!(rnp->ofl_seq & 0x1));
+ smp_mb(); // Pair with rcu_gp_cleanup()'s ->ofl_seq barrier().
raw_spin_lock_irqsave_rcu_node(rnp, flags);
WRITE_ONCE(rnp->qsmaskinitnext, rnp->qsmaskinitnext | mask);
newcpu = !(rnp->expmaskinitnext & mask);
rcu_gpnum_ovf(rnp, rdp); /* Offline-induced counter wrap? */
rdp->rcu_onl_gp_seq = READ_ONCE(rcu_state.gp_seq);
rdp->rcu_onl_gp_flags = READ_ONCE(rcu_state.gp_flags);
- if (rnp->qsmask & mask) { /* RCU waiting on incoming CPU? */
+
+ /* An incoming CPU should never be blocking a grace period. */
+ if (WARN_ON_ONCE(rnp->qsmask & mask)) { /* RCU waiting on incoming CPU? */
rcu_disable_urgency_upon_qs(rdp);
/* Report QS -after- changing ->qsmaskinitnext! */
rcu_report_qs_rnp(mask, rnp, rnp->gp_seq, flags);
} else {
raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
}
+ smp_mb(); // Pair with rcu_gp_cleanup()'s ->ofl_seq barrier().
+ WRITE_ONCE(rnp->ofl_seq, rnp->ofl_seq + 1);
+ WARN_ON_ONCE(rnp->ofl_seq & 0x1);
smp_mb(); /* Ensure RCU read-side usage follows above initialization. */
}
struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu);
struct rcu_node *rnp = rdp->mynode; /* Outgoing CPU's rdp & rnp. */
++++ ++++ // Do any dangling deferred wakeups.
++++ ++++ do_nocb_deferred_wakeup(rdp);
++++ ++++
/* QS for any half-done expedited grace period. */
preempt_disable();
rcu_report_exp_rdp(this_cpu_ptr(&rcu_data));
/* Remove outgoing CPU from mask in the leaf rcu_node structure. */
mask = rdp->grpmask;
+ WRITE_ONCE(rnp->ofl_seq, rnp->ofl_seq + 1);
+ WARN_ON_ONCE(!(rnp->ofl_seq & 0x1));
+ smp_mb(); // Pair with rcu_gp_cleanup()'s ->ofl_seq barrier().
raw_spin_lock(&rcu_state.ofl_lock);
raw_spin_lock_irqsave_rcu_node(rnp, flags); /* Enforce GP memory-order guarantee. */
rdp->rcu_ofl_gp_seq = READ_ONCE(rcu_state.gp_seq);
WRITE_ONCE(rnp->qsmaskinitnext, rnp->qsmaskinitnext & ~mask);
raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
raw_spin_unlock(&rcu_state.ofl_lock);
+ smp_mb(); // Pair with rcu_gp_cleanup()'s ->ofl_seq barrier().
+ WRITE_ONCE(rnp->ofl_seq, rnp->ofl_seq + 1);
+ WARN_ON_ONCE(rnp->ofl_seq & 0x1);
rdp->cpu_started = false;
}
for_each_possible_cpu(cpu) {
struct kfree_rcu_cpu *krcp = per_cpu_ptr(&krc, cpu);
- struct kvfree_rcu_bulk_data *bnode;
for (i = 0; i < KFREE_N_BATCHES; i++) {
INIT_RCU_WORK(&krcp->krw_arr[i].rcu_work, kfree_rcu_work);
krcp->krw_arr[i].krcp = krcp;
}
- for (i = 0; i < rcu_min_cached_objs; i++) {
- bnode = (struct kvfree_rcu_bulk_data *)
- __get_free_page(GFP_NOWAIT | __GFP_NOWARN);
-
- if (bnode)
- put_cached_bnode(krcp, bnode);
- else
- pr_err("Failed to preallocate for %d CPU!\n", cpu);
- }
-
INIT_DELAYED_WORK(&krcp->monitor_work, kfree_rcu_monitor);
+ INIT_WORK(&krcp->page_cache_work, fill_page_cache_func);
krcp->initialized = true;
}
if (register_shrinker(&kfree_rcu_shrinker))
set_tsk_need_resched(current);
set_preempt_need_resched();
if (IS_ENABLED(CONFIG_IRQ_WORK) && irqs_were_disabled &&
- !rdp->defer_qs_iw_pending && exp) {
+ !rdp->defer_qs_iw_pending && exp && cpu_online(rdp->cpu)) {
// Get scheduler to re-evaluate and call hooks.
// If !IRQ_WORK, FQS scan will eventually IPI.
init_irq_work(&rdp->defer_qs_iw,
{
struct task_struct *t = current;
+ +++++++ lockdep_assert_irqs_disabled();
if (user || rcu_is_cpu_rrupt_from_idle()) {
rcu_note_voluntary_context_switch(current);
}
static void wake_nocb_gp_defer(struct rcu_data *rdp, int waketype,
const char *reason)
{
++++ ++++ if (rdp->nocb_defer_wakeup == RCU_NOCB_WAKE_OFF)
++++ ++++ return;
if (rdp->nocb_defer_wakeup == RCU_NOCB_WAKE_NOT)
mod_timer(&rdp->nocb_timer, jiffies + 1);
if (rdp->nocb_defer_wakeup < waketype)
__call_rcu_nocb_wake(rdp, true, flags);
}
++++ ++++/*
++++ ++++ * Check if we ignore this rdp.
++++ ++++ *
++++ ++++ * We check that without holding the nocb lock but
++++ ++++ * we make sure not to miss a freshly offloaded rdp
++++ ++++ * with the current ordering:
++++ ++++ *
++++ ++++ * rdp_offload_toggle() nocb_gp_enabled_cb()
++++ ++++ * ------------------------- ----------------------------
++++ ++++ * WRITE flags LOCK nocb_gp_lock
++++ ++++ * LOCK nocb_gp_lock READ/WRITE nocb_gp_sleep
++++ ++++ * READ/WRITE nocb_gp_sleep UNLOCK nocb_gp_lock
++++ ++++ * UNLOCK nocb_gp_lock READ flags
++++ ++++ */
++++ ++++static inline bool nocb_gp_enabled_cb(struct rcu_data *rdp)
++++ ++++{
++++ ++++ u8 flags = SEGCBLIST_OFFLOADED | SEGCBLIST_KTHREAD_GP;
++++ ++++
++++ ++++ return rcu_segcblist_test_flags(&rdp->cblist, flags);
++++ ++++}
++++ ++++
++++ ++++static inline bool nocb_gp_update_state(struct rcu_data *rdp, bool *needwake_state)
++++ ++++{
++++ ++++ struct rcu_segcblist *cblist = &rdp->cblist;
++++ ++++
++++ ++++ if (rcu_segcblist_test_flags(cblist, SEGCBLIST_OFFLOADED)) {
++++ ++++ if (!rcu_segcblist_test_flags(cblist, SEGCBLIST_KTHREAD_GP)) {
++++ ++++ rcu_segcblist_set_flags(cblist, SEGCBLIST_KTHREAD_GP);
++++ ++++ if (rcu_segcblist_test_flags(cblist, SEGCBLIST_KTHREAD_CB))
++++ ++++ *needwake_state = true;
++++ ++++ }
++++ ++++ return true;
++++ ++++ }
++++ ++++
++++ ++++ /*
++++ ++++ * De-offloading. Clear our flag and notify the de-offload worker.
++++ ++++ * We will ignore this rdp until it ever gets re-offloaded.
++++ ++++ */
++++ ++++ WARN_ON_ONCE(!rcu_segcblist_test_flags(cblist, SEGCBLIST_KTHREAD_GP));
++++ ++++ rcu_segcblist_clear_flags(cblist, SEGCBLIST_KTHREAD_GP);
++++ ++++ if (!rcu_segcblist_test_flags(cblist, SEGCBLIST_KTHREAD_CB))
++++ ++++ *needwake_state = true;
++++ ++++ return false;
++++ ++++}
++++ ++++
++++ ++++
/*
* No-CBs GP kthreads come here to wait for additional callbacks to show up
* or for grace periods to end.
*/
WARN_ON_ONCE(my_rdp->nocb_gp_rdp != my_rdp);
for (rdp = my_rdp; rdp; rdp = rdp->nocb_next_cb_rdp) {
++++ ++++ bool needwake_state = false;
++++ ++++
++++ ++++ if (!nocb_gp_enabled_cb(rdp))
++++ ++++ continue;
trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("Check"));
rcu_nocb_lock_irqsave(rdp, flags);
++++ ++++ if (!nocb_gp_update_state(rdp, &needwake_state)) {
++++ ++++ rcu_nocb_unlock_irqrestore(rdp, flags);
++++ ++++ if (needwake_state)
++++ ++++ swake_up_one(&rdp->nocb_state_wq);
++++ ++++ continue;
++++ ++++ }
bypass_ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass);
if (bypass_ncbs &&
(time_after(j, READ_ONCE(rdp->nocb_bypass_first) + 1) ||
bypass_ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass);
} else if (!bypass_ncbs && rcu_segcblist_empty(&rdp->cblist)) {
rcu_nocb_unlock_irqrestore(rdp, flags);
++++ ++++ if (needwake_state)
++++ ++++ swake_up_one(&rdp->nocb_state_wq);
continue; /* No callbacks here, try next. */
}
if (bypass_ncbs) {
}
if (needwake_gp)
rcu_gp_kthread_wake();
++++ ++++ if (needwake_state)
++++ ++++ swake_up_one(&rdp->nocb_state_wq);
}
my_rdp->nocb_gp_bypass = bypass;
return 0;
}
++++ ++++static inline bool nocb_cb_can_run(struct rcu_data *rdp)
++++ ++++{
++++ ++++ u8 flags = SEGCBLIST_OFFLOADED | SEGCBLIST_KTHREAD_CB;
++++ ++++ return rcu_segcblist_test_flags(&rdp->cblist, flags);
++++ ++++}
++++ ++++
++++ ++++static inline bool nocb_cb_wait_cond(struct rcu_data *rdp)
++++ ++++{
++++ ++++ return nocb_cb_can_run(rdp) && !READ_ONCE(rdp->nocb_cb_sleep);
++++ ++++}
++++ ++++
/*
* Invoke any ready callbacks from the corresponding no-CBs CPU,
* then, if there are no more, wait for more to appear.
*/
static void nocb_cb_wait(struct rcu_data *rdp)
{
++++ ++++ struct rcu_segcblist *cblist = &rdp->cblist;
unsigned long cur_gp_seq;
unsigned long flags;
++++ ++++ bool needwake_state = false;
bool needwake_gp = false;
struct rcu_node *rnp = rdp->mynode;
local_bh_enable();
lockdep_assert_irqs_enabled();
rcu_nocb_lock_irqsave(rdp, flags);
---- ---- if (rcu_segcblist_nextgp(&rdp->cblist, &cur_gp_seq) &&
++++ ++++ if (rcu_segcblist_nextgp(cblist, &cur_gp_seq) &&
rcu_seq_done(&rnp->gp_seq, cur_gp_seq) &&
raw_spin_trylock_rcu_node(rnp)) { /* irqs already disabled. */
needwake_gp = rcu_advance_cbs(rdp->mynode, rdp);
raw_spin_unlock_rcu_node(rnp); /* irqs remain disabled. */
}
---- ---- if (rcu_segcblist_ready_cbs(&rdp->cblist)) {
---- ---- rcu_nocb_unlock_irqrestore(rdp, flags);
---- ---- if (needwake_gp)
---- ---- rcu_gp_kthread_wake();
---- ---- return;
---- ---- }
---- ---- trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("CBSleep"));
WRITE_ONCE(rdp->nocb_cb_sleep, true);
++++ ++++
++++ ++++ if (rcu_segcblist_test_flags(cblist, SEGCBLIST_OFFLOADED)) {
++++ ++++ if (!rcu_segcblist_test_flags(cblist, SEGCBLIST_KTHREAD_CB)) {
++++ ++++ rcu_segcblist_set_flags(cblist, SEGCBLIST_KTHREAD_CB);
++++ ++++ if (rcu_segcblist_test_flags(cblist, SEGCBLIST_KTHREAD_GP))
++++ ++++ needwake_state = true;
++++ ++++ }
++++ ++++ if (rcu_segcblist_ready_cbs(cblist))
++++ ++++ WRITE_ONCE(rdp->nocb_cb_sleep, false);
++++ ++++ } else {
++++ ++++ /*
++++ ++++ * De-offloading. Clear our flag and notify the de-offload worker.
++++ ++++ * We won't touch the callbacks and keep sleeping until we ever
++++ ++++ * get re-offloaded.
++++ ++++ */
++++ ++++ WARN_ON_ONCE(!rcu_segcblist_test_flags(cblist, SEGCBLIST_KTHREAD_CB));
++++ ++++ rcu_segcblist_clear_flags(cblist, SEGCBLIST_KTHREAD_CB);
++++ ++++ if (!rcu_segcblist_test_flags(cblist, SEGCBLIST_KTHREAD_GP))
++++ ++++ needwake_state = true;
++++ ++++ }
++++ ++++
++++ ++++ if (rdp->nocb_cb_sleep)
++++ ++++ trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("CBSleep"));
++++ ++++
rcu_nocb_unlock_irqrestore(rdp, flags);
if (needwake_gp)
rcu_gp_kthread_wake();
---- ---- swait_event_interruptible_exclusive(rdp->nocb_cb_wq,
---- ---- !READ_ONCE(rdp->nocb_cb_sleep));
---- ---- if (!smp_load_acquire(&rdp->nocb_cb_sleep)) { /* VVV */
---- ---- /* ^^^ Ensure CB invocation follows _sleep test. */
---- ---- return;
---- ---- }
---- ---- WARN_ON(signal_pending(current));
---- ---- trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("WokeEmpty"));
++++ ++++
++++ ++++ if (needwake_state)
++++ ++++ swake_up_one(&rdp->nocb_state_wq);
++++ ++++
++++ ++++ do {
++++ ++++ swait_event_interruptible_exclusive(rdp->nocb_cb_wq,
++++ ++++ nocb_cb_wait_cond(rdp));
++++ ++++
++++ ++++ // VVV Ensure CB invocation follows _sleep test.
++++ ++++ if (smp_load_acquire(&rdp->nocb_cb_sleep)) { // ^^^
++++ ++++ WARN_ON(signal_pending(current));
++++ ++++ trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("WokeEmpty"));
++++ ++++ }
++++ ++++ } while (!nocb_cb_can_run(rdp));
}
/*
/* Is a deferred wakeup of rcu_nocb_kthread() required? */
static int rcu_nocb_need_deferred_wakeup(struct rcu_data *rdp)
{
---- ---- return READ_ONCE(rdp->nocb_defer_wakeup);
++++ ++++ return READ_ONCE(rdp->nocb_defer_wakeup) > RCU_NOCB_WAKE_NOT;
}
/* Do a deferred wakeup of rcu_nocb_kthread(). */
do_nocb_deferred_wakeup_common(rdp);
}
++++ ++++static int rdp_offload_toggle(struct rcu_data *rdp,
++++ ++++ bool offload, unsigned long flags)
++++ ++++ __releases(rdp->nocb_lock)
++++ ++++{
++++ ++++ struct rcu_segcblist *cblist = &rdp->cblist;
++++ ++++ struct rcu_data *rdp_gp = rdp->nocb_gp_rdp;
++++ ++++ bool wake_gp = false;
++++ ++++
++++ ++++ rcu_segcblist_offload(cblist, offload);
++++ ++++
++++ ++++ if (rdp->nocb_cb_sleep)
++++ ++++ rdp->nocb_cb_sleep = false;
++++ ++++ rcu_nocb_unlock_irqrestore(rdp, flags);
++++ ++++
++++ ++++ /*
++++ ++++ * Ignore former value of nocb_cb_sleep and force wake up as it could
++++ ++++ * have been spuriously set to false already.
++++ ++++ */
++++ ++++ swake_up_one(&rdp->nocb_cb_wq);
++++ ++++
++++ ++++ raw_spin_lock_irqsave(&rdp_gp->nocb_gp_lock, flags);
++++ ++++ if (rdp_gp->nocb_gp_sleep) {
++++ ++++ rdp_gp->nocb_gp_sleep = false;
++++ ++++ wake_gp = true;
++++ ++++ }
++++ ++++ raw_spin_unlock_irqrestore(&rdp_gp->nocb_gp_lock, flags);
++++ ++++
++++ ++++ if (wake_gp)
++++ ++++ wake_up_process(rdp_gp->nocb_gp_kthread);
++++ ++++
++++ ++++ return 0;
++++ ++++}
++++ ++++
++++ ++++static int __rcu_nocb_rdp_deoffload(struct rcu_data *rdp)
++++ ++++{
++++ ++++ struct rcu_segcblist *cblist = &rdp->cblist;
++++ ++++ unsigned long flags;
++++ ++++ int ret;
++++ ++++
++++ ++++ pr_info("De-offloading %d\n", rdp->cpu);
++++ ++++
++++ ++++ rcu_nocb_lock_irqsave(rdp, flags);
++++ ++++ /*
++++ ++++ * If there are still pending work offloaded, the offline
++++ ++++ * CPU won't help much handling them.
++++ ++++ */
++++ ++++ if (cpu_is_offline(rdp->cpu) && !rcu_segcblist_empty(&rdp->cblist)) {
++++ ++++ rcu_nocb_unlock_irqrestore(rdp, flags);
++++ ++++ return -EBUSY;
++++ ++++ }
++++ ++++
++++ ++++ ret = rdp_offload_toggle(rdp, false, flags);
++++ ++++ swait_event_exclusive(rdp->nocb_state_wq,
++++ ++++ !rcu_segcblist_test_flags(cblist, SEGCBLIST_KTHREAD_CB |
++++ ++++ SEGCBLIST_KTHREAD_GP));
++++ ++++ rcu_nocb_lock_irqsave(rdp, flags);
++++ ++++ /* Make sure nocb timer won't stay around */
++++ ++++ WRITE_ONCE(rdp->nocb_defer_wakeup, RCU_NOCB_WAKE_OFF);
++++ ++++ rcu_nocb_unlock_irqrestore(rdp, flags);
++++ ++++ del_timer_sync(&rdp->nocb_timer);
++++ ++++
++++ ++++ /*
++++ ++++ * Flush bypass. While IRQs are disabled and once we set
++++ ++++ * SEGCBLIST_SOFTIRQ_ONLY, no callback is supposed to be
++++ ++++ * enqueued on bypass.
++++ ++++ */
++++ ++++ rcu_nocb_lock_irqsave(rdp, flags);
++++ ++++ rcu_nocb_flush_bypass(rdp, NULL, jiffies);
++++ ++++ rcu_segcblist_set_flags(cblist, SEGCBLIST_SOFTIRQ_ONLY);
++++ ++++ /*
++++ ++++ * With SEGCBLIST_SOFTIRQ_ONLY, we can't use
++++ ++++ * rcu_nocb_unlock_irqrestore() anymore. Theoretically we
++++ ++++ * could set SEGCBLIST_SOFTIRQ_ONLY with cb unlocked and IRQs
++++ ++++ * disabled now, but let's be paranoid.
++++ ++++ */
++++ ++++ raw_spin_unlock_irqrestore(&rdp->nocb_lock, flags);
++++ ++++
++++ ++++ return ret;
++++ ++++}
++++ ++++
++++ ++++static long rcu_nocb_rdp_deoffload(void *arg)
++++ ++++{
++++ ++++ struct rcu_data *rdp = arg;
++++ ++++
++++ ++++ WARN_ON_ONCE(rdp->cpu != raw_smp_processor_id());
++++ ++++ return __rcu_nocb_rdp_deoffload(rdp);
++++ ++++}
++++ ++++
++++ ++++int rcu_nocb_cpu_deoffload(int cpu)
++++ ++++{
++++ ++++ struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu);
++++ ++++ int ret = 0;
++++ ++++
++++ ++++ if (rdp == rdp->nocb_gp_rdp) {
++++ ++++ pr_info("Can't deoffload an rdp GP leader (yet)\n");
++++ ++++ return -EINVAL;
++++ ++++ }
++++ ++++ mutex_lock(&rcu_state.barrier_mutex);
++++ ++++ cpus_read_lock();
++++ ++++ if (rcu_segcblist_is_offloaded(&rdp->cblist)) {
++++ ++++ if (cpu_online(cpu))
++++ ++++ ret = work_on_cpu(cpu, rcu_nocb_rdp_deoffload, rdp);
++++ ++++ else
++++ ++++ ret = __rcu_nocb_rdp_deoffload(rdp);
++++ ++++ if (!ret)
++++ ++++ cpumask_clear_cpu(cpu, rcu_nocb_mask);
++++ ++++ }
++++ ++++ cpus_read_unlock();
++++ ++++ mutex_unlock(&rcu_state.barrier_mutex);
++++ ++++
++++ ++++ return ret;
++++ ++++}
++++ ++++EXPORT_SYMBOL_GPL(rcu_nocb_cpu_deoffload);
++++ ++++
++++ ++++static int __rcu_nocb_rdp_offload(struct rcu_data *rdp)
++++ ++++{
++++ ++++ struct rcu_segcblist *cblist = &rdp->cblist;
++++ ++++ unsigned long flags;
++++ ++++ int ret;
++++ ++++
++++ ++++ /*
++++ ++++ * For now we only support re-offload, ie: the rdp must have been
++++ ++++ * offloaded on boot first.
++++ ++++ */
++++ ++++ if (!rdp->nocb_gp_rdp)
++++ ++++ return -EINVAL;
++++ ++++
++++ ++++ pr_info("Offloading %d\n", rdp->cpu);
++++ ++++ /*
++++ ++++ * Can't use rcu_nocb_lock_irqsave() while we are in
++++ ++++ * SEGCBLIST_SOFTIRQ_ONLY mode.
++++ ++++ */
++++ ++++ raw_spin_lock_irqsave(&rdp->nocb_lock, flags);
++++ ++++ /* Re-enable nocb timer */
++++ ++++ WRITE_ONCE(rdp->nocb_defer_wakeup, RCU_NOCB_WAKE_NOT);
++++ ++++ /*
++++ ++++ * We didn't take the nocb lock while working on the
++++ ++++ * rdp->cblist in SEGCBLIST_SOFTIRQ_ONLY mode.
++++ ++++ * Every modifications that have been done previously on
++++ ++++ * rdp->cblist must be visible remotely by the nocb kthreads
++++ ++++ * upon wake up after reading the cblist flags.
++++ ++++ *
++++ ++++ * The layout against nocb_lock enforces that ordering:
++++ ++++ *
++++ ++++ * __rcu_nocb_rdp_offload() nocb_cb_wait()/nocb_gp_wait()
++++ ++++ * ------------------------- ----------------------------
++++ ++++ * WRITE callbacks rcu_nocb_lock()
++++ ++++ * rcu_nocb_lock() READ flags
++++ ++++ * WRITE flags READ callbacks
++++ ++++ * rcu_nocb_unlock() rcu_nocb_unlock()
++++ ++++ */
++++ ++++ ret = rdp_offload_toggle(rdp, true, flags);
++++ ++++ swait_event_exclusive(rdp->nocb_state_wq,
++++ ++++ rcu_segcblist_test_flags(cblist, SEGCBLIST_KTHREAD_CB) &&
++++ ++++ rcu_segcblist_test_flags(cblist, SEGCBLIST_KTHREAD_GP));
++++ ++++
++++ ++++ return ret;
++++ ++++}
++++ ++++
++++ ++++static long rcu_nocb_rdp_offload(void *arg)
++++ ++++{
++++ ++++ struct rcu_data *rdp = arg;
++++ ++++
++++ ++++ WARN_ON_ONCE(rdp->cpu != raw_smp_processor_id());
++++ ++++ return __rcu_nocb_rdp_offload(rdp);
++++ ++++}
++++ ++++
++++ ++++int rcu_nocb_cpu_offload(int cpu)
++++ ++++{
++++ ++++ struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu);
++++ ++++ int ret = 0;
++++ ++++
++++ ++++ mutex_lock(&rcu_state.barrier_mutex);
++++ ++++ cpus_read_lock();
++++ ++++ if (!rcu_segcblist_is_offloaded(&rdp->cblist)) {
++++ ++++ if (cpu_online(cpu))
++++ ++++ ret = work_on_cpu(cpu, rcu_nocb_rdp_offload, rdp);
++++ ++++ else
++++ ++++ ret = __rcu_nocb_rdp_offload(rdp);
++++ ++++ if (!ret)
++++ ++++ cpumask_set_cpu(cpu, rcu_nocb_mask);
++++ ++++ }
++++ ++++ cpus_read_unlock();
++++ ++++ mutex_unlock(&rcu_state.barrier_mutex);
++++ ++++
++++ ++++ return ret;
++++ ++++}
++++ ++++EXPORT_SYMBOL_GPL(rcu_nocb_cpu_offload);
++++ ++++
void __init rcu_init_nohz(void)
{
int cpu;
rdp = per_cpu_ptr(&rcu_data, cpu);
if (rcu_segcblist_empty(&rdp->cblist))
rcu_segcblist_init(&rdp->cblist);
---- ---- rcu_segcblist_offload(&rdp->cblist);
++++ ++++ rcu_segcblist_offload(&rdp->cblist, true);
++++ ++++ rcu_segcblist_set_flags(&rdp->cblist, SEGCBLIST_KTHREAD_CB);
++++ ++++ rcu_segcblist_set_flags(&rdp->cblist, SEGCBLIST_KTHREAD_GP);
}
rcu_organize_nocb_kthreads();
}
{
init_swait_queue_head(&rdp->nocb_cb_wq);
init_swait_queue_head(&rdp->nocb_gp_wq);
++++ ++++ init_swait_queue_head(&rdp->nocb_state_wq);
raw_spin_lock_init(&rdp->nocb_lock);
raw_spin_lock_init(&rdp->nocb_bypass_lock);
raw_spin_lock_init(&rdp->nocb_gp_lock);
}
EXPORT_SYMBOL_GPL(rcu_bind_current_to_nocb);
++++ ++++// The ->on_cpu field is available only in CONFIG_SMP=y, so...
++++ ++++#ifdef CONFIG_SMP
++++ ++++static char *show_rcu_should_be_on_cpu(struct task_struct *tsp)
++++ ++++{
++++ ++++ return tsp && tsp->state == TASK_RUNNING && !tsp->on_cpu ? "!" : "";
++++ ++++}
++++ ++++#else // #ifdef CONFIG_SMP
++++ ++++static char *show_rcu_should_be_on_cpu(struct task_struct *tsp)
++++ ++++{
++++ ++++ return "";
++++ ++++}
++++ ++++#endif // #else #ifdef CONFIG_SMP
++++ ++++
/*
* Dump out nocb grace-period kthread state for the specified rcu_data
* structure.
{
struct rcu_node *rnp = rdp->mynode;
---- ---- pr_info("nocb GP %d %c%c%c%c%c%c %c[%c%c] %c%c:%ld rnp %d:%d %lu\n",
++++ ++++ pr_info("nocb GP %d %c%c%c%c%c%c %c[%c%c] %c%c:%ld rnp %d:%d %lu %c CPU %d%s\n",
rdp->cpu,
"kK"[!!rdp->nocb_gp_kthread],
"lL"[raw_spin_is_locked(&rdp->nocb_gp_lock)],
".B"[!!rdp->nocb_gp_bypass],
".G"[!!rdp->nocb_gp_gp],
(long)rdp->nocb_gp_seq,
---- ---- rnp->grplo, rnp->grphi, READ_ONCE(rdp->nocb_gp_loops));
++++ ++++ rnp->grplo, rnp->grphi, READ_ONCE(rdp->nocb_gp_loops),
++++ ++++ rdp->nocb_gp_kthread ? task_state_to_char(rdp->nocb_gp_kthread) : '.',
++++ ++++ rdp->nocb_cb_kthread ? (int)task_cpu(rdp->nocb_gp_kthread) : -1,
++++ ++++ show_rcu_should_be_on_cpu(rdp->nocb_cb_kthread));
}
/* Dump out nocb kthread state for the specified rcu_data structure. */
static void show_rcu_nocb_state(struct rcu_data *rdp)
{
++++ ++++ char bufw[20];
++++ ++++ char bufr[20];
struct rcu_segcblist *rsclp = &rdp->cblist;
bool waslocked;
bool wastimer;
if (rdp->nocb_gp_rdp == rdp)
show_rcu_nocb_gp_state(rdp);
---- ---- pr_info(" CB %d->%d %c%c%c%c%c%c F%ld L%ld C%d %c%c%c%c%c q%ld\n",
++++ ++++ sprintf(bufw, "%ld", rsclp->gp_seq[RCU_WAIT_TAIL]);
++++ ++++ sprintf(bufr, "%ld", rsclp->gp_seq[RCU_NEXT_READY_TAIL]);
++++ ++++ pr_info(" CB %d^%d->%d %c%c%c%c%c%c F%ld L%ld C%d %c%c%s%c%s%c%c q%ld %c CPU %d%s\n",
rdp->cpu, rdp->nocb_gp_rdp->cpu,
++++ ++++ rdp->nocb_next_cb_rdp ? rdp->nocb_next_cb_rdp->cpu : -1,
"kK"[!!rdp->nocb_cb_kthread],
"bB"[raw_spin_is_locked(&rdp->nocb_bypass_lock)],
"cC"[!!atomic_read(&rdp->nocb_lock_contended)],
jiffies - rdp->nocb_nobypass_last,
rdp->nocb_nobypass_count,
".D"[rcu_segcblist_ready_cbs(rsclp)],
---- ---- ".W"[!rcu_segcblist_restempty(rsclp, RCU_DONE_TAIL)],
---- ---- ".R"[!rcu_segcblist_restempty(rsclp, RCU_WAIT_TAIL)],
---- ---- ".N"[!rcu_segcblist_restempty(rsclp, RCU_NEXT_READY_TAIL)],
++++ ++++ ".W"[!rcu_segcblist_segempty(rsclp, RCU_WAIT_TAIL)],
++++ ++++ rcu_segcblist_segempty(rsclp, RCU_WAIT_TAIL) ? "" : bufw,
++++ ++++ ".R"[!rcu_segcblist_segempty(rsclp, RCU_NEXT_READY_TAIL)],
++++ ++++ rcu_segcblist_segempty(rsclp, RCU_NEXT_READY_TAIL) ? "" : bufr,
++++ ++++ ".N"[!rcu_segcblist_segempty(rsclp, RCU_NEXT_TAIL)],
".B"[!!rcu_cblist_n_cbs(&rdp->nocb_bypass)],
---- ---- rcu_segcblist_n_cbs(&rdp->cblist));
++++ ++++ rcu_segcblist_n_cbs(&rdp->cblist),
++++ ++++ rdp->nocb_cb_kthread ? task_state_to_char(rdp->nocb_cb_kthread) : '.',
++++ ++++ rdp->nocb_cb_kthread ? (int)task_cpu(rdp->nocb_gp_kthread) : -1,
++++ ++++ show_rcu_should_be_on_cpu(rdp->nocb_cb_kthread));
/* It is OK for GP kthreads to have GP state. */
if (rdp->nocb_gp_rdp == rdp)
/* panic() on RCU Stall sysctl. */
int sysctl_panic_on_rcu_stall __read_mostly;
+ int sysctl_max_rcu_stall_to_panic __read_mostly;
#ifdef CONFIG_PROVE_RCU
#define RCU_STALL_DELAY_DELTA (5 * HZ)
/* If so specified via sysctl, panic, yielding cleaner stall-warning output. */
static void panic_on_rcu_stall(void)
{
+ static int cpu_stall;
+
+ if (++cpu_stall < sysctl_max_rcu_stall_to_panic)
+ return;
+
if (sysctl_panic_on_rcu_stall)
panic("RCU Stall\n");
}
struct task_struct *t;
struct task_struct *ts[8];
+ +++++++ lockdep_assert_irqs_disabled();
if (!rcu_preempt_blocked_readers_cgp(rnp))
return 0;
pr_err("\tTasks blocked on level-%d rcu_node (CPUs %d-%d):",
".q"[rscr.rs.b.need_qs],
".e"[rscr.rs.b.exp_hint],
".l"[rscr.on_blkd_list]);
+ +++++++ lockdep_assert_irqs_disabled();
put_task_struct(t);
ndetected++;
}
rcu_for_each_leaf_node(rnp) {
raw_spin_lock_irqsave_rcu_node(rnp, flags);
for_each_leaf_node_possible_cpu(rnp, cpu)
------ -- if (rnp->qsmask & leaf_node_cpu_bit(rnp, cpu))
------ -- if (!trigger_single_cpu_backtrace(cpu))
++++++ ++ if (rnp->qsmask & leaf_node_cpu_bit(rnp, cpu)) {
++++++ ++ if (cpu_is_offline(cpu))
++++++ ++ pr_err("Offline CPU %d blocking current GP.\n", cpu);
++++++ ++ else if (!trigger_single_cpu_backtrace(cpu))
dump_cpu_task(cpu);
++++++ ++ }
raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
}
}
/* Complain about starvation of grace-period kthread. */
static void rcu_check_gp_kthread_starvation(void)
{
++++++ ++ int cpu;
struct task_struct *gpk = rcu_state.gp_kthread;
unsigned long j;
if (rcu_is_gp_kthread_starving(&j)) {
++++++ ++ cpu = gpk ? task_cpu(gpk) : -1;
pr_err("%s kthread starved for %ld jiffies! g%ld f%#x %s(%d) ->state=%#lx ->cpu=%d\n",
rcu_state.name, j,
(long)rcu_seq_current(&rcu_state.gp_seq),
data_race(rcu_state.gp_flags),
gp_state_getname(rcu_state.gp_state), rcu_state.gp_state,
------ -- gpk ? gpk->state : ~0, gpk ? task_cpu(gpk) : -1);
++++++ ++ gpk ? gpk->state : ~0, cpu);
if (gpk) {
pr_err("\tUnless %s kthread gets sufficient CPU time, OOM is now expected behavior.\n", rcu_state.name);
pr_err("RCU grace-period kthread stack dump:\n");
sched_show_task(gpk);
++++++ ++ if (cpu >= 0) {
++++++ ++ if (cpu_is_offline(cpu)) {
++++++ ++ pr_err("RCU GP kthread last ran on offline CPU %d.\n", cpu);
++++++ ++ } else {
++++++ ++ pr_err("Stack dump where RCU GP kthread last ran:\n");
++++++ ++ if (!trigger_single_cpu_backtrace(cpu))
++++++ ++ dump_cpu_task(cpu);
++++++ ++ }
++++++ ++ }
wake_up_process(gpk);
}
}
}
++++++ ++/* Complain about missing wakeups from expired fqs wait timer */
++++++ ++static void rcu_check_gp_kthread_expired_fqs_timer(void)
++++++ ++{
++++++ ++ struct task_struct *gpk = rcu_state.gp_kthread;
++++++ ++ short gp_state;
++++++ ++ unsigned long jiffies_fqs;
++++++ ++ int cpu;
++++++ ++
++++++ ++ /*
++++++ ++ * Order reads of .gp_state and .jiffies_force_qs.
++++++ ++ * Matching smp_wmb() is present in rcu_gp_fqs_loop().
++++++ ++ */
++++++ ++ gp_state = smp_load_acquire(&rcu_state.gp_state);
++++++ ++ jiffies_fqs = READ_ONCE(rcu_state.jiffies_force_qs);
++++++ ++
++++++ ++ if (gp_state == RCU_GP_WAIT_FQS &&
++++++ ++ time_after(jiffies, jiffies_fqs + RCU_STALL_MIGHT_MIN) &&
++++++ ++ gpk && !READ_ONCE(gpk->on_rq)) {
++++++ ++ cpu = task_cpu(gpk);
++++++ ++ pr_err("%s kthread timer wakeup didn't happen for %ld jiffies! g%ld f%#x %s(%d) ->state=%#lx\n",
++++++ ++ rcu_state.name, (jiffies - jiffies_fqs),
++++++ ++ (long)rcu_seq_current(&rcu_state.gp_seq),
++++++ ++ data_race(rcu_state.gp_flags),
++++++ ++ gp_state_getname(RCU_GP_WAIT_FQS), RCU_GP_WAIT_FQS,
++++++ ++ gpk->state);
++++++ ++ pr_err("\tPossible timer handling issue on cpu=%d timer-softirq=%u\n",
++++++ ++ cpu, kstat_softirqs_cpu(TIMER_SOFTIRQ, cpu));
++++++ ++ }
++++++ ++}
++++++ ++
static void print_other_cpu_stall(unsigned long gp_seq, unsigned long gps)
{
int cpu;
struct rcu_node *rnp;
long totqlen = 0;
+ +++++++ lockdep_assert_irqs_disabled();
+ +++++++
/* Kick and suppress, if so configured. */
rcu_stall_kick_kthreads();
if (rcu_stall_is_suppressed())
}
}
ndetected += rcu_print_task_stall(rnp, flags); // Releases rnp->lock.
+ +++++++ lockdep_assert_irqs_disabled();
}
for_each_possible_cpu(cpu)
WRITE_ONCE(rcu_state.jiffies_stall,
jiffies + 3 * rcu_jiffies_till_stall_check() + 3);
++++++ ++ rcu_check_gp_kthread_expired_fqs_timer();
rcu_check_gp_kthread_starvation();
panic_on_rcu_stall();
struct rcu_node *rnp = rcu_get_root();
long totqlen = 0;
+ +++++++ lockdep_assert_irqs_disabled();
+ +++++++
/* Kick and suppress, if so configured. */
rcu_stall_kick_kthreads();
if (rcu_stall_is_suppressed())
jiffies - gps,
(long)rcu_seq_current(&rcu_state.gp_seq), totqlen);
++++++ ++ rcu_check_gp_kthread_expired_fqs_timer();
rcu_check_gp_kthread_starvation();
rcu_dump_cpu_stacks();
unsigned long js;
struct rcu_node *rnp;
+ +++++++ lockdep_assert_irqs_disabled();
if ((rcu_stall_is_suppressed() && !READ_ONCE(rcu_kick_kthreads)) ||
!rcu_gp_in_progress())
return;
*
* Normal scheduling state is serialized by rq->lock. __schedule() takes the
* local CPU's rq->lock, it optionally removes the task from the runqueue and
- * always looks at the local rq data structures to find the most elegible task
+ * always looks at the local rq data structures to find the most eligible task
* to run next.
*
* Task enqueue is also under rq->lock, possibly taken from another CPU.
update_rq_clock_task(rq, delta);
}
- static inline void
- rq_csd_init(struct rq *rq, call_single_data_t *csd, smp_call_func_t func)
- {
- csd->flags = 0;
- csd->func = func;
- csd->info = rq;
- }
-
#ifdef CONFIG_SCHED_HRTICK
/*
* Use HR-timers to deliver accurate preemption points.
static void hrtick_rq_init(struct rq *rq)
{
#ifdef CONFIG_SMP
- rq_csd_init(rq, &rq->hrtick_csd, __hrtick_start);
+ INIT_CSD(&rq->hrtick_csd, __hrtick_start, rq);
#endif
hrtimer_init(&rq->hrtick_timer, CLOCK_MONOTONIC, HRTIMER_MODE_REL_HARD);
rq->hrtick_timer.function = hrtick;
/*
* Atomically grab the task, if ->wake_q is !nil already it means
- * its already queued (either by us or someone else) and will get the
+ * it's already queued (either by us or someone else) and will get the
* wakeup due to that.
*
* In order to ensure that a pending wakeup will observe our pending
return false;
/*
- * If there are more than one RR tasks, we need the tick to effect the
+ * If there are more than one RR tasks, we need the tick to affect the
* actual RR behaviour.
*/
if (rq->rt.rr_nr_running) {
* accounting was performed at enqueue time and we can just return
* here.
*
- * Need to be careful of the following enqeueue/dequeue ordering
+ * Need to be careful of the following enqueue/dequeue ordering
* problem too
*
* enqueue(taskA)
* // sched_uclamp_used gets enabled
* enqueue(taskB)
* dequeue(taskA)
- * // Must not decrement bukcet->tasks here
+ * // Must not decrement bucket->tasks here
* dequeue(taskB)
*
* where we could end up with stale data in uc_se and
static int uclamp_validate(struct task_struct *p,
const struct sched_attr *attr)
{
- unsigned int lower_bound = p->uclamp_req[UCLAMP_MIN].value;
- unsigned int upper_bound = p->uclamp_req[UCLAMP_MAX].value;
+ int util_min = p->uclamp_req[UCLAMP_MIN].value;
+ int util_max = p->uclamp_req[UCLAMP_MAX].value;
- if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MIN)
- lower_bound = attr->sched_util_min;
- if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MAX)
- upper_bound = attr->sched_util_max;
+ if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MIN) {
+ util_min = attr->sched_util_min;
- if (lower_bound > upper_bound)
- return -EINVAL;
- if (upper_bound > SCHED_CAPACITY_SCALE)
+ if (util_min + 1 > SCHED_CAPACITY_SCALE + 1)
+ return -EINVAL;
+ }
+
+ if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MAX) {
+ util_max = attr->sched_util_max;
+
+ if (util_max + 1 > SCHED_CAPACITY_SCALE + 1)
+ return -EINVAL;
+ }
+
+ if (util_min != -1 && util_max != -1 && util_min > util_max)
return -EINVAL;
/*
return 0;
}
+ static bool uclamp_reset(const struct sched_attr *attr,
+ enum uclamp_id clamp_id,
+ struct uclamp_se *uc_se)
+ {
+ /* Reset on sched class change for a non user-defined clamp value. */
+ if (likely(!(attr->sched_flags & SCHED_FLAG_UTIL_CLAMP)) &&
+ !uc_se->user_defined)
+ return true;
+
+ /* Reset on sched_util_{min,max} == -1. */
+ if (clamp_id == UCLAMP_MIN &&
+ attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MIN &&
+ attr->sched_util_min == -1) {
+ return true;
+ }
+
+ if (clamp_id == UCLAMP_MAX &&
+ attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MAX &&
+ attr->sched_util_max == -1) {
+ return true;
+ }
+
+ return false;
+ }
+
static void __setscheduler_uclamp(struct task_struct *p,
const struct sched_attr *attr)
{
enum uclamp_id clamp_id;
- /*
- * On scheduling class change, reset to default clamps for tasks
- * without a task-specific value.
- */
for_each_clamp_id(clamp_id) {
struct uclamp_se *uc_se = &p->uclamp_req[clamp_id];
+ unsigned int value;
- /* Keep using defined clamps across class changes */
- if (uc_se->user_defined)
+ if (!uclamp_reset(attr, clamp_id, uc_se))
continue;
/*
* at runtime.
*/
if (unlikely(rt_task(p) && clamp_id == UCLAMP_MIN))
- __uclamp_update_util_min_rt_default(p);
+ value = sysctl_sched_uclamp_util_min_rt_default;
else
- uclamp_se_set(uc_se, uclamp_none(clamp_id), false);
+ value = uclamp_none(clamp_id);
+
+ uclamp_se_set(uc_se, value, false);
}
if (likely(!(attr->sched_flags & SCHED_FLAG_UTIL_CLAMP)))
return;
- if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MIN) {
+ if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MIN &&
+ attr->sched_util_min != -1) {
uclamp_se_set(&p->uclamp_req[UCLAMP_MIN],
attr->sched_util_min, true);
}
- if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MAX) {
+ if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MAX &&
+ attr->sched_util_max != -1) {
uclamp_se_set(&p->uclamp_req[UCLAMP_MAX],
attr->sched_util_max, true);
}
#ifdef CONFIG_SMP
+ static void
+ __do_set_cpus_allowed(struct task_struct *p, const struct cpumask *new_mask, u32 flags);
+
+ static int __set_cpus_allowed_ptr(struct task_struct *p,
+ const struct cpumask *new_mask,
+ u32 flags);
+
+ static void migrate_disable_switch(struct rq *rq, struct task_struct *p)
+ {
+ if (likely(!p->migration_disabled))
+ return;
+
+ if (p->cpus_ptr != &p->cpus_mask)
+ return;
+
+ /*
+ * Violates locking rules! see comment in __do_set_cpus_allowed().
+ */
+ __do_set_cpus_allowed(p, cpumask_of(rq->cpu), SCA_MIGRATE_DISABLE);
+ }
+
+ void migrate_disable(void)
+ {
+ struct task_struct *p = current;
+
+ if (p->migration_disabled) {
+ p->migration_disabled++;
+ return;
+ }
+
+ preempt_disable();
+ this_rq()->nr_pinned++;
+ p->migration_disabled = 1;
+ preempt_enable();
+ }
+ EXPORT_SYMBOL_GPL(migrate_disable);
+
+ void migrate_enable(void)
+ {
+ struct task_struct *p = current;
+
+ if (p->migration_disabled > 1) {
+ p->migration_disabled--;
+ return;
+ }
+
+ /*
+ * Ensure stop_task runs either before or after this, and that
+ * __set_cpus_allowed_ptr(SCA_MIGRATE_ENABLE) doesn't schedule().
+ */
+ preempt_disable();
+ if (p->cpus_ptr != &p->cpus_mask)
+ __set_cpus_allowed_ptr(p, &p->cpus_mask, SCA_MIGRATE_ENABLE);
+ /*
+ * Mustn't clear migration_disabled() until cpus_ptr points back at the
+ * regular cpus_mask, otherwise things that race (eg.
+ * select_fallback_rq) get confused.
+ */
+ barrier();
+ p->migration_disabled = 0;
+ this_rq()->nr_pinned--;
+ preempt_enable();
+ }
+ EXPORT_SYMBOL_GPL(migrate_enable);
+
+ static inline bool rq_has_pinned_tasks(struct rq *rq)
+ {
+ return rq->nr_pinned;
+ }
+
/*
* Per-CPU kthreads are allowed to run on !active && online CPUs, see
* __set_cpus_allowed_ptr() and select_fallback_rq().
if (!cpumask_test_cpu(cpu, p->cpus_ptr))
return false;
- if (is_per_cpu_kthread(p))
+ if (is_per_cpu_kthread(p) || is_migration_disabled(p))
return cpu_online(cpu);
return cpu_active(cpu);
}
struct migration_arg {
- struct task_struct *task;
- int dest_cpu;
+ struct task_struct *task;
+ int dest_cpu;
+ struct set_affinity_pending *pending;
+ };
+
+ struct set_affinity_pending {
+ refcount_t refs;
+ struct completion done;
+ struct cpu_stop_work stop_work;
+ struct migration_arg arg;
};
/*
*/
static int migration_cpu_stop(void *data)
{
+ struct set_affinity_pending *pending;
struct migration_arg *arg = data;
struct task_struct *p = arg->task;
+ int dest_cpu = arg->dest_cpu;
struct rq *rq = this_rq();
+ bool complete = false;
struct rq_flags rf;
/*
* The original target CPU might have gone down and we might
* be on another CPU but it doesn't matter.
*/
- local_irq_disable();
+ local_irq_save(rf.flags);
/*
* We need to explicitly wake pending tasks before running
* __migrate_task() such that we will not miss enforcing cpus_ptr
raw_spin_lock(&p->pi_lock);
rq_lock(rq, &rf);
+
+ pending = p->migration_pending;
/*
* If task_rq(p) != rq, it cannot be migrated here, because we're
* holding rq->lock, if p->on_rq == 0 it cannot get enqueued because
* we're holding p->pi_lock.
*/
if (task_rq(p) == rq) {
+ if (is_migration_disabled(p))
+ goto out;
+
+ if (pending) {
+ p->migration_pending = NULL;
+ complete = true;
+ }
+
+ /* migrate_enable() -- we must not race against SCA */
+ if (dest_cpu < 0) {
+ /*
+ * When this was migrate_enable() but we no longer
+ * have a @pending, a concurrent SCA 'fixed' things
+ * and we should be valid again. Nothing to do.
+ */
+ if (!pending) {
+ WARN_ON_ONCE(!cpumask_test_cpu(task_cpu(p), &p->cpus_mask));
+ goto out;
+ }
+
+ dest_cpu = cpumask_any_distribute(&p->cpus_mask);
+ }
+
if (task_on_rq_queued(p))
- rq = __migrate_task(rq, &rf, p, arg->dest_cpu);
+ rq = __migrate_task(rq, &rf, p, dest_cpu);
else
- p->wake_cpu = arg->dest_cpu;
+ p->wake_cpu = dest_cpu;
+
+ } else if (dest_cpu < 0 || pending) {
+ /*
+ * This happens when we get migrated between migrate_enable()'s
+ * preempt_enable() and scheduling the stopper task. At that
+ * point we're a regular task again and not current anymore.
+ *
+ * A !PREEMPT kernel has a giant hole here, which makes it far
+ * more likely.
+ */
+
+ /*
+ * The task moved before the stopper got to run. We're holding
+ * ->pi_lock, so the allowed mask is stable - if it got
+ * somewhere allowed, we're done.
+ */
+ if (pending && cpumask_test_cpu(task_cpu(p), p->cpus_ptr)) {
+ p->migration_pending = NULL;
+ complete = true;
+ goto out;
+ }
+
+ /*
+ * When this was migrate_enable() but we no longer have an
+ * @pending, a concurrent SCA 'fixed' things and we should be
+ * valid again. Nothing to do.
+ */
+ if (!pending) {
+ WARN_ON_ONCE(!cpumask_test_cpu(task_cpu(p), &p->cpus_mask));
+ goto out;
+ }
+
+ /*
+ * When migrate_enable() hits a rq mis-match we can't reliably
+ * determine is_migration_disabled() and so have to chase after
+ * it.
+ */
+ task_rq_unlock(rq, p, &rf);
+ stop_one_cpu_nowait(task_cpu(p), migration_cpu_stop,
+ &pending->arg, &pending->stop_work);
+ return 0;
}
- rq_unlock(rq, &rf);
- raw_spin_unlock(&p->pi_lock);
+ out:
+ task_rq_unlock(rq, p, &rf);
+
+ if (complete)
+ complete_all(&pending->done);
+
+ /* For pending->{arg,stop_work} */
+ pending = arg->pending;
+ if (pending && refcount_dec_and_test(&pending->refs))
+ wake_up_var(&pending->refs);
- local_irq_enable();
+ return 0;
+ }
+
+ int push_cpu_stop(void *arg)
+ {
+ struct rq *lowest_rq = NULL, *rq = this_rq();
+ struct task_struct *p = arg;
+
+ raw_spin_lock_irq(&p->pi_lock);
+ raw_spin_lock(&rq->lock);
+
+ if (task_rq(p) != rq)
+ goto out_unlock;
+
+ if (is_migration_disabled(p)) {
+ p->migration_flags |= MDF_PUSH;
+ goto out_unlock;
+ }
+
+ p->migration_flags &= ~MDF_PUSH;
+
+ if (p->sched_class->find_lock_rq)
+ lowest_rq = p->sched_class->find_lock_rq(p, rq);
+
+ if (!lowest_rq)
+ goto out_unlock;
+
+ // XXX validate p is still the highest prio task
+ if (task_rq(p) == rq) {
+ deactivate_task(rq, p, 0);
+ set_task_cpu(p, lowest_rq->cpu);
+ activate_task(lowest_rq, p, 0);
+ resched_curr(lowest_rq);
+ }
+
+ double_unlock_balance(rq, lowest_rq);
+
+ out_unlock:
+ rq->push_busy = false;
+ raw_spin_unlock(&rq->lock);
+ raw_spin_unlock_irq(&p->pi_lock);
+
+ put_task_struct(p);
return 0;
}
* sched_class::set_cpus_allowed must do the below, but is not required to
* actually call this function.
*/
- void set_cpus_allowed_common(struct task_struct *p, const struct cpumask *new_mask)
+ void set_cpus_allowed_common(struct task_struct *p, const struct cpumask *new_mask, u32 flags)
{
+ if (flags & (SCA_MIGRATE_ENABLE | SCA_MIGRATE_DISABLE)) {
+ p->cpus_ptr = new_mask;
+ return;
+ }
+
cpumask_copy(&p->cpus_mask, new_mask);
p->nr_cpus_allowed = cpumask_weight(new_mask);
}
- void do_set_cpus_allowed(struct task_struct *p, const struct cpumask *new_mask)
+ static void
+ __do_set_cpus_allowed(struct task_struct *p, const struct cpumask *new_mask, u32 flags)
{
struct rq *rq = task_rq(p);
bool queued, running;
- lockdep_assert_held(&p->pi_lock);
+ /*
+ * This here violates the locking rules for affinity, since we're only
+ * supposed to change these variables while holding both rq->lock and
+ * p->pi_lock.
+ *
+ * HOWEVER, it magically works, because ttwu() is the only code that
+ * accesses these variables under p->pi_lock and only does so after
+ * smp_cond_load_acquire(&p->on_cpu, !VAL), and we're in __schedule()
+ * before finish_task().
+ *
+ * XXX do further audits, this smells like something putrid.
+ */
+ if (flags & SCA_MIGRATE_DISABLE)
+ SCHED_WARN_ON(!p->on_cpu);
+ else
+ lockdep_assert_held(&p->pi_lock);
queued = task_on_rq_queued(p);
running = task_current(rq, p);
if (running)
put_prev_task(rq, p);
- p->sched_class->set_cpus_allowed(p, new_mask);
+ p->sched_class->set_cpus_allowed(p, new_mask, flags);
if (queued)
enqueue_task(rq, p, ENQUEUE_RESTORE | ENQUEUE_NOCLOCK);
set_next_task(rq, p);
}
+ void do_set_cpus_allowed(struct task_struct *p, const struct cpumask *new_mask)
+ {
+ __do_set_cpus_allowed(p, new_mask, 0);
+ }
+
+ /*
+ * This function is wildly self concurrent; here be dragons.
+ *
+ *
+ * When given a valid mask, __set_cpus_allowed_ptr() must block until the
+ * designated task is enqueued on an allowed CPU. If that task is currently
+ * running, we have to kick it out using the CPU stopper.
+ *
+ * Migrate-Disable comes along and tramples all over our nice sandcastle.
+ * Consider:
+ *
+ * Initial conditions: P0->cpus_mask = [0, 1]
+ *
+ * P0@CPU0 P1
+ *
+ * migrate_disable();
+ * <preempted>
+ * set_cpus_allowed_ptr(P0, [1]);
+ *
+ * P1 *cannot* return from this set_cpus_allowed_ptr() call until P0 executes
+ * its outermost migrate_enable() (i.e. it exits its Migrate-Disable region).
+ * This means we need the following scheme:
+ *
+ * P0@CPU0 P1
+ *
+ * migrate_disable();
+ * <preempted>
+ * set_cpus_allowed_ptr(P0, [1]);
+ * <blocks>
+ * <resumes>
+ * migrate_enable();
+ * __set_cpus_allowed_ptr();
+ * <wakes local stopper>
+ * `--> <woken on migration completion>
+ *
+ * Now the fun stuff: there may be several P1-like tasks, i.e. multiple
+ * concurrent set_cpus_allowed_ptr(P0, [*]) calls. CPU affinity changes of any
+ * task p are serialized by p->pi_lock, which we can leverage: the one that
+ * should come into effect at the end of the Migrate-Disable region is the last
+ * one. This means we only need to track a single cpumask (i.e. p->cpus_mask),
+ * but we still need to properly signal those waiting tasks at the appropriate
+ * moment.
+ *
+ * This is implemented using struct set_affinity_pending. The first
+ * __set_cpus_allowed_ptr() caller within a given Migrate-Disable region will
+ * setup an instance of that struct and install it on the targeted task_struct.
+ * Any and all further callers will reuse that instance. Those then wait for
+ * a completion signaled at the tail of the CPU stopper callback (1), triggered
+ * on the end of the Migrate-Disable region (i.e. outermost migrate_enable()).
+ *
+ *
+ * (1) In the cases covered above. There is one more where the completion is
+ * signaled within affine_move_task() itself: when a subsequent affinity request
+ * cancels the need for an active migration. Consider:
+ *
+ * Initial conditions: P0->cpus_mask = [0, 1]
+ *
+ * P0@CPU0 P1 P2
+ *
+ * migrate_disable();
+ * <preempted>
+ * set_cpus_allowed_ptr(P0, [1]);
+ * <blocks>
+ * set_cpus_allowed_ptr(P0, [0, 1]);
+ * <signal completion>
+ * <awakes>
+ *
+ * Note that the above is safe vs a concurrent migrate_enable(), as any
+ * pending affinity completion is preceded by an uninstallation of
+ * p->migration_pending done with p->pi_lock held.
+ */
+ static int affine_move_task(struct rq *rq, struct task_struct *p, struct rq_flags *rf,
+ int dest_cpu, unsigned int flags)
+ {
+ struct set_affinity_pending my_pending = { }, *pending = NULL;
+ struct migration_arg arg = {
+ .task = p,
+ .dest_cpu = dest_cpu,
+ };
+ bool complete = false;
+
+ /* Can the task run on the task's current CPU? If so, we're done */
+ if (cpumask_test_cpu(task_cpu(p), &p->cpus_mask)) {
+ struct task_struct *push_task = NULL;
+
+ if ((flags & SCA_MIGRATE_ENABLE) &&
+ (p->migration_flags & MDF_PUSH) && !rq->push_busy) {
+ rq->push_busy = true;
+ push_task = get_task_struct(p);
+ }
+
+ pending = p->migration_pending;
+ if (pending) {
+ refcount_inc(&pending->refs);
+ p->migration_pending = NULL;
+ complete = true;
+ }
+ task_rq_unlock(rq, p, rf);
+
+ if (push_task) {
+ stop_one_cpu_nowait(rq->cpu, push_cpu_stop,
+ p, &rq->push_work);
+ }
+
+ if (complete)
+ goto do_complete;
+
+ return 0;
+ }
+
+ if (!(flags & SCA_MIGRATE_ENABLE)) {
+ /* serialized by p->pi_lock */
+ if (!p->migration_pending) {
+ /* Install the request */
+ refcount_set(&my_pending.refs, 1);
+ init_completion(&my_pending.done);
+ p->migration_pending = &my_pending;
+ } else {
+ pending = p->migration_pending;
+ refcount_inc(&pending->refs);
+ }
+ }
+ pending = p->migration_pending;
+ /*
+ * - !MIGRATE_ENABLE:
+ * we'll have installed a pending if there wasn't one already.
+ *
+ * - MIGRATE_ENABLE:
+ * we're here because the current CPU isn't matching anymore,
+ * the only way that can happen is because of a concurrent
+ * set_cpus_allowed_ptr() call, which should then still be
+ * pending completion.
+ *
+ * Either way, we really should have a @pending here.
+ */
+ if (WARN_ON_ONCE(!pending)) {
+ task_rq_unlock(rq, p, rf);
+ return -EINVAL;
+ }
+
+ if (flags & SCA_MIGRATE_ENABLE) {
+
+ refcount_inc(&pending->refs); /* pending->{arg,stop_work} */
+ p->migration_flags &= ~MDF_PUSH;
+ task_rq_unlock(rq, p, rf);
+
+ pending->arg = (struct migration_arg) {
+ .task = p,
+ .dest_cpu = -1,
+ .pending = pending,
+ };
+
+ stop_one_cpu_nowait(cpu_of(rq), migration_cpu_stop,
+ &pending->arg, &pending->stop_work);
+
+ return 0;
+ }
+
+ if (task_running(rq, p) || p->state == TASK_WAKING) {
+ /*
+ * Lessen races (and headaches) by delegating
+ * is_migration_disabled(p) checks to the stopper, which will
+ * run on the same CPU as said p.
+ */
+ task_rq_unlock(rq, p, rf);
+ stop_one_cpu(cpu_of(rq), migration_cpu_stop, &arg);
+
+ } else {
+
+ if (!is_migration_disabled(p)) {
+ if (task_on_rq_queued(p))
+ rq = move_queued_task(rq, rf, p, dest_cpu);
+
+ p->migration_pending = NULL;
+ complete = true;
+ }
+ task_rq_unlock(rq, p, rf);
+
+ do_complete:
+ if (complete)
+ complete_all(&pending->done);
+ }
+
+ wait_for_completion(&pending->done);
+
+ if (refcount_dec_and_test(&pending->refs))
+ wake_up_var(&pending->refs);
+
+ /*
+ * Block the original owner of &pending until all subsequent callers
+ * have seen the completion and decremented the refcount
+ */
+ wait_var_event(&my_pending.refs, !refcount_read(&my_pending.refs));
+
+ return 0;
+ }
+
/*
* Change a given task's CPU affinity. Migrate the thread to a
* proper CPU and schedule it away if the CPU it's executing on
* call is not atomic; no spinlocks may be held.
*/
static int __set_cpus_allowed_ptr(struct task_struct *p,
- const struct cpumask *new_mask, bool check)
+ const struct cpumask *new_mask,
+ u32 flags)
{
const struct cpumask *cpu_valid_mask = cpu_active_mask;
unsigned int dest_cpu;
rq = task_rq_lock(p, &rf);
update_rq_clock(rq);
- if (p->flags & PF_KTHREAD) {
+ if (p->flags & PF_KTHREAD || is_migration_disabled(p)) {
/*
- * Kernel threads are allowed on online && !active CPUs
+ * Kernel threads are allowed on online && !active CPUs.
+ *
+ * Specifically, migration_disabled() tasks must not fail the
+ * cpumask_any_and_distribute() pick below, esp. so on
+ * SCA_MIGRATE_ENABLE, otherwise we'll not call
+ * set_cpus_allowed_common() and actually reset p->cpus_ptr.
*/
cpu_valid_mask = cpu_online_mask;
}
* Must re-check here, to close a race against __kthread_bind(),
* sched_setaffinity() is not guaranteed to observe the flag.
*/
- if (check && (p->flags & PF_NO_SETAFFINITY)) {
+ if ((flags & SCA_CHECK) && (p->flags & PF_NO_SETAFFINITY)) {
ret = -EINVAL;
goto out;
}
- if (cpumask_equal(&p->cpus_mask, new_mask))
- goto out;
+ if (!(flags & SCA_MIGRATE_ENABLE)) {
+ if (cpumask_equal(&p->cpus_mask, new_mask))
+ goto out;
+
+ if (WARN_ON_ONCE(p == current &&
+ is_migration_disabled(p) &&
+ !cpumask_test_cpu(task_cpu(p), new_mask))) {
+ ret = -EBUSY;
+ goto out;
+ }
+ }
/*
* Picking a ~random cpu helps in cases where we are changing affinity
goto out;
}
- do_set_cpus_allowed(p, new_mask);
+ __do_set_cpus_allowed(p, new_mask, flags);
if (p->flags & PF_KTHREAD) {
/*
p->nr_cpus_allowed != 1);
}
- /* Can the task run on the task's current CPU? If so, we're done */
- if (cpumask_test_cpu(task_cpu(p), new_mask))
- goto out;
+ return affine_move_task(rq, p, &rf, dest_cpu, flags);
- if (task_running(rq, p) || p->state == TASK_WAKING) {
- struct migration_arg arg = { p, dest_cpu };
- /* Need help from migration thread: drop lock and wait. */
- task_rq_unlock(rq, p, &rf);
- stop_one_cpu(cpu_of(rq), migration_cpu_stop, &arg);
- return 0;
- } else if (task_on_rq_queued(p)) {
- /*
- * OK, since we're going to drop the lock immediately
- * afterwards anyway.
- */
- rq = move_queued_task(rq, &rf, p, dest_cpu);
- }
out:
task_rq_unlock(rq, p, &rf);
int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask)
{
- return __set_cpus_allowed_ptr(p, new_mask, false);
+ return __set_cpus_allowed_ptr(p, new_mask, 0);
}
EXPORT_SYMBOL_GPL(set_cpus_allowed_ptr);
* Clearly, migrating tasks to offline CPUs is a fairly daft thing.
*/
WARN_ON_ONCE(!cpu_online(new_cpu));
+
+ WARN_ON_ONCE(is_migration_disabled(p));
#endif
trace_sched_migrate_task(p, new_cpu);
}
fallthrough;
case possible:
+ /*
+ * XXX When called from select_task_rq() we only
+ * hold p->pi_lock and again violate locking order.
+ *
+ * More yuck to audit.
+ */
do_set_cpus_allowed(p, cpu_possible_mask);
state = fail;
break;
* The caller (fork, wakeup) owns p->pi_lock, ->cpus_ptr is stable.
*/
static inline
- int select_task_rq(struct task_struct *p, int cpu, int sd_flags, int wake_flags)
+ int select_task_rq(struct task_struct *p, int cpu, int wake_flags)
{
lockdep_assert_held(&p->pi_lock);
- if (p->nr_cpus_allowed > 1)
- cpu = p->sched_class->select_task_rq(p, cpu, sd_flags, wake_flags);
+ if (p->nr_cpus_allowed > 1 && !is_migration_disabled(p))
+ cpu = p->sched_class->select_task_rq(p, cpu, wake_flags);
else
cpu = cpumask_any(p->cpus_ptr);
void sched_set_stop_task(int cpu, struct task_struct *stop)
{
+ static struct lock_class_key stop_pi_lock;
struct sched_param param = { .sched_priority = MAX_RT_PRIO - 1 };
struct task_struct *old_stop = cpu_rq(cpu)->stop;
sched_setscheduler_nocheck(stop, SCHED_FIFO, ¶m);
stop->sched_class = &stop_sched_class;
+
+ /*
+ * The PI code calls rt_mutex_setprio() with ->pi_lock held to
+ * adjust the effective priority of a task. As a result,
+ * rt_mutex_setprio() can trigger (RT) balancing operations,
+ * which can then trigger wakeups of the stop thread to push
+ * around the current task.
+ *
+ * The stop task itself will never be part of the PI-chain, it
+ * never blocks, therefore that ->pi_lock recursion is safe.
+ * Tell lockdep about this by placing the stop->pi_lock in its
+ * own class.
+ */
+ lockdep_set_class(&stop->pi_lock, &stop_pi_lock);
}
cpu_rq(cpu)->stop = stop;
}
}
- #else
+ #else /* CONFIG_SMP */
static inline int __set_cpus_allowed_ptr(struct task_struct *p,
- const struct cpumask *new_mask, bool check)
+ const struct cpumask *new_mask,
+ u32 flags)
{
return set_cpus_allowed_ptr(p, new_mask);
}
- #endif /* CONFIG_SMP */
+ static inline void migrate_disable_switch(struct rq *rq, struct task_struct *p) { }
+
+ static inline bool rq_has_pinned_tasks(struct rq *rq)
+ {
+ return false;
+ }
+
+ #endif /* !CONFIG_SMP */
static void
ttwu_stat(struct task_struct *p, int cpu, int wake_flags)
#ifdef CONFIG_SMP
if (p->sched_class->task_woken) {
/*
- * Our task @p is fully woken up and running; so its safe to
+ * Our task @p is fully woken up and running; so it's safe to
* drop the rq->lock, hereafter rq is only used for statistics.
*/
rq_unpin_lock(rq, rf);
/*
* If the owning (remote) CPU is still in the middle of schedule() with
- * this task as prev, wait until its done referencing the task.
+ * this task as prev, wait until it's done referencing the task.
*
* Pairs with the smp_store_release() in finish_task().
*
*/
smp_cond_load_acquire(&p->on_cpu, !VAL);
- cpu = select_task_rq(p, p->wake_cpu, SD_BALANCE_WAKE, wake_flags);
+ cpu = select_task_rq(p, p->wake_cpu, wake_flags | WF_TTWU);
if (task_cpu(p) != cpu) {
if (p->in_iowait) {
delayacct_blkio_end(p);
/**
* try_invoke_on_locked_down_task - Invoke a function on task in fixed state
- ------- * @p: Process for which the function is to be invoked.
+ +++++++ * @p: Process for which the function is to be invoked, can be @current.
* @func: Function to invoke.
* @arg: Argument to function.
*
*/
bool try_invoke_on_locked_down_task(struct task_struct *p, bool (*func)(struct task_struct *t, void *arg), void *arg)
{
- ------- bool ret = false;
struct rq_flags rf;
+ +++++++ bool ret = false;
struct rq *rq;
- ------- lockdep_assert_irqs_enabled();
- ------- raw_spin_lock_irq(&p->pi_lock);
+ +++++++ raw_spin_lock_irqsave(&p->pi_lock, rf.flags);
if (p->on_rq) {
rq = __task_rq_lock(p, &rf);
if (task_rq(p) == rq)
ret = func(p, arg);
}
}
- ------- raw_spin_unlock_irq(&p->pi_lock);
+ +++++++ raw_spin_unlock_irqrestore(&p->pi_lock, rf.flags);
return ret;
}
init_numa_balancing(clone_flags, p);
#ifdef CONFIG_SMP
p->wake_entry.u_flags = CSD_TYPE_TTWU;
+ p->migration_pending = NULL;
#endif
}
*/
p->recent_used_cpu = task_cpu(p);
rseq_migrate(p);
- __set_task_cpu(p, select_task_rq(p, task_cpu(p), SD_BALANCE_FORK, 0));
+ __set_task_cpu(p, select_task_rq(p, task_cpu(p), WF_FORK));
#endif
rq = __task_rq_lock(p, &rf);
update_rq_clock(rq);
#ifdef CONFIG_SMP
if (p->sched_class->task_woken) {
/*
- * Nothing relies on rq->lock after this, so its fine to
+ * Nothing relies on rq->lock after this, so it's fine to
* drop it.
*/
rq_unpin_lock(rq, &rf);
#endif
}
+ #ifdef CONFIG_SMP
+
+ static void do_balance_callbacks(struct rq *rq, struct callback_head *head)
+ {
+ void (*func)(struct rq *rq);
+ struct callback_head *next;
+
+ lockdep_assert_held(&rq->lock);
+
+ while (head) {
+ func = (void (*)(struct rq *))head->func;
+ next = head->next;
+ head->next = NULL;
+ head = next;
+
+ func(rq);
+ }
+ }
+
+ static void balance_push(struct rq *rq);
+
+ struct callback_head balance_push_callback = {
+ .next = NULL,
+ .func = (void (*)(struct callback_head *))balance_push,
+ };
+
+ static inline struct callback_head *splice_balance_callbacks(struct rq *rq)
+ {
+ struct callback_head *head = rq->balance_callback;
+
+ lockdep_assert_held(&rq->lock);
+ if (head)
+ rq->balance_callback = NULL;
+
+ return head;
+ }
+
+ static void __balance_callbacks(struct rq *rq)
+ {
+ do_balance_callbacks(rq, splice_balance_callbacks(rq));
+ }
+
+ static inline void balance_callbacks(struct rq *rq, struct callback_head *head)
+ {
+ unsigned long flags;
+
+ if (unlikely(head)) {
+ raw_spin_lock_irqsave(&rq->lock, flags);
+ do_balance_callbacks(rq, head);
+ raw_spin_unlock_irqrestore(&rq->lock, flags);
+ }
+ }
+
+ #else
+
+ static inline void __balance_callbacks(struct rq *rq)
+ {
+ }
+
+ static inline struct callback_head *splice_balance_callbacks(struct rq *rq)
+ {
+ return NULL;
+ }
+
+ static inline void balance_callbacks(struct rq *rq, struct callback_head *head)
+ {
+ }
+
+ #endif
+
static inline void
prepare_lock_switch(struct rq *rq, struct task_struct *next, struct rq_flags *rf)
{
* prev into current:
*/
spin_acquire(&rq->lock.dep_map, 0, 0, _THIS_IP_);
+ __balance_callbacks(rq);
raw_spin_unlock_irq(&rq->lock);
}
# define finish_arch_post_lock_switch() do { } while (0)
#endif
+ static inline void kmap_local_sched_out(void)
+ {
+ #ifdef CONFIG_KMAP_LOCAL
+ if (unlikely(current->kmap_ctrl.idx))
+ __kmap_local_sched_out();
+ #endif
+ }
+
+ static inline void kmap_local_sched_in(void)
+ {
+ #ifdef CONFIG_KMAP_LOCAL
+ if (unlikely(current->kmap_ctrl.idx))
+ __kmap_local_sched_in();
+ #endif
+ }
+
/**
* prepare_task_switch - prepare to switch tasks
* @rq: the runqueue preparing to switch
perf_event_task_sched_out(prev, next);
rseq_preempt(prev);
fire_sched_out_preempt_notifiers(prev, next);
+ kmap_local_sched_out();
prepare_task(next);
prepare_arch_switch(next);
}
finish_lock_switch(rq);
finish_arch_post_lock_switch();
kcov_finish_switch(current);
+ /*
+ * kmap_local_sched_out() is invoked with rq::lock held and
+ * interrupts disabled. There is no requirement for that, but the
+ * sched out code does not have an interrupt enabled section.
+ * Restoring the maps on sched in does not require interrupts being
+ * disabled either.
+ */
+ kmap_local_sched_in();
fire_sched_in_preempt_notifiers(current);
/*
return rq;
}
- #ifdef CONFIG_SMP
-
- /* rq->lock is NOT held, but preemption is disabled */
- static void __balance_callback(struct rq *rq)
- {
- struct callback_head *head, *next;
- void (*func)(struct rq *rq);
- unsigned long flags;
-
- raw_spin_lock_irqsave(&rq->lock, flags);
- head = rq->balance_callback;
- rq->balance_callback = NULL;
- while (head) {
- func = (void (*)(struct rq *))head->func;
- next = head->next;
- head->next = NULL;
- head = next;
-
- func(rq);
- }
- raw_spin_unlock_irqrestore(&rq->lock, flags);
- }
-
- static inline void balance_callback(struct rq *rq)
- {
- if (unlikely(rq->balance_callback))
- __balance_callback(rq);
- }
-
- #else
-
- static inline void balance_callback(struct rq *rq)
- {
- }
-
- #endif
-
/**
* schedule_tail - first thing a freshly forked thread must call.
* @prev: the thread we just switched away from.
*/
rq = finish_task_switch(prev);
- balance_callback(rq);
preempt_enable();
if (current->set_child_tid)
}
/*
- * IO-wait accounting, and how its mostly bollocks (on SMP).
+ * IO-wait accounting, and how it's mostly bollocks (on SMP).
*
* The idea behind IO-wait account is to account the idle time that we could
* have spend running if it were not for IO. That is, if we were to improve the
int dest_cpu;
raw_spin_lock_irqsave(&p->pi_lock, flags);
- dest_cpu = p->sched_class->select_task_rq(p, task_cpu(p), SD_BALANCE_EXEC, 0);
+ dest_cpu = p->sched_class->select_task_rq(p, task_cpu(p), WF_EXEC);
if (dest_cpu == smp_processor_id())
goto unlock;
preempt_count_set(PREEMPT_DISABLED);
}
rcu_sleep_check();
+ SCHED_WARN_ON(ct_state() == CONTEXT_USER);
profile_hit(SCHED_PROFILING, __builtin_return_address(0));
/*
* Optimization: we know that if all tasks are in the fair class we can
* call that function directly, but only if the @prev task wasn't of a
- * higher scheduling class, because otherwise those loose the
+ * higher scheduling class, because otherwise those lose the
* opportunity to pull in more work from other CPUs.
*/
if (likely(prev->sched_class <= &fair_sched_class &&
*/
++*switch_count;
+ migrate_disable_switch(rq, prev);
psi_sched_switch(prev, next, !task_on_rq_queued(prev));
trace_sched_switch(preempt, prev, next);
rq = context_switch(rq, prev, next, &rf);
} else {
rq->clock_update_flags &= ~(RQCF_ACT_SKIP|RQCF_REQ_SKIP);
- rq_unlock_irq(rq, &rf);
- }
- balance_callback(rq);
+ rq_unpin_lock(rq, &rf);
+ __balance_callbacks(rq);
+ raw_spin_unlock_irq(&rq->lock);
+ }
}
void __noreturn do_task_dead(void)
} while (need_resched());
}
- #ifdef CONFIG_CONTEXT_TRACKING
+ #if defined(CONFIG_CONTEXT_TRACKING) && !defined(CONFIG_HAVE_CONTEXT_TRACKING_OFFSTACK)
asmlinkage __visible void __sched schedule_user(void)
{
/*
* right. rt_mutex_slowunlock()+rt_mutex_postunlock() work together to
* ensure a task is de-boosted (pi_task is set to NULL) before the
* task is allowed to run again (and can exit). This ensures the pointer
- * points to a blocked task -- which guaratees the task is present.
+ * points to a blocked task -- which guarantees the task is present.
*/
p->pi_top_task = pi_task;
out_unlock:
/* Avoid rq from going away on us: */
preempt_disable();
- __task_rq_unlock(rq, &rf);
- balance_callback(rq);
+ rq_unpin_lock(rq, &rf);
+ __balance_callbacks(rq);
+ raw_spin_unlock(&rq->lock);
+
preempt_enable();
}
#else
/*
* The RT priorities are set via sched_setscheduler(), but we still
* allow the 'normal' nice value to be set - but as expected
- * it wont have any effect on scheduling until the task is
+ * it won't have any effect on scheduling until the task is
* SCHED_DEADLINE, SCHED_FIFO or SCHED_RR:
*/
if (task_has_dl_policy(p) || task_has_rt_policy(p)) {
int retval, oldprio, oldpolicy = -1, queued, running;
int new_effective_prio, policy = attr->sched_policy;
const struct sched_class *prev_class;
+ struct callback_head *head;
struct rq_flags rf;
int reset_on_fork;
int queue_flags = DEQUEUE_SAVE | DEQUEUE_MOVE | DEQUEUE_NOCLOCK;
/* Avoid rq from going away on us: */
preempt_disable();
+ head = splice_balance_callbacks(rq);
task_rq_unlock(rq, p, &rf);
if (pi) {
}
/* Run balance callbacks after we've adjusted the PI chain: */
- balance_callback(rq);
+ balance_callbacks(rq, head);
preempt_enable();
return 0;
}
#endif
again:
- retval = __set_cpus_allowed_ptr(p, new_mask, true);
+ retval = __set_cpus_allowed_ptr(p, new_mask, SCA_CHECK);
if (!retval) {
cpuset_cpus_allowed(p, cpus_allowed);
return ret;
}
- /**
- * sys_sched_yield - yield the current processor to other threads.
- *
- * This function yields the current CPU to other tasks. If there are no
- * other threads running on this CPU then this function will return.
- *
- * Return: 0.
- */
static void do_sched_yield(void)
{
struct rq_flags rf;
schedstat_inc(rq->yld_count);
current->sched_class->yield_task(rq);
- /*
- * Since we are going to call schedule() anyway, there's
- * no need to preempt or enable interrupts:
- */
preempt_disable();
- rq_unlock(rq, &rf);
+ rq_unlock_irq(rq, &rf);
sched_preempt_enable_no_resched();
schedule();
}
+ /**
+ * sys_sched_yield - yield the current processor to other threads.
+ *
+ * This function yields the current CPU to other tasks. If there are no
+ * other threads running on this CPU then this function will return.
+ *
+ * Return: 0.
+ */
SYSCALL_DEFINE0(sched_yield)
{
do_sched_yield();
*
* The scheduler is at all times free to pick the calling task as the most
* eligible task to run, if removing the yield() call from your code breaks
- * it, its already broken.
+ * it, it's already broken.
*
* Typical broken usage is:
*
(unsigned long)task_thread_info(p)->flags);
print_worker_info(KERN_INFO, p);
+ print_stop_info(KERN_INFO, p);
show_stack(p, NULL, KERN_INFO);
put_task_stack(p);
}
#ifdef CONFIG_SMP
/*
- * Its possible that init_idle() gets called multiple times on a task,
+ * It's possible that init_idle() gets called multiple times on a task,
* in that case do_set_cpus_allowed() will not do the right thing.
*
* And since this is boot we can forgo the serialization.
*/
- set_cpus_allowed_common(idle, cpumask_of(cpu));
+ set_cpus_allowed_common(idle, cpumask_of(cpu), 0);
#endif
/*
* We're having a chicken and egg problem, even though we are
/* finish_cpu(), as ran on the BP, will clean up the active_mm state */
}
- /*
- * Since this CPU is going 'away' for a while, fold any nr_active delta
- * we might have. Assumes we're called after migrate_tasks() so that the
- * nr_active count is stable. We need to take the teardown thread which
- * is calling this into account, so we hand in adjust = 1 to the load
- * calculation.
- *
- * Also see the comment "Global load-average calculations".
- */
- static void calc_load_migrate(struct rq *rq)
+ static int __balance_push_cpu_stop(void *arg)
{
- long delta = calc_load_fold_active(rq, 1);
- if (delta)
- atomic_long_add(delta, &calc_load_tasks);
- }
+ struct task_struct *p = arg;
+ struct rq *rq = this_rq();
+ struct rq_flags rf;
+ int cpu;
- static struct task_struct *__pick_migrate_task(struct rq *rq)
- {
- const struct sched_class *class;
- struct task_struct *next;
+ raw_spin_lock_irq(&p->pi_lock);
+ rq_lock(rq, &rf);
- for_each_class(class) {
- next = class->pick_next_task(rq);
- if (next) {
- next->sched_class->put_prev_task(rq, next);
- return next;
- }
+ update_rq_clock(rq);
+
+ if (task_rq(p) == rq && task_on_rq_queued(p)) {
+ cpu = select_fallback_rq(rq->cpu, p);
+ rq = __migrate_task(rq, &rf, p, cpu);
}
- /* The idle class should always have a runnable task */
- BUG();
+ rq_unlock(rq, &rf);
+ raw_spin_unlock_irq(&p->pi_lock);
+
+ put_task_struct(p);
+
+ return 0;
}
+ static DEFINE_PER_CPU(struct cpu_stop_work, push_work);
+
/*
- * Migrate all tasks from the rq, sleeping tasks will be migrated by
- * try_to_wake_up()->select_task_rq().
- *
- * Called with rq->lock held even though we'er in stop_machine() and
- * there's no concurrency possible, we hold the required locks anyway
- * because of lock validation efforts.
+ * Ensure we only run per-cpu kthreads once the CPU goes !active.
*/
- static void migrate_tasks(struct rq *dead_rq, struct rq_flags *rf)
+ static void balance_push(struct rq *rq)
{
- struct rq *rq = dead_rq;
- struct task_struct *next, *stop = rq->stop;
- struct rq_flags orf = *rf;
- int dest_cpu;
+ struct task_struct *push_task = rq->curr;
+ lockdep_assert_held(&rq->lock);
+ SCHED_WARN_ON(rq->cpu != smp_processor_id());
/*
- * Fudge the rq selection such that the below task selection loop
- * doesn't get stuck on the currently eligible stop task.
- *
- * We're currently inside stop_machine() and the rq is either stuck
- * in the stop_machine_cpu_stop() loop, or we're executing this code,
- * either way we should never end up calling schedule() until we're
- * done here.
+ * Ensure the thing is persistent until balance_push_set(.on = false);
*/
- rq->stop = NULL;
+ rq->balance_callback = &balance_push_callback;
/*
- * put_prev_task() and pick_next_task() sched
- * class method both need to have an up-to-date
- * value of rq->clock[_task]
+ * Both the cpu-hotplug and stop task are in this case and are
+ * required to complete the hotplug process.
*/
- update_rq_clock(rq);
-
- for (;;) {
+ if (is_per_cpu_kthread(push_task) || is_migration_disabled(push_task)) {
/*
- * There's this thread running, bail when that's the only
- * remaining thread:
+ * If this is the idle task on the outgoing CPU try to wake
+ * up the hotplug control thread which might wait for the
+ * last task to vanish. The rcuwait_active() check is
+ * accurate here because the waiter is pinned on this CPU
+ * and can't obviously be running in parallel.
+ *
+ * On RT kernels this also has to check whether there are
+ * pinned and scheduled out tasks on the runqueue. They
+ * need to leave the migrate disabled section first.
*/
- if (rq->nr_running == 1)
- break;
+ if (!rq->nr_running && !rq_has_pinned_tasks(rq) &&
+ rcuwait_active(&rq->hotplug_wait)) {
+ raw_spin_unlock(&rq->lock);
+ rcuwait_wake_up(&rq->hotplug_wait);
+ raw_spin_lock(&rq->lock);
+ }
+ return;
+ }
- next = __pick_migrate_task(rq);
+ get_task_struct(push_task);
+ /*
+ * Temporarily drop rq->lock such that we can wake-up the stop task.
+ * Both preemption and IRQs are still disabled.
+ */
+ raw_spin_unlock(&rq->lock);
+ stop_one_cpu_nowait(rq->cpu, __balance_push_cpu_stop, push_task,
+ this_cpu_ptr(&push_work));
+ /*
+ * At this point need_resched() is true and we'll take the loop in
+ * schedule(). The next pick is obviously going to be the stop task
+ * which is_per_cpu_kthread() and will push this task away.
+ */
+ raw_spin_lock(&rq->lock);
+ }
- /*
- * Rules for changing task_struct::cpus_mask are holding
- * both pi_lock and rq->lock, such that holding either
- * stabilizes the mask.
- *
- * Drop rq->lock is not quite as disastrous as it usually is
- * because !cpu_active at this point, which means load-balance
- * will not interfere. Also, stop-machine.
- */
- rq_unlock(rq, rf);
- raw_spin_lock(&next->pi_lock);
- rq_relock(rq, rf);
+ static void balance_push_set(int cpu, bool on)
+ {
+ struct rq *rq = cpu_rq(cpu);
+ struct rq_flags rf;
- /*
- * Since we're inside stop-machine, _nothing_ should have
- * changed the task, WARN if weird stuff happened, because in
- * that case the above rq->lock drop is a fail too.
- */
- if (WARN_ON(task_rq(next) != rq || !task_on_rq_queued(next))) {
- raw_spin_unlock(&next->pi_lock);
- continue;
- }
+ rq_lock_irqsave(rq, &rf);
+ if (on)
+ rq->balance_callback = &balance_push_callback;
+ else
+ rq->balance_callback = NULL;
+ rq_unlock_irqrestore(rq, &rf);
+ }
- /* Find suitable destination for @next, with force if needed. */
- dest_cpu = select_fallback_rq(dead_rq->cpu, next);
- rq = __migrate_task(rq, rf, next, dest_cpu);
- if (rq != dead_rq) {
- rq_unlock(rq, rf);
- rq = dead_rq;
- *rf = orf;
- rq_relock(rq, rf);
- }
- raw_spin_unlock(&next->pi_lock);
- }
+ /*
+ * Invoked from a CPUs hotplug control thread after the CPU has been marked
+ * inactive. All tasks which are not per CPU kernel threads are either
+ * pushed off this CPU now via balance_push() or placed on a different CPU
+ * during wakeup. Wait until the CPU is quiescent.
+ */
+ static void balance_hotplug_wait(void)
+ {
+ struct rq *rq = this_rq();
+
+ rcuwait_wait_event(&rq->hotplug_wait,
+ rq->nr_running == 1 && !rq_has_pinned_tasks(rq),
+ TASK_UNINTERRUPTIBLE);
+ }
+
+ #else
- rq->stop = stop;
+ static inline void balance_push(struct rq *rq)
+ {
+ }
+
+ static inline void balance_push_set(int cpu, bool on)
+ {
}
+
+ static inline void balance_hotplug_wait(void)
+ {
+ }
+
#endif /* CONFIG_HOTPLUG_CPU */
void set_rq_online(struct rq *rq)
struct rq *rq = cpu_rq(cpu);
struct rq_flags rf;
+ balance_push_set(cpu, false);
+
#ifdef CONFIG_SCHED_SMT
/*
* When going up, increment the number of cores with SMT present.
int sched_cpu_deactivate(unsigned int cpu)
{
+ struct rq *rq = cpu_rq(cpu);
+ struct rq_flags rf;
int ret;
set_cpu_active(cpu, false);
*/
synchronize_rcu();
+ balance_push_set(cpu, true);
+
+ rq_lock_irqsave(rq, &rf);
+ if (rq->rd) {
+ update_rq_clock(rq);
+ BUG_ON(!cpumask_test_cpu(cpu, rq->rd->span));
+ set_rq_offline(rq);
+ }
+ rq_unlock_irqrestore(rq, &rf);
+
#ifdef CONFIG_SCHED_SMT
/*
* When going down, decrement the number of cores with SMT present.
ret = cpuset_cpu_inactive(cpu);
if (ret) {
+ balance_push_set(cpu, false);
set_cpu_active(cpu, true);
return ret;
}
}
#ifdef CONFIG_HOTPLUG_CPU
+
+ /*
+ * Invoked immediately before the stopper thread is invoked to bring the
+ * CPU down completely. At this point all per CPU kthreads except the
+ * hotplug thread (current) and the stopper thread (inactive) have been
+ * either parked or have been unbound from the outgoing CPU. Ensure that
+ * any of those which might be on the way out are gone.
+ *
+ * If after this point a bound task is being woken on this CPU then the
+ * responsible hotplug callback has failed to do it's job.
+ * sched_cpu_dying() will catch it with the appropriate fireworks.
+ */
+ int sched_cpu_wait_empty(unsigned int cpu)
+ {
+ balance_hotplug_wait();
+ return 0;
+ }
+
+ /*
+ * Since this CPU is going 'away' for a while, fold any nr_active delta we
+ * might have. Called from the CPU stopper task after ensuring that the
+ * stopper is the last running task on the CPU, so nr_active count is
+ * stable. We need to take the teardown thread which is calling this into
+ * account, so we hand in adjust = 1 to the load calculation.
+ *
+ * Also see the comment "Global load-average calculations".
+ */
+ static void calc_load_migrate(struct rq *rq)
+ {
+ long delta = calc_load_fold_active(rq, 1);
+
+ if (delta)
+ atomic_long_add(delta, &calc_load_tasks);
+ }
+
int sched_cpu_dying(unsigned int cpu)
{
struct rq *rq = cpu_rq(cpu);
sched_tick_stop(cpu);
rq_lock_irqsave(rq, &rf);
- if (rq->rd) {
- BUG_ON(!cpumask_test_cpu(cpu, rq->rd->span));
- set_rq_offline(rq);
- }
- migrate_tasks(rq, &rf);
- BUG_ON(rq->nr_running != 1);
+ BUG_ON(rq->nr_running != 1 || rq_has_pinned_tasks(rq));
rq_unlock_irqrestore(rq, &rf);
calc_load_migrate(rq);
rq->last_blocked_load_update_tick = jiffies;
atomic_set(&rq->nohz_flags, 0);
- rq_csd_init(rq, &rq->nohz_csd, nohz_csd_func);
+ INIT_CSD(&rq->nohz_csd, nohz_csd_func, rq);
+ #endif
+ #ifdef CONFIG_HOTPLUG_CPU
+ rcuwait_init(&rq->hotplug_wait);
#endif
#endif /* CONFIG_SMP */
hrtick_rq_init(rq);
add_taint(TAINT_WARN, LOCKDEP_STILL_OK);
}
EXPORT_SYMBOL_GPL(__cant_sleep);
+
+ #ifdef CONFIG_SMP
+ void __cant_migrate(const char *file, int line)
+ {
+ static unsigned long prev_jiffy;
+
+ if (irqs_disabled())
+ return;
+
+ if (is_migration_disabled(current))
+ return;
+
+ if (!IS_ENABLED(CONFIG_PREEMPT_COUNT))
+ return;
+
+ if (preempt_count() > 0)
+ return;
+
+ if (time_before(jiffies, prev_jiffy + HZ) && prev_jiffy)
+ return;
+ prev_jiffy = jiffies;
+
+ pr_err("BUG: assuming non migratable context at %s:%d\n", file, line);
+ pr_err("in_atomic(): %d, irqs_disabled(): %d, migration_disabled() %u pid: %d, name: %s\n",
+ in_atomic(), irqs_disabled(), is_migration_disabled(current),
+ current->pid, current->comm);
+
+ debug_show_held_locks(current);
+ dump_stack();
+ add_taint(TAINT_WARN, LOCKDEP_STILL_OK);
+ }
+ EXPORT_SYMBOL_GPL(__cant_migrate);
+ #endif
#endif
#ifdef CONFIG_MAGIC_SYSRQ
return -EINVAL;
#endif
/*
- * Serialize against wake_up_new_task() such that if its
+ * Serialize against wake_up_new_task() such that if it's
* running, we're sure to observe its full state.
*/
raw_spin_lock_irq(&task->pi_lock);
projects / linux.git / commitdiff