task_work: Use TIF_NOTIFY_SIGNAL if available

If the arch supports TIF_NOTIFY_SIGNAL, then use that for TWA_SIGNAL as
it's more efficient than using the signal delivery method. This is
especially true on threaded applications, where ->sighand is shared across
threads, but it's also lighter weight on non-shared cases.

io_uring is a heavy consumer of TWA_SIGNAL based task_work. A test with
threads shows a nice improvement running an io_uring based echo server.

stock kernel:
0.01% <= 0.1 milliseconds
95.86% <= 0.2 milliseconds
98.27% <= 0.3 milliseconds
99.71% <= 0.4 milliseconds
100.00% <= 0.5 milliseconds
100.00% <= 0.6 milliseconds
100.00% <= 0.7 milliseconds
100.00% <= 0.8 milliseconds
100.00% <= 0.9 milliseconds
100.00% <= 1.0 milliseconds
100.00% <= 1.1 milliseconds
100.00% <= 2 milliseconds
100.00% <= 3 milliseconds
100.00% <= 3 milliseconds
1378930.00 requests per second
~1600% CPU

1.38M requests/second, and all 16 CPUs are maxed out.

patched kernel:
0.01% <= 0.1 milliseconds
98.24% <= 0.2 milliseconds
99.47% <= 0.3 milliseconds
99.99% <= 0.4 milliseconds
100.00% <= 0.5 milliseconds
100.00% <= 0.6 milliseconds
100.00% <= 0.7 milliseconds
100.00% <= 0.8 milliseconds
100.00% <= 0.9 milliseconds
100.00% <= 1.2 milliseconds
1666111.38 requests per second
~1450% CPU

1.67M requests/second, and we're no longer just hammering on the sighand
lock. The original reporter states:

"For 5.7.15 my benchmark achieves 1.6M qps and system cpu is at ~80%.
 for 5.7.16 or later it achieves only 1M qps and the system cpu is is
 at ~100%"

with the only difference there being that TWA_SIGNAL is used
unconditionally in 5.7.16, since it's required to be able to handle the
inability to run task_work if the application is waiting in the kernel
already on an event that needs task_work run to be satisfied. Also see
commit 0ba9c9edcd15.

Reported-by: Roman Gershman <[email protected]>
Signed-off-by: Jens Axboe <[email protected]>
Signed-off-by: Thomas Gleixner <[email protected]>
Reviewed-by: Oleg Nesterov <[email protected]>
Link: https://lore.kernel.org/r/[email protected]

diff --git a/kernel/task_work.c b/kernel/task_work.c
index 613b2d634af82fb6c3d4505df7159c9396e10e89..ae058893913ce57aa0d23d8a6fa84e8c75cc56a0 100644 (file)
--- a/kernel/task_work.c
+++ b/kernel/task_work.c
@@ -5,6 +5,34 @@
 
 static struct callback_head work_exited; /* all we need is ->next == NULL */
 
+/*
+ * TWA_SIGNAL signaling - use TIF_NOTIFY_SIGNAL, if available, as it's faster
+ * than TIF_SIGPENDING as there's no dependency on ->sighand. The latter is
+ * shared for threads, and can cause contention on sighand->lock. Even for
+ * the non-threaded case TIF_NOTIFY_SIGNAL is more efficient, as no locking
+ * or IRQ disabling is involved for notification (or running) purposes.
+ */
+static void task_work_notify_signal(struct task_struct *task)
+{
+#if defined(TIF_NOTIFY_SIGNAL)
+       set_notify_signal(task);
+#else
+       unsigned long flags;
+
+       /*
+        * Only grab the sighand lock if we don't already have some
+        * task_work pending. This pairs with the smp_store_mb()
+        * in get_signal(), see comment there.
+        */
+       if (!(READ_ONCE(task->jobctl) & JOBCTL_TASK_WORK) &&
+           lock_task_sighand(task, &flags)) {
+               task->jobctl |= JOBCTL_TASK_WORK;
+               signal_wake_up(task, 0);
+               unlock_task_sighand(task, &flags);
+       }
+#endif
+}
+
 /**
  * task_work_add - ask the @task to execute @work->func()
  * @task: the task which should run the callback
@@ -28,7 +56,6 @@ int
 task_work_add(struct task_struct *task, struct callback_head *work, int notify)
 {
        struct callback_head *head;
-       unsigned long flags;
 
        do {
                head = READ_ONCE(task->task_works);
@@ -42,17 +69,7 @@ task_work_add(struct task_struct *task, struct callback_head *work, int notify)
                set_notify_resume(task);
                break;
        case TWA_SIGNAL:
-               /*
-                * Only grab the sighand lock if we don't already have some
-                * task_work pending. This pairs with the smp_store_mb()
-                * in get_signal(), see comment there.
-                */
-               if (!(READ_ONCE(task->jobctl) & JOBCTL_TASK_WORK) &&
-                   lock_task_sighand(task, &flags)) {
-                       task->jobctl |= JOBCTL_TASK_WORK;
-                       signal_wake_up(task, 0);
-                       unlock_task_sighand(task, &flags);
-               }
+               task_work_notify_signal(task);
                break;
        }