sched/idle/x86: Optimize unnecessary mwait_idle() resched IPIs

To fully take advantage of MWAIT, apparently the CLFLUSH instruction needs
another quirk on certain CPUs: proper barriers around it on certain machines.

On a Q6600 SMP system, pipe-test scheduling performance, cross core,
improves significantly:

  3.8.13                   487.2 KHz    1.000
  3.13.0-master            415.5 KHz     .852
  3.13.0-master+           415.2 KHz     .852     + restore mwait_idle
  3.13.0-master++          488.5 KHz    1.002     + restore mwait_idle + IPI fix

Since X86_BUG_CLFLUSH_MONITOR is already a quirk, don't create a separate
quirk for the extra smp_mb()s.

Signed-off-by: Mike Galbraith <[email protected]>
Cc: <[email protected]> # 3.10+
Cc: Borislav Petkov <[email protected]>
Cc: H. Peter Anvin <[email protected]>
Cc: Ian Malone <[email protected]>
Cc: Josh Boyer <[email protected]>
Cc: Len Brown <[email protected]>
Cc: Len Brown <[email protected]>
Cc: Linus Torvalds <[email protected]>
Cc: Mike Galbraith <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Thomas Gleixner <[email protected]>
Link: http://lkml.kernel.org/r/[email protected]
[ Ported to recent kernel, added comments about the quirk. ]
Signed-off-by: Ingo Molnar <[email protected]>

diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c
index da06f741d2a61008af15a8bb395833086b8df1ae..6ad8a6396b75f35d8e4a8dbd58f3a972aca4b5a9 100644 (file)
--- a/arch/x86/kernel/process.c
+++ b/arch/x86/kernel/process.c
@@ -428,18 +428,22 @@ static int prefer_mwait_c1_over_halt(const struct cpuinfo_x86 *c)
 
 static void mwait_idle(void)
 {
-       if (!need_resched()) {
-               if (this_cpu_has(X86_BUG_CLFLUSH_MONITOR))
+       if (!current_set_polling_and_test()) {
+               if (this_cpu_has(X86_BUG_CLFLUSH_MONITOR)) {
+                       smp_mb(); /* quirk */
                        clflush((void *)&current_thread_info()->flags);
+                       smp_mb(); /* quirk */
+               }
 
                __monitor((void *)&current_thread_info()->flags, 0, 0);
-               smp_mb();
                if (!need_resched())
                        __sti_mwait(0, 0);
                else
                        local_irq_enable();
-       } else
+       } else {
                local_irq_enable();
+       }
+       __current_clr_polling();
 }
 
 void select_idle_routine(const struct cpuinfo_x86 *c)