percpu-refcount: fix usage of this_cpu_ops

The percpu-refcount infrastructure uses the underscore variants of
this_cpu_ops in order to modify percpu reference counters.
(e.g. __this_cpu_inc()).

However the underscore variants do not atomically update the percpu
variable, instead they may be implemented using read-modify-write
semantics (more than one instruction).  Therefore it is only safe to
use the underscore variant if the context is always the same (process,
softirq, or hardirq). Otherwise it is possible to lose updates.

This problem is something that Sebastian has seen within the aio
subsystem which uses percpu refcounters both in process and softirq
context leading to reference counts that never dropped to zeroes; even
though the number of "get" and "put" calls matched.

Fix this by using the non-underscore this_cpu_ops variant which
provides correct per cpu atomic semantics and fixes the corrupted
reference counts.

Cc: Kent Overstreet <[email protected]>
Cc: <[email protected]> # v3.11+
Reported-by: Sebastian Ott <[email protected]>
Signed-off-by: Heiko Carstens <[email protected]>
Signed-off-by: Tejun Heo <[email protected]>
References: http://lkml.kernel.org/g/alpine.LFD.2.11.1406041540520.21183@denkbrett

diff --git a/include/linux/percpu-refcount.h b/include/linux/percpu-refcount.h
index 95961f0bf62d4212002a63efb29f28dfe5e6ab1b..0afb48fd449d3f952afdfd64c170715cb90f6a08 100644 (file)
--- a/include/linux/percpu-refcount.h
+++ b/include/linux/percpu-refcount.h
@@ -110,7 +110,7 @@ static inline void percpu_ref_get(struct percpu_ref *ref)
        pcpu_count = ACCESS_ONCE(ref->pcpu_count);
 
        if (likely(REF_STATUS(pcpu_count) == PCPU_REF_PTR))
-               __this_cpu_inc(*pcpu_count);
+               this_cpu_inc(*pcpu_count);
        else
                atomic_inc(&ref->count);
 
@@ -139,7 +139,7 @@ static inline bool percpu_ref_tryget(struct percpu_ref *ref)
        pcpu_count = ACCESS_ONCE(ref->pcpu_count);
 
        if (likely(REF_STATUS(pcpu_count) == PCPU_REF_PTR)) {
-               __this_cpu_inc(*pcpu_count);
+               this_cpu_inc(*pcpu_count);
                ret = true;
        }
 
@@ -164,7 +164,7 @@ static inline void percpu_ref_put(struct percpu_ref *ref)
        pcpu_count = ACCESS_ONCE(ref->pcpu_count);
 
        if (likely(REF_STATUS(pcpu_count) == PCPU_REF_PTR))
-               __this_cpu_dec(*pcpu_count);
+               this_cpu_dec(*pcpu_count);
        else if (unlikely(atomic_dec_and_test(&ref->count)))
                ref->release(ref);