mm: introduce get_user_pages_fast
Introduce a new get_user_pages_fast mm API, which is basically a
get_user_pages with a less general API (but still tends to be suited to
the common case):
- task and mm are always current and current->mm
- force is always 0
- pages is always non-NULL
- don't pass back vmas
This restricted API can be implemented in a much more scalable way on many
architectures when the ptes are present, by walking the page tables
locklessly (no mmap_sem or page table locks). When the ptes are not
populated, get_user_pages_fast() could be slower.
This is implemented locklessly on x86, and used in some key direct IO call
sites, in later patches, which provides nearly 10% performance improvement
on a threaded database workload.
Lots of other code could use this too, depending on use cases (eg. grep
drivers/). And it might inspire some new and clever ways to use it.
[
[email protected]: build fix]
[
[email protected]: coding-style fixes]
Signed-off-by: Nick Piggin <[email protected]>
Cc: Dave Kleikamp <[email protected]>
Cc: Andy Whitcroft <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: Thomas Gleixner <[email protected]>
Cc: Andi Kleen <[email protected]>
Cc: Dave Kleikamp <[email protected]>
Cc: Badari Pulavarty <[email protected]>
Cc: Zach Brown <[email protected]>
Cc: Jens Axboe <[email protected]>
Reviewed-by: Peter Zijlstra <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>