mm: vmscan: avoid split during shrink_folio_list()

Now that swap supports storing all mTHP sizes, avoid splitting large
folios before swap-out.  This benefits performance of the swap-out path by
eliding split_folio_to_list(), which is expensive, and also sets us up for
swapping in large folios in a future series.

If the folio is partially mapped, we continue to split it since we want to
avoid the extra IO overhead and storage of writing out pages
uneccessarily.

THP_SWPOUT and THP_SWPOUT_FALLBACK counters should continue to count
events only for PMD-mappable folios to avoid user confusion.  THP_SWPOUT
already has the appropriate guard.  Add a guard for THP_SWPOUT_FALLBACK.
It may be appropriate to add per-size counters in future.

Link: https://lkml.kernel.org/r/[email protected]
Signed-off-by: Ryan Roberts <[email protected]>
Reviewed-by: David Hildenbrand <[email protected]>
Reviewed-by: Barry Song <[email protected]>
Cc: Barry Song <[email protected]>
Cc: Chris Li <[email protected]>
Cc: Gao Xiang <[email protected]>
Cc: "Huang, Ying" <[email protected]>
Cc: Kefeng Wang <[email protected]>
Cc: Lance Yang <[email protected]>
Cc: Matthew Wilcox (Oracle) <[email protected]>
Cc: Michal Hocko <[email protected]>
Cc: Yang Shi <[email protected]>
Cc: Yu Zhao <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 7554adc9c824ed71ea65b05db29f8c71b743adff..da93213af981e99dc58aa8fcc2b2d91e8d1db2fd 100644 (file)
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -1206,25 +1206,25 @@ retry:
                                        if (!can_split_folio(folio, NULL))
                                                goto activate_locked;
                                        /*
-                                        * Split folios without a PMD map right
-                                        * away. Chances are some or all of the
-                                        * tail pages can be freed without IO.
+                                        * Split partially mapped folios right away.
+                                        * We can free the unmapped pages without IO.
                                         */
-                                       if (!folio_entire_mapcount(folio) &&
-                                           split_folio_to_list(folio,
-                                                               folio_list))
+                                       if (data_race(!list_empty(&folio->_deferred_list)) &&
+                                           split_folio_to_list(folio, folio_list))
                                                goto activate_locked;
                                }
                                if (!add_to_swap(folio)) {
                                        if (!folio_test_large(folio))
                                                goto activate_locked_split;
                                        /* Fallback to swap normal pages */
-                                       if (split_folio_to_list(folio,
-                                                               folio_list))
+                                       if (split_folio_to_list(folio, folio_list))
                                                goto activate_locked;
 #ifdef CONFIG_TRANSPARENT_HUGEPAGE
-                                       count_memcg_folio_events(folio, THP_SWPOUT_FALLBACK, 1);
-                                       count_vm_event(THP_SWPOUT_FALLBACK);
+                                       if (nr_pages >= HPAGE_PMD_NR) {
+                                               count_memcg_folio_events(folio,
+                                                       THP_SWPOUT_FALLBACK, 1);
+                                               count_vm_event(THP_SWPOUT_FALLBACK);
+                                       }
 #endif
                                        if (!add_to_swap(folio))
                                                goto activate_locked_split;