memcg/hugetlb: add hugeTLB counters to memcg

This patch introduces a new counter to memory.stat that tracks hugeTLB
usage, only if hugeTLB accounting is done to memory.current.  This feature
is enabled the same way hugeTLB accounting is enabled, via the
memory_hugetlb_accounting mount flag for cgroupsv2.

1. Why is this patch necessary?
Currently, memcg hugeTLB accounting is an opt-in feature [1] that adds
hugeTLB usage to memory.current.  However, the metric is not reported in
memory.stat.  Given that users often interpret memory.stat as a breakdown
of the value reported in memory.current, the disparity between the two
reports can be confusing.  This patch solves this problem by including the
metric in memory.stat as well, but only if it is also reported in
memory.current (it would also be confusing if the value was reported in
memory.stat, but not in memory.current)

Aside from the consistency between the two files, we also see benefits in
observability.  Userspace might be interested in the hugeTLB footprint of
cgroups for many reasons.  For instance, system admins might want to
verify that hugeTLB usage is distributed as expected across tasks: i.e.
memory-intensive tasks are using more hugeTLB pages than tasks that don't
consume a lot of memory, or are seen to fault frequently.  Note that this
is separate from wanting to inspect the distribution for limiting purposes
(in which case, hugeTLB controller makes more sense).

2. We already have a hugeTLB controller. Why not use that?
It is true that hugeTLB tracks the exact value that we want.  In fact, by
enabling the hugeTLB controller, we get all of the observability benefits
that I mentioned above, and users can check the total hugeTLB usage,
verify if it is distributed as expected, etc.

With this said, there are 2 problems:
(a) They are still not reported in memory.stat, which means the
    disparity between the memcg reports are still there.
(b) We cannot reasonably expect users to enable the hugeTLB controller
    just for the sake of hugeTLB usage reporting, especially since
    they don't have any use for hugeTLB usage enforcing [2].

3. Implementation Details:
In the alloc / free hugetlb functions, we call lruvec_stat_mod_folio
regardless of whether memcg accounts hugetlb.  mem_cgroup_commit_charge
which is called from alloc_hugetlb_folio will set memcg for the folio only
if the CGRP_ROOT_MEMORY_HUGETLB_ACCOUNTING cgroup mount option is used, so
lruvec_stat_mod_folio accounts per-memcg hugetlb counters only if the
feature is enabled.  Regardless of whether memcg accounts for hugetlb, the
newly added global counter is updated and shown in /proc/vmstat.

The global counter is added because vmstats is the preferred framework for
cgroup stats.  It makes stat items consistent between global and cgroups.
It also provides a per-node breakdown, which is useful.  Because it does
not use cgroup-specific hooks, we also keep generic MM code separate from
memcg code.

[1] https://lore.kernel.org/all/20231006184629[email protected]/
[2] Of course, we can't make a new patch for every feature that can be
    duplicated. However, since the existing solution of enabling the
    hugeTLB controller is an imperfect solution that still leaves a
    discrepancy between memory.stat and memory.curent, I think that it
    is reasonable to isolate the feature in this case.

Link: https://lkml.kernel.org/r/[email protected]
Signed-off-by: Joshua Hahn <[email protected]>
Suggested-by: Nhat Pham <[email protected]>
Suggested-by: Shakeel Butt <[email protected]>
Suggested-by: Johannes Weiner <[email protected]>
Acked-by: Shakeel Butt <[email protected]>
Acked-by: Johannes Weiner <[email protected]>
Acked-by: Chris Down <[email protected]>
Acked-by: Michal Hocko <[email protected]>
Reviewed-by: Roman Gushchin <[email protected]>
Reviewed-by: Nhat Pham <[email protected]>
Cc: Jonathan Corbet <[email protected]>
Cc: Michal Koutný <[email protected]>
Cc: Muchun Song <[email protected]>
Cc: Zefan Li <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>

diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst
index 6d02168d78bed67bfe2c7aa17f5a26f61346138f..02dc54fe1f53047a1b49ab6600cb410d7007a278 100644 (file)
--- a/Documentation/admin-guide/cgroup-v2.rst
+++ b/Documentation/admin-guide/cgroup-v2.rst
@@ -1655,6 +1655,11 @@ The following nested keys are defined.
          pgdemote_khugepaged
                Number of pages demoted by khugepaged.
 
+         hugetlb
+               Amount of memory used by hugetlb pages. This metric only shows
+               up if hugetlb usage is accounted for in memory.current (i.e.
+               cgroup is mounted with the memory_hugetlb_accounting option).
+
   memory.numa_stat
        A read-only nested-keyed file which exists on non-root cgroups.
 

diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 5e8f567753bdd18922dbd8f143ad3f8f836994fc..b36124145a16f2778b0466a1a7fa73557f79f2ad 100644 (file)
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -220,6 +220,9 @@ enum node_stat_item {
        PGDEMOTE_KSWAPD,
        PGDEMOTE_DIRECT,
        PGDEMOTE_KHUGEPAGED,
+#ifdef CONFIG_HUGETLB_PAGE
+       NR_HUGETLB,
+#endif
        NR_VM_NODE_STAT_ITEMS
 };
 

diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 2c8c5da0f5d324c511b695ab685ff4986bb889b3..ea2ed8e301ef2c22dec0b0a32f9a1ff747a7c3b4 100644 (file)
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -1925,6 +1925,7 @@ void free_huge_folio(struct folio *folio)
                                     pages_per_huge_page(h), folio);
        hugetlb_cgroup_uncharge_folio_rsvd(hstate_index(h),
                                          pages_per_huge_page(h), folio);
+       lruvec_stat_mod_folio(folio, NR_HUGETLB, -pages_per_huge_page(h));
        mem_cgroup_uncharge(folio);
        if (restore_reserve)
                h->resv_huge_pages++;
@@ -3093,6 +3094,7 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
 
        if (!memcg_charge_ret)
                mem_cgroup_commit_charge(folio, memcg);
+       lruvec_stat_mod_folio(folio, NR_HUGETLB, pages_per_huge_page(h));
        mem_cgroup_put(memcg);
 
        return folio;

diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index ed2dd88437cab9aefd5ffa68bfd5cc8dee79c09d..7b3503d12aaf1af121b6a65d792ddfface164fae 100644 (file)
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -315,6 +315,9 @@ static const unsigned int memcg_node_stat_items[] = {
        PGDEMOTE_KSWAPD,
        PGDEMOTE_DIRECT,
        PGDEMOTE_KHUGEPAGED,
+#ifdef CONFIG_HUGETLB_PAGE
+       NR_HUGETLB,
+#endif
 };
 
 static const unsigned int memcg_stat_items[] = {
@@ -1366,6 +1369,9 @@ static const struct memory_stat memory_stats[] = {
        { "unevictable",                NR_UNEVICTABLE                  },
        { "slab_reclaimable",           NR_SLAB_RECLAIMABLE_B           },
        { "slab_unreclaimable",         NR_SLAB_UNRECLAIMABLE_B         },
+#ifdef CONFIG_HUGETLB_PAGE
+       { "hugetlb",                    NR_HUGETLB                      },
+#endif
 
        /* The memory events */
        { "workingset_refault_anon",    WORKINGSET_REFAULT_ANON         },
@@ -1461,6 +1467,11 @@ static void memcg_stat_format(struct mem_cgroup *memcg, struct seq_buf *s)
        for (i = 0; i < ARRAY_SIZE(memory_stats); i++) {
                u64 size;
 
+#ifdef CONFIG_HUGETLB_PAGE
+               if (unlikely(memory_stats[i].idx == NR_HUGETLB) &&
+                   !(cgrp_dfl_root.flags & CGRP_ROOT_MEMORY_HUGETLB_ACCOUNTING))
+                       continue;
+#endif
                size = memcg_page_state_output(memcg, memory_stats[i].idx);
                seq_buf_printf(s, "%s %llu\n", memory_stats[i].name, size);
 

diff --git a/mm/vmstat.c b/mm/vmstat.c
index 11a37c528395ea03d25afc76a50f602a3bdd5849..4d016314a56c95e6247f8970511afb364e686278 100644 (file)
--- a/mm/vmstat.c
+++ b/mm/vmstat.c
@@ -1273,6 +1273,9 @@ const char * const vmstat_text[] = {
        "pgdemote_kswapd",
        "pgdemote_direct",
        "pgdemote_khugepaged",
+#ifdef CONFIG_HUGETLB_PAGE
+       "nr_hugetlb",
+#endif
        /* system-wide enum vm_stat_item counters */
        "nr_dirty_threshold",
        "nr_dirty_background_threshold",