mmu_notifiers: call invalidate_range() when invalidating TLBs

mmu_notifiers: call invalidate_range() when invalidating TLBs

The invalidate_range() is going to become an architecture specific mmu
notifier used to keep the TLB of secondary MMUs such as an IOMMU in sync
with the CPU page tables.  Currently it is called from separate code paths
to the main CPU TLB invalidations.  This can lead to a secondary TLB not
getting invalidated when required and makes it hard to reason about when
exactly the secondary TLB is invalidated.

To fix this move the notifier call to the architecture specific TLB
maintenance functions for architectures that have secondary MMUs requiring
explicit software invalidations.

This fixes a SMMU bug on ARM64.  On ARM64 PTE permission upgrades require
a TLB invalidation.  This invalidation is done by the architecture
specific ptep_set_access_flags() which calls flush_tlb_page() if required.
However this doesn't call the notifier resulting in infinite faults being
generated by devices using the SMMU if it has previously cached a
read-only PTE in it's TLB.

Moving the invalidations into the TLB invalidation functions ensures all
invalidations happen at the same time as the CPU invalidation.  The
architecture specific flush_tlb_all() routines do not call the notifier as
none of the IOMMUs require this.

Link: https://lkml.kernel.org/r/0287ae32d91393a582897d6c4db6f7456b1001f2.1690292440.git-series.apopple@nvidia.com
Signed-off-by: Alistair Popple <[email protected]>
Suggested-by: Jason Gunthorpe <[email protected]>
Tested-by: SeongJae Park <[email protected]>
Acked-by: Catalin Marinas <[email protected]>
Reviewed-by: Jason Gunthorpe <[email protected]>
Tested-by: Luis Chamberlain <[email protected]>
Cc: Andrew Donnellan <[email protected]>
Cc: Chaitanya Kumar Borah <[email protected]>
Cc: Frederic Barrat <[email protected]>
Cc: John Hubbard <[email protected]>
Cc: Kevin Tian <[email protected]>
Cc: Michael Ellerman <[email protected]>
Cc: Nicholas Piggin <[email protected]>
Cc: Nicolin Chen <[email protected]>
Cc: Robin Murphy <[email protected]>
Cc: Sean Christopherson <[email protected]>
Cc: Tvrtko Ursulin <[email protected]>
Cc: Will Deacon <[email protected]>
Cc: Zhi Wang <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>