kevin/linux
1
Fork 0
Code

riscv: Disable preemption while handling PR_RISCV_CTX_SW_FENCEI_OFF

Browse Source 
The icache will be flushed in switch_to() if force_icache_flush is true,
or in flush_icache_deferred() if icache_stale_mask is set. Between
setting force_icache_flush to false and calculating the new
icache_stale_mask, preemption needs to be disabled. There are two
reasons for this:

1. If CPU migration happens between force_icache_flush = false, and the
   icache_stale_mask is set, an icache flush will not be emitted.
2. smp_processor_id() is used in set_icache_stale_mask() to mark the
   current CPU as not needing another flush since a flush will have
   happened either by userspace or by the kernel when performing the
   migration. smp_processor_id() is currently called twice with preemption
   enabled which causes a race condition. It allows
   icache_stale_mask to be populated with inconsistent CPU ids.

Resolve these two issues by setting the icache_stale_mask before setting
force_icache_flush to false, and using get_cpu()/put_cpu() to obtain the
smp_processor_id().

Signed-off-by: Charlie Jenkins <charlie@rivosinc.com>
Fixes: 6b9391b581 ("riscv: Include riscv_set_icache_flush_ctx prctl")
Link: https://lore.kernel.org/r/20240903-fix_fencei_optimization-v2-1-8025f20171fc@rivosinc.com
Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
This commit is contained in:
Charlie Jenkins 2024-09-03 15:52:34 -07:00 committed by Palmer Dabbelt
parent 2840dadf0d
commit 7c1e5b9690
No known key found for this signature in database
GPG Key ID: 2E1319F35FBB1889
1 changed files with 6 additions and 6 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

12
arch/riscv/mm/cacheflush.c
View File

@ -158,6 +158,7 @@ void __init riscv_init_cbo_blocksizes(void)
#ifdef CONFIG_SMP
static void set_icache_stale_mask(void)
{
int cpu = get_cpu();
cpumask_t *mask;
bool stale_cpu;
@ -168,10 +169,11 @@ static void set_icache_stale_mask(void)
* concurrently on different harts.
*/
mask = &current->mm->context.icache_stale_mask;
stale_cpu = cpumask_test_cpu(smp_processor_id(), mask);
stale_cpu = cpumask_test_cpu(cpu, mask);
cpumask_setall(mask);
cpumask_assign_cpu(smp_processor_id(), mask, stale_cpu);
cpumask_assign_cpu(cpu, mask, stale_cpu);
put_cpu();
}
#endif
@ -239,14 +241,12 @@ int riscv_set_icache_flush_ctx(unsigned long ctx, unsigned long scope)
case PR_RISCV_CTX_SW_FENCEI_OFF:
switch (scope) {
case PR_RISCV_SCOPE_PER_PROCESS:
current->mm->context.force_icache_flush = false;
set_icache_stale_mask();
current->mm->context.force_icache_flush = false;
break;
case PR_RISCV_SCOPE_PER_THREAD:
current->thread.force_icache_flush = false;
set_icache_stale_mask();
current->thread.force_icache_flush = false;
break;
default:
return -EINVAL;