kevin/linux
1
Fork 0
Code
d595d423d0
linux/kernel
History
Linus Torvalds 3510ca20ec Minor page waitqueue cleanups 
Tim Chen and Kan Liang have been battling a customer load that shows
extremely long page wakeup lists.  The cause seems to be constant NUMA
migration of a hot page that is shared across a lot of threads, but the
actual root cause for the exact behavior has not been found.

Tim has a patch that batches the wait list traversal at wakeup time, so
that we at least don't get long uninterruptible cases where we traverse
and wake up thousands of processes and get nasty latency spikes.  That
is likely 4.14 material, but we're still discussing the page waitqueue
specific parts of it.

In the meantime, I've tried to look at making the page wait queues less
expensive, and failing miserably.  If you have thousands of threads
waiting for the same page, it will be painful.  We'll need to try to
figure out the NUMA balancing issue some day, in addition to avoiding
the excessive spinlock hold times.

That said, having tried to rewrite the page wait queues, I can at least
fix up some of the braindamage in the current situation. In particular:

 (a) we don't want to continue walking the page wait list if the bit
     we're waiting for already got set again (which seems to be one of
     the patterns of the bad load).  That makes no progress and just
     causes pointless cache pollution chasing the pointers.

 (b) we don't want to put the non-locking waiters always on the front of
     the queue, and the locking waiters always on the back.  Not only is
     that unfair, it means that we wake up thousands of reading threads
     that will just end up being blocked by the writer later anyway.

Also add a comment about the layout of 'struct wait_page_key' - there is
an external user of it in the cachefiles code that means that it has to
match the layout of 'struct wait_bit_key' in the two first members.  It
so happens to match, because 'struct page *' and 'unsigned long *' end
up having the same values simply because the page flags are the first
member in struct page.

Cc: Tim Chen <tim.c.chen@linux.intel.com>
Cc: Kan Liang <kan.liang@intel.com>
Cc: Mel Gorman <mgorman@techsingularity.net>
Cc: Christopher Lameter <cl@linux.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: Davidlohr Bueso <dave@stgolabs.net>
Cc: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2017-08-27 13:55:12 -07:00
..
bpf bpf: fix bpf_prog_get_info_by_fd to dump correct xlated_prog_len 2017-07-29 23:29:41 -07:00
cgroup cpuset: fix a deadlock due to incomplete patching of cpusets_enabled() 2017-08-02 17:16:12 -07:00
configs config: android-base: disable CONFIG_NFSD and CONFIG_NFS_FS 2017-06-09 11:47:38 +02:00
debug sched/headers: Prepare for new header dependencies before moving code to <linux/sched/debug.h> 2017-03-02 08:42:34 +01:00
events perf/core: Fix group {cpu,task} validation 2017-08-25 11:00:34 +02:00
gcov gcov: support GCC 7.1 2017-05-12 15:57:15 -07:00
irq genirq/ipi: Fixup checks against nr_cpu_ids 2017-08-20 10:49:05 +02:00
livepatch livepatch: Fix stacking of patches with respect to RCU 2017-06-20 10:42:19 +02:00
locking Merge branch 'locking-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip 2017-07-21 11:11:23 -07:00
power mm: fix global NR_SLAB_.*CLAIMABLE counter reads 2017-08-10 15:54:06 -07:00
printk Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/pmladek/printk 2017-07-05 11:11:26 -07:00
rcu rcu: Remove RCU CPU stall warnings from Tiny RCU 2017-06-08 18:52:45 -07:00
sched Minor page waitqueue cleanups 2017-08-27 13:55:12 -07:00
time timers: Fix excessive granularity of new timers after a nohz idle 2017-08-24 11:40:18 +02:00
trace Various bug fixes: 2017-08-24 14:08:22 -07:00
.gitignore
acct.c sched/headers: Prepare to move cputime functionality from <linux/sched.h> into <linux/sched/cputime.h> 2017-03-02 08:42:39 +01:00
async.c async: Adjust system_state checks 2017-05-23 10:01:37 +02:00
audit_fsnotify.c Merge branch 'fsnotify' of git://git.kernel.org/pub/scm/linux/kernel/git/jack/linux-fs 2017-05-03 11:05:15 -07:00
audit_tree.c Merge branch 'fsnotify' of git://git.kernel.org/pub/scm/linux/kernel/git/jack/linux-fs 2017-05-03 11:05:15 -07:00
audit_watch.c audit/stable-4.13 PR 20170816 2017-08-16 16:48:34 -07:00
audit.c Merge branch 'stable-4.13' of git://git.infradead.org/users/pcmoore/audit 2017-07-20 10:22:26 -07:00
audit.h audit: style fix 2017-06-12 18:07:43 -04:00
auditfilter.c audit: kernel generated netlink traffic should have a portid of 0 2017-05-02 10:16:05 -04:00
auditsc.c Merge branch 'stable-4.13' of git://git.infradead.org/users/pcmoore/audit 2017-07-05 11:24:05 -07:00
backtracetest.c
bounds.c
capability.c capability: export has_capability 2017-01-12 07:01:56 -07:00
compat.c Merge branch 'misc.compat' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs 2017-07-06 20:57:13 -07:00
configs.c Replace <asm/uaccess.h> with <linux/uaccess.h> globally 2016-12-24 11:46:01 -08:00
context_tracking.c
cpu_pm.c
cpu.c smp/hotplug: Replace BUG_ON and react useful 2017-07-11 22:25:44 +02:00
crash_core.c kdump: protect vmcoreinfo data under the crash memory 2017-07-12 16:26:00 -07:00
crash_dump.c
cred.c doc: ReSTify credentials.txt 2017-05-18 10:30:19 -06:00
delayacct.c sched/headers: Prepare to move cputime functionality from <linux/sched.h> into <linux/sched/cputime.h> 2017-03-02 08:42:39 +01:00
dma.c
elfcore.c
exec_domain.c
exit.c kernel/exit.c: avoid undefined behaviour when calling wait4() 2017-07-10 16:32:36 -07:00
extable.c lib/extable.c: use bsearch() library function in search_extable() 2017-07-10 16:32:35 -07:00
fork.c fork: fix incorrect fput of ->exe_file causing use-after-free 2017-08-25 16:12:46 -07:00
freezer.c
futex_compat.c Replace <asm/uaccess.h> with <linux/uaccess.h> globally 2016-12-24 11:46:01 -08:00
futex.c futex: Remove unnecessary warning from get_futex_key 2017-08-09 14:00:54 -07:00
groups.c kernel/groups.c: use sort library function 2017-07-10 16:32:34 -07:00
hung_task.c kernel/hung_task.c: defer showing held locks 2017-05-08 17:15:10 -07:00
irq_work.c
jump_label.c jump_label: Reorder hotplug lock and jump_label_lock 2017-05-26 10:10:45 +02:00
kallsyms.c kernel/kallsyms.c: replace all_var with IS_ENABLED(CONFIG_KALLSYMS_ALL) 2017-07-10 16:32:34 -07:00
kcmp.c kcmp: add KCMP_EPOLL_TFD mode to compare epoll target files 2017-07-12 16:26:01 -07:00
Kconfig.freezer
Kconfig.hz
Kconfig.locks
Kconfig.preempt
kcov.c kcov: simplify interrupt check 2017-05-08 17:15:12 -07:00
kexec_core.c kdump: protect vmcoreinfo data under the crash memory 2017-07-12 16:26:00 -07:00
kexec_file.c kexec_file: adjust declaration of kexec_purgatory 2017-07-12 16:26:02 -07:00
kexec_internal.h kexec_file: adjust declaration of kexec_purgatory 2017-07-12 16:26:02 -07:00
kexec.c kdump: protect vmcoreinfo data under the crash memory 2017-07-12 16:26:00 -07:00
kmod.c kmod: fix wait on recursive loop 2017-08-18 15:32:01 -07:00
kprobes.c kprobes: Ensure that jprobe probepoints are at function entry 2017-07-08 11:05:35 +02:00
ksysfs.c kexec: move vmcoreinfo out of the kernel's .bss section 2017-07-12 16:25:59 -07:00
kthread.c cgroup, kthread: close race window where new kthreads can be migrated to non-root cgroups 2017-03-17 10:18:47 -04:00
latencytop.c sched/headers: Prepare to move sched_info_on() and force_schedstat_enabled() from <linux/sched.h> to <linux/sched/stat.h> 2017-03-02 08:42:39 +01:00
Makefile kernel/watchdog: split up config options 2017-07-12 16:26:02 -07:00
membarrier.c Fix: Disable sys_membarrier when nohz_full is enabled 2017-01-23 11:32:16 -08:00
memremap.c mm, memory_hotplug: replace for_device by want_memblock in arch_add_memory 2017-07-06 16:24:32 -07:00
module_signing.c
module-internal.h
module.c Modules updates for v4.13 2017-07-12 17:22:01 -07:00
notifier.c kernel/notifier.c: simplify expression 2017-02-24 17:46:56 -08:00
nsproxy.c perf: Add PERF_RECORD_NAMESPACES to include namespaces related info 2017-03-13 15:57:41 -03:00
padata.c padata: Avoid nested calls to cpus_read_lock() in pcrypt_init_padata() 2017-05-26 10:10:37 +02:00
panic.c sched/headers: Prepare for new header dependencies before moving code to <linux/sched/debug.h> 2017-03-02 08:42:34 +01:00
params.c boot/param: Move next_arg() function to lib/cmdline.c for later reuse 2017-04-18 10:37:13 +02:00
pid_namespace.c pid_ns: Sleep in TASK_INTERRUPTIBLE in zap_pid_ns_processes 2017-05-13 17:26:01 -05:00
pid.c pids: make task_tgid_nr_ns() safe 2017-08-21 12:47:31 -07:00
profile.c sched/headers: Prepare to move sched_info_on() and force_schedstat_enabled() from <linux/sched.h> to <linux/sched/stat.h> 2017-03-02 08:42:39 +01:00
ptrace.c ptrace: Properly initialize ptracer_cred on fork 2017-05-23 07:40:44 -05:00
range.c
reboot.c
relay.c Merge branch 'work.splice' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs 2017-05-02 11:38:06 -07:00
resource.c
seccomp.c seccomp: Switch from atomic_t to recount_t 2017-06-26 09:24:00 -07:00
signal.c signal: don't remove SIGNAL_UNKILLABLE for traced tasks. 2017-08-18 15:32:02 -07:00
smp.c smp, cpumask: Use non-atomic cpumask_{set,clear}_cpu() 2017-05-23 10:01:32 +02:00
smpboot.c sched/headers: Prepare for new header dependencies before moving code to <linux/sched/task.h> 2017-03-02 08:42:35 +01:00
smpboot.h
softirq.c sched/core: Remove 'task' parameter and rename tsk_restore_flags() to current_restore_flags() 2017-04-11 09:06:32 +02:00
stacktrace.c stacktrace/x86: add function for detecting reliable stack traces 2017-03-08 09:18:02 +01:00
stop_machine.c stop_machine: Provide stop_machine_cpuslocked() 2017-05-26 10:10:36 +02:00
sys_ni.c move aio compat to fs/aio.c 2016-12-22 22:58:37 -05:00
sys.c fix a braino in compat_sys_getrlimit() 2017-07-12 09:15:00 -07:00
sysctl_binary.c kernel/sysctl_binary.c: check name array length in deprecated_sysctl_warning() 2017-07-12 16:26:00 -07:00
sysctl.c kernel/watchdog: split up config options 2017-07-12 16:26:02 -07:00
task_work.c
taskstats.c taskstats: add e/u/stime for TGID command 2017-05-08 17:15:12 -07:00
test_kprobes.c
torture.c sched/headers: Prepare for new header dependencies before moving code to <linux/sched/clock.h> 2017-03-02 08:42:27 +01:00
tracepoint.c sched/headers: Prepare for new header dependencies before moving code to <linux/sched/task.h> 2017-03-02 08:42:35 +01:00
tsacct.c sched/headers: Prepare to move cputime functionality from <linux/sched.h> into <linux/sched/cputime.h> 2017-03-02 08:42:39 +01:00
ucount.c ucount: Remove the atomicity from ucount->count 2017-03-06 15:26:37 -06:00
uid16.c sched/headers: Prepare to remove <linux/cred.h> inclusion from <linux/sched.h> 2017-03-02 08:42:31 +01:00
up.c
user_namespace.c sched/headers: Prepare for new header dependencies before moving code to <linux/sched/signal.h> 2017-03-02 08:42:29 +01:00
user-return-notifier.c
user.c sched/headers: Prepare for new header dependencies before moving code to <linux/sched/user.h> 2017-03-02 08:42:29 +01:00
utsname_sysctl.c sched/headers: Remove <linux/rwsem.h> from <linux/sched.h> 2017-03-03 01:45:36 +01:00
utsname.c sched/headers: Prepare to move the task_lock()/unlock() APIs to <linux/sched/task.h> 2017-03-02 08:42:38 +01:00
watchdog_hld.c kernel/watchdog: Prevent false positives with turbo modes 2017-08-18 12:35:02 +02:00
watchdog.c kernel/watchdog: Prevent false positives with turbo modes 2017-08-18 12:35:02 +02:00
workqueue_internal.h
workqueue.c workqueue: Work around edge cases for calc of pool's cpumask 2017-07-28 11:05:52 -04:00