bd78432c8f
Background:
The MCE handler has several paths that it can take, depending on various
conditions of the MCE status and the value of the 'tolerant' knob. The
exact semantics are not well defined and the code is a bit twisty.
Description:
This patch makes the MCE handler's behavior more clear by documenting the
behavior for various 'tolerant' levels. It also fixes or enhances
several small things in the handler. Specifically:
* If RIPV is set it is not safe to restart, so set the 'no way out'
flag rather than the 'kill it' flag.
* Don't panic() on correctable MCEs.
* If the _OVER bit is set *and* the _UC bit is set (meaning possibly
dropped uncorrected errors), set the 'no way out' flag.
* Use EIPV for testing whether an app can be killed (SIGBUS) rather
than RIPV. According to docs, EIPV indicates that the error is
related to the IP, while RIPV simply means the IP is valid to
restart from.
* Don't clear the MCi_STATUS registers until after the panic() path.
This leaves the status bits set after the panic() so clever BIOSes
can find them (and dumb BIOSes can do nothing).
This patch also calls nonseekable_open() in mce_open (as suggested by akpm).
Result:
Tolerant levels behave almost identically to how they always have, but
not it's well defined. There's a slightly higher chance of panic()ing
when multiple errors happen (a good thing, IMHO). If you take an MBE and
panic(), the error status bits are not cleared.
Alternatives:
None.
Testing:
I used software to inject correctable and uncorrectable errors. With
tolerant = 3, the system usually survives. With tolerant = 2, the system
usually panic()s (PCC) but not always. With tolerant = 1, the system
always panic()s. When the system panic()s, the BIOS is able to detect
that the cause of death was an MC4. I was not able to reproduce the
case of a non-PCC error in userspace, with EIPV, with (tolerant < 3).
That will be rare at best.
Signed-off-by: Tim Hockin <thockin@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Andi Kleen <ak@suse.de>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
78 lines
3.0 KiB
Plaintext
78 lines
3.0 KiB
Plaintext
|
|
Configurable sysfs parameters for the x86-64 machine check code.
|
|
|
|
Machine checks report internal hardware error conditions detected
|
|
by the CPU. Uncorrected errors typically cause a machine check
|
|
(often with panic), corrected ones cause a machine check log entry.
|
|
|
|
Machine checks are organized in banks (normally associated with
|
|
a hardware subsystem) and subevents in a bank. The exact meaning
|
|
of the banks and subevent is CPU specific.
|
|
|
|
mcelog knows how to decode them.
|
|
|
|
When you see the "Machine check errors logged" message in the system
|
|
log then mcelog should run to collect and decode machine check entries
|
|
from /dev/mcelog. Normally mcelog should be run regularly from a cronjob.
|
|
|
|
Each CPU has a directory in /sys/devices/system/machinecheck/machinecheckN
|
|
(N = CPU number)
|
|
|
|
The directory contains some configurable entries:
|
|
|
|
Entries:
|
|
|
|
bankNctl
|
|
(N bank number)
|
|
64bit Hex bitmask enabling/disabling specific subevents for bank N
|
|
When a bit in the bitmask is zero then the respective
|
|
subevent will not be reported.
|
|
By default all events are enabled.
|
|
Note that BIOS maintain another mask to disable specific events
|
|
per bank. This is not visible here
|
|
|
|
The following entries appear for each CPU, but they are truly shared
|
|
between all CPUs.
|
|
|
|
check_interval
|
|
How often to poll for corrected machine check errors, in seconds
|
|
(Note output is hexademical). Default 5 minutes. When the poller
|
|
finds MCEs it triggers an exponential speedup (poll more often) on
|
|
the polling interval. When the poller stops finding MCEs, it
|
|
triggers an exponential backoff (poll less often) on the polling
|
|
interval. The check_interval variable is both the initial and
|
|
maximum polling interval.
|
|
|
|
tolerant
|
|
Tolerance level. When a machine check exception occurs for a non
|
|
corrected machine check the kernel can take different actions.
|
|
Since machine check exceptions can happen any time it is sometimes
|
|
risky for the kernel to kill a process because it defies
|
|
normal kernel locking rules. The tolerance level configures
|
|
how hard the kernel tries to recover even at some risk of
|
|
deadlock. Higher tolerant values trade potentially better uptime
|
|
with the risk of a crash or even corruption (for tolerant >= 3).
|
|
|
|
0: always panic on uncorrected errors, log corrected errors
|
|
1: panic or SIGBUS on uncorrected errors, log corrected errors
|
|
2: SIGBUS or log uncorrected errors, log corrected errors
|
|
3: never panic or SIGBUS, log all errors (for testing only)
|
|
|
|
Default: 1
|
|
|
|
Note this only makes a difference if the CPU allows recovery
|
|
from a machine check exception. Current x86 CPUs generally do not.
|
|
|
|
trigger
|
|
Program to run when a machine check event is detected.
|
|
This is an alternative to running mcelog regularly from cron
|
|
and allows to detect events faster.
|
|
|
|
TBD document entries for AMD threshold interrupt configuration
|
|
|
|
For more details about the x86 machine check architecture
|
|
see the Intel and AMD architecture manuals from their developer websites.
|
|
|
|
For more details about the architecture see
|
|
see http://one.firstfloor.org/~andi/mce.pdf
|