Linux – “Memory corruption detected in low memory” kernel messages

linux-kernelpower-managementsuspend

I have some memory corruption detected in low memory messages. I have read about it here, but I'm afraid I can't really make anything out of the discussion. It does indeed happen when I suspend/resume, but unlike others who had similar problems, I do not have a broadcom wifi card. This is all marvell ethernet and no wifi.

This is a desktop tower with:

Linux gentoomyuser3x86_64 3.10.25-gentoo #1 SMP 2014 x86_64 Intel(R)
Core(TM)2 Quad CPU Q6600 @ 2.40GHz GenuineIntel GNU/Linux

Here is the log output:

[12960.735034] Corrupted low memory at ffff88000000fff0 (fff0 phys) = 40000000000000
[12960.735035] Corrupted low memory at ffff88000000fff8 (fff8 phys) = 250c60840020
[12960.735036] ------------[ cut here ]------------
[12960.735042] WARNING: at arch/x86/kernel/check.c:140 check_for_bios_corruption+0x10f/0x120()
[12960.735043] Memory corruption detected in low memory
[12960.735056] Modules linked in: nouveau cfbfillrect cfbimgblt video mxm_wmi wmi cfbcopyarea ttm fbcon bitblit softcursor font tileblit drm_kms_helper backlight fb fbdev fuse
[12960.735060] CPU: 0 PID: 9096 Comm: kworker/0:1 Not tainted 3.10.25-gentoo #1
[12960.735061] Hardware name: System manufacturer Maximus Formula/Maximus Formula, BIOS 1403    09/10/2009
[12960.735064] Workqueue: events check_corruption
[12960.735068]  0000000000000009 ffff8801165fdcf8 ffffffff8157431b ffff8801165fdd38
[12960.735070]  ffffffff81045e9b 0000000000000000 0000000000000000 ffff880000010000
[12960.735073]  ffffffff819b20b0 0000000000000001 ffff880000000000 ffff8801165fdd98
[12960.735074] Call Trace:
[12960.735079]  [<ffffffff8157431b>] dump_stack+0x19/0x1b
[12960.735082]  [<ffffffff81045e9b>] warn_slowpath_common+0x6b/0xa0
[12960.735085]  [<ffffffff81045f71>] warn_slowpath_fmt+0x41/0x50
[12960.735087]  [<ffffffff8102bc2f>] check_for_bios_corruption+0x10f/0x120
[12960.735090]  [<ffffffff8102bc49>] check_corruption+0x9/0x40
[12960.735094]  [<ffffffff81061f8f>] process_one_work+0x17f/0x420
[12960.735096]  [<ffffffff81062679>] worker_thread+0x119/0x370
[12960.735099]  [<ffffffff81062560>] ? rescuer_thread+0x2f0/0x2f0
[12960.735102]  [<ffffffff81068c4b>] kthread+0xbb/0xc0
[12960.735105]  [<ffffffff81068b90>] ? kthread_create_on_node+0x120/0x120
[12960.735109]  [<ffffffff815819ac>] ret_from_fork+0x7c/0xb0
[12960.735111]  [<ffffffff81068b90>] ? kthread_create_on_node+0x120/0x120
[12960.735113] ---[ end trace a3687665afd39e38 ]---

And verbose lspci -v and some lshw output to better understand my setup, if required:

00:00.0 Host bridge: Intel Corporation 82X38/X48 Express DRAM Controller (rev 01)
    Subsystem: ASUSTeK Computer Inc. Device 8295
    Flags: bus master, fast devsel, latency 0
    Capabilities: [e0] Vendor Specific Information: Len=0c <?>

00:06.0 PCI bridge: Intel Corporation 82X38/X48 Express Host-Secondary PCI Express Bridge (rev 01) (prog-if 00 [Normal decode])
    Flags: bus master, fast devsel, latency 0
    Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
    I/O behind bridge: 0000d000-0000dfff
    Memory behind bridge: fd000000-feafffff
    Prefetchable memory behind bridge: 00000000f0000000-00000000f9ffffff
    Capabilities: [88] Subsystem: Intel Corporation Device 0000
    Capabilities: [80] Power Management version 3
    Capabilities: [90] MSI: Enable+ Count=1/1 Maskable- 64bit-
    Capabilities: [a0] Express Root Port (Slot+), MSI 00
    Capabilities: [100] Virtual Channel
    Capabilities: [140] Root Complex Link
    Kernel driver in use: pcieport

00:1a.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #4 (rev 02) (prog-if 00 [UHCI])
    Subsystem: ASUSTeK Computer Inc. P5K PRO Motherboard
    Flags: bus master, medium devsel, latency 0, IRQ 16
    I/O ports at c800 [size=32]
    Capabilities: [50] PCI Advanced Features
    Kernel driver in use: uhci_hcd

00:1a.1 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #5 (rev 02) (prog-if 00 [UHCI])
    Subsystem: ASUSTeK Computer Inc. P5K PRO Motherboard
    Flags: bus master, medium devsel, latency 0, IRQ 21
    I/O ports at c880 [size=32]
    Capabilities: [50] PCI Advanced Features
    Kernel driver in use: uhci_hcd

00:1a.2 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #6 (rev 02) (prog-if 00 [UHCI])
    Subsystem: ASUSTeK Computer Inc. P5K PRO Motherboard
    Flags: bus master, medium devsel, latency 0, IRQ 18
    I/O ports at cc00 [size=32]
    Capabilities: [50] PCI Advanced Features
    Kernel driver in use: uhci_hcd

00:1a.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #2 (rev 02) (prog-if 20 [EHCI])
    Subsystem: ASUSTeK Computer Inc. P5K PRO Motherboard
    Flags: bus master, medium devsel, latency 0, IRQ 18
    Memory at fcfffc00 (32-bit, non-prefetchable) [size=1K]
    Capabilities: [50] Power Management version 2
    Capabilities: [58] Debug port: BAR=1 offset=00a0
    Capabilities: [98] PCI Advanced Features
    Kernel driver in use: ehci-pci

00:1b.0 Audio device: Intel Corporation 82801I (ICH9 Family) HD Audio Controller (rev 02)
    Subsystem: ASUSTeK Computer Inc. Device 8277
    Flags: bus master, fast devsel, latency 0, IRQ 45
    Memory at fcff8000 (64-bit, non-prefetchable) [size=16K]
    Capabilities: [50] Power Management version 2
    Capabilities: [60] MSI: Enable+ Count=1/1 Maskable- 64bit+
    Capabilities: [70] Express Root Complex Integrated Endpoint, MSI 00
    Capabilities: [100] Virtual Channel
    Capabilities: [130] Root Complex Link
    Kernel driver in use: snd_hda_intel

00:1c.0 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 1 (rev 02) (prog-if 00 [Normal decode])
    Flags: bus master, fast devsel, latency 0
    Bus: primary=00, secondary=03, subordinate=03, sec-latency=0
    I/O behind bridge: 00001000-00001fff
    Memory behind bridge: fa000000-fa3fffff
    Prefetchable memory behind bridge: 00000000fbf00000-00000000fbffffff
    Capabilities: [40] Express Root Port (Slot+), MSI 00
    Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
    Capabilities: [90] Subsystem: ASUSTeK Computer Inc. P5K PRO Motherboard
    Capabilities: [a0] Power Management version 2
    Capabilities: [100] Virtual Channel
    Capabilities: [180] Root Complex Link
    Kernel driver in use: pcieport

00:1c.2 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 3 (rev 02) (prog-if 00 [Normal decode])
    Flags: bus master, fast devsel, latency 0
    Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
    I/O behind bridge: 0000e000-0000efff
    Memory behind bridge: feb00000-febfffff
    Prefetchable memory behind bridge: 00000000fa400000-00000000fa5fffff
    Capabilities: [40] Express Root Port (Slot+), MSI 00
    Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
    Capabilities: [90] Subsystem: ASUSTeK Computer Inc. Device 8277
    Capabilities: [a0] Power Management version 2
    Capabilities: [100] Virtual Channel
    Capabilities: [180] Root Complex Link
    Kernel driver in use: pcieport

00:1d.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 (rev 02) (prog-if 00 [UHCI])
    Subsystem: ASUSTeK Computer Inc. P5K PRO Motherboard
    Flags: bus master, medium devsel, latency 0, IRQ 23
    I/O ports at c080 [size=32]
    Capabilities: [50] PCI Advanced Features
    Kernel driver in use: uhci_hcd

00:1d.1 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 (rev 02) (prog-if 00 [UHCI])
    Subsystem: ASUSTeK Computer Inc. P5K PRO Motherboard
    Flags: bus master, medium devsel, latency 0, IRQ 19
    I/O ports at c400 [size=32]
    Capabilities: [50] PCI Advanced Features
    Kernel driver in use: uhci_hcd

00:1d.2 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 (rev 02) (prog-if 00 [UHCI])
    Subsystem: ASUSTeK Computer Inc. P5K PRO Motherboard
    Flags: bus master, medium devsel, latency 0, IRQ 18
    I/O ports at c480 [size=32]
    Capabilities: [50] PCI Advanced Features
    Kernel driver in use: uhci_hcd

00:1d.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 (rev 02) (prog-if 20 [EHCI])
    Subsystem: ASUSTeK Computer Inc. P5K PRO Motherboard
    Flags: bus master, medium devsel, latency 0, IRQ 23
    Memory at fcfff800 (32-bit, non-prefetchable) [size=1K]
    Capabilities: [50] Power Management version 2
    Capabilities: [58] Debug port: BAR=1 offset=00a0
    Capabilities: [98] PCI Advanced Features
    Kernel driver in use: ehci-pci

00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92) (prog-if 01 [Subtractive decode])
    Flags: bus master, fast devsel, latency 0
    Bus: primary=00, secondary=04, subordinate=04, sec-latency=32
    Capabilities: [50] Subsystem: ASUSTeK Computer Inc. P5K PRO Motherboard

00:1f.0 ISA bridge: Intel Corporation 82801IR (ICH9R) LPC Interface Controller (rev 02)
    Subsystem: ASUSTeK Computer Inc. P5K PRO Motherboard
    Flags: bus master, medium devsel, latency 0
    Capabilities: [e0] Vendor Specific Information: Len=0c <?>

00:1f.2 SATA controller: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] (rev 02) (prog-if 01 [AHCI 1.0])
    Subsystem: ASUSTeK Computer Inc. Device 8277
    Flags: bus master, 66MHz, medium devsel, latency 0, IRQ 43
    I/O ports at bc00 [size=8]
    I/O ports at b880 [size=4]
    I/O ports at b800 [size=8]
    I/O ports at b480 [size=4]
    I/O ports at b400 [size=32]
    Memory at fcffe800 (32-bit, non-prefetchable) [size=2K]
    Capabilities: [80] MSI: Enable+ Count=1/16 Maskable- 64bit-
    Capabilities: [70] Power Management version 3
    Capabilities: [a8] SATA HBA v1.0
    Capabilities: [b0] PCI Advanced Features
    Kernel driver in use: ahci

00:1f.3 SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller (rev 02)
    Subsystem: ASUSTeK Computer Inc. P5K PRO Motherboard
    Flags: medium devsel, IRQ 18
    Memory at fcfff400 (64-bit, non-prefetchable) [size=256]
    I/O ports at 0400 [size=32]

01:00.0 VGA compatible controller: NVIDIA Corporation GF119 [GeForce GT 610] (rev a1) (prog-if 00 [VGA controller])
    Subsystem: Gigabyte Technology Co., Ltd Device 3546
    Flags: bus master, fast devsel, latency 0, IRQ 16
    Memory at fd000000 (32-bit, non-prefetchable) [size=16M]
    Memory at f0000000 (64-bit, prefetchable) [size=128M]
    Memory at f8000000 (64-bit, prefetchable) [size=32M]
    I/O ports at dc00 [size=128]
    Expansion ROM at fea00000 [disabled] [size=512K]
    Capabilities: [60] Power Management version 3
    Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
    Capabilities: [78] Express Endpoint, MSI 00
    Capabilities: [b4] Vendor Specific Information: Len=14 <?>
    Capabilities: [100] Virtual Channel
    Capabilities: [128] Power Budgeting <?>
    Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
    Kernel driver in use: nouveau
    Kernel modules: nouveau

01:00.1 Audio device: NVIDIA Corporation GF119 HDMI Audio Controller (rev a1)
    Subsystem: Gigabyte Technology Co., Ltd Device 3546
    Flags: bus master, fast devsel, latency 0, IRQ 17
    Memory at feafc000 (32-bit, non-prefetchable) [size=16K]
    Capabilities: [60] Power Management version 3
    Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
    Capabilities: [78] Express Endpoint, MSI 00
    Kernel driver in use: snd_hda_intel

02:00.0 Ethernet controller: Marvell Technology Group Ltd. 88E8056 PCI-E Gigabit Ethernet Controller (rev 12)
    Subsystem: ASUSTeK Computer Inc. Motherboard
    Flags: bus master, fast devsel, latency 0, IRQ 44
    Memory at febfc000 (64-bit, non-prefetchable) [size=16K]
    I/O ports at e800 [size=256]
    Expansion ROM at febc0000 [disabled] [size=128K]
    Capabilities: [48] Power Management version 3
    Capabilities: [50] Vital Product Data
    Capabilities: [5c] MSI: Enable+ Count=1/1 Maskable- 64bit+
    Capabilities: [e0] Express Legacy Endpoint, MSI 00
    Capabilities: [100] Advanced Error Reporting
    Kernel driver in use: sky2

lshw

        gentoomyuser3x86_64          
        description: Desktop Computer
        product: Maximus Formula (To Be Filled By O.E.M.)
        vendor: System manufacturer
        version: System Version
        serial: System Serial Number
        width: 64 bits
        capabilities: smbios-2.4 dmi-2.4
        configuration: boot=normal chassis=desktop family=To Be Filled By O.E.M. sku=To Be Filled By O.E.M. uuid=00020003-0004-0005-0006-000700080009
      *-core
           description: Motherboard
           product: Maximus Formula
           vendor: ASUSTeK Computer INC.
           physical id: 0
           version: Rev 1.xx
           serial: MB-1234567890
           slot: To Be Filled By O.E.M.
         *-firmware
              description: BIOS
              vendor: American Megatrends Inc.
              physical id: 0
              version: 1403
              date: 09/10/2009
              size: 64KiB
              capacity: 1984KiB
              capabilities: isa pci pnp apm upgrade shadowing escd cdboot bootselect socketedrom edd int13floppy1200 int13floppy720 int13floppy2880 int5printscreen int9keyboard int14serial int17printer int10video acpi usb ls120boot zipboot biosbootspecification
         *-cpu
              description: CPU
              product: Intel(R) Core(TM)2 Quad CPU    Q6600  @ 2.40GHz
              vendor: Intel Corp.
              physical id: 4
              bus info: cpu@0
              version: Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz
              serial: To Be Filled By O.E.M.
              slot: LGA775
              size: 2394MHz
              capacity: 3800MHz
              width: 64 bits
              clock: 266MHz
              capabilities: fpu fpu_exception wp vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx x86-64 constant_tsc arch_perfmon pebs bts rep_good nopl aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm dtherm tpr_shadow vnmi flexpriority cpufreq
            *-cache:0
                 description: L1 cache
                 physical id: 5
                 slot: L1-Cache
                 size: 128KiB
                 capacity: 128KiB
                 capabilities: internal write-back data
            *-cache:1
                 description: L2 cache
                 physical id: 6
                 slot: L2-Cache
                 size: 8MiB
                 capacity: 8MiB
                 capabilities: internal write-back instruction

This is Gentoo but on Arch, I can find this. There is mention somewhere of a powerdown package on Arch which allows "graceful" suspend/resume. I suspend with:

echo -n "mem" > /sys/power/state

So should I worry about anything and is there anything wrong with the way I suspend my system?

As suggested, I have run one full pass with memtest86+ (15mins, 12 tests I think) and no errors came out. This is 2x2gb Corsair. This rig is burnt in and very stable. I have compiled over 1000 Gentoo packages on this.


Further testing

After a community member identified a kernel patch which had been deployed to solve exactly this error(Revert "epoll: use freezable blocking call"), I tried to apply the patch to my sources. It turns out I already have the patch installed!?

I also installed the suspend package which contains s2ram etc. This utility seemingly does nothing different in the way it suspends the computer. I compared logs of suspend and wake up events from using s2ram and then from using the command I use and there is no difference. I conclude that in that respect there is nothing wrong in using echo -n "mem" > /sys/power/state.

I also validated with ethtool that wake on lan is disabled (supports wake on: pg / wake on: d) so it should not interfere with power events if I understand correctly.

Maybe a setting in the bios? Or that bios is no longer updated and will trigger that exception and we'll have to live with that.

Best Answer

By default, Linux kernel reserves lowest 64K of system memory for BIOS and repeatedly scans that part of the memory for unexpected changes. If background scanning process notices that the memory has been unexpectedly changed (corrupted) it spams the kernel log with something along the lines

Corrupted low memory at <virtual address> (<address> phys) = <memory contents after corruption>
...
Memory corruption detected in low memory
...
Hardware name: <BIOS info>

where virtual address is the virtual address that points to the corrupted memory. This can be practically anything and unless you know that some other software is using said address you can safely ignore this. The physical address is the interesting part. As long as that address (always printed in hexadecimal) is lower than 64K (2^16 bytes) you will be fine and the message is really trying to say

The motherboard manufacturer is incompetent and the system BIOS is writing to low memory area (0-64K of physical memory) without reserving that area using ACPI memory tables. Don't worry, Linux kernel expected this bug and did not store anything important there. If your BIOS ever gets fixed, you could use the last 64K of your real RAM, too. However, you will probably need to re-compile your kernel to disable the expectation that the BIOS has this bug.

The kernel developers were trying to collect a list of known bad BIOS versions and only disable the usage of lowest 64K of RAM if BIOS was known buggy. In practice, it turns out that BIOS manufacturers are incompetent enough that it would make more sense to create a list of known good BIOS versions. That list would be much shorter. In long run, it will be much easier to just declare lowest 64K of memory as always bad and ignore the whole mess. That would cause lost 64K of RAM for all i386/amd64 compatible linux computers but would allow developers to spend their time for something more productive.

If you know that your BIOS is bad (above warning is emitted to kernel log) and just want to skip the scanning for bad BIOS and keep reserving the memory you can reserve that memory area and disable scanning. Just add following kernel flags:

memmap=64K$0 memory_corruption_check=0

The flag memmap reserves 64K of RAM at physical address 0 (start of the physical RAM) and flag memory_corruption_check disables background scanning for corruption.

If your system has unexpected RAM corruption and you suspect the BIOS, adding memmap=1024K$0 will reserve 1MB at the start of physical RAM from Linux kernel. Windows 10 is rumoured to do this by default so I would not be surprised if future BIOS versions corrupted all over the first 1MB if that corruption did not cause any issues using Windows 10.

Related Question