Oracle 12c Database swapping after set up huge pages

memoryoracle

I need to setup a Oracle Database 12c with 64GB RAM on Linux RedHet. I decided to use huge pages. Something must have went wrong. Altertlog:

WARNING: Heavy swapping observed on system in last 5 mins. pct of
memory swapped in [0.42%] pct of memory swapped out [1.82%]. Please
make sure there is no memory pressure and the SGA and PGA are
configured correctly. Look at DBRM trace file for more details.

Supported system pagesize(s):   PAGESIZE  AVAILABLE_PAGES  EXPECTED_PAGES  ALLOCATED_PAGES  ERROR(s) 
                               4K          Configured           10                    8192010          NONE 
                            2048K          15946             16001                    0                NONE

Seems like the Database is not using the huge pages? What could be the issue?

Update:

 grep Huge /proc/meminfo
AnonHugePages:     40960 kB
HugePages_Total:   16001
HugePages_Free:    16001
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB

Best Answer

The steps I did to setup this hugepage configuration on ReDHat 7.3 64GBRAM and the mistake. (Link from oracle base posted above provides a better Manual for reuse) Check total RAM of OS:

grep MemTotal /proc/meminfo
MemTotal:       65809456 kB

Check Oracle Database – AMM needs to be off: MEMORY_TARGET und MEMORY_MAX_TARGETneed to be set to zero.

SQL> select value from v$parameter where name = 'memory_target';
VALUE
---------------------------
0

Calculate SGA size, for example 32GB:

select value/1024 from v$parameter where name = 'sga_target';

VALUE/1024
----------
  32768000

Calculate Huge Pages, for a x86_64 Red Hat Enterprise Linux Server we can use a default 2MB hugepagesize: 

SGA / Hugepagesize=Hugepage +1
  32768000/2048=16001

This need to be set in /etc/sysctl.conf as root:

vm.nr_hugepages =16001

Also we need to set the /etc/security/limits.conf. This value hard and soft memlock should be greater than the the SGA and smaller than the total RAM. 

Hugepages * Hugepagesize = minimum Memlock or something like 90% of total RAM.

Setting memlock to a lower value then SGA leads to unpredictable behavior of the Oracle Database. As commented above Oracle Database will allocate memory without using the huge pages. In my case this leaded to fatal errors when using impdp. Datapump did not provide errors which made sense. 

grep oracle /etc/security/limits.conf
oracle  soft    nproc   2047
oracle  hard    nproc   16384
oracle  soft    nofile  1024
oracle  hard    nofile  65536
oracle  soft    stack   10240
#oracle  soft    memlock 19857408
#oracle  hard    memlock 19857408
oracle  soft    memlock 57671680
oracle  hard    memlock 57671680

For RedHat we also need to turn off Transparent Hugepages

cat /sys/kernel/mm/redhat_transparent_hugepage/enabled
[always] madvise never

echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled
echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag

cat /sys/kernel/mm/redhat_transparent_hugepage/enabled
always madvise [never]

Reboot System and start Oracle Database check:

grep Huge /proc/meminfo
AnonHugePages:     40960 kB
HugePages_Total:   16001
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Related Question