MariaDB – Why MariaDB is Not Using All Available Memory

mariadbmemory

I have a joomla web site with about 160.000 articles. I use 10.4.6-MariaDB, nginx and php 7.3.

When I check my server with top, I see a maximum of 3,3 MB of memory usage by MariaDB while I have given it 25GB, or at least this is what I think.

Bellow is my excerpt 50-server.cnf:

# If you use the same .cnf file for MariaDB of different versions,
# use this group for options that older servers don't understand
[mariadb-10.1]
innodb_log_buffer_size=2G
innodb_buffer_pool_size=25G
innodb_file_per_table= ON
sql_bin_log=0;

The above are on a Debian 8, nginx, hosting a joomla 4 alpha 7 cms.

In response to Rick James comments:

The query that takes long is 7 sec:

SELECT DISTINCT a.id, a.title, a.alias, a.checked_out, a.checked_out_time, a.catid, a.state, a.access, a.created, a.created_by, a.created_by_alias, a.modified, a.ordering, a.featured, a.language, a.hits, a.publish_up, a.publish_down, a.introtext,l.title AS language_title, l.image AS language_image,uc.name AS editor,ag.title AS access_level,c.title AS category_title,ua.name AS author_name,`wa`.`stage_id` AS `stage_id`,`ws`.`title` AS `stage_title`,`ws`.`condition` AS `stage_condition`,`ws`.`workflow_id` AS `workflow_id`,COUNT(asso2.id)>1 as association
FROM bwn_content AS a
LEFT JOIN `bwn_languages` AS l ON l.lang_code = a.language
LEFT JOIN bwn_users AS uc ON uc.id=a.checked_out
LEFT JOIN bwn_viewlevels AS ag ON ag.id = a.access
LEFT JOIN bwn_categories AS c ON c.id = a.catid
LEFT JOIN bwn_users AS ua ON ua.id = a.created_by
INNER JOIN `bwn_workflow_associations` AS `wa` ON `wa`.`item_id` = `a`.`id`
INNER JOIN `bwn_workflow_stages` AS `ws` ON `ws`.`id` = `wa`.`stage_id`
LEFT JOIN bwn_associations AS asso ON asso.id = a.id AND asso.context='com_content.item'
LEFT JOIN bwn_associations AS asso2 ON asso2.key = asso.key
WHERE `ws`.`condition` IN (1, 0) AND `wa`.`extension`='com_content'
GROUP BY `a`.`id`,`a`.`title`,`a`.`alias`,`a`.`checked_out`,`a`.`checked_out_time`,`a`.`state`,`a`.`access`,`a`.`created`,`a`.`created_by`,`a`.`created_by_alias`,`a`.`modified`,`a`.`ordering`,`a`.`featured`,`a`.`language`,`a`.`hits`,`a`.`publish_up`,`a`.`publish_down`,`a`.`catid`,`l`.`title`,`l`.`image`,`uc`.`name`,`ag`.`title`,`c`.`title`,`ua`.`name`,`ws`.`title`,`ws`.`workflow_id`,`ws`.`condition`,`wa`.`stage_id`
ORDER BY a.id DESC

The main table at the above query is

bwn_content is about 1.5G and has about 160.000 records

Show create statement for the above is:

CREATE TABLE `bwn_content` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `asset_id` int(10) unsigned NOT NULL DEFAULT 0 COMMENT 'FK to the #__assets table.',
  `title` varchar(255) COLLATE utf8mb4_unicode_ci NOT NULL DEFAULT '',
  `alias` varchar(400) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL DEFAULT '',
  `introtext` mediumtext COLLATE utf8mb4_unicode_ci NOT NULL,
  `fulltext` mediumtext COLLATE utf8mb4_unicode_ci NOT NULL,
  `state` tinyint(3) NOT NULL DEFAULT 0,
  `catid` int(10) unsigned NOT NULL DEFAULT 0,
  `created` datetime NOT NULL DEFAULT '0000-00-00 00:00:00',
  `created_by` int(10) unsigned NOT NULL DEFAULT 0,
  `created_by_alias` varchar(255) COLLATE utf8mb4_unicode_ci NOT NULL DEFAULT '',
  `modified` datetime NOT NULL DEFAULT '0000-00-00 00:00:00',
  `modified_by` int(10) unsigned NOT NULL DEFAULT 0,
  `checked_out` int(10) unsigned NOT NULL DEFAULT 0,
  `checked_out_time` datetime NOT NULL DEFAULT '0000-00-00 00:00:00',
  `publish_up` datetime NOT NULL DEFAULT '0000-00-00 00:00:00',
  `publish_down` datetime NOT NULL DEFAULT '0000-00-00 00:00:00',
  `images` text COLLATE utf8mb4_unicode_ci NOT NULL,
  `urls` text COLLATE utf8mb4_unicode_ci NOT NULL,
  `attribs` varchar(5120) COLLATE utf8mb4_unicode_ci NOT NULL,
  `version` int(10) unsigned NOT NULL DEFAULT 1,
  `ordering` int(11) NOT NULL DEFAULT 0,
  `metakey` text COLLATE utf8mb4_unicode_ci NOT NULL,
  `metadesc` text COLLATE utf8mb4_unicode_ci NOT NULL,
  `access` int(10) unsigned NOT NULL DEFAULT 0,
  `hits` int(10) unsigned NOT NULL DEFAULT 0,
  `metadata` text COLLATE utf8mb4_unicode_ci NOT NULL,
  `featured` tinyint(3) unsigned NOT NULL DEFAULT 0 COMMENT 'Set if article is featured.',
  `language` char(7) COLLATE utf8mb4_unicode_ci NOT NULL COMMENT 'The language code for the article.',
  `xreference` varchar(50) COLLATE utf8mb4_unicode_ci NOT NULL DEFAULT '' COMMENT 'A reference to enable linkages to external data sets.',
  PRIMARY KEY (`id`),
  KEY `idx_access` (`access`),
  KEY `idx_checkout` (`checked_out`),
  KEY `idx_state` (`state`),
  KEY `idx_catid` (`catid`),
  KEY `idx_createdby` (`created_by`),
  KEY `idx_featured_catid` (`featured`,`catid`),
  KEY `idx_language` (`language`),
  KEY `idx_xreference` (`xreference`),
  KEY `idx_alias` (`alias`(191))
) ENGINE=InnoDB AUTO_INCREMENT=165248 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_c

Best Answer

Your settings are in a [mariadb-10.1] section, so this will not be used by MariaDB 10.4. You need to either move them to e.g. a [mariadb-10.4] or a [mariadb] section.

You can check the actual values used by MariaDB from the command-line client with commands such as:

SHOW VARIABLES LIKE 'innodb_log_buffer_size';
SHOW VARIABLES LIKE 'innodb_buffer_pool_size';

To find the size of your data + indexes, run e.g. one of these queries:

SELECT 
  round(sum(data_length + index_length) / 1024 / 1024, 1) "DB size (MB)"
FROM information_schema.tables;

This will show the size per schema + the total:

SELECT table_schema "DB name", 
  round(sum(data_length + index_length) / 1024 / 1024, 1) "DB size (MB)"
FROM information_schema.tables
GROUP BY table_schema WITH ROLLUP;

This will show the size per table + total in a particular database:

SELECT table_name "table name",        
  round(sum(data_length + index_length) / 1024 / 1024, 1) "DB Size in MB"
FROM information_schema.tables
WHERE table_schema = 'joomla' 
GROUP BY table_name WITH ROLLUP;

Related Solutions

MongoDB Not Using All Available RAM

The resident memory size represents the number of pages in memory actually touched by the mongod process. If that is significantly lower than the available memory and data exceeds the available memory (yours does), then it could be a case of simply not having actively touched enough pages yet.

To determine if this is the case, you should run free -m, the output should look something like this:

free -m
             total       used       free     shared    buffers     cached
Mem:          3709       3484        224          0         84       2412
-/+ buffers/cache:        987       2721
Swap:         3836        156       3680

In my example, cached is not close to the total, which means that not only has mongod not touched enough pages, the filesystem cache has not yet even been filled by pages being read from the disk in general.

A quick remedy for this would be the touch command (added in 2.2) - it should be used with caution on large data sets as it will attempt to load everything into RAM even if the data is far too large to fit (causing a lot of disk IO and page faults). It will certainly fill up the memory effectively though :)

If your cached value is close to the total available, then your issue is that a large number of pages being read into memory from disk are not relevant to (and hence not touched by) the mongod process. The usual candidate for this kind of discrepency is readahead. I've already covered that particular topic elsewhere in detail, so I'll just link those two answers for future reading if necessary.

SQL Server 2008 R2 – Not Using Available Memory

Standard Edition is limited to a 64 GB buffer pool.

To use all the available memory of the server, you'll need to upgrade to either Enterprise or Datacenter Edition, or scale out to multiple instances of Standard Edition.

Best Answer

Related Solutions

MongoDB Not Using All Available RAM

SQL Server 2008 R2 – Not Using Available Memory

Related Question