Thesql load data infile takes too long and db file grows too large

MySQL

I am trying to import pretty large file (3.5G with 39mlm rows) with

set foreign_key_checks=0; 
set sql_log_bin=0; 
set unique_checks=0;
load data infile '/var/lib/mysql-files/2.csv' ignore 
into table main_data fields terminated by ','  
optionally enclosed by '"' 
lines terminated by '\n' 
ignore 1 rows (col, col2, col3);

It already passed like 24 hours, and database file in /var/lib/mysql/MyDB grow to 11G, I am not sure why, because the file I import from is less than 4G.
It is mysql 8.0. The only things I added to config is

innodb_buffer_pool_size = 22G
character-set-server=utf8mb4
collation-server=utf8mb4_unicode_ci
innodb_autoinc_lock_mode=2

I think if I batch insert these data it would be faster…

As suggester in 1st comment here is creat table syntax.
I now think that so many indexes can be the issue with size and time :

CREATE TABLE `main_data` (
  `file_name` tinyint(4) NOT NULL,
  `line_number` int(11) NOT NULL,
  `link` varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
  `title` varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
  `doc_type` varchar(45) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
  `meta_keywords` varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
  `meta_description` text COLLATE utf8mb4_unicode_ci,
  `language` varchar(5) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
  `done` timestamp NULL DEFAULT NULL,
  PRIMARY KEY (`file_name`,`line_number`),
  UNIQUE KEY `link_UNIQUE` (`link`),
  KEY `done_idx` (`done`),
  KEY `lang_idx` (`language`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

I only import 1st three columns (file_name, line_number, link), rest are nulls.

Best Answer

24 hours is a very long time- the most important factor when running LOAD DATA is the buffer pool, and yours (22GB) should be enough to accommodate the imported 3.5G file. Something else must be going on, please share more data like the SHOW CREATE TABLE output for the used table and SHOW PROCESSLIST- maybe there is something blocking the table and you are waiting for the process to start. You can check at any time the state of the load by checking the GLOBAL STATUS for innodb%' as well as theHandler_writes`.

To give you an idea of how much I would expect this to take, a 3.7 GB file should not take much more than 200 seconds to safely load on an SSD desktop-grade machine. See these tests I did not a long time ago on all recent MariaDB and MySQL versions on a non-server grade machine, even on a default (bad) config: https://dbahire.com/testing-again-load-data-on-mysql-5-6-5-7-8-0-non-ga-and-mariadb-10-0-10-1-and-10-2-non-ga/

YOUR QUERY

SELECT post.postid, post.attach FROM newbb_innopost AS post WHERE post.threadid = 51506;

At first glance, that query should only touches 1.1597% (62510 out of 5390146) of the table. It should be fast given the key distribution of threadid 51506.

REALITY CHECK

No matter which version of MySQL (Oracle, Percona, MariaDB) you use, none of them can fight to one enemy they all have in common : The InnoDB Architecture.

InnoDB Architecture

CLUSTERED INDEX

Please keep in mind that the each threadid entry has a primary key attached. This means that when you read from the index, it must do a primary key lookup within the ClusteredIndex (internally named gen_clust_index). In the ClusteredIndex, each InnoDB page contains both data and PRIMARY KEY index info. See my post Best of MyISAM and InnoDB for more info.

REDUNDANT INDEXES

You have a lot of clutter in the table because some indexes have the same leading columns. MySQL and InnoDB has to navigate through the index clutter to get to needed BTREE nodes. You should reduced that clutter by running the following:

ALTER TABLE newbb_innopost
    DROP INDEX threadid,
    DROP INDEX threadid_2,
    DROP INDEX threadid_visible_dateline,
    ADD INDEX threadid_visible_dateline_index (`threadid`,`visible`,`dateline`,`userid`)
;

Why strip down these indexes ?

The first three indexes start with threadid
threadid_2 and threadid_visible_dateline start with the same three columns
threadid_visible_dateline does not need postid since it's the PRIMARY KEY and it's embedded

BUFFER CACHING

The InnoDB Buffer Pool caches data and index pages. MyISAM only caches index pages.

Just in this area alone, MyISAM does not waste time caching data. That's because it's not designed to cache data. InnoDB caches every data page and index page (and its grandmother) it touches. If your InnoDB Buffer Pool is too small, you could be caching pages, invalidating pages, and removing pages all in one query.

TABLE LAYOUT

You could shave of some space from the row by considering importthreadid and importpostid. You have them as BIGINTs. They take up 16 bytes in the ClusteredIndex per row.

You should run this

SELECT importthreadid,importpostid FROM newbb_innopost PROCEDURE ANALYSE();

This will recommend what data types these columns should be for the given dataset.

CONCLUSION

MyISAM has a lot less to contend with than InnoDB, especially in the area of caching.

While you revealed the amount of RAM (32GB) and the version of MySQL (Server version: 10.0.12-MariaDB-1~trusty-wsrep-log mariadb.org binary distribution, wsrep_25.10.r4002), there are still other pieces to this puzzle you have not revealed

The InnoDB settings
The Number of Cores
Other settings from my.cnf

If you can add these things to the question, I can further elaborate.

UPDATE 2014-08-28 11:27 EDT

You should increase threading

innodb_read_io_threads = 64
innodb_write_io_threads = 16
innodb_log_buffer_size = 256M

I would consider disabling the query cache (See my recent post Why query_cache_type is disabled by default start from MySQL 5.6?)

query_cache_size = 0

I would preserve the Buffer Pool

innodb_buffer_pool_dump_at_shutdown=1
innodb_buffer_pool_load_at_startup=1

Increase purge threads (if you do DML on multiple tables)

innodb_purge_threads = 4

GIVE IT A TRY !!!

Mysql – Load csv file to thesql using load data local infile

Your problem is on using the variable $name on your LOAD DATA LOCAL INFILE statement, you should use the variable $file in your statement..

Your statement should change to;

$import = "LOAD DATA LOCAL INFILE '" . $file. "' INTO TABLE temporal_load FIELDS TERMINATED BY ',' optionally ENCLOSED BY '\"' LINES TERMINATED BY '\r\n' IGNORE 1 LINES (num,ticker,company,sector,industry) ";

mysql_query($import) or die(mysql_error());