Based on my experience, I would use LOAD DATA INFILE to import your CSV File.
The LOAD DATA INFILE statement reads rows from a text file into a
table at a very high speed.
Example I found on the internet Load Data example. I tested this example on my box and worked fine
Example Table
CREATE TABLE example (
`Id` int(11) NOT NULL AUTO_INCREMENT,
`Column2` varchar(14) NOT NULL,
`Column3` varchar(14) NOT NULL,
`Column4` varchar(14) NOT NULL,
`Column5` DATE NOT NULL,
PRIMARY KEY (`Id`)
) ENGINE=InnoDB
Example CSV File
# more /tmp/example.csv
Column1,Column2,Column3,Column4,Column5
1,A,Foo,sdsdsd,4/13/2013
2,B,Bar,sdsa,4/12/2013
3,C,Foo,wewqe,3/12/2013
4,D,Bar,asdsad,2/1/2013
5,E,FOObar,wewqe,5/1/2013
Import Statement to be run from MySQL console
LOAD DATA LOCAL INFILE '/tmp/example.csv'
-> INTO TABLE example
-> FIELDS TERMINATED BY ','
-> LINES TERMINATED BY '\n'
-> IGNORE 1 LINES
-> (id, Column3,Column4, @Column5)
-> set
-> Column5 = str_to_date(@Column5, '%m/%d/%Y');
Result
MySQL [testcsv]> select * from example;
+----+---------+---------+---------+------------+
| Id | Column2 | Column3 | Column4 | Column5 |
+----+---------+---------+---------+------------+
| 1 | | Column2 | Column3 | 0000-00-00 |
| 2 | | B | Bar | 0000-00-00 |
| 3 | | C | Foo | 0000-00-00 |
| 4 | | D | Bar | 0000-00-00 |
| 5 | | E | FOObar | 0000-00-00 |
+----+---------+---------+---------+------------+
IGNORE just simply ignores the first line which are column headers.
After IGNORE, we are specifying the columns (skipping column2), to import, which matches one of the criteria in your question.
Here is another example directly from Oracle : LOAD DATA INFILE example
This should be enough to get you started.
If you are concerned about issuing an UDPATE
with lots of columns that don't happen to change, don't worry about it.
If you don't have all the possible "new values" and want to build a custom UPDATE
every time, then do just that.
$sets = array();
$binds = array();
if (...) {
$sets[] = "foo = :foo";
$binds['foo'] = $foo_value;
}
if (...) {
$sets[] = "bar = :bar";
$binds['bar'] = $bar_value;
}
...
$sql = "UPDATE ... SET " .
implode(', ', $sets) .
" WHERE ...";
$sth = $pdo->prepare($sql);
$sth=>execute($binds);
Stored Procedure
When using CONCAT()
to construct a query, be sure to use QUOTE()
around any values that as suspect -- such as might have quotes in them, or might be subject to 'sql injection'.
See: String Functions in the MySQL 5.7 Reference Manual.
Best Answer
See http://mysql.rjweb.org/doc.php/deletebig#deleting_in_chunks for a technique of chunking a table. It is aimed at
DELETE
, but can easily be adapted toUPDATE
.This should avoid the timeouts and hangs.