Mysql – merge multiple fields of csv file while running thesqlimport command

csvMySQLmysql-5.1

I have a mysql table like this

+----------------+--------------+------+-----+---------+----------------+
| Field          | Type         | Null | Key | Default | Extra          |
+----------------+--------------+------+-----+---------+----------------+
| id             | int(11)      | NO   | PRI | NULL    | auto_increment |
| roll_no        | varchar(50)  | NO   | MUL | NULL    |                |
| marks          | varchar(100) | YES  |     | NULL    |                |
| academy        | varchar(100) | YES  |     | NULL    |                |
+----------------+--------------+------+-----+---------+----------------+

the marks column can have multiple numbers separated by comma like this

45,67,78,80, # First Example
34,56,  # Second Example

I have to run mysqlimport command such that the fields after the first fields till second-last fields should merge into marks column

I tried something like this

mysqlimport --fields-terminated-by=, --columns='roll_no, marks, academy' --ignore --local -u root -pxxxx result results.csv

But extra fields (after 3rd one) gets truncated. How to handle such scenarios where the csv fields are dynamic

Best Answer

Because it is a CSV file (Comma Separated Values), it thinks that the each of the marks is a different field. Your initial export should either separate values by a different char, like "#", or quot the values bu ".

Related Solutions

MySQL – Get Column from Multiple Tables

If all the tables use the MyISAM Storage Engine and have the same table structure, I have some good news for you.

You can create a single table that consumes no additional space except a .frm file and some mapping info. The key is to take advantage of the MERGE (MRG_MyISAM) Storage Engine.

Here is how you can do this:

CREATE TABLE XMerge LIKE X1;
ALTER TABLE XMerge ENGINE=MRG_MYISAM
UNION=(X1,X2,X3,X4) INSERT_METHOD=LAST;

Using this method, you can query the 4 tables at the same time like this:

SELECT ReqF FROM XMerge WHERE EmpName='John';

Was that simple, or what ???

In your case, you have 75 tables. You would do this:

CREATE TABLE XMerge LIKE X1;
ALTER TABLE XMerge ENGINE=MRG_MYISAM
UNION=(X1,X2,X3,X4,X5,X6,X7,X8,X9,
X10,X11,X12,X13,X14,X15,X16,X17,X18,X19,
X20,X21,X22,X23,X24,X25,X26,X27,X28,X29,
X30,X31,X32,X33,X34,X35,X36,X37,X38,X39,
X40,X41,X42,X43,X44,X45,X46,X47,X48,X49,
X50,X51,X52,X53,X54,X55,X56,X57,X58,X59,
X60,X61,X62,X63,X64,X65,X66,X67,X68,X69,
X70,X71,X72,X73,X74,X75) INSERT_METHOD=LAST;
SELECT ReqF FROM XMerge WHERE EmpName='John';

The beauty of this is that creating a MERGE table takes milliseconds. Just make sure every table has an index on EmpName. Better to do 75 indexed lookups that 75 full table scans. If there is no index on EmpName, you need to do this:

ALTER TABLE X1 ADD UNIQUE KEY (EmpName);
ALTER TABLE X2 ADD UNIQUE KEY (EmpName);
.
.
.
ALTER TABLE X75 ADD UNIQUE KEY (EmpName);
CREATE TABLE XMerge LIKE X1;
ALTER TABLE XMerge ENGINE=MRG_MYISAM
UNION=(X1,X2,X3,X4,X5,X6,X7,X8,X9,
X10,X11,X12,X13,X14,X15,X16,X17,X18,X19,
X20,X21,X22,X23,X24,X25,X26,X27,X28,X29,
X30,X31,X32,X33,X34,X35,X36,X37,X38,X39,
X40,X41,X42,X43,X44,X45,X46,X47,X48,X49,
X50,X51,X52,X53,X54,X55,X56,X57,X58,X59,
X60,X61,X62,X63,X64,X65,X66,X67,X68,X69,
X70,X71,X72,X73,X74,X75) INSERT_METHOD=LAST;
SELECT ReqF FROM XMerge WHERE EmpName='John';

Give it a Try !!!

Mysql – “Lost connection to MySQL server during query” error

I am running MySQL 5.7.26 on Windows and when doing just a simple query from the MySQL command line client I get that same error you describing. I found out it only occurs when the query length is exact 64 characters! Just by adding a space fixed the problem for me.
It doesn't matter if it's a select or insert/delete/update query.
I have run 66.000 queries on my server and only the queries with exact 64 characters failed.
I checked the database for corruption. But I could also reproduce same problem when making a new database and table on that server.
I could only reproduce this problem on that production server. I created a new server with same version with exact same settings as a VMWare machine but could not recreate. Used the same MySQL client.

Best Answer

Related Solutions

MySQL – Get Column from Multiple Tables

Mysql – “Lost connection to MySQL server during query” error

Related Question