Mysql – Importing CSV file to MySQL via PHPMyAdmin

Assuming you can't further increase the memory size then there are a couple of options.

• If you aren't familiar with the command line and you really want to stick with a GUI style import then you can use BigDump (http://www.ozerov.de/bigdump/). I used it once but it's been a while. From what I remember you will download a file named bigdump.php (with some instructions) and put it on your webserver in the directory with the MySQL DB dump file that is too large to import via PHPMYADMIN. Then navigate to it using your browser - something like http://your-website.com/bigdump.php.

• If you are familiar with the command line and using a Linux based system then you can use code like like this:

  mysql -u USERNAME -p DATABASENAME < FILENAME.sql

The database and user (with privileges to the database) will need to exist prior to running this command though. Note I copied the above command from another source. I always do my dumps and restores like this: DUMP: mysqldump -u DB_USER -h DB_HOST -pDB_PASSWORD DB_NAME | gzip -9 > DB_NAME.sql.gz RESTORE: gunzip < PATH_TO_DUMP/DB_NAME.sql.gz | mysql -u DB_USER -pDB_PASSWORD DB_NAME

• Finally, and most painfully, you could choose to dump individual tables or groups of tables into small enough dumps. Then restore these individual dumps one at a time via PHPMYADMIN.

FTP & RESTORE instructions:

Download FileZilla here: http://filezilla-project.org/download.php?type=client

Typically you can FTP into your server (especially a shared host) using the same login and password you would for SSH (you command line login). Just make sure to set up the connection to your server properly in FileZilla.

Once connected your default path will likely be your $HOME folder (on nix machines). If you aren't planning on doing a lot of FTPing just dump what you want in there. You can always move it from the command line.

If you FTP'd a MySQL Dump into your $HOME folder then the RESTORE command for a DB named mydb and a db dump file named mydb.sql.gz (in the $HOME folder) would be:

gunzip < $HOME/mydb.sql.gz | mysql -u DB_USER -pDB_PASSWORD mydb

where you would obviously need change the DB_USER and DB_PASSWORD values to what user and password you setup for the DB.

Remember:

• The DB must already exist (but be empty) before the RESTORE command is executed
• The DB_USER and DB_PASSWORD must be setup for the DB before the RESTORE command (you can do this from PHPMYADMIN - Privileges tab)
• When you set up DB_USER from PHPMYADMIN it will default to only allowing DB_USER to access the DB from localhost (the server the DB is hosted on). I'm assuming this is fine for your situation grant. When you get into more advanced DUMPs and RESTOREs you may want to do it from another machine entirely which isn't hard but requires a couple more pieces of information be provided and DB_USER with the proper privileges.

FTP the DB and RESTORE it... Step by Step

1. Locate zipped MySQL dump file on your desktop (Example: C:\Web\DB\Dumps\mydb.sql.zip)
2. Download and install FileZilla
3. In FileZilla go to File->Site Manager and click on the New Site button (lower left quadrant of GUI). Set the host to a domain (Example: dest.com) on that resides on your DESTINATION server. Set Protocol to FTP or SFTP (this depends on your hosting, ask them). Set the User and Password (Example: User = grant, Password = grant's password) - if the FTP protocol is SFTP then any set of valid SSH login credentials should work.
4. (In FileZilla) Click Connect (bottom left most button)
5. (In FileZilla) Once connected to the remote server (should be your DESTINATION server) use the left side of FileZilla to navigate to the directory where the dump file you located in Step 1 resides. (The path for the local machine is on the far left, about half way down and it is labeled Local site:. In our example the textbox next to Local site: would read C:\Web\DB\Dumps.)
6. (In FileZilla) Locate the Remote site: label directly to the right of Local Site:.
7. (In FileZilla) There is no technical reason you can't put the DB DUMP in the folder tha FileZilla opens up to. It will likely default to your $HOME directory. If you don't have any reason to put the DB Dump in a specific directory - such as for your own organizational purposes - then leave the path on the Remote site alone. Write down that path (Linux Example: /usr/grant).
8. (In FileZilla) Locate the file on your local machine on the left half of FileZilla just under the Tree shown beneath the Local site: label you found in Step 5. Double click on that file and FileZilla will FTP it to your Remote Site.
9. Within PHPMYADMIN on your DESTINATION server, create a new database (type mydb into the Create new database textbox and Click the Create button).
10. (In PHPMYADMIN) Choose mydb from the menu on the left.
11. (In PHPMYADMIN) Click on the Privileges tab (upper right). Once on that page click on Add New User.
12. (In PHPMYADMIN->Privileges->Add New User) Provide a username (Example: mydb_grant) and password (Example: mydb_password). In the Host dropdown choose Local. Make sure that the radio button next to "Grant all privileges on database" is selected. Then click Go (bottom right of screen).
13. SSH into your DESTINATION server.
14. (In your SSH terminal) Navigate to the directory your wrote down in Step 7. Using a command like - cd $HOME or cd /usr/grant. (On Linux machines $HOME is an environment variable that will equal something like /usr/grant.)
15. (In your SSH terminal) Execute the following command: gunzip < /usr/grant/mydb.sql.gz | mysql -u mydb_grant -pmydb_password mydb (there is no space between -p and the password because that is the way it needs to be, that is not a typo).

Once that last command completes your DB has been RESTORED. I don't recommend using the DB user and password I provided simply replace them with more secure choices of your own.

DUMP and RESTORE on different servers without FTP

If you wish to skip the FTP stuff and don't mind redoing the DUMP then you can do it this way:

1. Login into the DESTINATION server (where you want the DB to reside - we'll call it destination.com)
2. Create the DB, User, and proper privileges as mentioned above on the DESTINATION server
3. Make sure the privileges on the SOURCE server allows for SOURCE_DB_USER to access the DB remotely - either allow access from dest.com or from % (% - everyone and everywhere if they have the right username and password)
4. On the DESTINATION server run: mysqldump -u SOURCE_DB_USER -h source.com -pSOURCE_DB_PASSWORD mydb| gzip -9 > $HOME/mydb.sql.gz where source.com is the SOURCE server (where the DB currently resides)
5. Then (on the DESTINATION server) run: gunzip < $HOME/mydb.sql.gz | mysql -u DEST_DB_USER -pDEST_DB_PASSWORD mydb

That should be all you need to do to move the DB from one server to another. Caveats are that I've only ever done this on my own dedicated Ubuntu (with standard LAMP) and Dreamhost's VPS servers (which I believe run CentOS but I could be wrong).

If your table has a table with more than 1000 columns, it cannot be converted to InnoDB. In that case, run this query

SELECT CEILING(SUM(index_length)/POWER(1024,2)) num
FROM information_schema.tables WHERE engine='MyISAM';

This will give you the correct size for key_buffer_size in MB.

Since you are doing an UPSERT, you should set concurrent_insert to 2 to make INSERTs go faster. You may want to consider changing the table's row format to Fixed. I wrote about why to do both in StackOverflow. In essence, if you make the table's row format Fixed, all table rows are the same size. Thus, INSERTs and UPDATEs would operate on the exact same length of data. Management of row access is far more reasonable.

Since MyISAM only caches indexes (in the key buffer), all data must be read from disk. anything you can do to getting better RAID performance (as asked by @TomTom) would help your cause as well.