MySQL does have the nasty habit being swap-happy. Jeremy Cole has best addressed this in his blog : http://blog.jcole.us/2012/04/16/a-brief-update-on-numa-and-mysql/. From that blog, you learn that there is something you can do: Add numactl --interleave=all inside /etc/init.d/mysql.

SUGGESTIONS

If the Server is dedicated to doing only MySQL, please change the following in /etc/my.cnf:

[mysqld]
innodb_open_files=1000
innodb_flush_method=O_DIRECT
innodb_buffer_pool_size=768M
innodb_log_file_size=192M

If the server is at least dual-core, add these

innodb_buffer_pool_instances=2
innodb_read_io_threads=16
innodb_write_io_threads=16
innodb_io_capacity=2000

Next, login to mysql run SET GLOBAL innodb_fast_shutdown = 0;

Next, run the following in the OS

cd /var/lib/mysql
service mysql stop
mv ib_logfile0 ib_logfile0.bak
mv ib_logfile1 ib_logfile1.bak
service mysql start

Give it a Try !!!

UPDATE 2012-12-31 08:30 EDT

From your last comment

It stopped climbing around 1 gb. I removed unused databases and it just seems mysql 5.5 stores a lot of data in memory as this didn't happen in 5.0. Did mysql change a lot?

Yes, MySQL has changed a lot. In fact, there are many cases where upgrading from MySQL 5.0 to MySQL 5.5 and resulted in performance degradation. InnoDB 5.5 is now equipped to do hyperthreading and multicore engagement.

Percona actually tested this out awhile ago.

Please read me past posts on this subject

• Jul 16, 2012 : decreased performance of stored procedure when migrated from mysql server 5.0 to 5.5
• Nov 24, 2011 : Why mysql 5.5 slower than 5.1 (linux,using mysqlslap)
• Oct 05, 2011 : Query runs a long time in some newer MySQL versions
• Sep 20, 2011 : Multi cores and MySQL Performance
• Jun 19, 2011 : How do I properly perform a MySQL bake-off?

I also wrote about this in ServerFault and StackOverflow

• Feb 22, 2012 : https://serverfault.com/questions/362388/mysql-5-1-vs-mysql-5-5-5-1-twice-as-fast/362706#362706
• Feb 08, 2012 : https://stackoverflow.com/questions/9202825/mysql-5-5-perfomance/9202860#9202860

Workarounds

BigDump http://www.ozerov.de/bigdump/

It executes only a small part of the huge dump and restarts itself. The next session starts where the last was stopped to keep you from running into your server’s limits. Instructions for using it are on the BigDump website. Basically you place your SQL file in a folder on your server, along with the bigdump.php file. You edit that file with your database information and then visit the page on your server and set the import to go. This is a fairly quick process and will save you a lot of time.

SQLDumpSpliter http://www.rusiczki.net/2007/01/24/sql-dump-file-splitter/

Create the splits and then upload them to a directory on your server.

If you want to restore the dump, you have to execute the yourdatabase_DataStructure.sql first because it contains the Tables structure. After that, you can execute the other .sql-files as they contain the data of the now existing tables. Using SSH, CD to your directory again and make sure to send this first:

mysql -u db_user -p db_name < yourbackup_DataStructure.sql

Then your splits:

mysql -u db_user -p db_name < yourbackup_1.sql
mysql -u db_user -p db_name < yourbackup_2.sql

Source: http://www.ontwerps.nl/methods-importing-large-sql-files