Mysql – How to store logs correctly

innodbMySQLmysql-5.5

(This is somewhat a follow up on my last question)

I am using MySQL 5.5 with innodb.

My Log table consists of two columns. One for user number, and the other for a string consisting of IP, request method, request body, and so on. The two columns use no index.

Usually the content of the logs are below 200 letters. However, there are special cases that the content go up to more than 5000 words.

I've thought of three ways to log long contents. First is to split the content into 200 words and put it in several rows. Second is to set the data type to TEXT or VARCHAR(10000). Third is to make a new table only for storing long contents.

What is the normal approach to storing logs?

Best Answer

I would not use any of those options: I would properly split out each part of the log entry, putting each in its own column of appropriate maximum length. This is more work of course as you need to break the data down for insert if you are getting it as a single string (it'll also consume a bit more disk space, probably) but it will greatly increase the analysis you can do on the logs later.

Also define useful indexes. At very least something on that ordering column and/or if there is a date+time in those logs there should be an index covering that.

If you are just storing logs as blocks of text per line then there is very little point pushing them into the database: just keep them in their original flat files where you can use grep and other such tools. If you want to take advantage of the database's features at all, create a table optimised for making best use of the DB instead of optimising solely for easiest insert convenience.

You could also break down the consistent parts of query strings, form parts, and cookies (I'm assuming these are HTTP(S) logs) into their own sub-tables, but unlike separating basic log variables that is probably going to be considerable overkill.