I know when a program crashes, some files creating in /var/crash/. But I don't know what these files are!
Is that error logs?
What are the files located in `/var/crash/`
crashdirectory-structurelogs
Related Solutions
The information you are looking for is not found in usual syslog logs. For viewing performance history from the command line, sysstat is an excellent tool.
With sysstat, the sadc collects system information and writes them to a log file. The log file is a binary format, but can be viewed with the sar command.
Here is an example of sar output with no options:
$ sar
09:15:01 AM CPU %user %nice %system %iowait %steal %idle
10:05:01 AM all 77.49 0.37 22.13 0.00 0.00 0.00
10:15:01 AM all 77.30 0.40 22.29 0.00 0.00 0.00
10:25:01 AM all 77.19 0.38 22.42 0.00 0.00 0.00
10:35:01 AM all 39.31 0.35 23.80 0.01 0.00 36.53
10:45:01 AM all 32.22 0.34 24.26 0.03 0.00 43.15
10:55:01 AM all 32.80 0.33 23.78 0.01 0.00 43.08
11:05:01 AM all 32.70 0.33 23.76 0.00 0.00 43.20
Average: all 63.90 0.39 22.79 0.00 0.00 12.91
The information you see is the same information provided by top, but is historical data. You can also see detailed information about RAM, network, and disk utilization. Here is an example for RAM usage:
$ sar -r
09:15:01 AM kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit
02:15:01 PM 457076 1357116 74.81 277876 810948 205520 5.40
02:25:01 PM 456836 1357356 74.82 277876 811168 205384 5.40
02:35:01 PM 456976 1357216 74.81 277876 811256 204728 5.38
02:45:01 PM 457036 1357156 74.81 277876 811368 204840 5.38
02:55:01 PM 456588 1357604 74.83 277896 811492 204924 5.38
Average: 332452 1481740 81.67 277720 793953 416953 10.96
Outside of running sar locally, there are many monitoring systems that show performance trending data. Munin, cacti, and zabbix are some examples. These have the benefit of graphing and keeping the data for multiple servers in a centralized location.
Update to answer from comments:
The sar command will tell you if you ran out of RAM prior to the crash. This will be obvious as kbbuffers and kbcached will drop dramatically. You can also check dmesg for OOM (out of memory) killer, but dmesg is only written to logs if klogd is installed. You won't see any logs about out of disk space, unless an application specifically reports its failure to write to disk. However, if the disk is full, syslog won't be able to write the log to disk either.
The numbers in the file name normally represent the data for that specific day on the month.
You can extract the data from the files with:
sar -f <filename>
The sa files are data files that can only be read an interrogated using the sar package. The sar files are just ASCII files that can be read with something like cat.
Best Answer
No, they are crash dumps of the kernel, i.e. files that contains useful information to recover the state of the kernel when it was crashing. These files are for your kernel what core dumps are for ordinary executables. There exists an utility that can be used to analyze those files : crash.