GIT – Any  system will generate many log files by default, containing various information about the operation of the system (normal actions, debugging information, security/authorization messages, web/email events, etc). If no rotation would occur on the various log files, then they will just grow bigger and bigger, filling up the space (on high traffic sites) but most importantly making it very difficult to find any information that we might be looking for in those log files. Fortunately this is handled in most Linux distributions by defaultand we don’t have anything special to do to set it up… if will function out of the box, rotating the log files by default.

This little how-to will show you how the default log rotation works, based on syslog andlogrotate. There are other ways to achieve this, like using syslog-ng instead of syslog that I will cover in a future article.
Understanding how the default setup works, will help you have an idea of what will happen andwhat logs will be rotated, when will this occur, and how long will they be kept. Also it will show you the places where you can make changes in case you need to do that (if you want to save some log for a longer time, or if you want to rotate it differently from the default). The examples I will present are taken from a  system, so if you are running a different system, they might differ a little, but not drastically (like you might have the cron setup to run at a different time, or keep a different number of log iterations by default).

I have split this article in two parts based on the way how the log rotation is handled:

  • system log files: most of the system log files are rotated by syslog itself and not using logrotate. You will see here what are these log files, and how they are rotated.
  • application log fileslogrotate is the default choice to rotate all the other log files. It can rotate the logs based on various parameters: daily, weekly, monthly, based on the size of the log, it can compress the logs to save space, etc.

Rotating Linux Log Files – Part 1: syslog

Syslog is the default logging application installed in most Linux distributions. It can be replaced with syslog-ng for better functionality, but about this in a future article. As I explained in the introduction, the log files that are managed by syslog are not rotated with logrotate, but bysyslog itself. In the second part I will cover the log files that are handled by logrotate.

What files are handled by syslog? We can find out what are those files simply by inspecting the syslog configuration file (/etc/syslog.conf) that defines each log file, and also what kind of information is saved to each particular file. Let’s see how the configuration file looks on a fresh Debian system (I have removed most of the comments and kept only the relevant log definitions):

# /etc/syslog.conf Configuration file for syslogd.
#
# For more information see syslog.conf(5)
# manpage.

auth,authpriv.* /var/log/auth.log
*.*;auth,authpriv.none -/var/log/syslog
#cron.* /var/log/cron.log
daemon.* -/var/log/daemon.log
kern.* -/var/log/kern.log
lpr.* -/var/log/lpr.log
mail.* -/var/log/mail.log
user.* -/var/log/user.log
uucp.* /var/log/uucp.log

mail.info -/var/log/mail.info
mail.warn -/var/log/mail.warn
mail.err /var/log/mail.err
news.crit /var/log/news/news.crit
news.err /var/log/news/news.err
news.notice -/var/log/news/news.notice
*.=debug;
auth,authpriv.none;
news.none;mail.none -/var/log/debug
*.=info;*.=notice;*.=warn;
auth,authpriv.none;
cron,daemon.none;
mail,news.none -/var/log/messages

Normally, I would change this and configure based on my preferences, but his is not the point now. As you can see there are various log files that will contain the information specified by the configured facility (authpriv, cron, daemon, ftp, kern, lpr, mail, mark, news, security, syslog, user, uucp and local0 through local7).

How are these files rotated? As I previously said this is handled by syslog itself, and it is done using 2 cron scripts: daily and weekly.

daily rotation: (handled by /etc/cron.daily/sysklogd)

  • any files that contains the *.* facility in the syslog configuration are rotated daily. The reason for this, is that they will log all the information regardless of the facility, and can become quite quickly very big.
  • if we will look inside the daily syslog cron we will see that it finds the logs it need to rotate by launching the file syslogd-listfiles:
    /usr/sbin/syslogd-listfiles 
    /var/log/syslog <- the result on my default system 
  • the actual rotation is handled by the savelog program as it can be seen in this line:
    savelog -g adm -m 640 -u  -c 7 $LOG >/dev/null

So we can see here that by default my debian system will keep 7 archives of previous logs (7 days). If I would want to change this, then all I have to do is to replace the -c 7 parameter with what I need.
When does this rotation occur? Since it is launched from /etc/cron.daily/ it is defined in the system wide crontab:

# /etc/crontab: system-wide crontab
...
25 6 * * * root test -x /usr/sbin/anacron || run-parts --report /etc/cron.daily
47 6 * * 7 root test -x /usr/sbin/anacron || run-parts --report /etc/cron.weekly
so based on the default cron job, this will be done daily at 6:25AM.

weekly rotation: (handled by /etc/cron.weekly/sysklogd)

  • the rest of the syslog generated log files (different from *.* facility) will be rotatedweekly.

if we will look inside the weekly syslog cron we will see it finds the logs it needs to rotate by running:

/usr/sbin/syslogd-listfiles –weekly
/var/log/mail.warn
/var/log/uucp.log
/var/log/user.log
/var/log/daemon.log
/var/log/messages
/var/log/debug
/var/log/auth.log
/var/log/mail.err
/var/log/mail.log
/var/log/kern.log
/var/log/lpr.log
/var/log/mail.info

  • as we can see all the logs defined in the syslog configuration file will appear, except the news.*, that can be included by adding syslogd-listfiles –news, if needed.
  • the rotation is again handled by the savelog program:
    savelog -g adm -m 640 -u root -c 4 $LOG >/dev/null

So by default it will keep 4 archives of old logs (without counting the current log); the archives will have the extension: *.0-*.3 (with the first archive not compressed by default). If I would want to change this, I would need to modify accordingly the -c 4 parameter based on my needs.
As seen above in the system crontab, this rotation will take place at 6:47AM each Sunday (the weekly cronjob).

For example, the rotated log files for the messages log file, will look like this:

/var/log/messages /var/log/messages.0 /var/log/messages.1.gz /var/log/messages.2.gz /var/log/messages.3.gz 

Note: On RedHat based systems (RHEL, , Fedora, etc.) the functionality covered above doesn’t exist by default (even though I don’t see why it could not be implemented if someone wants it). On these operating systems, this is handled also by logrotate as shown in the next part.
This covers the basics on how system logs are rotated. In part 2 we will be looking at how application logs are rotated.

 

Rotating Linux Log Files – Part 2: logrotate

logrotate is the default application used to rotate all other log files not handled by syslog itself (details on rotating system log files can be found in part 1 of the article). It allows automatic rotation, compression, removal, and mailing of log files. Each log file may be handled daily, weekly, monthly, or when it grows too large.

Normally, logrotate is run as a daily cron job. Let’s look into the script that was installed in /etc/cron.daily for this:

cat /etc/cron.daily/logrotate 
#!/bin/sh 
test -x /usr/sbin/logrotate || exit 0 
/usr/sbin/logrotate /etc/logrotate.conf

Logrotate will look into /etc/logrotate.conf for its configuration directives.

cat /etc/logrotate.conf
# see “man logrotate” for details
# rotate log files weekly
weekly

# keep 4 weeks worth of backlogs
rotate 4

# create new (empty) log files after rotating old ones
create

# uncomment this if you want your log files compressed
#compress

# packages drop log rotation information into this directory
include /etc/logrotate.d

# no packages own wtmp, or btmp — we’ll rotate them here
/var/log/wtmp {
missingok
monthly
create 0664 root utmp
rotate 1
}

/var/log/btmp {
missingok
monthly
create 0664 root utmp
rotate 1
}

# system-specific logs may be configured here

So we can see it defines some default parameters (weekly, rotate 4, create, compress) and includes all the files from /etc/logrotate.d/. Also it defines the rotation for some files that are not handled by syslog itself, like wtmp. For example, I would want to keep more than one month of old wtmp logs, then I would have to change the parameter rotate 1.

Inside the /etc/logrotate.d/ various packages will install their own configuration file that will ensure their logs are properly rotate (on my fresh Debian install I have the following files:acpid apache2 aptitude base-config dpkg exim4-base). As long as you don’t change the paths to those logs the rotation will work out of the box. But in case you change them you might want to look inside this folder and make the proper adjustments to the log file definitions, to assure they will be rotated. For example, let’s look at the apache rotation file created here by the apache2 package:

cat /etc/logrotate.d/apache2
/var/log/apache2/*.log {
weekly
missingok
rotate 52
compress
delaycompress
notifempty
create 640 root adm
sharedscripts
postrotate
if [ -f /var/run/apache2.pid ]; then
/etc/init.d/apache2 restart > /dev/null
fi
endscript
}

We can see that by default it will rotate apache logs found in /var/log/apache2/ that have the extension *.log, on a weekly basis and keep 52 archives (about 1 year) of the old data. Once the rotation is completed it will restart the apache daemon. You can check logrotate manual page for all the available parameters, as they are self-explanatory.
Now, if I would like to keep my own apache log files in a different location (/var/weblogs for example) and rotate them monthly then I will need to make the following changes:

/var/weblogs/*.log { monthly ...

Probably, I will also want to change the default hour when the daily cron is running to have it on midnight. Anyway this is just an example and you will most certainly configure this based on your needs.

Even though I didn’t intended with this article to describe what each configuration parameter of logrotate means (as you can easily find out yourself), but to show what is the logic and its functionality, I would like to add that while configuring and testing this you might find very useful the debug option:

logrotate -d file

This will show you what it will do, without actually rotating anything, and this is most valuable while testing complex setups that you don’t want to ‘play’ with the logs to see if your configuration will work as you want it.

Also logrotate -f file will force the rotation even if that would have normally not occurred (logrotate will only assume it need to run and rotate logs once per day).

Note: as mentioned also in part 1, RedHat based systems (RHEL, Centos, Fedora, etc.) will also rotate by default the ‘system logs’ using logrotate and not syslog’s internal method asDebian systems. This is handled by default with the logrotate configuration file:

cat /etc/logrotate.d/syslog
/var/log/messages

/var/log/secure

/var/log/maillog

/var/log/spooler

/var/log/boot.log

/var/log/cron {
sharedscripts
postrotate
/bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
endscript
}

The sharedscripts parameter means that the postrotate script will only be run once (after the old logs have been compressed), not once for each log which is rotated.
So nothing special defined here, besides the log files that will be rotated, and it will use the defaults from /etc/logrotate.conf.

Print Friendly

Comments

comments

Bài viết liên quan