Exim Internet Mailer

1. Finding out what Exim processes are doing (exiwhat)

On operating systems that can restart a system call after receiving a signal (most modern OS), an Exim process responds to the SIGUSR1 signal by writing a line describing what it is doing to the file exim-process.info in the Exim spool directory. The exiwhat script sends the signal to all Exim processes it can find, having first emptied the file. It then waits for one second to allow the Exim processes to react before displaying the results. In order to run exiwhat successfully you have to have sufficient privilege to send the signal to the Exim processes, so it is normally run as root.

Warning: This is not an efficient process. It is intended for occasional use by system administrators. It is not sensible, for example, to set up a script that sends SIGUSR1 signals to Exim processes at short intervals.

Unfortunately, the ps command that exiwhat uses to find Exim processes varies in different operating systems. Not only are different options used, but the format of the output is different. For this reason, there are some system configuration options that configure exactly how exiwhat works. If it doesn’t seem to be working for you, check the following compile-time options:

EXIWHAT_PS_CMD     the command for running ps
EXIWHAT_PS_ARG     the argument for ps
EXIWHAT_EGREP_ARG  the argument for egrep to select from ps output
EXIWHAT_KILL_ARG   the argument for the kill command

An example of typical output from exiwhat is

164 daemon: -q1h, listening on port 25
10483 running queue: waiting for 0tAycK-0002ij-00 (10492)
10492 delivering 0tAycK-0002ij-00 to mail.ref.example
  [10.19.42.42] (editor@ref.example)
10592 handling incoming call from [192.168.243.242]
10628 accepting a local non-SMTP message

The first number in the output line is the process number. The third line has been split here, in order to fit it on the page.

2. Selective queue listing (exiqgrep)

This utility is a Perl script contributed by Matt Hubbard. It runs

exim -bpu

or (in case -a switch is specified)

exim -bp

The -C option is used to specify an alternate exim.conf which might contain alternate exim configuration the queue management might be using.

to obtain a queue listing, and then greps the output to select messages that match given criteria. The following selection options are available:

-f <regex>

Match the sender address using a case-insensitive search. The field that is tested is enclosed in angle brackets, so you can test for bounce messages with

exiqgrep -f '^<>$'

-r <regex>

Match a recipient address using a case-insensitive search. The field that is tested is not enclosed in angle brackets.

-s <regex>

Match against the size field.

-y <seconds>

Match messages that are younger than the given time.

-o <seconds>

Match messages that are older than the given time.

-z

Match only frozen messages.

-x

Match only non-frozen messages.

The following options control the format of the output:

-c

Display only the count of matching messages.

-l

Long format – display the full message information as output by Exim. This is the default.

-i

Display message ids only.

-b

Brief format – one line per message.

-R

Display messages in reverse order.

-a

Include delivered recipients in queue listing.

There is one more option, -h, which outputs a list of options.

3. Summarizing the queue (exiqsumm)

The exiqsumm utility is a Perl script which reads the output of exim -bp and produces a summary of the messages on the queue. Thus, you use it by running a command such as

exim -bp | exiqsumm

The output consists of one line for each domain that has messages waiting for it, as in the following example:

3   2322   74m   66m  msn.com.example

Each line lists the number of pending deliveries for a domain, their total volume, and the length of time that the oldest and the newest messages have been waiting. Note that the number of pending deliveries is greater than the number of messages when messages have more than one recipient.

A summary line is output at the end. By default the output is sorted on the domain name, but exiqsumm has the options -a and -c, which cause the output to be sorted by oldest message and by count of messages, respectively. There are also three options that split the messages for each domain into two or more subcounts: -b separates bounce messages, -f separates frozen messages, and -s separates messages according to their sender.

The output of exim -bp contains the original addresses in the message, so this also applies to the output from exiqsumm. No domains from addresses generated by aliasing or forwarding are included (unless the one_time option of the redirect router has been used to convert them into “top level” addresses).

4. Extracting specific information from the log (exigrep)

The exigrep utility is a Perl script that searches one or more main log files for entries that match a given pattern. When it finds a match, it extracts all the log entries for the relevant message, not just those that match the pattern. Thus, exigrep can extract complete log entries for a given message, or all mail for a given user, or for a given host, for example. The input files can be in Exim log format or syslog format. If a matching log line is not associated with a specific message, it is included in exigrep’s output without any additional lines. The usage is:

exigrep [-t<n>] [-I] [-l] [-M] [-v] <pattern> [<log file>] ...

If no log file names are given on the command line, the standard input is read.

The -t argument specifies a number of seconds. It adds an additional condition for message selection. Messages that are complete are shown only if they spent more than <n> seconds on the queue.

By default, exigrep does case-insensitive matching. The -I option makes it case-sensitive. This may give a performance improvement when searching large log files. Without -I, the Perl pattern matches use Perl’s /i option; with -I they do not. In both cases it is possible to change the case sensitivity within the pattern by using (?i) or (?-i).

The -l option means “literal”, that is, treat all characters in the pattern as standing for themselves. Otherwise the pattern must be a Perl regular expression.

The -v option inverts the matching condition. That is, a line is selected if it does not match the pattern.

The -M options means “related messages”. exigrep will show messages that are generated as a result/response to a message that exigrep matched normally.

Example of -M: user_a sends a message to user_b, which generates a bounce back to user_b. If exigrep is used to search for “user_a”, only the first message will be displayed. But if exigrep is used to search for “user_b”, the first and the second (bounce) message will be displayed. Using -M with exigrep when searching for “user_a” will show both messages since the bounce is “related” to or a “result” of the first message that was found by the search term.

If the location of a zcat command is known from the definition of ZCAT_COMMAND in Local/Makefile, exigrep automatically passes any file whose name ends in COMPRESS_SUFFIX through zcat as it searches it. If the ZCAT_COMMAND is not executable, exigrep tries to use autodetection of some well known compression extensions.

5. Selecting messages by various criteria (exipick)

John Jetmore’s exipick utility is included in the Exim distribution. It lists messages from the queue according to a variety of criteria. For details of exipick’s facilities, visit the web page at http://www.exim.org/eximwiki/ToolExipickManPage or run exipick with the --help option.

6. Cycling log files (exicyclog)

The exicyclog script can be used to cycle (rotate) mainlog and rejectlog files. This is not necessary if only syslog is being used, or if you are using log files with datestamps in their names (see section 52.3). Some operating systems have their own standard mechanisms for log cycling, and these can be used instead of exicyclog if preferred. There are two command line options for exicyclog:

• -k <count> specifies the number of log files to keep, overriding the default that is set when Exim is built. The default default is 10.

• -l <path> specifies the log file path, in the same format as Exim’s log_file_path option (for example, /var/log/exim_%slog), again overriding the script’s default, which is to find the setting from Exim’s configuration.

Each time exicyclog is run the file names get “shuffled down” by one. If the main log file name is mainlog (the default) then when exicyclog is run mainlog becomes mainlog.01, the previous mainlog.01 becomes mainlog.02 and so on, up to the limit that is set in the script or by the -k option. Log files whose numbers exceed the limit are discarded. Reject logs are handled similarly.

If the limit is greater than 99, the script uses 3-digit numbers such as mainlog.001, mainlog.002, etc. If you change from a number less than 99 to one that is greater, or vice versa, you will have to fix the names of any existing log files.

If no mainlog file exists, the script does nothing. Files that “drop off” the end are deleted. All files with numbers greater than 01 are compressed, using a compression command which is configured by the COMPRESS_COMMAND setting in Local/Makefile. It is usual to run exicyclog daily from a root crontab entry of the form

1 0 * * * su exim -c /usr/exim/bin/exicyclog

assuming you have used the name “exim” for the Exim user. You can run exicyclog as root if you wish, but there is no need.

7. Mail statistics (eximstats)

A Perl script called eximstats is provided for extracting statistical information from log files. The output is either plain text, or HTML. Exim log files are also supported by the Lire system produced by the LogReport Foundation http://www.logreport.org.

The eximstats script has been hacked about quite a bit over time. The latest version is the result of some extensive revision by Steve Campbell. A lot of information is given by default, but there are options for suppressing various parts of it. Following any options, the arguments to the script are a list of files, which should be main log files. For example:

eximstats -nr /var/spool/exim/log/mainlog.01

By default, eximstats extracts information about the number and volume of messages received from or delivered to various hosts. The information is sorted both by message count and by volume, and the top fifty hosts in each category are listed on the standard output. Similar information, based on email addresses or domains instead of hosts can be requested by means of various options. For messages delivered and received locally, similar statistics are also produced per user.

The output also includes total counts and statistics about delivery errors, and histograms showing the number of messages received and deliveries made in each hour of the day. A delivery with more than one address in its envelope (for example, an SMTP transaction with more than one RCPT command) is counted as a single delivery by eximstats.

Though normally more deliveries than receipts are reported (as messages may have multiple recipients), it is possible for eximstats to report more messages received than delivered, even though the queue is empty at the start and end of the period in question. If an incoming message contains no valid recipients, no deliveries are recorded for it. A bounce message is handled as an entirely separate message.

eximstats always outputs a grand total summary giving the volume and number of messages received and deliveries made, and the number of hosts involved in each case. It also outputs the number of messages that were delayed (that is, not completely delivered at the first attempt), and the number that had at least one address that failed.

The remainder of the output is in sections that can be independently disabled or modified by various options. It consists of a summary of deliveries by transport, histograms of messages received and delivered per time interval (default per hour), information about the time messages spent on the queue, a list of relayed messages, lists of the top fifty sending hosts, local senders, destination hosts, and destination local users by count and by volume, and a list of delivery errors that occurred.

The relay information lists messages that were actually relayed, that is, they came from a remote host and were directly delivered to some other remote host, without being processed (for example, for aliasing or forwarding) locally.

There are quite a few options for eximstats to control exactly what it outputs. These are documented in the Perl script itself, and can be extracted by running the command perldoc on the script. For example:

perldoc /usr/exim/bin/eximstats

8. Checking access policy (exim_checkaccess)

The -bh command line argument allows you to run a fake SMTP session with debugging output, in order to check what Exim is doing when it is applying policy controls to incoming SMTP mail. However, not everybody is sufficiently familiar with the SMTP protocol to be able to make full use of -bh, and sometimes you just want to answer the question “Does this address have access?” without bothering with any further details.

The exim_checkaccess utility is a “packaged” version of -bh. It takes two arguments, an IP address and an email address:

exim_checkaccess 10.9.8.7 A.User@a.domain.example

The utility runs a call to Exim with the -bh option, to test whether the given email address would be accepted in a RCPT command in a TCP/IP connection from the host with the given IP address. The output of the utility is either the word “accepted”, or the SMTP error response, for example:

Rejected:
550 Relay not permitted

When running this test, the utility uses <> as the envelope sender address for the MAIL command, but you can change this by providing additional options. These are passed directly to the Exim command. For example, to specify that the test is to be run with the sender address himself@there.example you can use:

exim_checkaccess 10.9.8.7 A.User@a.domain.example \
                 -f himself@there.example

Note that these additional Exim command line items must be given after the two mandatory arguments.

Because the exim_checkaccess uses -bh, it does not perform callouts while running its checks. You can run checks that include callouts by using -bhc, but this is not yet available in a “packaged” form.

9. Making DBM files (exim_dbmbuild)

The exim_dbmbuild program reads an input file containing keys and data in the format used by the lsearch lookup (see section 9.3). It writes a DBM file using the lower-cased alias names as keys and the remainder of the information as data. The lower-casing can be prevented by calling the program with the -nolc option.

A terminating zero is included as part of the key string. This is expected by the dbm lookup type. However, if the option -nozero is given, exim_dbmbuild creates files without terminating zeroes in either the key strings or the data strings. The dbmnz lookup type can be used with such files.

The program requires two arguments: the name of the input file (which can be a single hyphen to indicate the standard input), and the name of the output file. It creates the output under a temporary name, and then renames it if all went well.

If the native DB interface is in use (USE_DB is set in a compile-time configuration file – this is common in free versions of Unix) the two file names must be different, because in this mode the Berkeley DB functions create a single output file using exactly the name given. For example,

exim_dbmbuild /etc/aliases /etc/aliases.db

reads the system alias file and creates a DBM version of it in /etc/aliases.db.

In systems that use the ndbm routines (mostly proprietary versions of Unix), two files are used, with the suffixes .dir and .pag. In this environment, the suffixes are added to the second argument of exim_dbmbuild, so it can be the same as the first. This is also the case when the Berkeley functions are used in compatibility mode (though this is not recommended), because in that case it adds a .db suffix to the file name.

If a duplicate key is encountered, the program outputs a warning, and when it finishes, its return code is 1 rather than zero, unless the -noduperr option is used. By default, only the first of a set of duplicates is used – this makes it compatible with lsearch lookups. There is an option -lastdup which causes it to use the data for the last duplicate instead. There is also an option -nowarn, which stops it listing duplicate keys to stderr. For other errors, where it doesn’t actually make a new file, the return code is 2.

10. Finding individual retry times (exinext)

A utility called exinext (mostly a Perl script) provides the ability to fish specific information out of the retry database. Given a mail domain (or a complete address), it looks up the hosts for that domain, and outputs any retry information for the hosts or for the domain. At present, the retry information is obtained by running exim_dumpdb (see below) and post-processing the output. For example:

$ exinext piglet@milne.fict.example
kanga.milne.example:192.168.8.1 error 146: Connection refused
  first failed: 21-Feb-1996 14:57:34
  last tried:   21-Feb-1996 14:57:34
  next try at:  21-Feb-1996 15:02:34
roo.milne.example:192.168.8.3 error 146: Connection refused
  first failed: 20-Jan-1996 13:12:08
  last tried:   21-Feb-1996 11:42:03
  next try at:  21-Feb-1996 19:42:03
  past final cutoff time

You can also give exinext a local part, without a domain, and it will give any retry information for that local part in your default domain. A message id can be used to obtain retry information pertaining to a specific message. This exists only when an attempt to deliver a message to a remote host suffers a message-specific error (see section 48.2). exinext is not particularly efficient, but then it is not expected to be run very often.

The exinext utility calls Exim to find out information such as the location of the spool directory. The utility has -C and -D options, which are passed on to the exim commands. The first specifies an alternate Exim configuration file, and the second sets macros for use within the configuration file. These features are mainly to help in testing, but might also be useful in environments where more than one configuration file is in use.

11. Hints database maintenance

Three utility programs are provided for maintaining the DBM files that Exim uses to contain its delivery hint information. Each program requires two arguments. The first specifies the name of Exim’s spool directory, and the second is the name of the database it is to operate on. These are as follows:

• retry: the database of retry information

• wait-<transport name>: databases of information about messages waiting for remote hosts

• callout: the callout cache

• ratelimit: the data for implementing the ratelimit ACL condition

• misc: other hints data

The misc database is used for

• Serializing ETRN runs (when smtp_etrn_serialize is set)

• Serializing delivery to a specific host (when serialize_hosts is set in an smtp transport)

• Limiting the concurrency of specific transports (when max_parallel is set in a transport)

12. exim_dumpdb

The entire contents of a database are written to the standard output by the exim_dumpdb program, which has no options or arguments other than the spool and database names. For example, to dump the retry database:

exim_dumpdb /var/spool/exim retry

Two lines of output are produced for each entry:

T:mail.ref.example:192.168.242.242 146 77 Connection refused
31-Oct-1995 12:00:12 02-Nov-1995 12:21:39 02-Nov-1995 20:21:39 *

The first item on the first line is the key of the record. It starts with one of the letters R, or T, depending on whether it refers to a routing or transport retry. For a local delivery, the next part is the local address; for a remote delivery it is the name of the remote host, followed by its failing IP address (unless retry_include_ip_address is set false on the smtp transport). If the remote port is not the standard one (port 25), it is added to the IP address. Then there follows an error code, an additional error code, and a textual description of the error.

The three times on the second line are the time of first failure, the time of the last delivery attempt, and the computed time for the next attempt. The line ends with an asterisk if the cutoff time for the last retry rule has been exceeded.

Each output line from exim_dumpdb for the wait-xxx databases consists of a host name followed by a list of ids for messages that are or were waiting to be delivered to that host. If there are a very large number for any one host, continuation records, with a sequence number added to the host name, may be seen. The data in these records is often out of date, because a message may be routed to several alternative hosts, and Exim makes no effort to keep cross-references.

13. exim_tidydb

The exim_tidydb utility program is used to tidy up the contents of a hints database. If run with no options, it removes all records that are more than 30 days old. The age is calculated from the date and time that the record was last updated. Note that, in the case of the retry database, it is not the time since the first delivery failure. Information about a host that has been down for more than 30 days will remain in the database, provided that the record is updated sufficiently often.

The cutoff date can be altered by means of the -t option, which must be followed by a time. For example, to remove all records older than a week from the retry database:

exim_tidydb -t 7d /var/spool/exim retry

Both the wait-xxx and retry databases contain items that involve message ids. In the former these appear as data in records keyed by host – they were messages that were waiting for that host – and in the latter they are the keys for retry information for messages that have suffered certain types of error. When exim_tidydb is run, a check is made to ensure that message ids in database records are those of messages that are still on the queue. Message ids for messages that no longer exist are removed from wait-xxx records, and if this leaves any records empty, they are deleted. For the retry database, records whose keys are non-existent message ids are removed. The exim_tidydb utility outputs comments on the standard output whenever it removes information from the database.

Certain records are automatically removed by Exim when they are no longer needed, but others are not. For example, if all the MX hosts for a domain are down, a retry record is created for each one. If the primary MX host comes back first, its record is removed when Exim successfully delivers to it, but the records for the others remain because Exim has not tried to use those hosts.

It is important, therefore, to run exim_tidydb periodically on all the hints databases. You should do this at a quiet time of day, because it requires a database to be locked (and therefore inaccessible to Exim) while it does its work. Removing records from a DBM file does not normally make the file smaller, but all the common DBM libraries are able to re-use the space that is released. After an initial phase of increasing in size, the databases normally reach a point at which they no longer get any bigger, as long as they are regularly tidied.

Warning: If you never run exim_tidydb, the space used by the hints databases is likely to keep on increasing.

14. exim_fixdb

The exim_fixdb program is a utility for interactively modifying databases. Its main use is for testing Exim, but it might also be occasionally useful for getting round problems in a live system. It has no options, and its interface is somewhat crude. On entry, it prompts for input with a right angle-bracket. A key of a database record can then be entered, and the data for that record is displayed.

If “d” is typed at the next prompt, the entire record is deleted. For all except the retry database, that is the only operation that can be carried out. For the retry database, each field is output preceded by a number, and data for individual fields can be changed by typing the field number followed by new data, for example:

> 4 951102:1000

resets the time of the next delivery attempt. Time values are given as a sequence of digit pairs for year, month, day, hour, and minute. Colons can be used as optional separators.

15. Mailbox maintenance (exim_lock)

The exim_lock utility locks a mailbox file using the same algorithm as Exim. For a discussion of locking issues, see section 26.3. Exim_lock can be used to prevent any modification of a mailbox by Exim or a user agent while investigating a problem. The utility requires the name of the file as its first argument. If the locking is successful, the second argument is run as a command (using C’s system() function); if there is no second argument, the value of the SHELL environment variable is used; if this is unset or empty, /bin/sh is run. When the command finishes, the mailbox is unlocked and the utility ends. The following options are available:

-fcntl

Use fcntl() locking on the open mailbox.

-flock

Use flock() locking on the open mailbox, provided the operating system supports it.

-interval

This must be followed by a number, which is a number of seconds; it sets the interval to sleep between retries (default 3).

-lockfile

Create a lock file before opening the mailbox.

-mbx

Lock the mailbox using MBX rules.

-q

Suppress verification output.

-retries

This must be followed by a number; it sets the number of times to try to get the lock (default 10).

-restore_time

This option causes exim_lock to restore the modified and read times to the locked file before exiting. This allows you to access a locked mailbox (for example, to take a backup copy) without disturbing the times that the user subsequently sees.

-timeout

This must be followed by a number, which is a number of seconds; it sets a timeout to be used with a blocking fcntl() lock. If it is not set (the default), a non-blocking call is used.

-v

Generate verbose output.

If none of -fcntl, -flock, -lockfile or -mbx are given, the default is to create a lock file and also to use fcntl() locking on the mailbox, which is the same as Exim’s default. The use of -flock or -fcntl requires that the file be writeable; the use of -lockfile requires that the directory containing the file be writeable. Locking by lock file does not last for ever; Exim assumes that a lock file is expired if it is more than 30 minutes old.

The -mbx option can be used with either or both of -fcntl or -flock. It assumes -fcntl by default. MBX locking causes a shared lock to be taken out on the open mailbox, and an exclusive lock on the file /tmp/.n.m where n and m are the device number and inode number of the mailbox file. When the locking is released, if an exclusive lock can be obtained for the mailbox, the file in /tmp is deleted.

The default output contains verification of the locking that takes place. The -v option causes some additional information to be given. The -q option suppresses all output except error messages.

A command such as

exim_lock /var/spool/mail/spqr

runs an interactive shell while the file is locked, whereas

exim_lock -q /var/spool/mail/spqr <<End
<some commands>
End

runs a specific non-interactive sequence of commands while the file is locked, suppressing all verification output. A single command can be run by a command such as

exim_lock -q /var/spool/mail/spqr \
  "cp /var/spool/mail/spqr /some/where"

Note that if a command is supplied, it must be entirely contained within the second argument – hence the quotes.