Exim Internet Mailer

1. Overall philosophy

Exim is designed to work efficiently on systems that are permanently connected to the Internet and are handling a general mix of mail. In such circumstances, most messages can be delivered immediately. Consequently, Exim does not maintain independent queues of messages for specific domains or hosts, though it does try to send several messages in a single SMTP connection after a host has been down, and it also maintains per-host retry information.

2. Policy control

Policy controls are now an important feature of MTAs that are connected to the Internet. Perhaps their most important job is to stop MTAs from being abused as “open relays” by misguided individuals who send out vast amounts of unsolicited junk and want to disguise its source. Exim provides flexible facilities for specifying policy controls on incoming mail:

• Exim 4 (unlike previous versions of Exim) implements policy controls on incoming mail by means of Access Control Lists (ACLs). Each list is a series of statements that may either grant or deny access. ACLs can be used at several places in the SMTP dialogue while receiving a message from a remote host. However, the most common places are after each RCPT command, and at the very end of the message. The sysadmin can specify conditions for accepting or rejecting individual recipients or the entire message, respectively, at these two points (see chapter 44). Denial of access results in an SMTP error code.

• An ACL is also available for locally generated, non-SMTP messages. In this case, the only available actions are to accept or deny the entire message.

• When Exim is compiled with the content-scanning extension, facilities are provided in the ACL mechanism for passing the message to external virus and/or spam scanning software. The result of such a scan is passed back to the ACL, which can then use it to decide what to do with the message.

• When a message has been received, either from a remote host or from the local host, but before the final acknowledgment has been sent, a locally supplied C function called local_scan() can be run to inspect the message and decide whether to accept it or not (see chapter 46). If the message is accepted, the list of recipients can be modified by the function.

• Using the local_scan() mechanism is another way of calling external scanner software. The SA-Exim add-on package works this way. It does not require Exim to be compiled with the content-scanning extension.

• After a message has been accepted, a further checking mechanism is available in the form of the system filter (see chapter 47). This runs at the start of every delivery process.

3. User filters

In a conventional Exim configuration, users are able to run private filters by setting up appropriate .forward files in their home directories. See chapter 22 (about the redirect router) for the configuration needed to support this, and the separate document entitled Exim’s interfaces to mail filtering for user details. Two different kinds of filtering are available:

• Sieve filters are written in the standard filtering language that is defined by RFC 3028.

• Exim filters are written in a syntax that is unique to Exim, but which is more powerful than Sieve, which it pre-dates.

User filters are run as part of the routing process, described below.

4. Message identification

Every message handled by Exim is given a message id which is 23 characters long. It is divided into three parts, separated by hyphens, for example 16VDhn-000000001bo-D342. Each part is a sequence of letters and digits, normally encoding numbers in base 62. However, in the Darwin operating system (Mac OS X) and when Exim is compiled to run under Cygwin, base 36 (avoiding the use of lower case letters) is used instead, because the message id is used to construct filenames, and the names of files in those systems are not always case-sensitive.

The detail of the contents of the message id have changed as Exim has evolved. Earlier versions relied on the operating system not re-using a process id (pid) within one second. On modern operating systems, this assumption can no longer be made, so the algorithm had to be changed. To retain backward compatibility, the format of the message id was retained, which is why the following rules are somewhat eccentric:

• The first six characters of the message id are the time at which the message started to be received, to a granularity of one second. That is, this field contains the number of seconds since the start of the epoch (the normal Unix way of representing the date and time of day).

• After the first hyphen, the next

  eleven

  characters are the id of the process that received the message.

• There are two different possibilities for the final four characters:

  1. If localhost_number is not set, this value is the fractional part of the time of reception, normally in units of microseconds. but for systems that must use base 36 instead of base 62 (because of case-insensitive file systems), the units are 2 us.

  2. If localhost_number is set, it is multiplied by 500000 (250000) and added to the fractional part of the time, which in this case is in units of 2 us (4 us).

After a message has been received, Exim waits for the clock to tick at the appropriate resolution before proceeding, so that if another message is received by the same process, or by another process with the same (re-used) pid, it is guaranteed that the time will be different. In most cases, the clock will already have ticked while the message was being received.

The exim_msgdate utility (see section 54.16) can be used to display the date, and optionally the process id, of an Exim Message ID.

5. Receiving mail

The only way Exim can receive mail from another host is using SMTP over TCP/IP, in which case the sender and recipient addresses are transferred using SMTP commands. However, from a locally running process (such as a user’s MUA), there are several possibilities:

• If the process runs Exim with the -bm option, the message is read non-interactively (usually via a pipe), with the recipients taken from the command line, or from the body of the message if -t is also used.

• If the process runs Exim with the -bS option, the message is also read non-interactively, but in this case the recipients are listed at the start of the message in a series of SMTP RCPT commands, terminated by a DATA command. This is called “batch SMTP” format, but it isn’t really SMTP. The SMTP commands are just another way of passing envelope addresses in a non-interactive submission.

• If the process runs Exim with the -bs option, the message is read interactively, using the SMTP protocol. A two-way pipe is normally used for passing data between the local process and the Exim process. This is “real” SMTP and is handled in the same way as SMTP over TCP/IP. For example, the ACLs for SMTP commands are used for this form of submission.

• A local process may also make a TCP/IP call to the host’s loopback address (127.0.0.1) or any other of its IP addresses. When receiving messages, Exim does not treat the loopback address specially. It treats all such connections in the same way as connections from other hosts.

In the three cases that do not involve TCP/IP, the sender address is constructed from the login name of the user that called Exim and a default qualification domain (which can be set by the qualify_domain configuration option). For local or batch SMTP, a sender address that is passed using the SMTP MAIL command is ignored. However, the system administrator may allow certain users (“trusted users”) to specify a different sender addresses unconditionally, or all users to specify certain forms of different sender address. The -f option or the SMTP MAIL command is used to specify these different addresses. See section 5.2 for details of trusted users, and the untrusted_set_sender option for a way of allowing untrusted users to change sender addresses.

Messages received by either of the non-interactive mechanisms are subject to checking by the non-SMTP ACL if one is defined. Messages received using SMTP (either over TCP/IP or interacting with a local process) can be checked by a number of ACLs that operate at different times during the SMTP session. Either individual recipients or the entire message can be rejected if local policy requirements are not met. The local_scan() function (see chapter 46) is run for all incoming messages.

Exim can be configured not to start a delivery process when a message is received; this can be unconditional, or depend on the number of incoming SMTP connections or the system load. In these situations, new messages wait on the queue until a queue runner process picks them up. However, in standard configurations under normal conditions, delivery is started as soon as a message is received.

6. Handling an incoming message

When Exim accepts a message, it writes two files in its spool directory. The first contains the envelope information, the current status of the message, and the header lines, and the second contains the body of the message. The names of the two spool files consist of the message id, followed by -H for the file containing the envelope and header, and -D for the data file.

By default, all these message files are held in a single directory called input inside the general Exim spool directory. Some operating systems do not perform very well if the number of files in a directory gets large; to improve performance in such cases, the split_spool_directory option can be used. This causes Exim to split up the input files into 62 sub-directories whose names are single letters or digits. When this is done, the queue is processed one sub-directory at a time instead of all at once, which can improve overall performance even when there are not enough files in each directory to affect file system performance.

The envelope information consists of the address of the message’s sender and the addresses of the recipients. This information is entirely separate from any addresses contained in the header lines. The status of the message includes a list of recipients who have already received the message. The format of the first spool file is described in chapter 57.

Address rewriting that is specified in the rewrite section of the configuration (see chapter 31) is done once and for all on incoming addresses, both in the header lines and the envelope, at the time the message is accepted. If during the course of delivery additional addresses are generated (for example, via aliasing), these new addresses are rewritten as soon as they are generated. At the time a message is actually delivered (transported) further rewriting can take place; because this is a transport option, it can be different for different forms of delivery. It is also possible to specify the addition or removal of certain header lines at the time the message is delivered (see chapters 15 and 24).

7. Life of a message

A message remains in the spool directory until it is completely delivered to its recipients or to an error address, or until it is deleted by an administrator or by the user who originally created it. In cases when delivery cannot proceed – for example when a message can neither be delivered to its recipients nor returned to its sender, the message is marked “frozen” on the spool, and no more deliveries are attempted.

An administrator can “thaw” such messages when the problem has been corrected, and can also freeze individual messages by hand if necessary. In addition, an administrator can force a delivery error, causing a bounce message to be sent.

There are options called ignore_bounce_errors_after and timeout_frozen_after, which discard frozen messages after a certain time. The first applies only to frozen bounces, the second to all frozen messages.

While Exim is working on a message, it writes information about each delivery attempt to its main log file. This includes successful, unsuccessful, and delayed deliveries for each recipient (see chapter 53). The log lines are also written to a separate message log file for each message. These logs are solely for the benefit of the administrator and are normally deleted along with the spool files when processing of a message is complete. The use of individual message logs can be disabled by setting no_message_logs; this might give an improvement in performance on very busy systems.

All the information Exim itself needs to set up a delivery is kept in the first spool file, along with the header lines. When a successful delivery occurs, the address is immediately written at the end of a journal file, whose name is the message id followed by -J. At the end of a delivery run, if there are some addresses left to be tried again later, the first spool file (the -H file) is updated to indicate which these are, and the journal file is then deleted. Updating the spool file is done by writing a new file and renaming it, to minimize the possibility of data loss.

Should the system or Exim crash after a successful delivery but before the spool file has been updated, the journal is left lying around. The next time Exim attempts to deliver the message, it reads the journal file and updates the spool file before proceeding. This minimizes the chances of double deliveries caused by crashes.

8. Processing an address for delivery

The main delivery processing elements of Exim are called routers and transports, and collectively these are known as drivers. Code for a number of them is provided in the source distribution, and compile-time options specify which ones are included in the binary. Runtime options specify which ones are actually used for delivering messages.

Each driver that is specified in the runtime configuration is an instance of that particular driver type. Multiple instances are allowed; for example, you can set up several different smtp transports, each with different option values that might specify different ports or different timeouts. Each instance has its own identifying name. In what follows we will normally use the instance name when discussing one particular instance (that is, one specific configuration of the driver), and the generic driver name when discussing the driver’s features in general.

A router is a driver that operates on an address, either determining how its delivery should happen, by assigning it to a specific transport, or converting the address into one or more new addresses (for example, via an alias file). A router may also explicitly choose to fail an address, causing it to be bounced.

A transport is a driver that transmits a copy of the message from Exim’s spool to some destination. There are two kinds of transport: for a local transport, the destination is a file or a pipe on the local host, whereas for a remote transport the destination is some other host. A message is passed to a specific transport as a result of successful routing. If a message has several recipients, it may be passed to a number of different transports.

An address is processed by passing it to each configured router instance in turn, subject to certain preconditions, until a router accepts the address or specifies that it should be bounced. We will describe this process in more detail shortly. First, as a simple example, we consider how each recipient address in a message is processed in a small configuration of three routers.

To make this a more concrete example, it is described in terms of some actual routers, but remember, this is only an example. You can configure Exim’s routers in many different ways, and there may be any number of routers in a configuration.

The first router that is specified in a configuration is often one that handles addresses in domains that are not recognized specifically by the local host. Typically these are addresses for arbitrary domains on the Internet. A precondition is set up which looks for the special domains known to the host (for example, its own domain name), and the router is run for addresses that do not match. Typically, this is a router that looks up domains in the DNS in order to find the hosts to which this address routes. If it succeeds, the address is assigned to a suitable SMTP transport; if it does not succeed, the router is configured to fail the address.

The second router is reached only when the domain is recognized as one that “belongs” to the local host. This router does redirection – also known as aliasing and forwarding. When it generates one or more new addresses from the original, each of them is routed independently from the start. Otherwise, the router may cause an address to fail, or it may simply decline to handle the address, in which case the address is passed to the next router.

The final router in many configurations is one that checks to see if the address belongs to a local mailbox. The precondition may involve a check to see if the local part is the name of a login account, or it may look up the local part in a file or a database. If its preconditions are not met, or if the router declines, we have reached the end of the routers. When this happens, the address is bounced.

9. Processing an address for verification

As well as being used to decide how to deliver to an address, Exim’s routers are also used for address verification. Verification can be requested as one of the checks to be performed in an ACL for incoming messages, on both sender and recipient addresses, and it can be tested using the -bv and -bvs command line options.

When an address is being verified, the routers are run in “verify mode”. This does not affect the way the routers work, but it is a state that can be detected. By this means, a router can be skipped or made to behave differently when verifying. A common example is a configuration in which the first router sends all messages to a message-scanning program unless they have been previously scanned. Thus, the first router accepts all addresses without any checking, making it useless for verifying. Normally, the no_verify option would be set for such a router, causing it to be skipped in verify mode.

10. Running an individual router

As explained in the example above, a number of preconditions are checked before running a router. If any are not met, the router is skipped, and the address is passed to the next router. When all the preconditions on a router are met, the router is run. What happens next depends on the outcome, which is one of the following:

• accept: The router accepts the address, and either assigns it to a transport or generates one or more “child” addresses. Processing the original address ceases unless the unseen option is set on the router. This option can be used to set up multiple deliveries with different routing (for example, for keeping archive copies of messages). When unseen is set, the address is passed to the next router. Normally, however, an accept return marks the end of routing.

  Any child addresses generated by the router are processed independently, starting with the first router by default. It is possible to change this by setting the redirect_router option to specify which router to start at for child addresses. Unlike pass_router (see below) the router specified by redirect_router may be anywhere in the router configuration.

• pass: The router recognizes the address, but cannot handle it itself. It requests that the address be passed to another router. By default, the address is passed to the next router, but this can be changed by setting the pass_router option. However, (unlike redirect_router) the named router must be below the current router (to avoid loops).

• decline: The router declines to accept the address because it does not recognize it at all. By default, the address is passed to the next router, but this can be prevented by setting the no_more option. When no_more is set, all the remaining routers are skipped. In effect, no_more converts decline into fail.

• fail: The router determines that the address should fail, and queues it for the generation of a bounce message. There is no further processing of the original address unless unseen is set on the router.

• defer: The router cannot handle the address at the present time. (A database may be offline, or a DNS lookup may have timed out.) No further processing of the address happens in this delivery attempt. It is tried again next time the message is considered for delivery.

• error: There is some error in the router (for example, a syntax error in its configuration). The action is as for defer.

If an address reaches the end of the routers without having been accepted by any of them, it is bounced as unrouteable. The default error message in this situation is “unrouteable address”, but you can set your own message by making use of the cannot_route_message option. This can be set for any router; the value from the last router that “saw” the address is used.

Sometimes while routing you want to fail a delivery when some conditions are met but others are not, instead of passing the address on for further routing. You can do this by having a second router that explicitly fails the delivery when the relevant conditions are met. The redirect router has a “fail” facility for this purpose.

11. Duplicate addresses

Once routing is complete, Exim scans the addresses that are assigned to local and remote transports and discards any duplicates that it finds. During this check, local parts are treated case-sensitively. This happens only when actually delivering a message; when testing routers with -bt, all the routed addresses are shown.

12. Router preconditions

The preconditions that are tested for each router are listed below, in the order in which they are tested. The individual configuration options are described in more detail in chapter 15.

1. The local_part_prefix and local_part_suffix options can specify that the local parts handled by the router may or must have certain prefixes and/or suffixes. If a mandatory affix (prefix or suffix) is not present, the router is skipped. These conditions are tested first. When an affix is present, it is removed from the local part before further processing, including the evaluation of any other conditions.

2. Routers can be designated for use only when not verifying an address, that is, only when routing it for delivery (or testing its delivery routing). If the verify option is set false, the router is skipped when Exim is verifying an address. Setting the verify option actually sets two options, verify_sender and verify_recipient, which independently control the use of the router for sender and recipient verification. You can set these options directly if you want a router to be used for only one type of verification. Note that cutthrough delivery is classed as a recipient verification for this purpose.

3. If the address_test option is set false, the router is skipped when Exim is run with the -bt option to test an address routing. This can be helpful when the first router sends all new messages to a scanner of some sort; it makes it possible to use -bt to test subsequent delivery routing without having to simulate the effect of the scanner.

4. Routers can be designated for use only when verifying an address, as opposed to routing it for delivery. The verify_only option controls this. Again, cutthrough delivery counts as a verification.

5. Individual routers can be explicitly skipped when running the routers to check an address given in the SMTP EXPN command (see the expn option).

6. If the domains option is set, the domain of the address must be in the set of domains that it defines. A match verifies the variable $domain (which carries tainted data) and assigns an untainted value to the $domain_data variable. Such an untainted value is often needed in the transport. For specifics of the matching operation and the resulting untainted value, refer to section 10.3.

   When an untainted value is wanted, use this option rather than the generic condition option.

7. If the local_parts option is set, the local part of the address must be in the set of local parts that it defines. A match verifies the variable $local_part (which carries tainted data) and assigns an untainted value to the $local_part_data variable. Such an untainted value is often needed in the transport. For specifics of the matching operation and the resulting untainted value, refer to section 10.6.

   When an untainted value is wanted, use this option rather than the generic condition option.

   If local_part_prefix or local_part_suffix is in use, the prefix or suffix is removed from the local part before this check. If you want to do precondition tests on local parts that include affixes, you can do so by using a condition option (see below) that uses the variables $local_part, $local_part_prefix, $local_part_prefix_v, $local_part_suffix and $local_part_suffix_v as necessary.

8. If the check_local_user option is set, the local part must be the name of an account on the local host. If this check succeeds, the uid and gid of the local user are placed in $local_user_uid and $local_user_gid and the user’s home directory is placed in $home; these values can be used in the remaining preconditions.

9. If the router_home_directory option is set, it is expanded at this point, because it overrides the value of $home. If this expansion were left till later, the value of $home as set by check_local_user would be used in subsequent tests. Having two different values of $home in the same router could lead to confusion.

10. If the senders option is set, the envelope sender address must be in the set of addresses that it defines.

11. If the require_files option is set, the existence or non-existence of specified files is tested.

12. If the condition option is set, it is evaluated and tested. This option uses an expanded string to allow you to set up your own custom preconditions. Expanded strings are described in chapter 11.

    Note that while using this option for address matching technically works, it does not set any de-tainted values. Such values are often needed, either for router-specific options or for transport options. Using the domains and local_parts options is usually the most convenient way to obtain them.

Note that require_files comes near the end of the list, so you cannot use it to check for the existence of a file in which to lookup up a domain, local part, or sender. However, as these options are all expanded, you can use the exists expansion condition to make such tests within each condition. The require_files option is intended for checking files that the router may be going to use internally, or which are needed by a specific transport (for example, .procmailrc).

13. Delivery in detail

When a message is to be delivered, the sequence of events is as follows:

1. If a system-wide filter file is specified, the message is passed to it. The filter may add recipients to the message, replace the recipients, discard the message, cause a new message to be generated, or cause the message delivery to fail. The format of the system filter file is the same as for Exim user filter files, described in the separate document entitled Exim’s interfaces to mail filtering. (Note: Sieve cannot be used for system filter files.)

   Some additional features are available in system filters – see chapter 47 for details. Note that a message is passed to the system filter only once per delivery attempt, however many recipients it has. However, if there are several delivery attempts because one or more addresses could not be immediately delivered, the system filter is run each time. The filter condition first_delivery can be used to detect the first run of the system filter.

2. Each recipient address is offered to each configured router, in turn, subject to its preconditions, until one is able to handle it. If no router can handle the address, that is, if they all decline, the address is failed. Because routers can be targeted at particular domains, several locally handled domains can be processed entirely independently of each other.

3. A router that accepts an address may assign it to a local or a remote transport. However, the transport is not run at this time. Instead, the address is placed on a list for the particular transport, which will be run later. Alternatively, the router may generate one or more new addresses (typically from alias, forward, or filter files). New addresses are fed back into this process from the top, but in order to avoid loops, a router ignores any address which has an identically-named ancestor that was processed by itself.

4. When all the routing has been done, addresses that have been successfully handled are passed to their assigned transports. When local transports are doing real local deliveries, they handle only one address at a time, but if a local transport is being used as a pseudo-remote transport (for example, to collect batched SMTP messages for transmission by some other means) multiple addresses can be handled. Remote transports can always handle more than one address at a time, but can be configured not to do so, or to restrict multiple addresses to the same domain.

5. Each local delivery to a file or a pipe runs in a separate process under a non-privileged uid, and these deliveries are run one at a time. Remote deliveries also run in separate processes, normally under a uid that is private to Exim (“the Exim user”), but in this case, several remote deliveries can be run in parallel. The maximum number of simultaneous remote deliveries for any one message is set by the remote_max_parallel option. The order in which deliveries are done is not defined, except that all local deliveries happen before any remote deliveries.

6. When it encounters a local delivery during a queue run, Exim checks its retry database to see if there has been a previous temporary delivery failure for the address before running the local transport. If there was a previous failure, Exim does not attempt a new delivery until the retry time for the address is reached. However, this happens only for delivery attempts that are part of a queue run. Local deliveries are always attempted when delivery immediately follows message reception, even if retry times are set for them. This makes for better behaviour if one particular message is causing problems (for example, causing quota overflow, or provoking an error in a filter file).

7. Remote transports do their own retry handling, since an address may be deliverable to one of a number of hosts, each of which may have a different retry time. If there have been previous temporary failures and no host has reached its retry time, no delivery is attempted, whether in a queue run or not. See chapter 32 for details of retry strategies.

8. If there were any permanent errors, a bounce message is returned to an appropriate address (the sender in the common case), with details of the error for each failing address. Exim can be configured to send copies of bounce messages to other addresses.

9. If one or more addresses suffered a temporary failure, the message is left on the queue, to be tried again later. Delivery of these addresses is said to be deferred.

10. When all the recipient addresses have either been delivered or bounced, handling of the message is complete. The spool files and message log are deleted, though the message log can optionally be preserved if required.

14. Retry mechanism

Exim’s mechanism for retrying messages that fail to get delivered at the first attempt is the queue runner process. You must either run an Exim daemon that uses the -q option with a time interval to start queue runners at regular intervals or use some other means (such as cron) to start them. If you do not arrange for queue runners to be run, messages that fail temporarily at the first attempt will remain in your queue forever. A queue runner process works its way through the queue, one message at a time, trying each delivery that has passed its retry time. You can run several queue runners at once.

Exim uses a set of configured rules to determine when next to retry the failing address (see chapter 32). These rules also specify when Exim should give up trying to deliver to the address, at which point it generates a bounce message. If no retry rules are set for a particular host, address, and error combination, no retries are attempted, and temporary errors are treated as permanent.