Exim Internet Mailer

1. Multiple messages on a single connection

The sending of multiple messages over a single TCP/IP connection can arise in two ways:

• If a message contains more than max_rcpt (see below) addresses that are routed to the same host, more than one copy of the message has to be sent to that host. In this situation, multiple copies may be sent in a single run of the smtp transport over a single TCP/IP connection. (What Exim actually does when it has too many addresses to send in one message also depends on the value of the global remote_max_parallel option. Details are given in section 45.1.)

• When a message has been successfully delivered over a TCP/IP connection, Exim looks in its hints database to see if there are any other messages awaiting a connection to the same host. If there are, a new delivery process is started for one of them, and the current TCP/IP connection is passed on to it. The new process may in turn send multiple copies and possibly create yet another process.

For each copy sent over the same TCP/IP connection, a sequence counter is incremented, and if it ever gets to the value of connection_max_messages, no further messages are sent over that connection.

2. Use of the $host variable

At the start of a run of the smtp transport, the values of $host and $host_address are the name and IP address of the first host on the host list passed by the router. However, when the transport is about to connect to a specific host, and while it is connected to that host, $host and $host_address are set to the values for that host. These are the values that are in force when the helo_data, hosts_try_auth, interface, serialize_hosts, and the various TLS options are expanded.

3. Private options for smtp

The private options of the smtp transport are as follows:

When an address is delayed because of a 4xx response to a RCPT command, it is the combination of sender and recipient that is delayed in subsequent queue runs until the retry time is reached. You can delay the recipient without reference to the sender (which is what earlier versions of Exim did), by setting address_retry_include_sender false. However, this can lead to problems with servers that regularly issue 4xx responses to RCPT commands.

When a host specified in hosts or fallback_hosts (see below) turns out to be the local host, or is listed in hosts_treat_as_local, delivery is deferred by default. However, if allow_localhost is set, Exim goes on to do the delivery anyway. This should be used only in special cases when the configuration ensures that no looping will result (for example, a differently configured Exim is listening on the port to which the message is sent).

When Exim has authenticated as a client, or if authenticated_sender_force is true, this option sets a value for the AUTH= item on outgoing MAIL commands, overriding any existing authenticated sender value. If the string expansion is forced to fail, the option is ignored. Other expansion failures cause delivery to be deferred. If the result of expansion is an empty string, that is also ignored.

If the SMTP session is not authenticated, the expansion of authenticated_sender still happens (and can cause the delivery to be deferred if it fails), but no AUTH= item is added to MAIL commands unless authenticated_sender_force is true.

This option allows you to use the smtp transport in LMTP mode to deliver mail to Cyrus IMAP and provide the proper local part as the “authenticated sender”, via a setting such as:

authenticated_sender = $local_part

This removes the need for IMAP subfolders to be assigned special ACLs to allow direct delivery to those subfolders.

Because of expected uses such as that just described for Cyrus (when no domain is involved), there is no checking on the syntax of the provided value.

If this option is set true, the authenticated_sender option’s value is used for the AUTH= item on outgoing MAIL commands, even if Exim has not authenticated as a client.

This sets a timeout for receiving a response to an SMTP command that has been sent out. It is also used when waiting for the initial banner line from the remote host. Its value must not be zero.

This sets a timeout for the connect() function, which sets up a TCP/IP call to a remote host. A setting of zero allows the system timeout (typically several minutes) to act. To have any effect, the value of this option must be less than the system timeout. However, it has been observed that on some systems there is no system timeout, which is why the default value for this option is 5 minutes, a value recommended by RFC 1123.

This controls the maximum number of separate message deliveries that are sent over a single TCP/IP connection. If the value is zero, there is no limit. For testing purposes, this value can be overridden by the -oB command line option.

This sets a timeout for the transmission of each block in the data portion of the message. As a result, the overall timeout for a message depends on the size of the message. Its value must not be zero. See also final_timeout.

This option controls what happens when all remote IP addresses for a given domain have been inaccessible for so long that they have passed their retry cutoff times.

In the default state, if the next retry time has not been reached for any of them, the address is bounced without trying any deliveries. In other words, Exim delays retrying an IP address after the final cutoff time until a new retry time is reached, and can therefore bounce an address without ever trying a delivery, when machines have been down for a long time. Some people are unhappy at this prospect, so...

If delay_after_cutoff is set false, Exim behaves differently. If all IP addresses are past their final cutoff time, Exim tries to deliver to those IP addresses that have not been tried since the message arrived. If there are none, of if they all fail, the address is bounced. In other words, it does not delay when a new message arrives, but immediately tries those expired IP addresses that haven’t been tried since the message arrived. If there is a continuous stream of messages for the dead hosts, unsetting delay_after_cutoff means that there will be many more attempts to deliver to them.

If the hosts or fallback_hosts option is being used, and the gethostbyname option is false, the RES_DEFNAMES resolver option is set. See the qualify_single option in chapter 17 for more details.

If the hosts or fallback_hosts option is being used, and the gethostbyname option is false, the RES_DNSRCH resolver option is set. See the search_parents option in chapter 17 for more details.

String expansion is not applied to this option. The argument must be a colon-separated list of host names or IP addresses, optionally also including port numbers, though the separator can be changed, as described in section 6.19. Each individual item in the list is the same as an item in a route_list setting for the manualroute router, as described in section 20.5.

Fallback hosts can also be specified on routers, which associate them with the addresses they process. As for the hosts option without hosts_override, fallback_hosts specified on the transport is used only if the address does not have its own associated fallback host list. Unlike hosts, a setting of fallback_hosts on an address is not overridden by hosts_override. However, hosts_randomize does apply to fallback host lists.

If Exim is unable to deliver to any of the hosts for a particular address, and the errors are not permanent rejections, the address is put on a separate transport queue with its host list replaced by the fallback hosts, unless the address was routed via MX records and the current host was in the original MX list. In that situation, the fallback host list is not used.

Once normal deliveries are complete, the fallback queue is delivered by re-running the same transports with the new host lists. If several failing addresses have the same fallback hosts (and max_rcpt permits it), a single copy of the message is sent.

The resolution of the host names on the fallback list is controlled by the gethostbyname option, as for the hosts option. Fallback hosts apply both to cases when the host list comes with the address and when it is taken from hosts. This option provides a “use a smart host only if delivery fails” facility.

This is the timeout that applies while waiting for the response to the final line containing just “.” that terminates a message. Its value must not be zero.

If this option is true when the hosts and/or fallback_hosts options are being used, names are looked up using gethostbyname() (or getipnodebyname() when available) instead of using the DNS. Of course, that function may in fact use the DNS, but it may also consult other sources of information such as /etc/hosts.

The value of this option is expanded, and used as the argument for the EHLO, HELO, or LHLO command that starts the outgoing SMTP or LMTP session. The variables $host and $host_address are set to the identity of the remote host, and can be used to generate different values for different servers.

Hosts are associated with an address by a router such as dnslookup, which finds the hosts by looking up the address domain in the DNS, or by manualroute, which has lists of hosts in its configuration. However, email addresses can be passed to the smtp transport by any router, and not all of them can provide an associated list of hosts.

The hosts option specifies a list of hosts to be used if the address being processed does not have any hosts associated with it. The hosts specified by hosts are also used, whether or not the address has its own hosts, if hosts_override is set.

The string is first expanded, before being interpreted as a colon-separated list of host names or IP addresses, possibly including port numbers. The separator may be changed to something other than colon, as described in section 6.19. Each individual item in the list is the same as an item in a route_list setting for the manualroute router, as described in section 20.5. However, note that the /MX facility of the manualroute router is not available here.

If the expansion fails, delivery is deferred. Unless the failure was caused by the inability to complete a lookup, the error is logged to the panic log as well as the main log. Host names are looked up either by searching directly for address records in the DNS or by calling gethostbyname() (or getipnodebyname() when available), depending on the setting of the gethostbyname option. When Exim is compiled with IPv6 support, if a host that is looked up in the DNS has both IPv4 and IPv6 addresses, both types of address are used.

During delivery, the hosts are tried in order, subject to their retry status, unless hosts_randomize is set.

This option is for use with broken hosts that announce ESMTP facilities (for example, PIPELINING) and then fail to implement them properly. When a host matches hosts_avoid_esmtp, Exim sends HELO rather than EHLO at the start of the SMTP session. This means that it cannot use any of the ESMTP facilities such as AUTH, PIPELINING, SIZE, and STARTTLS.

Exim will not try to start a TLS session when delivering to any host that matches this list. See chapter 39 for details of TLS.

This option limits the number of IP addresses that are tried for any one delivery in cases where there are temporary delivery errors. Section 30.4 describes in detail how the value of this option is used.

This is an additional check on the maximum number of IP addresses that Exim tries for any one delivery. Section 30.4 describes its use and why it exists.

For any host that matches this list, a connection on which a TLS session has been started will not be passed to a new delivery process for sending another message on the same connection. See section 39.10 for an explanation of when this might be needed.

If this option is set and the hosts option is also set, any hosts that are attached to the address are ignored, and instead the hosts specified by the hosts option are always used. This option does not apply to fallback_hosts.

If this option is set, and either the list of hosts is taken from the hosts or the fallback_hosts option, or the hosts supplied by the router were not obtained from MX records (this includes fallback hosts from the router), and were not randomizied by the router, the order of trying the hosts is randomized each time the transport runs. Randomizing the order of a host list can be used to do crude load sharing.

When hosts_randomize is true, a host list may be split into groups whose order is separately randomized. This makes it possible to set up MX-like behaviour. The boundaries between groups are indicated by an item that is just + in the host list. For example:

hosts = host1:host2:host3:+:host4:host5

The order of the first three hosts and the order of the last two hosts is randomized for each use, but the first three always end up before the last two. If hosts_randomize is not set, a + item in the list is ignored.

This option provides a list of servers for which authentication must succeed before Exim will try to transfer a message. If authentication fails for servers which are not in this list, Exim tries to send unauthenticated. If authentication fails for one of these servers, delivery is deferred. This temporary error is detectable in the retry rules, so it can be turned into a hard failure if required. See also hosts_try_auth, and chapter 33 for details of authentication.

Exim will insist on using a TLS session when delivering to any host that matches this list. See chapter 39 for details of TLS. Note: This option affects outgoing mail only. To insist on TLS for incoming messages, use an appropriate ACL.

This option provides a list of servers to which, provided they announce authentication support, Exim will attempt to authenticate as a client when it connects. If authentication fails, Exim will try to transfer the message unauthenticated. See also hosts_require_auth, and chapter 33 for details of authentication.

This option specifies which interface to bind to when making an outgoing SMTP call. Note: Do not confuse this with the interface address that was used when a message was received, which is in $received_ip_address, formerly known as $interface_address. The name was changed to minimize confusion with the outgoing interface address. There is no variable that contains an outgoing interface address because, unless it is set by this option, its value is unknown.

During the expansion of the interface option the variables $host and $host_address refer to the host to which a connection is about to be made during the expansion of the string. Forced expansion failure, or an empty string result causes the option to be ignored. Otherwise, after expansion, the string must be a list of IP addresses, colon-separated by default, but the separator can be changed in the usual way. For example:

interface = <; 192.168.123.123 ; 3ffe:ffff:836f::fe86:a061

The first interface of the correct type (IPv4 or IPv6) is used for the outgoing connection. If none of them are the correct type, the option is ignored. If interface is not set, or is ignored, the system’s IP functions choose which interface to use if the host has more than one.

This option controls the setting of SO_KEEPALIVE on outgoing TCP/IP socket connections. When set, it causes the kernel to probe idle connections periodically, by sending packets with “old” sequence numbers. The other end of the connection should send a acknowledgement if the connection is still okay or a reset if the connection has been aborted. The reason for doing this is that it has the beneficial effect of freeing up certain types of connection that can get stuck when the remote host is disconnected without tidying up the TCP/IP call properly. The keepalive mechanism takes several hours to detect unreachable hosts.

If this option is set true when the protocol option is set to “lmtp”, the string IGNOREQUOTA is added to RCPT commands, provided that the LMTP server has advertised support for IGNOREQUOTA in its response to the LHLO command.

This option limits the number of RCPT commands that are sent in a single SMTP message transaction. Each set of addresses is treated independently, and so can cause parallel connections to the same host if remote_max_parallel permits this.

When this option is set, the smtp transport can handle a number of addresses containing a mixture of different domains provided they all resolve to the same list of hosts. Turning the option off restricts the transport to handling only one domain at a time. This is useful if you want to use $domain in an expansion for the transport, because it is set only when there is a single domain involved in a remote delivery.

This option specifies the TCP/IP port on the server to which Exim connects. Note: Do not confuse this with the port that was used when a message was received, which is in $received_port, formerly known as $interface_port. The name was changed to minimize confusion with the outgoing port. There is no variable that contains an outgoing port.

If the value of this option begins with a digit it is taken as a port number; otherwise it is looked up using getservbyname(). The default value is normally “smtp”, but if protocol is set to “lmtp”, the default is “lmtp”. If the expansion fails, or if a port number cannot be found, delivery is deferred.

If this option is set to “lmtp” instead of “smtp”, the default value for the port option changes to “lmtp”, and the transport operates the LMTP protocol (RFC 2033) instead of SMTP. This protocol is sometimes used for local deliveries into closed message stores. Exim also has support for running LMTP over a pipe to a local process – see chapter 28.

Exim normally includes both the host name and the IP address in the key it constructs for indexing retry data after a temporary delivery failure. This means that when one of several IP addresses for a host is failing, it gets tried periodically (controlled by the retry rules), but use of the other IP addresses is not affected.

However, in some dialup environments hosts are assigned a different IP address each time they connect. In this situation the use of the IP address as part of the retry key leads to undesirable behaviour. Setting this option false causes Exim to use only the host name. This should normally be done on a separate instance of the smtp transport, set up specially to handle the dialup hosts.

Because Exim operates in a distributed manner, if several messages for the same host arrive at around the same time, more than one simultaneous connection to the remote host can occur. This is not usually a problem except when there is a slow link between the hosts. In that situation it may be helpful to restrict Exim to one connection at a time. This can be done by setting serialize_hosts to match the relevant hosts.

Exim implements serialization by means of a hints database in which a record is written whenever a process connects to one of the restricted hosts. The record is deleted when the connection is completed. Obviously there is scope for records to get left lying around if there is a system or program crash. To guard against this, Exim ignores any records that are more than six hours old.

If you set up this kind of serialization, you should also arrange to delete the relevant hints database whenever your system reboots. The names of the files start with misc and they are kept in the spool/db directory. There may be one or two files, depending on the type of DBM in use. The same files are used for ETRN serialization.

If a remote SMTP server indicates that it supports the SIZE option of the MAIL command, Exim uses this to pass over the message size at the start of an SMTP transaction. It adds the value of size_addition to the value it sends, to allow for headers and other text that may be added during delivery by configuration options or in a transport filter. It may be necessary to increase this if a lot of text is added to messages.

Alternatively, if the value of size_addition is set negative, it disables the use of the SIZE option altogether.

The value of this option must be the absolute path to a file which contains the client’s certificate, for possible use when sending a message over an encrypted connection. The values of $host and $host_address are set to the name and address of the server during the expansion. See chapter 39 for details of TLS.

Note: This option must be set if you want Exim to be able to use a TLS certificate when sending messages as a client. The global option of the same name specifies the certificate for Exim as a server; it is not automatically assumed that the same certificate should be used when Exim is operating as a client.

This option specifies a certificate revocation list. The expanded value must be the name of a file that contains a CRL in PEM format.

The value of this option must be the absolute path to a file which contains the client’s private key. This is used when sending a message over an encrypted connection using a client certificate. The values of $host and $host_address are set to the name and address of the server during the expansion. If this option is unset, or the expansion is forced to fail, or the result is an empty string, the private key is assumed to be in the same file as the certificate. See chapter 39 for details of TLS.

The value of this option must be a list of permitted cipher suites, for use when setting up an outgoing encrypted connection. (There is a global option of the same name for controlling incoming connections.) The values of $host and $host_address are set to the name and address of the server during the expansion. See chapter 39 for details of TLS; note that this option is used in different ways by OpenSSL and GnuTLS (see sections 39.4 and 39.5). For GnuTLS, the order of the ciphers is a preference order.

When the server host is not in hosts_require_tls, and there is a problem in setting up a TLS session, this option determines whether or not Exim should try to deliver the message unencrypted. If it is set false, delivery to the current host is deferred; if there are other hosts, they are tried. If this option is set true, Exim attempts to deliver unencrypted after a 4xx response to STARTTLS. Also, if STARTTLS is accepted, but the subsequent TLS negotiation fails, Exim closes the current connection (because it is in an unknown state), opens a new one to the same host, and then tries the delivery in clear.

The value of this option must be the absolute path to a file containing permitted server certificates, for use when setting up an encrypted connection. Alternatively, if you are using OpenSSL, you can set tls_verify_certificates to the name of a directory containing certificate files. This does not work with GnuTLS; the option must be set to the name of a single file if you are using GnuTLS. The values of $host and $host_address are set to the name and address of the server during the expansion of this option. See chapter 39 for details of TLS.

4. How the limits for the number of hosts to try are used

There are two options that are concerned with the number of hosts that are tried when an SMTP delivery takes place. They are hosts_max_try and hosts_max_try_hardlimit.

The hosts_max_try option limits the number of hosts that are tried for a single delivery. However, despite the term “host” in its name, the option actually applies to each IP address independently. In other words, a multihomed host is treated as several independent hosts, just as it is for retrying.

Many of the larger ISPs have multiple MX records which often point to multihomed hosts. As a result, a list of a dozen or more IP addresses may be created as a result of routing one of these domains.

Trying every single IP address on such a long list does not seem sensible; if several at the top of the list fail, it is reasonable to assume there is some problem that is likely to affect all of them. Roughly speaking, the value of hosts_max_try is the maximum number that are tried before deferring the delivery. However, the logic cannot be quite that simple.

Firstly, IP addresses that are skipped because their retry times have not arrived do not count, and in addition, addresses that are past their retry limits are also not counted, even when they are tried. This means that when some IP addresses are past their retry limits, more than the value of hosts_max_retry may be tried. The reason for this behaviour is to ensure that all IP addresses are considered before timing out an email address (but see below for an exception).

Secondly, when the hosts_max_try limit is reached, Exim looks down the host list to see if there is a subsequent host with a different (higher valued) MX. If there is, that host is considered next, and the current IP address is used but not counted. This behaviour helps in the case of a domain with a retry rule that hardly ever delays any hosts, as is now explained:

Consider the case of a long list of hosts with one MX value, and a few with a higher MX value. If hosts_max_try is small (the default is 5) only a few hosts at the top of the list are tried at first. With the default retry rule, which specifies increasing retry times, the higher MX hosts are eventually tried when those at the top of the list are skipped because they have not reached their retry times.

However, it is common practice to put a fixed short retry time on domains for large ISPs, on the grounds that their servers are rarely down for very long. Unfortunately, these are exactly the domains that tend to resolve to long lists of hosts. The short retry time means that the lowest MX hosts are tried every time. The attempts may be in a different order because of random sorting, but without the special MX check, the higher MX hosts would never be tried until all the lower MX hosts had timed out (which might be several days), because there are always some lower MX hosts that have reached their retry times. With the special check, Exim considers at least one IP address from each MX value at every delivery attempt, even if the hosts_max_try limit has already been reached.

The above logic means that hosts_max_try is not a hard limit, and in particular, Exim normally eventually tries all the IP addresses before timing out an email address. When hosts_max_try was implemented, this seemed a reasonable thing to do. Recently, however, some lunatic DNS configurations have been set up with hundreds of IP addresses for some domains. It can take a very long time indeed for an address to time out in these cases.

The hosts_max_try_hardlimit option was added to help with this problem. Exim never tries more than this number of IP addresses; if it hits this limit and they are all timed out, the email address is bounced, even though not all possible IP addresses have been tried.