The “retry” section of the run time configuration file contains a list of retry rules which control how often Exim tries to deliver messages that cannot be delivered at the first attempt. If there are no retry rules, temporary errors are treated as permanent. The -brt command line option can be used to test which retry rule will be used for a given address or domain.
The most common cause of retries is temporary failure to deliver to a remote host because the host is down, or inaccessible because of a network problem. Exim's retry processing in this case is applied on a per-host (strictly, per IP address) basis, not on a per-message basis. Thus, if one message has recently been delayed, delivery of a new message to the same host is not immediately tried, but waits for the host's retry time to arrive. If the retry_defer log selector is set, the message “retry time not reached” is written to the main log whenever a delivery is skipped for this reason. Section 44.2 contains more details of the handling of errors during remote deliveries.
Retry processing applies to routing as well as to delivering, except as covered in the next paragraph. The retry rules do not distinguish between these actions. It is not possible, for example, to specify different behaviour for failures to route the domain snark.fict.example and failures to deliver to the host snark.fict.example. I didn't think anyone would ever need this added complication, so did not implement it. However, although they share the same retry rule, the actual retry times for routing and transporting a given domain are maintained independently.
When a delivery is not part of a queue run (typically an immediate delivery on receipt of a message), the routers are always run, and local deliveries are always attempted, even if retry times are set for them. This makes for better behaviour if one particular message is causing problems (for example, causing quota overflow, or provoking an error in a filter file). If such a delivery suffers a temporary failure, the retry data is updated as normal, and subsequent delivery attempts from queue runs occur only when the retry time for the local address is reached.
Each retry rule occupies one line and consists of three or four parts, separated by white space: a pattern, an error name, an optional list of sender addresses, and a list of retry parameters. The pattern and sender lists must be enclosed in double quotes if they contain white space. The rules are searched in order until one is found where the pattern, error name, and sender list (if present) match the failing host or address, the error that occurred, and the message's sender, respectively.
The pattern is any single item that may appear in an address list (see section 10.18). It is in fact processed as a one-item address list, which means that it is expanded before being tested against the address that has been delayed. Address list processing treats a plain domain name as if it were preceded by “*@”, which makes it possible for many retry rules to start with just a domain. For example,
lookingglass.fict.example * F,24h,30m;
provides a rule for any address in the lookingglass.fict.example domain, whereas
alice@lookingglass.fict.example * F,24h,30m;
applies only to temporary failures involving the local part alice. In practice, almost all rules start with a domain name pattern without a local part.
Warning: If you use a regular expression in a routing rule pattern, it must match a complete address, not just a domain, because that is how regular expressions work in address lists.
^\Nxyz\d+\.abc\.example$\N * G,1h,10m,2 Wrong
^\N[^@]+@xyz\d+\.abc\.example$\N * G,1h,10m,2 Right
When Exim is looking for a retry rule after a routing attempt has failed (for example, after a DNS timeout), each line in the retry configuration is tested against the complete address only if retry_use_local_part is set for the router. Otherwise, only the domain is used, except when matching against a regular expression, when the local part of the address is replaced with “*”. A domain on its own can match a domain pattern, or a pattern that starts with “*@”. By default, retry_use_local_part is true for routers where check_local_user is true, and false for other routers.
Similarly, when Exim is looking for a retry rule after a local delivery has failed (for example, after a mailbox full error), each line in the retry configuration is tested against the complete address only if retry_use_local_part is set for the transport (it defaults true for all local transports).
When Exim is looking for a retry rule after a remote delivery attempt has failed, what happens depends on the type of failure. After a 4xx SMTP response for a recipient address, the whole address is used when searching the retry rules. The rule that is found is used to create a retry time for the failing address.
For a temporary error that is not related to an individual address, (for example, a connection timeout), each line in the retry configuration is checked twice. First, the name of the remote host is used as a domain name (preceded by “*@” when matching a regular expression). If this does not match the line, the domain from the email address is tried in a similar fashion. For example, suppose the MX records for a.b.c.example are
a.b.c.example MX 5 x.y.z.example MX 6 p.q.r.example MX 7 m.n.o.example
and the retry rules are
p.q.r.example * F,24h,30m; a.b.c.example * F,4d,45m;
and a delivery to the host x.y.z.example fails. The first rule matches neither the host nor the domain, so Exim looks at the second rule. This does not match the host, but it does match the domain, so it is used to calculate the retry time for the host x.y.z.example. Meanwhile, Exim tries to deliver to p.q.r.example. If this fails, the first retry rule is used, because it matches the host.
In other words, failures to deliver to host p.q.r.example use the first rule to determine retry times, but for all the other hosts for the domain a.b.c.example, the second rule is used. The second rule is also used if routing to a.b.c.example suffers a temporary failure.
The second field in a retry rule is the name of a particular error, or an asterisk, which matches any error. The errors that can be tested for are:
auth_failed: Authentication failed when trying to send to a host in the hosts_require_auth list in an smtp transport.
rcpt_4xx: A 4xx error was received for an outgoing RCPT command. Either the first or both of the x's can be given as specific digits, for example: rcpt_45x or rcpt_436. For example, to recognize 452 errors given to RCPT commands by a particular host, and have retries every ten minutes and a one-hour timeout, you could set up a retry rule of this form:
the.host.name rcpt_452 F,1h,10m
These errors apply to both outgoing SMTP (the smtp transport) and outgoing LMTP (either the lmtp transport, or the smtp transport in LMTP mode). Note, however, that they apply only to responses to RCPT commands.
refused_MX: A connection to a host obtained from an MX record was refused.
refused_A: A connection to a host not obtained from an MX record was refused.
refused: A connection was refused.
timeout_connect_MX: A connection attempt to a host obtained from an MX record timed out.
timeout_connect_A: A connection attempt to a host not obtained from an MX record timed out.
timeout_connect: A connection attempt timed out.
timeout_MX: There was a timeout while connecting or during an SMTP session with a host obtained from an MX record.
timeout_A: There was a timeout while connecting or during an SMTP session with a host not obtained from an MX record.
timeout: There was a timeout while connecting or during an SMTP session.
quota: A mailbox quota was exceeded in a local delivery by the appendfile transport.
quota_<time>: A mailbox quota was exceeded in a local delivery by the appendfile transport, and the mailbox has not been accessed for <time>. For example, quota_4d applies to a quota error when the mailbox has not been accessed for four days.
The idea of quota_<time> is to make it possible to have shorter timeouts when the mailbox is full and is not being read by its owner. Ideally, it should be based on the last time that the user accessed the mailbox. However, it is not always possible to determine this. Exim uses the following heuristic rules:
If the mailbox is a single file, the time of last access (the “atime”) is used. As no new messages are being delivered (because the mailbox is over quota), Exim does not access the file, so this is the time of last user access.
For a maildir delivery, the time of last modification of the new subdirectory is used. As the mailbox is over quota, no new files are created in the new subdirectory, because no new messages are being delivered. Any change to the new subdirectory is therefore assumed to be the result of an MUA moving a new message to the cur directory when it is first read. The time that is used is therefore the last time that the user read a new message.
For other kinds of multi-file mailbox, the time of last access cannot be obtained, so a retry rule that uses this type of error field is never matched.
The quota errors apply both to system-enforced quotas and to Exim's own quota mechanism in the appendfile transport. The quota error also applies when a local delivery is deferred because a partition is full (the ENOSPC error).
You can specify retry rules that apply only when the failing message has a specific sender. In particular, this can be used to define retry rules that apply only to bounce messages. The third item in a retry rule can be of this form:
senders=<address list>
The retry timings themselves are then the fourth item. For example:
* * senders=: F,1h,30m
matches all temporary errors for bounce messages sent to any host. If the address list contains white space, it must be enclosed in quotes. For example:
a.domain timeout senders="xb.dom : yc.dom" G,8h,10m,1.5
When testing retry rules using -brt, you can supply a sender using the -f command line option, like this:
exim -f "" -brt user@dom.ain
If you do not set -f with -brt, a retry rule that contains a senders list is never matched.
The third (or fourth, if a senders list is present) field in a retry rule is a sequence of retry parameter sets, separated by semicolons. Each set consists of
<letter>,<cutoff time>,<arguments>
The letter identifies the algorithm for computing a new retry time; the cutoff time is the time beyond which this algorithm no longer applies, and the arguments vary the algorithm's action. The cutoff time is measured from the time that the first failure for the domain (combined with the local part if relevant) was detected, not from the time the message was received. The available algorithms are:
F: retry at fixed intervals. There is a single time parameter specifying the interval.
G: retry at geometrically increasing intervals. The first argument specifies a starting value for the interval, and the second a multiplier, which is used to increase the size of the interval at each retry.
When computing the next retry time, the algorithm definitions are scanned in order until one whose cutoff time has not yet passed is reached. This is then used to compute a new retry time that is later than the current time. In the case of fixed interval retries, this simply means adding the interval to the current time. For geometrically increasing intervals, retry intervals are computed from the rule's parameters until one that is greater than the previous interval is found. The main configuration variable retry_interval_max limits the maximum interval between retries.
A single remote domain may have a number of hosts associated with it, and each host may have more than one IP address. Retry algorithms are selected on the basis of the domain name, but are applied to each IP address independently. If, for example, a host has two IP addresses and one is unusable, Exim will generate retry times for it and will not try to use it until its next retry time comes. Thus the good IP address is likely to be tried first most of the time.
Retry times are hints rather than promises. Exim does not make any attempt to run deliveries exactly at the computed times. Instead, a queue runner process starts delivery processes for delayed messages periodically, and these attempt new deliveries only for those addresses that have passed their next retry time. If a new message arrives for a deferred address, an immediate delivery attempt occurs only if the address has passed its retry time. In the absence of new messages, the minimum time between retries is the interval between queue runner processes. There is not much point in setting retry times of five minutes if your queue runners happen only once an hour, unless there are a significant number of incoming messages (which might be the case on a system that is sending everything to a smart host, for example).
The data in the retry hints database can be inspected by using the exim_dumpdb or exim_fixdb utility programs (see chapter 49). The latter utility can also be used to change the data. The exinext utility script can be used to find out what the next retry times are for the hosts associated with a particular mail domain, and also for local deliveries that have been deferred.
Here are some example retry rules:
alice@wonderland.fict.example quota_5d F,7d,3h wonderland.fict.example quota_5d wonderland.fict.example * F,1h,15m; G,2d,1h,2; lookingglass.fict.example * F,24h,30m; * refused_A F,2h,20m; * * F,2h,15m; G,16h,1h,1.5; F,5d,8h
The first rule sets up special handling for mail to alice@wonderland.fict.example when there is an over-quota error and the mailbox has not been read for at least 5 days. Retries continue every three hours for 7 days. The second rule handles over-quota errors for all other local parts at wonderland.fict.example; the absence of a local part has the same effect as supplying “*@”. As no retry algorithms are supplied, messages that fail are bounced immediately if the mailbox has not been read for at least 5 days.
The third rule handles all other errors at wonderland.fict.example; retries happen every 15 minutes for an hour, then with geometrically increasing intervals until two days have passed since a delivery first failed. After the first hour there is a delay of one hour, then two hours, then four hours, and so on (this is a rather extreme example).
The fourth rule controls retries for the domain lookingglass.fict.example. They happen every 30 minutes for 24 hours only. The remaining two rules handle all other domains, with special action for connection refusal from hosts that were not obtained from an MX record.
The final rule in a retry configuration should always have asterisks in the first two fields so as to provide a general catch-all for any addresses that do not have their own special handling. This example tries every 15 minutes for 2 hours, then with intervals starting at one hour and increasing by a factor of 1.5 up to 16 hours, then every 8 hours up to 5 days.
Exim timestamps the data that it writes to its retry hints database. When it consults the data during a delivery it ignores any that is older than the value set in retry_data_expire (default 7 days). If, for example, a host hasn't been tried for 7 days, Exim will try to deliver to it immediately a message arrives, and if that fails, it will calculate a retry time as if it were failing for the first time.
This improves the behaviour for messages routed to rarely-used hosts such as MX backups. If such a host was down at one time, and happens to be down again when Exim tries a month later, using the old retry data would imply that it had been down all the time, which is not a justified assumption.
If a host really is permanently dead, this behaviour causes a burst of retries every now and again, but only if messages routed to it are rare. It there is a message at least once every 7 days the retry data never expires.
Special processing happens when an email address has been failing for so long that the cutoff time for the last algorithm is reached. For example, using the default retry rule:
* * F,2h,15m; G,16h,1h,1.5; F,4d,6h
the cutoff time is four days. Reaching the retry cutoff is independent of how long any specific message has been failing; it is the length of continuous failure for the recipient address that counts.
When the cutoff time is reached for a local delivery, or for all the IP addresses associated with a remote delivery, a subsequent delivery failure causes Exim to give up on the address, and a bounce message is generated. In order to cater for new messages that use the failing address, a next retry time is still computed from the final algorithm, and is used as follows:
For local deliveries, one delivery attempt is always made for any subsequent messages. If this delivery fails, the address fails immediately. The post-cutoff retry time is not used.
If the delivery is remote, there are two possibilities, controlled by the delay_after_cutoff option of the smtp transport. The option is true by default and in that case:
Until the post-cutoff retry time for one of the IP addresses is reached, the failing email address is bounced immediately, without a delivery attempt taking place. After that time, one new delivery attempt is made to those IP addresses that are past their retry times, and if that still fails, the address is bounced and new retry times are computed.
In other words, when all the hosts for a given email address have been failing for a long time, Exim bounces rather then defers until one of the hosts' retry times is reached. Then it tries once, and bounces if that attempt fails. This behaviour ensures that few resources are wasted in repeatedly trying to deliver to a broken destination, but if the host does recover, Exim will eventually notice.
If delay_after_cutoff is set false, Exim behaves differently. If all IP addresses are past their final cutoff time, Exim tries to deliver to those IP addresses that have not been tried since the message arrived. If there are no suitable IP addresses, or if they all fail, the address is bounced. In other words, it does not delay when a new message arrives, but tries the expired addresses immediately, unless they have been tried since the message arrived. If there is a continuous stream of messages for the failing domains, setting delay_after_cutoff false means that there will be many more attempts to deliver to permanently failing IP addresses than when delay_after_cutoff is true.
An additional rule is needed to cope with cases where a host is intermittently available, or when a message has some attribute that prevents its delivery when others to the same address get through. In this situation, because some messages are successfully delivered, the “retry clock” for the address keeps getting restarted, and so a message could remain on the queue for ever. To prevent this, if a message has been on the queue for longer than the cutoff time of any applicable retry rule for a given address, a delivery is attempted for that address, even if it is not yet time, and if this delivery fails, the address is timed out. A new retry time is not computed in this case, so that other messages for the same address are considered immediately.