The appendfile transport delivers a message by appending it to a file in the local file system, or by creating an entirely new file in a specified directory. Single files to which messages are appended can be in the traditional Unix mailbox format, or optionally in the MBX format supported by the Pine MUA and University of Washington IMAP daemon, inter alia. When each message is being delivered as a separate file, ``maildir'' format can optionally be used to give added protection against failures that happen part-way through the delivery. A third form of separate-file delivery known as ``mailstore'' is also supported. For all file formats, Exim attempts to create as many levels of directory as necessary, provided that create_directory is set.
The code for the optional formats is not included in the Exim binary by default. It is necessary to set SUPPORT_MBX, SUPPORT_MAILDIR and/or SUPPORT_MAILSTORE in Local/Makefile to have the appropriate code included.
Exim recognises system quota errors, and generates an appropriate message. Exim also supports its own quota control within the transport, for use when the system facility is unavailable or cannot be used for some reason.
If there is an error while appending to a file (for example, quota exceeded or partition filled), Exim attempts to reset the file's length and last modification time back to what they were before. If there is an error while creating an entirely new file, the new file is removed.
appendfile is most commonly used for local deliveries to users' mailboxes. However, it can also be used as a pseudo-remote transport for putting messages into files for remote delivery by some means other than Exim. ``Batch SMTP'' format is often used in this case (see the use_bsmtp option). appendfile is also used for delivering messages to files or directories whose names are obtained directly from alias, forwarding, or filtering operations. In these cases, $local_part contains the local part that was aliased or forwarded, while $address_file contains the name of the file or directory.
Before appending to a file, a number of security checks are made, and the file is locked. A detailed description is given below, after the list of private options.
Setting this option permits delivery to named pipes (FIFOs) as well as to regular files. If no process is reading the named pipe at delivery time, the delivery is deferred.
By default, appendfile will not deliver if the path name for the file is that of a symbolic link. Setting this option relaxes that constraint, but there are security issues involved in the use of symbolic links. Be sure you know what you are doing if you set this. Details of exactly what this option affects are included in the discussion which follows this list of options.
See the description of local delivery batching in chapter 24.
See the description of local delivery batching in chapter 24.
When this option is set, the group owner of the file defined by the file option is checked to see that it is the same as the group under which the delivery process is running. The default setting is false because the default file mode is 0600, which means that the group is irrelevant.
When this option is set, the owner of the file defined by the file option is checked to ensure that it is the same as the user under which the delivery process is running.
As appendfile writes the message, the start of each line is tested for matching check_string, and if it does, the initial matching characters are replaced by the contents of escape_string. The value of check_string is a literal string, not a regular expression, and the case of any letters it contains is significant.
If use_bsmtp is set the values of check_string and escape_string are forced to ``.'' and ``..'' respectively, and any settings in the configuration are ignored. Otherwise, they default to ``From '' and ``>From '' when the file option is set, and unset when the directory option is set.
The default settings, along with message_prefix and message_suffix, are suitable for traditional ``BSD'' mailboxes, where a line beginning with ``From '' indicates the start of a new message. All four options need changing if another format is used. For example, to deliver to mailboxes in MMDF format:
check_string = "\1\1\1\1\n" escape_string = "\1\1\1\1 \n" message_prefix = "\1\1\1\1\n" message_suffix = "\1\1\1\1\n"
When this option is true, Exim attempts to create any missing superior directories for the file that it is about to write. A created directory's mode is given by the directory_mode option.
This option constrains the location of files and directories that are created by this transport. It applies to files defined by the file option and directories defined by the directory option. In the case of maildir delivery, it applies to the top level directory, not the maildir directories beneath.
The option must be set to one of the words ``anywhere'', ``inhome'', or ``belowhome''. In the second and third cases, a home directory must have been set for the transport. This option is not useful when an explicit file name is given for normal mailbox deliveries. It is intended for the case when file names are generated from users' .forward files. These are usually handled by an appendfile transport called address_file. See also file_must_exist.
This option is mutually exclusive with the file option. When it is set, the string is expanded, and the message is delivered into a new file or files in or below the given directory, instead of being appended to a single mailbox file. A number of different formats are provided (see maildir_format and mailstore_format), and see section 25.3 for further details of this form of delivery.
When directory is set, but neither maildir_format nor mailstore_format is set, appendfile delivers each message into a file whose name is obtained by expanding this string. The default value generates a unique name from the current time, in base 62 form, and the inode of the file. The variable $inode is available only when expanding this option.
If appendfile creates any directories as a result of the create_directory option, their mode is specified by this option.
See check_string above.
This option is mutually exclusive with the directory option. It need not be set when appendfile is being used to deliver to files whose names are obtained from forwarding, filtering, or aliasing address expansions (by default under the instance name address_file), because in those cases the file name is associated with the address. Otherwise, the file option must be set unless the directory option is set. Either use_fcntl_lock or use_lockfile (or both) must be set with file. If you are using more than one host to deliver over NFS into the same mailboxes, you should always use lock files.
The string value is expanded for each delivery, and must yield an absolute path. The most common settings of this option are variations on one of these examples:
file = /var/spool/mail/$local_part file = /home/$local_part/inbox file = $home/inbox
In the first example, all deliveries are done into the same directory. If Exim is configured to use lock files (see use_lockfile below) it must be able to create a file in the directory, so the ``sticky'' bit must be turned on for deliveries to be possible, or alternatively the group option can be used to run the delivery under a group id which has write access to the directory.
This option requests the transport to check the format of an existing file before adding to it. The check consists of matching a specific string at the start of the file. The value of the option consists of an even number of colon-separated strings. The first of each pair is the test string, and the second is the name of a transport. If the transport associated with a matched string is not the current transport, control is passed over to the other transport. For example, suppose the standard local_delivery transport has this added to it:
file_format = "From : local_delivery :\ \1\1\1\1\n : local_mmdf_delivery"
Mailboxes that begin with ``From'' are still handled by this transport, but if a mailbox begins with four binary ones followed by a newline, control is passed to a transport called local_mmdf_delivery, which presumably is configured to do the delivery in MMDF format. If a mailbox does not exist or is empty, it is assumed to match the current transport. If the start of a mailbox doesn't match any string, or if the transport named for a given string is not defined, delivery is deferred.
If this option is true, the file specified by the file option must exist, and an error occurs if it does not. Otherwise, it is created if it does not exist.
By default, the appendfile transport uses non-blocking calls to fcntl() when locking an open mailbox file. If the call fails, the delivery process sleeps for lock_interval and tries again, up to lock_retries times. Non-blocking calls are used so that the file is not kept open during the wait for the lock; the reason for this is to make it as safe as possible for deliveries over NFS in the case when processes might be accessing an NFS mailbox without using a lock file. This should not be done, but misunderstandings and hence misconfigurations are not unknown.
On a busy system, however, the performance of a non-blocking lock approach is not as good as using a blocking lock with a timeout. In this case, the waiting is done inside the system call, and Exim's delivery process acquires the lock and can proceed as soon as the previous lock holder releases it.
If lock_fcntl_timeout is set to a non-zero time, blocking locks, with that timeout, are used. There may still be some retrying: the maximum number of retries is
(lock_retries * lock_interval) / lock_fcntl_timeout
rounded up to the next whole number. In other words, the total time during which appendfile is trying to get a lock is roughly the same, unless lock_fcntl_timeout is set very large.
You should consider setting this option if you are getting a lot of delayed local deliveries because of errors of the form
failed to lock mailbox /some/file (fcntl)
This specifies the time to wait between attempts to lock the file. See below for details of locking.
This specifies the maximum number of attempts to lock the file. A value of zero is treated as 1. See below for details of locking.
This specifies the mode of the created lock file, when a lock file is being used (see use_lockfile).
When a lock file is being used (see use_lockfile), if a lock file already exists and is older than this value, it is assumed to have been left behind by accident, and Exim attempts to remove it.
If this option is set with the directory option, the delivery is into a new file in the ``maildir'' format that is used by other mail software. The option is available only if SUPPORT_MAILDIR is present in Local/Makefile. See section 25.3 below for further details.
This option specifies the number of times to retry when writing a file in ``maildir'' format. See section 25.3 below.
This option applies only to deliveries in maildir format, and is described in section 25.3 below.
If this option is set with the directory option, the delivery is into two new files in ``mailstore'' format. The option is available only if SUPPORT_MAILSTORE is present in Local/Makefile. See section 25.3 below for further details.
This option applies only to deliveries in mailstore format, and is described in section 25.3 below.
This option applies only to deliveries in mailstore format, and is described in section 25.3 below.
This option is available only if Exim has been compiled with SUPPORT_MBX set in Local/Makefile. If mbx_format is set with the file option, the message is appended to the mailbox file in MBX format instead of traditional Unix format. This format is supported by Pine4 and its associated IMAP and POP daemons, by means of the c-client library that they all use. The message_prefix and message_suffix options are not automatically changed by the use of mbx_format; they should normally be set empty.
If none of the locking options are mentioned in the configuration, use_mbx_lock is assumed and the other locking options default to false. It is possible to specify the other kinds of locking with mbx_format, but use_fcntl_lock and use_mbx_lock are mutually exclusive. MBX locking interworks with c-client, providing for shared access to the mailbox. It should not be used if any program that does not use this form of locking is going to access the mailbox, nor should it be used if the mailbox file is NFS mounted, because it works only when the mailbox is accessed from a single host.
If you set use_fcntl_lock with an MBX-format mailbox, you cannot use the standard version of c-client, because as long as it has a mailbox open (this means for the whole of a Pine or IMAP session), Exim will not be able to append messages to it.
The string specified here is expanded and output at the start of every message. The default is unset unless file is specified and use_bsmtp is not set, in which case it is:
message_prefix = "From ${if def:return_path{$return_path}\
{MAILER-DAEMON}} ${tod_bsdinbox}\n"
The string specified here is expanded and output at the end of every message. The default is unset unless file is specified and use_bsmtp is not set, in which case it is a single newline character. The suffix can be suppressed by setting
message_suffix =
If the output file is created, it is given this mode. If it already exists and has wider permissions, they are reduced to this mode. If it has narrower permissions, an error occurs unless mode_fail_narrower is false. However, if the delivery is the result of a save command in a filter file specifing a particular mode, the mode of the output file is always forced to take that value, and this option is ignored.
This option applies in the case when an existing mailbox file has a narrower mode than that specified by the mode option. If mode_fail_narrower is true, the delivery is deferred (``mailbox has the wrong mode''); otherwise Exim continues with the delivery attempt, using the existing mode of the file.
If this option is true, the comsat daemon is notified after every successful delivery to a user mailbox. This is the daemon that notifies logged on users about incoming mail.
This option imposes a limit on the size of the file to which Exim is appending, or to the total space used in the directory tree when the directory option is set. In the latter case, computation of the space used is expensive, because all the files in the directory (and any sub-directories) have to be individually inspected and their sizes summed (but see quota_size_regex below). Also, there is no interlock against two simultaneous deliveries into a multi-file mailbox. For single-file mailboxes, of course, an interlock is a necessity.
A file's size is take as its used value. Because of blocking effects, this may be a lot less than the actual amount of disc space allocated to the file. If the sizes of a number of files are being added up, the rounding effect can become quite noticeable, especially on systems that have large block sizes. Nevertheless, it seems best to stick to the used figure, because this is the obvious value which users understand most easily.
The value of the option is expanded, and must then be a numerical value (decimal point allowed), optionally followed by one of the letters K or M. A value of zero unsets the option. The expansion happens while Exim is running as root, before it changes uid for the delivery. This means that files which are inaccessible to the end user can be used to hold quota values that are looked up in the expansion. When delivery fails because this quota is exceeded, the handling of the error is as for system quota failures.
By default, Exim's quota checking mimics system quotas, and restricts the mailbox to the specified maximum size, though the value is not accurate to the last byte, owing to separator lines and additional headers that may get added during message delivery. When a mailbox is nearly full, large messages may get refused even though small ones are accepted, because the size of the current message is added to the quota when the check is made. This behaviour can be changed by setting quota_is_inclusive false. When this is done, the check for exceeding the quota does not include the current message. Thus, deliveries continue until the quota has been exceeded; thereafter, no further messages are delivered. See also quota_warn_threshold.
This option applies when the directory option is set. It limits the total number of files in the directory (compare the inode limit in system quotas). It can only be used if quota is also set. The value is expanded; an expansion failure causes delivery to be deferred.
See quota above.
This option applies when one of the delivery modes that writes a separate file for each message is being used. When Exim wants to find the size of one of these files in order to test the quota, it first checks quota_size_regex. If this is set to a regular expression that matches the file name, and it captures one string, that string is interpreted as a representation of the file's size. This feature is useful only when users have no shell access to their mailboxes - otherwise they could defeat the quota simply by renaming the files. This facility can be used with maildir deliveries, by setting maildir_tag to add the file length to the file name. For example:
maildir_tag = ,S=$message_size quota_size_regex = S=(\d+)$
The string is not expanded.
See below for the use of this option. If it is not set when quota_warn_threshold is set, it defaults to
quota_warn_message = "\ To: $local_part@$domain\n\ Subject: Your mailbox\n\n\ This message is automatically created \ by mail delivery software.\n\n\ The size of your mailbox has exceeded \ a warning threshold that is\n\ set by the system administrator.\n"
This option is expanded in the same way as quota (see above). If the resulting value is greater than zero, and delivery of the message causes the size of the file or total space in the directory tree to cross the given threshold, a warning message is sent. If quota is also set, the threshold may be specified as a percentage of it by following the value with a percent sign. For example:
quota = 10M quota_warn_threshold = 75%
If quota is not set, a setting of quota_warn_threshold that ends with a percent sign is ignored.
The warning message itself is specified by the quota_warn_message option, and it must start with a To: header line containing the recipient(s). A Subject: line should also normally be supplied. The quota option does not have to be set in order to use this option; they are independent of one another except when the threshold is specified as a percentage.
If this option is set true, appendfile writes messages in ``batch SMTP'' format, with the envelope sender and recipient(s) included as SMTP commands. If you want to include a leading HELO command with such messages, you can do so by setting the message_prefix option. See section 42.8 for details of batch SMTP.
This option causes lines to be terminated with the two-character CRLF sequence (carriage return, linefeed) instead of just a linefeed character. In the case of batched SMTP, the byte sequence written to the file is then an exact image of what would be sent down a real SMTP connection.
The contents of the message_prefix and message_suffix options are written verbatim, so must contain their own carriage return characters if these are needed. In cases where these options have non-empty defaults, the values end with a single linefeed, so they almost always need to be changed if use_crlf is set.
This option controls the use of the fcntl() function to lock a file for exclusive use when a message is being appended. It is set by default unless use_mbx_lock is set. Otherwise, it should be turned off only if you know that all your MUAs use lock file locking. When use_fcntl_lock is off, use_lockfile must be on if mbx_format is not set.
If this option is turned off, Exim does not attempt to create a lock file when appending to a mailbox file. In this situation, the only locking is by fcntl(). You should only turn use_lockfile off if you are absolutely sure that every MUA that is ever going to look at your users' mailboxes uses fcntl() rather than a lock file, and even then only when you are not delivering over NFS from more than one host.
In order to append to an NFS file safely from more than one host, it is necessary to take out a lock before opening the file, and the lock file achieves this. Otherwise, even with fcntl() locking, there is a risk of file corruption.
The use_lockfile option is set by default unless use_mbx_lock is set. It is not possible to turn both use_lockfile and use_fcntl_lock off, except when mbx_format is set.
This option is available only if Exim has been compiled with SUPPORT_MBX set in Local/Makefile. Setting the option specifies that special MBX locking rules be used. It is set by default if mbx_format is set and none of the locking options are mentioned in the configuration. The locking rules are the same as are used by the c-client library that underlies Pine and the IMAP4 and POP daemons that come with it (see the discussion below). The rules allow for shared access to the mailbox. However, this kind of locking does not work when the mailbox is NFS mounted.
Before appending to a file, the following preparations are made:
If the name of the file is /dev/null, no action is taken, and a success return is given.
If any directories on the file's path are missing, Exim creates them if the create_directory option is set. A created directory's mode is given by the directory_mode option.
If file_format is set, the format of an existing file is checked. If this indicates that a different transport should be used, control is passed to that transport.
If use_lockfile is set, a lock file is built in a way that will work reliably over NFS, as follows:
Create a ``hitching post'' file whose name is that of the lock file with the current time, primary host name, and process id added, by opening for writing as a new file. If this fails with an access error, delivery is deferred.
Close the hitching post file, and hard link it to the lock file name.
If the call to link() succeeds, creation of the lock file has succeeded. Unlink the hitching post name.
Otherwise, use stat() to get information about the hitching post file, and then unlink hitching post name. If the number of links is exactly two, creation of the lock file succeeded but something (for example, an NFS server crash and restart) caused this fact not to be communicated to the link() call.
If creation of the lock file failed, wait for lock_interval and try again, up to lock_retries times. However, since any program that writes to a mailbox should complete its task very quickly, it is reasonable to time out old lock files that are normally the result of user agent and system crashes. If an existing lock file is older than lockfile_timeout Exim attempts to unlink it before trying again.
A call is made to lstat() to discover whether the main file exists, and if so, what its characteristics are. If lstat() fails for any reason other than non-existence, delivery is deferred.
If the file does exist and is a symbolic link, delivery is deferred, unless the allow_symlinks option is set, in which case the ownership of the link is checked, and then stat() is called to find out about the real file, which is then subjected to the checks below. The check on the top-level link ownership prevents one user creating a link for another's mailbox in a sticky directory, though allowing symbolic links in this case is definitely not a good idea. If there is a chain of symbolic links, the intermediate ones are not checked.
If the file already exists but is not a regular file, or if the file's owner and group (if the group is being checked - see check_group above) are different from the user and group under which the delivery is running, delivery is deferred.
If the file's permissions are more generous than specified, they are reduced. If they are insufficient, delivery is deferred, unless mode_fail_narrower is set false, in which case the delivery is tried using the existing permissions.
The file's inode number is saved, and the file is then opened for appending. If this fails because the file has vanished, appendfile behaves as if it hadn't existed (see below). For any other failures, delivery is deferred.
If the file is opened successfully, check that the inode number hasn't changed, that it is still a regular file, and that the owner and permissions have not changed. If anything is wrong, defer delivery and freeze the message.
If the file did not exist originally, defer delivery if the file_must_exist option is set. Otherwise, check that the file is being created in a permitted directory if the create_file option is set (deferring on failure), and then open for writing as a new file, with the O_EXCL and O_CREAT options, except when dealing with a symbolic link (the allow_symlinks option must be set). In this case, which can happen if the link points to a non-existent file, the file is opened for writing using O_CREAT but not O_EXCL, because that prevents link following.
If opening fails because the file exists, obey the tests given above for existing files. However, to avoid looping in a situation where the file is being continuously created and destroyed, the exists/not-exists loop is broken after 10 repetitions, and the message is then frozen.
If opening fails with any other error, defer delivery.
Once the file is open, unless both use_fcntl_lock and use_mbx_lock are false, it is locked using fcntl(). In the former case, an exclusive lock is requested, while in the latter, Exim takes out a shared lock on the open file, and an exclusive lock on the file whose name is
/tmp/.<device-number>.<inode-number>
using the device and inode numbers of the open mailbox file, in accordance with the MBX locking rules.
If fcntl() locking fails, there are two possible courses of action, depending on the value of lock_fcntl_timeout. If its value is zero, the file is closed, Exim waits for lock_interval, and then goes back and re-opens it as above and tries to lock it again. This happens up to lock_retries times, after which the delivery is deferred.
If lock_fcntl_timeout has a value greater than zero, a blocking call to fcntl() with that timeout is used, so there has already been some waiting involved. Nevertheless, Exim does not give up immediately. It retries up to
(lock_retries * lock_interval) / lock_fcntl_timeout
times (rounded up).
At the end of delivery, Exim closes the file (which releases the fcntl() lock) and then deletes the lock file if one was created.
When the directory option is set, each message is delivered into a newly-created file or set of files. No locking is required while writing the message, so the various locking options of the transport are ignored. The ``From'' line that by default separates messages in a single file is not normally needed, nor is the escaping of message lines that start with ``From'', and there is no need to ensure a newline at the end of each message. Consequently, the default values for check_string, message_prefix, and message_suffix are all unset when directory is set.
There are three different ways in which delivery to individual files can be done, controlled by the settings of the maildir_format and mailstore_format options. Note that code to support maildir or mailstore formats is not included in the binary unless SUPPORT_MAILDIR or SUPPORT_MAILSTORE, respectively, is set in Local/Makefile.
In all three cases an attempt is made to create the directory and any necessary sub-directories if they do not exist, provided that the create_directory option is set (the default). The location of a created directory can be constrained by setting create_file. A created directory's mode is given by the directory_mode option. If creation fails, or if the create_directory option is not set when creation is required, delivery is deferred.
The three different kinds of ``single file'' delivery are as follows:
If neither maildir_format nor mailstore_format is set, a single new file is created directly in the named directory. For example, when delivering messages into files in batched SMTP format for later delivery to some host (see section 42.8), a setting such as
directory = /var/bsmtp/$host
might be used. A message is written to a file with a temporary name, which is then renamed when the delivery is complete. The final name is obtained by expanding the contents of the directory_file option.
If the maildir_format option is true, Exim delivers each message by writing it to a file whose name is tmp/<time>.<pid>.<host> in the given directory, and then renaming it into the new sub-directory if all goes well.
Before opening the temporary file, Exim calls stat() on its name. If any response other than ENOENT (does not exist) is given, it waits 2 seconds and tries again, up to maildir_retries times.
If Exim is required to check a quota setting before a maildir delivery, it looks for a file called maildirfolder in the maildir directory (alongside new, cur, tmp). If this exists, Exim assumes the directory is a maildir++ folder directory, which is one level down from the user's top level mailbox directory. This causes it to start at the parent directory instead of the current directory when calculating the amount of space used.
If maildir_tag is set, the string is expanded for each delivery. This is done after the message has been written, so that the value of the $message_size variable can be set accurately during the expansion. If the expansion is forced to fail, the tag is ignored, but a non-forced failure causes delivery to be deferred. The expanded tag may contain any printing characters except ``/''. Non-printing characters in the string are ignored; if the resulting string is empty, it is ignored. If it starts with an alphanumeric character, a leading colon is inserted.
When the temporary maildir file is renamed into the new sub-directory, the tag is added to its name. However, if adding the tag takes the length of the name to the point where the test stat() call fails with ENAMETOOLONG, the tag is dropped and the maildir file is created with no tag. Tags can be used to encode the size of files in their names; see quota_size_regex above for an example.
If the mailstore_format option is true, each message is written as two files in the given directory. A unique base name is constructed from the message id and the current delivery process, and the files that are written use this base name plus the suffixes .env and .msg. The .env file contains the message's envelope, and the .msg file contains the message itself.
During delivery, the envelope is first written to a file with the suffix .tmp. The .msg file is then written, and when it is complete, the .tmp file is renamed as the .env file. Programs that access messages in mailstore format should wait for the presence of both a .msg and a .env file before accessing either of them. An alternative approach is to wait for the absence of a .tmp file.
The envelope file starts with any text defined by the mailstore_prefix option, expanded and terminated by a newline if there isn't one. Then follows the sender address on one line, then all the recipient addresses, one per line. There can be more than one recipient only if the batch_max option is set greater than one. Finally, mailstore_suffix is expanded and the result appended to the file, followed by a newline if it does not end with one.
If expansion of mailstore_prefix or mailstore_suffix ends with a forced failure, it is ignored. Other expansion errors are treated as serious configuration errors, and delivery is deferred.