4. Overview of ezmlm function

4.1 The basic setup.

In designing ezmlm, Dan J. Bernstein has used the unix philosophy of small component programs with limited and well defined functions. Requests for specific functions can then be met by the addition of new programs.

Thanks to the program execution mechanism Dan built into qmail, it is easy to execute several small programs per delivery in a defined sequence. It is also very easy to add shell scripts for further customization.

4.2 Inventions in ezmlm.

Dan J. Bernstein has written ezmlm in C. It is written for speed and reliability even in the face of power loss and NFS. These features are augmented to a large extent by the ruggedness of the qmail (also by Dan) delivery mechanism (see qmail-command(8)).

ezmlm uses some routines and techniques that still are not frequently seen in many mailing list managers. For example, subscriber E-mail addresses are stored in a hash so that searches require reading only, at most, 2% of the E-mail addresses. ezmlm has a optional message archive, where messages are stored 100 per directory, again to allow more efficient storage and retrieval. Important files are written under a new name and, only when safely written, moved in place, to assure that errors do not occur.

In addition, ezmlm has a number of new inventions. One of these is bounce detection, which generates an automatic warning containing information identifying the messages which have bounced, followed by a probe message to the E-mail addresses for which mail has bounced. If the probe bounces, the address is unsubscribed. Thus, the system won't remove E-mail addresses due to temporary bounces: it takes 12 days after the first bounce before a warning is sent, and another 12 days of bounces after the warning bounce before the probe message is set.

Another Dan J. Bernstein invention is the use of cryptographic cookies based on a timestamp, address, and action. These are used to assure that the user sending a request to subscribe or unsubscribe really controls the target address. It is also used to prevent forgery of warning or probe messages to make it exceedingly difficult to use the bounce detection mechanism to unsubscribe another user.

4.3 The qmail delivery mechanism.

See qmail(7), qmail-local(8), qmail-command(8), envelopes(5), and dot-qmail(5). Briefly, qmail having resolved the delivery address delivers it via the .qmail file that most completely matches the address. This file may be a link to another file, as is the case in ezmlm lists. qmail then delivers the message according to successive lines in this file forwarding it to an address, storing it, or piping it to a program. In the latter case, the program is expected to exit 0 leading delivery to proceed to the next line in the .qmail file, or 99 leading to success without delivery to succeeding lines. An exit code of 100 is a permanent error leading to an error message to the SENDER. An exit code of 111 is used for temporary errors, leading to re-delivery until successful or until the queue lifetime of the message has been exceeded.

Delivery granularity is the .qmail file and re-deliveries start at the top. Thus, if the message fails temporarily at a later line, the delivery according to an earlier line will be repeated. Similarly, qmail may have made deliveries successfully according to most of the .qmail file and then fail permanently. The SENDER is informed that the delivery failed, but not about at which point.

ezmlm takes advantage of these basic mechanisms to build a fast, efficient, and very configurable mailing list manager from a small set of independent programs.

4.4 What the different programs do.

See ezmlm(5) and the man pages for the different programs (listed in ezmlm(5)).

4.5 What the different files in the list directory do.

See ezmlm(5).

4.6 The paper path for posts

Messages to the list are delivered to a .qmail file, usually ~/.qmail-listname which is linked to DIR/editor. Here, the message is first delivered to ezmlm-reject(1) which can reject messages based on subject line contents, MIME content-type, and message body length. It also by default rejects all messages that do not have the list address in the ``To:'' or ``Cc:'' header. This eliminates most bulk spam. If the list is set up for restrictions based on envelope SENDER, the next delivery is to one or more instances of ezmlm-issubn(1). If the messages passed this check, it is usually delivered to ezmlm-send(1) for distribution. If the list is message moderated, it is instead delivered to ezmlm-store(1) which queues the message and sends out a moderation request. ezmlm-gate(1) is used by some other setups. It will for message moderated lists invoke ezmlm-send(1) directly if the message is from a specific set of SENDERs, and in other cases ezmlm-store(1) to send the message out for moderation.

If the list is configured for digests, DIR/editor also contains an ezmlm-tstdig(1) line followed by an ezmlm-get(1) line. If ezmlm-tstdig(1) determines that the criteria are met for digest generation, it exits with an exit code of 0, causing the ezmlm-get(1) line to be executed leading to a digest mailing. Otherwise, ezmlm-tstdig(1) exits 99, resulting in the remainder of the DIR/editor file to be ignored too long. The digest is not related to the message being delivered, but the delivery is used to trigger execution of the relevant programs.

In addition, DIR/editor contains a number of house-keeping functions. These are invocations of ezmlm-warn(1) to send out bounce warnings and ezmlm-clean(1) to clean the moderation queue of messages that have been ignored (if the list is message moderated). Again, these functions are not related to the specific message delivered, but the delivery itself is used as a convenient ``trigger'' for processing.

4.7 The ezmlm path for moderation messages.

Replies to moderation requests are channeled to DIR/moderator. This file contains an invocation of ezmlm-moderate(1) which invokes ezmlm-send(1) for accepted messages and sends out a rejection notice for rejected messages. It also sends error messages if the message is not found or already accepted/rejected contrary to the moderation message. Thus, if you accept a message already accepted, no error message is sent. ezmlm-clean(1) is also invoked from DIR/moderator for house keeping.

4.8 The ezmlm path for administrative messages.

Administrative requests for both list and digest lists are captured by ~/.qmail-listname-default linked to DIR/manager. Here they are delivered first to ezmlm-get(1) which processed archive retrieval requests, exiting 99 after successful completion which causes the rest of the delivery lines to be ignored. If the request is not for ezmlm-get(1) it rapidly exits 0. This leads to invocation of ezmlm-manage(1) which handles subscriber database functions, help messages, and (if configured) editing of DIR/text/ files. Again, ezmlm-warn(1) lines are included for bounce directory processing.

If configured, an ezmlm-request(1) line is present. This program constructs valid ezmlm requests from command in the subject lines of messages sent to listname-request@host and exits 99. These requests are mailed and will then return to be processed by one of the other programs.

4.9 The ezmlm path for bounces.

Bounces to the list and list-digest are handled by DIR/bouncer. The first delivery is to ezmlm-weed(1) which removes junk bounces. The second to ezmlm-return(1) which analyzes valid bounces storing the information in DIR/bounce/ for the list and DIR/digest/bounce/ for the digest. This is the information that ezmlm-warn(1) (invoked from DIR/editor and DIR/manager) uses and processes for automatic bounce handling. ezmlm-return(1) will also unsubscribe a subscriber from whom a probe message has bounced.

4.10 Messages to list-owner and list-digest-owner.

These are processed by DIR/owner and delivered to DIR/mailbox by default. It is better to put the real owner address in this location. This can be done manually, via editing of ezmlmrc(5), or via the ezmlm-make(1) -5 switch. Again, some house-keeping functions are also executed.

4.11 Structure of subscriber databases

ezmlm subscriber E-mail addresses are stored within DIR/subscribers/ as a hashed set of 53 files. The hash calculated from the address determines which of the 53 files and address is stored in. Thus, to find out if an address is a subscriber, ezmlm has to read at most about 2% of the E-mail addresses. The hash function insures that E-mail addresses are reasonably evenly distributed among the 53 files.

Addresses in the files in DIR/subscribers/ are stored as strings starting with ``T'', followed by the address, followed by a zero byte. This is the same format as taken by qmail-queue(8) on file descriptor 1. Thus, subscriber lists can be directly copied to qmail without any further processing.

4.12 Local case in E-mail addresses

RFC822 states that the host part of an address is case insensitive, but that case of the local part should be respected and the interpretation of it is the prerogative of the machine where the mailbox exists. Thus, ezmlm preserves the case of the local part, but converts the host part to lower case. ezmlm proper also bases the hash on the case of the local part, so that USER@host and user@host are not (usually) stored in the same file.

Locally, deliveries are most often case insensitive, i.e. mail to USER@host and user@host are delivered to the same mail box. A consequence of this is that many users use E-mail addresses with different case interchangeably. The problem is that when USER@host is subscribed, ezmlm will not find that address in response to an unsubscribe request from user@host. This is even more problematic when E-mail addresses have been added by hand to e.g. moderator lists.

ezmlm-idx>=0.22 changes address storage to make comparisons case insensitive and store E-mail addresses based on the hash of the all lower case address. Case is maintained for the local part. Thus, if USER@host is subscribed, mail is set to USER@host, but user@host is recognized as a subscriber and an unsubscribe request from user@host will remove USER@host from the subscriber list.

To maintain backwards compatibility with old subscriber lists, a second lookup is made for partially upper case E-mail addresses in some cases. This will find USER@host subscribed with a case sensitive hash as well.

If may be useful to move all old mixed case E-mail addresses to the ``new'' positions. Without this, USER@host subscribed with the old system will be able to unsubscribe as USER@host, but not as user@host. After the repositioning, s/he will be successfully able to use any case in an unsubscribe request, e.g. UsEr@host. To do this:

% ezmlm-list DIR | grep -G '[A-Z]' > tmp.tmp
% xargs ezmlm-sub DIR < tmp.tmp

This works, because subscribing an address, even if it already exists, will assure that it is stored with a case insensitive hash. On some systems, the grep ``-G'' switch need/should not be used.

4.13 Testing SENDER to insure posts are from list subscribers

This mode of operation is automatically set up if you specify the ezmlm-make(1) ``-u'' switch. Since there may be some addresses that should be allowed to post, but are not subscribers of list or list-digest, ezmlm-make(1) sets up an additional address database in DIR/extra/. Use ezmlm-sub(1), ezmlm-unsub(1), and ezmlm-list(1) to manipulate these addresses. If the list is configured for remote administration (see How remote administration works), you can add/remove addresses from the DIR/extra/ database by mailing list-allow-subscribe@listhost and list-allow-unsubscribe@listhost, respectively. Other commands that access subscriber databases work in the same manner.

To similarly restrict archive access, use the ezmlm-make(1) ``-g'' switch.

Since SENDER is under the control of a potential attacker, it is not secure to use tests of SENDER for anything important. However, when replies are always sent to SENDER (such as for archive access), a check of SENDER can prevent the sending of information to E-mail addresses not in the database.

To test sender, use the program ezmlm-issubn(1). It will return 0 (true for the shell, success for qmail deliveries) if SENDER is in at least one of a set of subscriber databases. If not, it will return 99 (false for the shell: success, but skip remainder of .qmail file for qmail deliveries). The basedirs of the subscriber lists (i.e. the directories in which the ``subscriber'' dirs are located) are given as arguments. ezmlm-issubn(1) can take any number of arguments.

Thus, to permit an action if SENDER is a subscriber to the list in any of DIR/, DIR/digest/, or DIR/extra/ and exit silently, put the following into the relevant .qmail file:

|/usr/local/bin/ezmlm/ezmlm-issubn DIR DIR/digest DIR/extra [...]
|/path/action_program

Restricting your list to posts from your subscribers is as easy as that. If your ezmlm binaries are in a different directory, you may have to modify the ezmlm-issubn(1) path.

ezmlm-issubn(1) has a ``-n'' switch which ``negates/reverses'' the exit code. To do an action if SENDER is NOT a subscriber of any of the lists:

|/usr/local/bin/ezmlm-ezmlm-issubn -n DIR/blacklist [dir2 ...]
|/path/other_program

To automatically configure the list with a blacklist address database in DIR/blacklist, use the ezmlm-make(1) ``-k'' switch. If the list is configured for remote administration (see How remote administration works) and if you are a remote administrator, you can manipulate the ``blacklist'' database remotely by sending mail to list-deny-subscribe-user=userhost@listhost, etc.

4.14 How cookies work

Each ezmlm list has it's own ``key'' created by ezmlm-make at setup time. This key is stored in DIR/key, and you can improve it by adding garbage of your own to it. However, changing the key will make all outstanding cookies invalid, so this should be done when the list is established.

When ezmlm receives an action request, such as ``subscribe'', it constructs a cookie as a function of:

• the request,
• the time,
• and the target address.

Dan has based these cookies on cryptographic functions that make it very unlikely that a change in any part of the cookie or the items will result in a valid combination. Thus, it is virtually impossible to forge a request even for someone who has a number of valid requests to analyze. Since the algorithm ezmlm uses is available, the security rests on the key (and the correctness of the algorithm). Anyone who knows the key for your lists can easily construct valid requests.

As ezmlm-make(1) doesn't use a truly random process to generate the key, it is theoretically possible that someone with sufficient knowledge about your system can guess your key. In practice, this is very unlikely, and the safety of the system is orders of magnitude higher than that of other mechanisms that you may rely on in your list management and mail transport (exclusive of strong encryption, such as PGP).

4.15 How moderator E-mail addresses are stored.

Moderator E-mail addresses are stored just like ezmlm subscriber addresses, in a set of up to 53 files within the subscribers subdirectory of the list's basedir/. For subscribers, the basedir/ is the list directory itself, i.e. DIR/. For moderators, the default is DIR/mod/, which can be overridden by placing a basedir name (starting with a ``/'') in DIR/modsub, DIR/remote, or DIR/modpost for subscription moderation, remote administration, and message moderation, respectively. This permits the use of one moderator database for multiple lists. Note: Subscription moderators and remote administrators are always the same addresses. If both DIR/modsub and DIR/remote contain paths, only the DIR/modsub path is used.

4.16 How subscription moderation works.

Subscription moderation is a simple extension of the ezmlm subscribe mechanism. Once the user has confirmed the subscribe request, a new request is constructed with a different action code. This is sent out to the moderator(s). When a moderator replies with a valid request and cookie combination, the user is subscribed. The user is then also welcomed to the list. Other moderators won't know that the request has already been approved. If other moderators reply to the request, no notification of the duplicate action is sent to the subscriber of the duplicate action. Ezmlm knows that this is a repeat request since the target address is already a subscriber.

The moderators are not informed about the result, unless there was an error (subscribing a target that is already a subscriber is not considered an error). This cuts down the number of messages a moderator receives. Any list moderator knows (or should know) the qmail/ezmlm/unix paradigm: if you're not told otherwise, your command was carried out successfully. This may be counterintuitive to those used to some other operating systems, but in our experience it doesn't take long to get used to the reliability and efficiency of U*ix/qmail/ezmlm.

Subscription moderation is enabled by creating DIR/modsub and adding the subscription moderator to DIR/mod/:

% ezmlm-sub DIR/mod moderator@host

4.17 How remote administration works.

The term ``remote administration'' is used to denote the ability of a list administrator by E-mail to add or remove any E-mail address from the subscriber list without the cooperation of the user. Normally, when user@userhost sends a message to list-subscribe-other=otherhost@listhost to subscribe other@otherhost, the confirmation request goes to other@otherhost. However, if remote administration is enabled and user@userhost is a moderator, a confirmation request (with a different action code) is sent back to user@userhost instead. The reply from the administrator is suppressed in the welcome message sent to the new subscriber (other@otherhost). This protects the identity of the remote administrator.

Remote administration is enabled by creating DIR/remote and adding the remote administrator E-mail address(es) to DIR/mod/:

% ezmlm-sub DIR/mod remoteadm@host

Remote administrators can add and remove addresses to the digest list, the ``allow'' list (user aliases for lists using SENDER restrictions on posting and archive access), and if used the ``deny'' list containing addresses that are denied posting rights to the list. The latter is easy to circumvent and intended to block errant mail robots, rather than human users.

4.18 How message moderation works.

ezmlm-store(1), invoked in DIR/editor, receives messages for message moderated lists. If DIR/modpost does not exist, ezmlm-store(1) just calls ezmlm-send(1) and the message is posted to the list as if it were not moderated. If DIR/modpost exists, ezmlm-store(1) places the message in DIR/mod/pending/. It also sends a moderation request to all the moderators. Included with this request is a copy of the message. The ``From:'' and ``Reply-To:'' E-mail addresses contain codes for ``reject'' and ``accept'', together with a unique message name (derived from the message timestamp and process id) and a cookie based on these items. When a moderator replies, ezmlm-moderate(1) is invoked via DIR/moderator. ezmlm-moderate(1) (after verifying that the cookie is less that 11.6 days old), validates the request, and if the request is valid and the message is found in DIR/mod/pending/, it carries out the requested action.

If the request is ``reject'' the post is returned to SENDER with an explanation and an optional moderator comment. If the request is ``accept'' the message is posted to the list via ezmlm-send(1). As the request is processed, a stub for the message is created in DIR/mod/rejected/ or DIR/mod/accepted/ for ``reject'' and ``accept'' requests, respectively.

If a valid reply is received but the message is no longer in DIR/mod/pending/, ezmlm-moderate(1) looks for the corresponding stub in DIR/mod/rejected/ and DIR/mod/accepted/. If the stub is found and the fate of the message was the one dictated by the new request, no further action is taken. If, however, no stub is found or the request and the actual message fate do not match, a notification is sent to the moderator. This scheme was chosen to impart a maximum of information with a minimum of messages. Also, it is the least demoralizing setup for multiple moderator lists, where it is important not to notify subsequent moderators that their work was in vain since the action of the first responding moderator has already resulted in processing of the message.

If a message is not ``rejected'' or ``accepted'' it remains in DIR/mod/pending/ until it times out. Cleanup of both messages and stubs is accomplished by ezmlm-clean(1) which is invoked through both DIR/editor and DIR/moderator for message moderated lists. ezmlm-clean(1) looks at the timestamp used to generate the message/stub name. If it is older than 120 hours (configurable in a range of 24-240 hours, by placing the value in DIR/modtime) it is removed. Unless suppressed with the ezmlm-clean(1) ``-R'' switch, the SENDER of the message is notified.

By default, the E-mail addresses of message moderators are stored as a subscriber list with a basedir of DIR/mod/. This can be changed to any other basedir by placing the name of that directory with a leading ``/'' in DIR/modpost. Although the default basedirs for message moderation and subscription moderation/remote administration are the same, both the functions and actors are entirely independent.

4.19 How messages are stored in the archive.

The structure of the ezmlm list archive is described in the ezmlm(5) manual page. Basically, the message is stored in DIR/archive/n/m, where ``n'' is the message number divided by 100 and ``m'' the remainder (2 digits). The first message is stored in DIR/archive/0/01.

4.20 How the message index works.

The ezmlm-idx(1) adds the option (default) of a message index to ezmlm. The ``From:'' line, the subject, the author's E-mail address and name and the time of receipt are logged for each message as it is received. The subject is ``normalized'' by concatenating split lines and removing reply-indicators such as ``Re:''. A hash of the normalized subject with all white space removed is also stored. The hash for any message within a thread is almost always the same and is used together with the order of receipt to connect a set of messages into a ``thread''.

The message index is stored as DIR/archive/n/index, where ``n'' is the message number mod 100. Thus, the directory DIR/archive/52/ stores messages 5200 through 5299 and the file ``index'' which contains the index for those messages.

The message index can be retrieved with the -index command (see ezmlm-get(1)). You can also retrieve a range of messages, a specific thread, or generate a message digest (see ezmlm-get(1)). Each of these commands can be disabled or restricted as desired by the list owner.

The ezmlm-idx(1) can be used at any time to either reconstruct an existing index or create one an index for an existing list without one.

4.21 How threading works

A ezmlm thread is just a message number-ordered set of messages with identical ``normalized'' subject entries. This is a very reliable method for threading messages. It does not rely on any variably present ``In-Reply-To:'' or ``References:'' headers. If the subject changes, the continuation becomes a separate thread very close to the original thread in a digest. ezmlm uses this mechanism to return message sets threaded and with a thread and author index, unless specifically told not to do so with the ``n'' format specifier. Naturally, lists set up without a message index (using the ezmlm-make ``-I'' switch) do not maintain thread information.

4.22 How digests work.

A ``digest'' is just an ordered collection of messages from a list, usually sent out regularly depending on the time and traffic volume since the last digest. Digest subscribers thus can read messages as ``threads'' once daily, rather than receiving a constant trickle of messages.

As a major change in ezmlm-idx-0.30, the digest is no longer a totally separate ezmlm-list, but a part of the main list. This has security advantages, makes setup and administration easier, saves space, and allows a consistent way for subscribers of both ``list'' and ``list-digest'' to retrieve missed messages from a single archive.

The digest of the list ``list'' is always called ``list-digest''. To set up a list with a digest, simply use the ezmlm-make(1) ``-d'' switch. You subscribe to and unsubscribe from a digest the same way as for the main list, except that the request is sent to e.g. list-digest-subscribe@host rather than to list-subscribe@host.

Any option such as remote admin or subscription moderation that is active for the list applies also to the digest list. Any restrictions in posts or archive retrieval set up for the list, automatically accept both subscribers of the main list and of the digest list.

The changes in ezmlm-idx>=0.30 allow all programs to service both list and list-digest functions. All digest-specific files are stored in DIR/digest/. Digest list subscriber addresses in DIR/digest/subscribers/ and digest list bounce information in DIR/digest/bounce/. Text files are shared between list and digest. To get the local part of the list or list-digest name in a context sensitive manner, use ``<#l#>'' (lower case ``L'') in the text file.

In order to generate digest, the list needs to be archived and indexed (both default). You can retrieve sets of messages from the message archive. Such sets are always returned to the SENDER of the request. ``Digests'' are a special form of such a set/request. First, there are no restrictions on the number of messages that can be in a digest (which is balanced by the requirement for a ``digest code'' that needs to be specified in order to create a digest based on a mailed request). Second, special files (DIR/digissue and DIR/dignum) keep track of the digest issue and the message number, amount, and time when the last digest was created. Thus, the system is adapted to make it easy to create the regular collections of messages commonly referred to as ``digests''.

Digest can be generated in several different ways:

Command line

ezmlm-get can be invoked on the command line, or via a script from e.g. crond(8):

        % ezmlm-get DIR < /dev/null

If for some reason the digest should be disseminated via a separate list, the digest can be redirected to a ``target address'' with the ezmlm-get(1) ``-t'' switch. This may be useful if a non-standard digest list name is required. In this case, the list disseminating the digest must be set up as a sublist of the main list (see How sublists work).

from DIR/editor

This is the default and does not require and additional setup. It works well with most lists. The only possible advantage is for very low traffic lists and for lists where it is important that a digest be sent out at a specific time (as DIR/editor digests are triggered only when messages are received).

In DIR/editor, ezmlm-get(1) needs to be combined with ezmlm-tstdig(1) so that digests are generated only if certain criteria are met (in this case, more than 30 messages, 64 kbytes of message body or 48 hours since the latest digest). Add these lines after the ezmlm-send line in DIR/editor:

        |/usr/local/bin/ezmlm/ezmlm-tstdig -t48 -m30 -k64 DIR || exit 99
        |/usr/local/bin/ezmlm/ezmlm-get diglist@host DIR || exit 0

To set this up automatically when you create the list:

        % ezmlm-make -d DIR dot local host [code]

Again, the ezmlm-get(1) ``-t'' switch can be used for non-standard arrangements to redirect the digest.

from DIR/manager

Here ezmlm-get(1) is in it's normal place before ezmlm-manage(1), but a digest code is specified in the ezmlm-get(1) command line. To trigger digests requires a regular trigger messages generated from e.g. crond(8) (see below). ezmlm-make(1) sets up ezmlm-get(1) this way if a digest ``code'' is given as the 5th ezmlm-make(1) command line argument. However, you need to set up the trigger messages separately (see below):

        % ezmlm-make DIR dot local host code

To also test for message volume with this setup, generate trigger messages with the granularity you'd like, and add a ezmlm-tstdig(1) line to DIR/manager. E.g., use a trigger message every 3 hours and the following ezmlm-tstdig(1) line before ezmlm-get(1):

        |/usr/local/bin/ezmlm/ezmlm-tstdig -t24 -m30 -k64 DIR || exit 99

In general, a cron-triggered digest is preferred for very large lists and for lists with very low traffic. Again, the ezmlm-get(1) ``-t'' switch can be used for non-standard arrangements to redirect the digest. For most lists, the digesting from DIR/editor works very well, and does not require any extra setup work.

Combination setups

The default setup in the ezmlmrc(5) file included in the distribution is the DIR/editor triggered setup described above. If you in addition use ezmlm-cron(1) or crond(8) directly to generate trigger messages to list-dig.code@host, you can get regular digests (via the trigger messages and DIR/manager), with extra digest sent when traffic is unusually high (via the ezmlm-tstdig/ezmlm-get limits set in DIR/editor). This works best when the time argument on the ezmlm-tstdig(1) command line is the same as the trigger message interval, and the other ezmlm-tstdig(1) parameters are set so that they are only rarely exceeded within the normal digest interval.

4.23 How ezmlm-tstdig works.

ezmlm-tstdig(1) looks at DIR/num and DIR/dignum to determine how many messages and how much traffic (in terms of bytes of message body) has arrived to the list since the latest digest. It also determines how much time has passed since the last digest was generated. If any of the criteria specified by command line switches exists, ezmlm-tstdig(1) exits 0, causing the invocation of the next line in the .qmail file. If not, ezmlm-tstdig(1) exits 99 causing qmail to skip the rest of the .qmail file. ezmlm-tstdig(1) looks at LOCAL to determine if it is invoked in the command line, in DIR/editor, or in DIR/manager. In the latter two cases, ezmlm-tstdig(1) verifies that the list local address is correct. If invoked in DIR/manager, ezmlm-tstdig(1) exits 0 for all action requests except list-dig, so that is does not interfere with the normal functions of ezmlm-get(1) and ezmlm-manage(1).

4.24 How sublists work.

ezmlm uses the concept of sublists. Sublists are regular ezmlm lists, except that they only accept messages from their parent list, which is placed in the file DIR/sublist.

sublists are used to split the load of a large mailing list among several hosts. All you need to do to set up a local sublist of e.g. the djb-qmail@cr.yp.to list is to create a ezmlm list, and put ``djb-qmail@cr.yp.to'' into DIR/sublist of you list, and subscribe the sublist to the main qmail list. Now anyone can subscribe to your local list which handles its own bounces, subscribe requests, etc. The load on the main list is only the single message to your local list. The ezmlm-split package is an attempt to automate the distribution of subscriber requests when a list is distributed via a set of sublists (see ezmlm-split).

Sublists will not add their own mailing list header and they will not add a subject prefix. Normally, sublists will use their own message number, rather than that used by the main list. With ezmlm-idx>=0.23, sublists that are not archived and not indexed, will instead use the main list message number. This way, bounce messages from the sublist can refer the subscriber to the main list archive. This is not done for indexed/archived sublists for security reasons (an attacker could overwrite messages in the sublist archive).

With ezmlm-idx>=0.31, there is support for using ezmlm as a sublist of a mailing list run by another mailing list manager. To set this up, set up a normal ezmlm sublist, then edit DIR/editor so that the ezmlm-send line contains the command line option ``-h X-Listprocessor-Version:'' (before DIR). As the header text, you need to use a header that the main list manager adds to messages. Now your sublist will accept only messages from the main list requiring that they come from that list and contain the header specified.

ezmlm-idx>=0.313 also has added protection against subscription of the ezmlm list to mailing lists run by other list managers. If the ezmlm-reject line in DIR/editor has ``-h'' and ``DIR'' on it, ezmlm-reject(1) will read DIR/headerreject and reject messages that have any header specified in that file. See the ezmlm-reject(1) man page for suitable headers.

4.25 How to service commands in the subject line.

Commands on the ``Subject:'' line should not be encouraged, since this does not take advantage of the flexibility of the qmail delivery mechanism and list setup. However, when migrating from other mailing list managers which use this method to issue list commands, configuring ezmlm to respond to such commands may be useful. In addition, some software manufacturers sell MUAs and mail gateways that are unable to correctly transport RFC822-compliant Internet mail with certain characters in the local part of the address.

ezmlm-request(1) is usually invoked in DIR/manager before ezmlm-get(1) and ezmlm-manage(1). It ignores all requests that are not for the list-request address. For requests to the list-request@host address, ezmlm-request(1) parses the ``Subject:'' line. If a ezmlm command address starting with the contents of DIR/outlocal (e.g. list-get45) is on the command line, ezmlm-request(1) generates the corresponding full ezmlm request message. If the subject does not start with the contents of DIR/outlocal, ezmlm-request(1) prefixes the line with the contents of DIR/outlocal, thereby building a complete ezmlm command. If a host name is specified, it must match the contents of DIR/inhost and it will be replaced by the contents of DIR/outhost which is also added if no host name (``@'') is found on the subject line.

Thus, a subject of ``subscribe'' to list-request@host will be auto-magically rewritten as a message to list-subscribe@host. Similarly, any ezmlm command or ``Reply-To:'' address can be pasted into the subject field and sent to list-request@host. ezmlm-request(1) does not validate the command name, but invalid commands result in a ``help'' message in reply via ezmlm-manage(1). This allows ezmlm-request(1) to also service custom commands, like list-faq@host that you may have created for your list.

If the ``Subject:'' is empty or does not start with a letter, ezmlm-request(1) will attempt to interpret the first message body line starting with a letter in the first position.

When ezmlm-request(1) has successfully processed a ''request'' command, it exits 99 to skip the rest of DIR/manager.

To set up a list to include ezmlm-request processing, use the ezmlm-make(1) ``-q'' switch.

4.26 How to support alternative command names.

ezmlm-idx>=0.23 allows alternate names for all user commands. This can be used to e.g. make a message to list-remove@host to result in an ``unsubscribe'' action. This may help migration from other mailing list managers and in non-English environments. The use of aliases allows ezmlm to respond to new command names, while always responding correctly to the standard commands. If ezmlm-request(1) is used it will automatically be able to deal with any commands you set up for the list, within ezmlm or as separate programs. See Multiple language support on how to set up command aliases.

4.27 How to add your own commands.

The qmail/ezmlm mechanism makes it very easy to add your own commands. You can add them to DIR/manager, but this requires great care in terms of ordering and exit codes. Easier is to set them up separately with a .qmail-list-command file.

Let's assume you want to allow anyone to determine how many subscribers are subscribed to your list with the command list-count@host. Just create a program to do the work:

        #!/bin/sh
        DTLINE='Delivered-To: list-count@host processor'
        grep "$DTLINE" > /dev/null &&
                { echo "This message is looping"; exit 100; }
        {
          echo "$DTLINE"
          cat <<EOF
          From: list-help@host
          To: $SENDER
          Subject: list@host subscriber count

          Current number of subscribers:
          EOF
          ezmlm-list /DIR | wc -l
        } | /var/qmail/qmail-inject -f list-return- "$SENDER"
        exit 0

4.28 How remote administrators can retrieve a subscriber list

A user with shell access can always manipulate subscriber lists with ezmlm-sub(1), ezmlm-unsub(1), and ezmlm-list(1) for the lists s/he owns.

Sometimes a remote administrator requires a list of subscriber E-mail addresses. At the same time, the list should be kept out of the hands of spammers and all unauthorized entities. By default, ezmlm does not allow remote subscriber list retrieval. You can enable the ``-list'' command for remote retrieval of a subscriber list by using the ezmlm-make(1) ``-l'' switch or by adding the ``-l'' switch to the ezmlm-manage(1) line in DIR/manager. With this switch, ezmlm will permit retrieval of a subscriber list, but only to remote administrators. Subscribers cannot get the list membership, and any outsider would have to be able to read a remote administrator's mail to get the list. Note: This option is not functional unless the list is configured for remote administration, i.e. the ezmlm-make(1) ``-rl'' switches need to both be used.

The list returned is unsorted for efficiency reasons. You can easily sort it or use your mail reader to find a specific entry. The number of subscribers is shown at the bottom of the list. To get the number of subscribers from the command line, use:

        % ezmlm-list DIR | wc -l

4.29 How remote administrators can receive a subscription log

The same conditions that enable remote administrators to retrieve a subscriber list (see subscriber_list ) also enable the remote admin to retrieve the subscription log, i.e. the log of changes made to the subscriber list. The command is list-log@host. The entries are of the form ``date timestamp dir event address comment''. ``dir'' is ``+'' for addition of an address, ``-'' for removal, event is empty for normal (un)subscribe ``manual'' for changes made with ezmlm-(un)sub, and ``probe'' for removals via bounce handling. ``address'' is the subscription address, and ``comment'' is empty or the subscribers ``From:'' line. The log can be used to look at recent additions/removals and to try to track down a subscriber address from e.g. the name on the ``From:'' line. The log is written on a best-effort basis. In contrast to the subscriber database, entries in the log may be lost at a system crash.

4.30 How text file editing works.

If a list is set up with the ezmlm-make(1) ``-n'' switch, or the ``-e'' switch is added to the ezmlm-manage(1) line in DIR/manager, ezmlm allows remote administrators to edit the text files that make up most of the ezmlm responses. Of course, this will work only if remote administration is enabled for the list. Replies are sent only if the target address is a remote administrator. Thus, ezmlm does not rely on SENDER (easily forged) but on the notion that only the recipient receives the message. This is a reasonable assumption for remote administrators that receive mail on the local system.

With this switch, ezmlm replies to the -edit command with a list of the files in DIR/text/. Only files where editing seems reasonable are included in the list. The remote administrator can edit any file in DIR/text/ by sending e-mail containing the new text to -edit.file where ``file'' is the name of the file replaced (edited). The file must exist and the name consist of only lower case letters and '-'. Any '-' (hyphen) must be substituted by a '_' (underscore). For remote administrator convenience, the substitution has been made in the list of files sent in reply to the -edit command.

In reply to this command, ezmlm sends a message with the file and editing instructions. A ``cookie'' based on the date, file name, and contents of the file is added to the ``Reply-To:'' address. The cookie becomes invalid as soon as the file has been changed, or after 27 hours, whichever is shorter. Also, the cookie cannot be used to edit any other file, even if the other file has exactly the same contents. If you sent an edit request, and decide not to edit the file, you can simply delete the message.

To apply standard changes to all your text files it is easier to edit ~/.ezmlmrc. To reset the list's text files back to their default contents (as specified by ezmlmrc(5)), use the ezmlm-make(1) ``-e'' switch together with any other switches used to set up the list.

4.31 How subject line prefixes work.

First of all, it is against a number of RFCs to modify the ``Subject:'' header of messages. However, it is frequently requested by users who have seen it on other list managers. Second, it is many times worse to have a prefix that changes from message to message, such as a prefix with the message number. However, a number of lists, especially in Japan, use this feature and in its absence these lists might be unable to take advantage of ezmlm. Thus, while we recommend against using a prefix, ezmlm-idx supports it.

To add a subject prefix, just put the text into dir/prefix. The only format that makes any sense is ``list:'' or ``(list)'' or such. This is configured if the ezmlm-make(1) ``-x'' switch is used.

The message number prefix is activated by putting e.g. ``(list-#)'' into DIR/prefix. ``#'' is replaced by the message number. ezmlm refuses to make more drastic changes in the subject of a message. As a consequence, the message number prefix is added only when the subject does not already contain a prefix. Thus, replies will have the message number of the original message. Doing anything else and still supporting RFC2047-encoded subjects in the archive threading (much more important) would require decoding the subject, removing/editing the prefix, and re-encoding the subject. This is far too invasive.

The entire thread can always be retrieved by sending a message to list-thread-x where ``x'' is the message number in the prefix of any message in the thread.

4.32 How bounces are handled.

Ezmlm messages are sent with a return-path that directs bounces to DIR/bouncer and also via ``VERP'' contain information about the intended recipient. Thus, programs run from DIR/bouncer know the subscriber for whom the message bounced. ezmlm-weed(1) is used to weed out non-informative bounces. For others ezmlm-return(1) decides if the address is a subscriber. If so, it saves the first bounce message and a list of bounced-message numbers. ezmlm-warn(1) executed from e.g. DIR/editor goes through these bounce files. If it finds any that are older than 1000000 seconds (about 11.6 days) it sends a warning message to the subscriber. If this warning message bounces, ezmlm-return(1) sets up a "warning flag" for the subscriber. If ezmlm-warn(1) finds a warning flag older than 11.6 days, it sends a "probe" to the subscriber. If ezmlm-return(1) receives a bounced probe, the subscriber is automatically unsubscribed.

The ezmlm-warn(1) ``-t'' switch can be used to change the time-out (in days). The ezmlm-warn(1) ``-d'' switch causes processing of ``list-digest'' bounces rather than ``list'' bounces. ezmlm-weed(1) and ezmlm-return(1) can handle bounces for either list.

ezmlm-warn(1) also removes any files in the bounce directory that are older than 3 times the bounce time-out.

4.33 How the info and faq commands work.

The -info and -faq commands simply reply with the contents of the DIR/text/info and DIR/text/faq files. Edit these files directly or remotely (see How to remotely edit dir/text files). The DIR/text/info file should start with a single line that is meaningful as is and describes the list. This will be used in later versions to allow automatic assembly of the global ``list-of-lists'' (see How to set up a global list address like majordomo@host or listserv@host).

4.34 How the global ezmlm list address works.

ezmlm-request(1) can be used to set up a global address, such as ezmlm@host which allows the user to see and interact with a number of different mailing lists. This is especially useful when your users are used to other mailing list managers, such as ``majordomo'' or ``listproc''. ezmlm-request(1) is set up to answer requests to the address (see How to set up a global list address like majordomo@host or listserv@host). There, it interprets the first line of the message body as a command. It will reply directly to ``lists'' and ``which'' commands. All other commands will be used to construct messages to the respective lists. Where other mailing list managers use synonyms of ezmlm commands, ezmlm-request(1) recognizes these and translates them to the corresponding ezmlm commands. ezmlm-request(1) will build commands also of unrecognized commands. Thus, if you create new commands for a list, ezmlm-request(1) will automatically support them.

If the user does not specify the complete list address, ezmlm-request(1) will attempt to complete the name. See the ezmlm-reject(1) man page for more info.

Sometimes, it is desirable to have a host- or user-wide address that can list available mailing lists