--- name: "filtermail" section: "1" platform: "linux" package: "filtermail" version: "1.06.00-1" release: "Debian 13 (Trixie)" platforms: ["linux"] url: "https://new.carta.tech/filtermail" --- # filtermail(1) Filter incoming e-mail to accepted, spam or ignored ## NAME filtermail - Filter incoming e-mail to accepted, spam or ignored ## SYNOPSIS **filtermail** \[OPTIONS\] *base* \ \[OPTIONS\] - cf. section **OPTIONS**\ *base*: the absolute path to the user’s home directory When specified paths with options or in the configuration file specifications may start with *~/*, in which case *~* is replaced by *base*. ## DESCRIPTION **Filtermail** filters incoming e-mail as either accepted, spam, or ignored e-mail. It uses rule files, which are inspected in sequence until the incoming e-mail matches a rule. Once that happens the rule’s associated action (accept, spam, or ignore) is executed. If the e-mail is not matched by any rule then the e-mail is accepted. Alternatively, when using option *--inspect* **filtermail** can be used to find the domain name of the sender, its IP address, its country of origin and the cidr-range containing the received IP address (see sections **OPTIONS** and **FILTERMAIL INSPECT** below). Accepted e-mail normally is appended to the mail file which is used by the incoming mail server when receiving mail for the current user. E.g., if the user’s username is *frank* then incoming mail is appended to the file */var/mail/frank*. Users may also define directories to contain saved e-mails (e.g., *~/Mail*), and **filtermail** can be configured to append e-mail considered as spam to, e.g., *~/Mail/spam*. Likewise, e-mail matching the ’ignore’ criteria could be appended to *~/Mail/ignore*. Instead of appending the complete e-mail to its destination file the received e-mail’s *From:* and *Subject:* headers can be appended to its destination file. This is achieved by prefixing *:HDRS:* to the name of the destination file. Alternatively, such e-mail can also be ignored, losing it completely, by not specifying a destination file. The option to merely log the received e-mail’s *From:* and *Subject:* headers may come in handy if the received e-mail is also kept elsewhere (e.g., in another account, which forwards the received e-mail to the computer running **filtermail**) and the ignoring rules might result in occasional false positive decisions. (see also section **OPTIONS** below). **Filtermail** uses three types of files: - o The configuration file contains values of options with are generally used (see sections **CONFIGURATION** and **OPTIONS**); - o Mail filtering rules are hierarchically ordered in the *rules* file: incoming mail is sequentially matched against the patterns defined in files specified in the rules file until a match is found. Once a match has been found the rule’s action (accept, ignore or spam) is executed, ending the filtering process (see section **RULES**); - o Each file specified in the rules file defines matching patterns, which are tested sequentially. Testing those patterns ends once the incoming mail matches a pattern. The result of the matching is forwarded to the rules file which may result in executing the rule’s action (see section **PATTERNS**). If **filtermail** detects a syntax error in the rules file or in a rule specification file the incoming mail is accepted. To avoid this situation the *--syntax* option (see section **OPTIONS**) should be used when modifying, adding or removing rule files to verify that the specified rules were correctly formulated. To use **filtermail** the incoming mail server must recognize it as a valid mail handling program (see section **EXAMPLES**). ## CONFIGURATION Options (see section **OPTIONS**) not flagged with \`NO\_CONFIG’ can also be specified in a configutation file. By default the configuration file *~/.filtermail/config* is used. Command line options always take precedence over specifications in the configuration file. The configuration file *must* exist, but may be empty. It must exist because its directory defines the directory where the files defining the filtering rules are located. Empty lines and the content of lines starting at the *#*-character are ignored. Option *--expire* is used to remove patterns whose date stamps (cf. section **PATTERNS**) indicate dates before the date specified by *--expire*. The configuration file may contain *dont-expire:* lines. Each *dont-expire:* line specifies the name of a pattern file whose entries don’t expire. Files specified in *dont-expire:* lines may not exist, but to avoid inspecting pattern files for expired dates the name(s) of those **s** must be identical to the names of the **s** used in the rules file (cf. section **RULES** below). Other files in the configuration file may use the *\[~\]/path* format or must be plain filenames (i.e., not starting with */*-characters). Plain files are relative to the directory containing the configuration file (cf. section **EXAMPLES**). ## RULES All mail filtering rules are defined in the *rules* file. Mail filtering starts at the first rule until either the incoming e-mail matches a rule, or until all rules have been processed and the e-mail does not match any rule. In the latter case the e-mail is considered accepted. Empty lines and lines whose first non-blank character is a *#*-character are ignored. The rules themselves cannot contain *#*-characters. Rules are written according to the following syntax (elements between square brackets are optional, the content of bracketed sections followed by a *\** character may be repeated (not using the square brackets). Lowercase words are keywords and cannot be used otherwise. Capitalized words are described below. Each rule is specified on its own line. Line continuation (using, e.g., *\\* at the end of a line) is not supported). Here’s the rule’s syntax: ``` if Header File [and Header File]* Action ``` *Header* is the name of a mail header (e.g., *From:, Received:*). Header specifications must be identical to the first words of header lines of the received e-mail. So to match the *From* header in the e-mail’s first line specify *From* (i.e., no colon). Some headers have variants. E.g., *Received:* and *Received-SPF:*. To select all headers sharing their initial characters append a *+*-character to the initial part (e.g., to select all headers starting with *Received* use *Received+*, and use *From+* to select all *From:* headers including the e-mail’s first line). *File* is the name of the file containing patterns to inspect. Filenames must start with *./* and define the locations of files below the configuration file’s directory. E.g., *./spam/subject* specifies the file *subject* in a subdirectory *spam* containing patterns considered by the rule. *Action* specifies the action to execute when e-mail matches a rule. *Action* can be - o *accept*: - the e-mail is accepted (i.e., appended to the *accept* location spcified in the configuration file) - - o *ignore*: - the e-mail is ignored (i.e., appended to the *ignore* location spcified in the configuration file) - - o *spam*: - the e-mail is considered spam (i.e., appended to the *spam* location spcified in the configuration file) ## PATTERNS Files specified in *rules* files define patterns which may be found in the headers defined by the rules. The header lines which are selected by the *Header* specifications in the *rules* file are matched against those patterns after removing the header labels from those lines. So *Subject:* *hello world* is passed to the patterns as the (trimmed) line *hello* *world*. Once a header’s content matches a pattern inspection ends with a successful match (the rule itself may specify *not*, in which case a successful match results in a failing match of the rule). Pattern files may start with file-specific comment (i.e., empty lines and lines whose first non-blank character is *#*) up to a comment line equal to *#=*. The patterns themselves may also be preceded by comment lines. Once a pattern is matched it is moved one position upward in the pattern file (including its associated comment). Pattern specifications use the following syntax (elements between square brackets are optional (when used, the square brackets are not specified), capitalized words are described below, each pattern is defined on a single line, line continuation is not supported): ``` Nr Date Expression [and Expression]* ``` This pattern indicates a match when all *Expressions* match. This syntax uses the following elements: - o *Nr*: specifies the number of times e-mail was successfully matched by the pattern. At each successful match the count is incremented (up to a maximum of 999), and the pattern rises one position in the pattern file. The field width of the number field must be at least 3 character positions wide. When adding a pattern use 1; - o *Date*: shows the date of the most recent match. Its format is *yy-mm-dd*, e.g. 23-03-31. When adding a pattern the current date could be used. - o *Expression*: defines at least one matching mode and associated expression. The *Expressions* themselves use the following syntax: ``` MatchMode [not] Spec ``` The selected headers are matched against *Spec* using the specified *MatchMode*. The *not* keyword is optional. When specified (omit the square brackets) the result of the match is negated. When multiple *Expression* specifications are joined by *and* keywords, then the final pattern results in a match if all *Expression* specifications indicate a match. Once an *Expression* does not indicate a match, then subsequent *Expressions* are not evaluated and headers do not match the pattern. When using *not* the e-mail may not match the *Spec* specification. E.g, when e-mail should contain a *To:* header or a *Cc:* header the following rule can be used (cf. section **EXAMPLES**:) ``` if To: ./match/noto and Cc: ./match/noto spam ``` with *match/noto*: ``` 1 23-05-10 p not ’.’ ``` Note that *Spec* must be surrounded by single quotes. To use a single quote inside a *Spec* escape it (as *\\’*). In general: the character following a backslash is used as-is, removing the backslash from the *Spec* (e.g., to construct a *Spec* containing *’\\n* specify *\\’\\\\n*). There are five types of *MatchModes. MatchModes* using regular expressions use extended regular expression patterns: prefix multipliers and bounding-characters by backslashes when they should be interpreted as ordinary characters (i.e., *\*, +, ?, ^, $, \|, (, ), \[, \], {, }* should be escaped when used as literal characters). - o *c*: (cidr) matches *\[d{1.3}.d{1.3}.d{1.3}.d{1.3}\]* IP address patterns found in selected headers against the *Spec* pattern, e.g., *’1.22.333.0/24’*; - o *i*: (ignore case) matches the selected headers case-insensitively against the *Spec* pattern, e.g., *’yes’*; - o *n*: (no-case pattern) matches the selected headers case-insensitively against the *Spec* regular expression pattern, e.g., *’news.\*\\.\*@reply’*; - o *p*: (pattern) matches the selected headers (case-sensitively) against the *Spec* regular expression pattern; - o *s*: (string text) matches the selected headers as specified (case-sensitively) against the *Spec* pattern, e.g., *’noreply@’*. This matchmode can also be used to match IP version 6 patterns if the cidr-range uses multiples of 16-bit values, e.g., *’\[2a02:7a60:’*; - o *$*: (script or program): *Spec* calls a script or program. The selected headers are passed to the script or program, which must return 0 to indicate a match and any other value to indicate no match. The script or program may specify arguments, but in addition two additional arguments are passed to the script or program, the first one being the name of a file containing the selected e-mail headers, the second one being the name of the file containing the e-mail’s content. The script or program does not use the *PATH* environment variable, so the path to the script or program (possibly starting with *~/*) must be specified. When *Spec* starts with the word *SHELL\]* then the script or program is called as */bin/sh -c* *’script/program \[arguments\]’*. Examples: ``` # call program in the user’s bin dir. with 4 arguments ’~/bin/program arg1, arg2’ # call script in the user’s bin dir. with 4 arguments # via /bin/sh ’SHELL ~/bin/script arg1, arg2’ ``` ## OPTIONS Short options, when defined, are provided between parentheses immediately following their long option equivalents. Several parameters specify locations of files written or used by **filtermail**. If a location specification starts with *~/* then the tilde-character is replaced by the *base* directory specified as **filtermail**’s argument. Otherwise, if the location does not start with a slash (*/*) character then the location is prefixed by the path of the directory containing the configuration file. Some options can also be specified in the configuration file (cf. section **CONFIGURATION**). Options that cannot be specified in the configuration file are marked as *NO-CONFIG*. - o **--accept**(\[:HDRS:\]path)\ the path receiving accept-marked mail. Mail not matching any rules is also accepted. Prepend *:HDRS:* to *path* to log the mail’s *From:* and *Subject:* headers to *path*. To completely ignore accept-marked mail omit this option (although ignoring accept-marked mail is probably undesirable). Examples (commonly *frank* (used in the example) is replaced by the user’s username): ``` --accept /var/mail/USER --accept :HDRS:/var/mail/USER ``` - - o **--cls**\ \ By default **filtermail** clears the terminal’s screen (calling *tput* *clear*). If the screen should not be cleared option *--cls no* can be specified. A default value can also be specified in the configuration file, using ``` cls: no ``` in which case option *--cls yes* can be specified to overrule the configuration file’s specification. This option is only used when the *--inspect* option is specified. - - o NO-CONFIG **--config**=*path* (**-c**)\ the path to the configuration file (default: ~/etc/mailfilter/config). (see also section *CONFIGURATION* below). - - o NO-CONFIG **--expire**=*date* (**-e**)\ no mail is read, but patterns having date stamps older than *date* are removed from their files and are stored in files having extension *.exp* (cf. section **PATTERNS** below). Use format *yy-mm-dd* when specifying *date*. This option and *--interactive* (see below) cannot both be specified. Example: ``` --date 23-01-31 ``` - - o NO-CONFIG **--help** (**-h**)\ a summary of **filtermail**’s usage is written to *cout* and **filtermail** ends, returning 0 to the shell. - - o **--ignore**(\[:HDRS:\]path)\ the path receiving ignore-marked mail. Prepend *:HDRS:* to *path* to log the mail’s *From:* and *Subject:* headers to *path*. To completely ignore ignore-marked mail omit this option. Examples: ``` --ignore ~/Mail/ignore --ignore :HDRS: ~/Mail/ignore ``` - - o NO-CONFIG **--inspect** (**-I**)\ when specifying this option **filtermail** expects a received e-mail file at its standard input showing the domain name of the sender, its IP address, its country of origin and the cidr-range containing the received IP address. E.g., ``` from renxincj.com (unknown [104.223.188.228]) IP = `104.223.188.228’, Country: US, CIDR = 104.223.128.0/17 ``` See section **FILTERMAIL INSPECT** for additional info. - - o NO-CONFIG **--interactive** (**-i**)\ the matching result of matching patterns is interactively simulated (no mail is read). This option and *--expire* cannot both be specified. - - o **--IP4-pattern**=*regex* (**-p**)\ by default IP4 patterns are recognized using the regular expression pattern *\\\[((\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3}))\\\]*, matching addresses like *\[123.11.22.33\]* in specified headers. This format is commonly encountered in *Received:* headers. When different IP4-patterns should be recognized then this option can be used to specify another regular expression for matching IP4-patterns in headers. - The required regex is an extended regular expression in which the characters *\*, +, ?, ^, $, \|, (, ), \[, \], {,* and *}* must be escaped when used as literal characters. Perl-like escape sequences *\\b* (word-boundary), *\\d* (digit character), *\\s* (white-space character), and *\\w* (alpha-numeric character) are supported. Their capital variants represent their complementary sets. - The regexes specified using this option must have at least five elements, addressable using indices 1..5:\ \* element 1 defines the complete IP4 pattern,\ \* element 2 defines the most significant (4th) octet (available in bits 24 thru 31 in the converted binary address),\ \* element 3 defines the 3th IP4 octet (available in bits 16 thru 23 in the converted binary address),\ \* element 4 defines the 2nd IP4 octet (available in bits 8 thru 15 in the converted binary address),\ \* element 5 defines the least significant (1st) IP4 octet (available in bits 0 thru 7 in the converted binary address). - - o **--log**=*spec* (**-l**)\ specify the log-facility. By default no logging is used.\ The *spec* argument can be the location of a file receiving the log messages.\ If *spec* uses the format *facility:level* (*facility* and *level* being the names defined in the [**syslog**(3)](/syslog/3.md) man-page omitting their *LOG\_* prefixes) then the log messages are written by *syslog*. Examples: ``` --log log/log # uses log/log in the configuration # file’s directory --log USER:NOTICE # uses syslog ``` - - o **--preamble**\ merely verify that the basic options are correctly specified. No mail is read, but all options and configuration file specifications are processed. When the verification successfully ends the message ``` Preamble successfully completed ``` is written to *cout* and **filtermail** returns 0. - - o **--received**=*content* (**-R**)\ before incoming mail reaches your computer your, e.g., organization’s mail server may have processed the incoming mail, and it may be OK to inspect only the *Received:* headers following that mail server’s *Received:* header. The content of the *Received:* header may (partially) be specified using this option, which can also be provided in the configuration file. E.g., if the *Received:* header originating from your organization’s mail server contains *ourorganization.org* then the option *-r ourorganization.org* causes all *Received:* headers to be ignored until the *Received:* header containing *ourorganization.org* has been seen. If the incoming mail doesn’t contain such a *Received:* header then all *Received:* headers are inspected. This option is only used when the *--inspect* option is specified. - - o **--rules**=*path* (**-r**)\ the location of the rules specification file (cf. section **RULES** below). - - o **--spam** *\[:HDRS:\]path*\ the path receiving spam-marked mail. Prepend *:HDRS:* to *path* to log the mail’s *From:* and *Subject:* headers to *path*. To completely ignore spam-marked mail omit this option. Examples: ``` --spam ~/Mail/spam --spam :HDRS: ~/Mail/spam ``` - - o NO-CONFIG **--syntax**\ perform a syntax check of the specified rules. Encountered syntax errors are logged in the log-file. The begin and end of the syntax check is also logged in the log file. No mail is read from the standard input stream. - - o NO-CONFIG **--version** (**-v**)\ writes **filtermail**’s version to *cout* and ends, returning 0. ## FILTERMAIL INSPECT When specifying the *--inspect* (*-I*) option **filtermail** expects a received e-mail file at its standard input showing the domain name of the sender, its IP address, its country of origin and the cidr-range containing the received IP address. E.g., ``` from renxincj.com (unknown [104.223.188.228]) IP = `104.223.188.228’, Country: US, CIDR = 104.223.128.0/17 ``` Mail handling programs (e.g., [**mutt**(1)](/mutt/1.md)) allow its users to pipe an e-mail file to a program, so the received e-mail can be inspected from inside the mail handling program. E.g., with *mutt* typing *\|* shows the prompt ``` Pipe to command: ``` and assuming that the **filtermail** program is available in the user’s *PATH* environment variable enter \`**filtermail** -I’ to pass the received e-mail to **filtermail**: ``` Pipe to command: filtermail -I ``` Depending on the content of the *Received:* headers **filtermail**’s output shows the domain name of the sender, its IP address, its country of origin and the cidr-range containing the received IP address. E.g., ``` from renxincj.com (unknown [104.223.188.228]) IP = `104.223.188.228’, Country: US, CIDR = 104.223.128.0/17 ``` IP version 6 addresses are also inspected, producing output like ``` from mail.resoascijournal.info (s857e6ba3.fastvps-server.com \ [2a03:f480:2:8::3f]) IP = `2a03:f480:2:8::3f’, Country: EE, CIDR = 2a03:f480:2::/48 ``` If the received e-mail is considered conspicuous (e.g., spam or mail to ignore) then the cidr range could be added to a file like *suspect.cidr*. Once more e-mails from the suspected cidr-range are received, the range could be added to, e.g., *~/etc/filtermail/spam/cidr* or to *~/etc/filtermail/ignore/cidr*, using a pattern line like ``` 1 23-05-10 s ’2a03:f480:2:’ ``` When the option *--cls* is specified as *yes* (either as command-line option or in the configuration file) then the terminal screen will be cleared before showing *--inspect’s* output. When the option *--received* is specified *Received:* headers appearing before the *Received:* header containing the content specified at the *--received* option are ignored. ## EXAMPLES - o *configure* **filtermail** *as a valid mail handling program:* Commonly incoming mail servers define a directory where valid mail handling programs (or links to those programs) are listed. E.g., [**sendmail**(8)](/sendmail/8.md) uses the \`sendmail restricted shell’ (*/etc/mail/smrsh*) directory. If **filtermail** is installed in a standard user-accessible directory (e.g., */usr/bin*) then the *smrsh* directory should contain the link ``` filtermail -> /usr/bin/filtermail ``` Once the **filtermail** program is recognized by the incoming mail server users may filter incoming e-mail through **filtermail** using, e.g., a *~/.forward* file. Such *.forward* files ignore empty lines and end-of-line comment (starting at *#*). Assuming a standard **filtermail**-configuration (cf. section **CONFIGURATION**) and assuming that user *frank’s* home-directory is */home/frank*, then */home/frank/.forward* should contain the following line: ``` "|/usr/bin/filtermail /home/USER" ``` Note the double quotes: they are required because **filtermail** is called with an argument. - o *a configuration file*: The following configuration file specifies that the *rules* and log files are located in the configuration file’s directory and defines paths for all three mail categories: ``` rules: rules log: log/log accept: /var/spool/mail/USER spam: ~/Mail/spam ignore: ~/Mail/ignore # as illustration of a ’dont-expire:’ specification: #dont-expire: ./ignore/from ``` - o *rule specifications*: Filtering rules are defined in the file specified by the *--rules* option or in the *rules:* line of the configuration file. Note that the pattern files must start with *./*. ``` if From: ./ignore/from ignore if Subject: ./spam/nolowercase spam # inspect all Received... headers: if Received+ ./spam/cidr spam # a To: or Cc: header is required: if To: ./match/noto and Cc: ./match/noto ignore ``` The final rule uses the pattern in *./match/tocc* (shown below) specifying the \`any character’ (*’.’*) regular expression: if the *To:* header is empty (which is also true if there is no *To:* header, then the *not To:* condition matches. The same holds true for the second condition. So if neither condition matches there is neither a *To:* nor a *Cc:* header, in which case the e-mail is sent to the *spam* destination. Also note that, according to *De Morgan’s* rule, a *not X and* *not Y* rule is identical to an *X or Y* rule. - o *pattern specifications*: ``` # the ./spam/nolowercase pattern: 1 23-05-10 not p ’[a-z]’ # e.g., the ./match/noto as used in the above example # requiring either a To: or a Cc: header: 1 23-05-10 p not ’.’ ``` ## FILES By default the configuration file is expected in the subdirectory *etc/filtermail* of the directory specified as **filtermail**’s argument. ## SEE ALSO [**mutt**(1)](/mutt/1.md), [**pattern**(3bobcat)](/pattern/3bobcat.md), [**regcomp**(3)](/regcomp/3.md), [**sendmail**(8)](/sendmail/8.md), [**syslog**(3)](/syslog/3.md), [**tput**(1)](/tput/1.md), [**whois**(1)](/whois/1.md) ## BUGS None reported. ## AUTHOR Frank B. Brokken (f.b.brokken@rug.nl).