an effective anti-spam with milter - milter manager

milter development with Ruby

milter development with Ruby — A tutorial for Ruby bindings

About this document

This document describes how to develop a milter with Ruby bindings for the milter library provided by milter manager.

See Developer center abount milter protocol.

Install

You can specify --enable-ruby-milter option to configure script if you want to develop milter writing Ruby. You can install packages on Debian GNU/Linux, Ubuntu and CentOS because there are deb/rpm packages for those platforms.

Debian GNU/Linux or Ubuntu:

% sudo aptitude -V -D -y install ruby-milter-core ruby-milter-client ruby-milter-server

CentOS:

% sudo yum install -y ruby-milter-core ruby-milter-client ruby-milter-server

You can specify --enable-ruby-milter option to configure script if there are no package in your environment.

% ./configure --enable-ruby-milter

You can confirm installed library version.

% ruby -r milter -e 'p Milter::VERSION'
[1, 8, 0]

You have succeeded to install ruby-milter if you can see version information.

Summary

Milter written in Ruby is followings:

require 'milter/client'

class Session < Milter::ClientSession
  def initialize(context)
    super(context)
    # Initialize
  end

  def connect(host, address)
    # ...
  end

  # Other callback definitions
end

command_line = Milter::Client::CommandLine.new
command_line.run do |client, _options|
  client.register(Session)
end

Let's write the milter that can reject a mail includes specified regular expression.

Callbacks

Milter callback methods are called for each event. Almost events have additional information. You can pass additional information via callback parameters or macro. This document describes callback parameters.

This is the list of callback methods and parameters.

connect(host, address)

This method is called when SMTP client connects to SMTP server.

host is hostname of SMTP client. address is IP Address of SMTP client.

For example, connetct from localhost.

host

"localhost"

address

This is a Milter::SocketAddress::IPv4 object represents inet:45875@[127.0.0.1] .

helo(fqdn)

This method is called when SMTP client sends HELO or EHLO command.

fqdn is FQDN reported via HELO/EHLO.

For example, SMTP client sends "EHLO mail.example.com".

fqdn

"mail.example.com"

envelope_from(from)

This method is called when SMTP client sends MAIL command.

from is sender mail address reported via MAIL command.

For exapme, SMTP client sends "MAIL FROM: <user@example.com>"

from

"<user@example.com>"

envelope_recipient(to)

This method is called when SMTP client send RCPT command. This method is called twice if SMTP client send RCPT command twice.

to is recipient mail address reported via RCPT command.

For example, SMTP client sends "RCPT TO: <user@example.com>"

to

"<user@example.com>"

data

This method is called when SMTP client sends DATA command.

header(name, value)

This method is called N times. N is the number of headers included in the mail.

name is header name. value is header value.

For example, there is a header which is "Subject: Hello!"

name

"Subject"

value

"Hello!"

end_of_header

This method is called when milter has finished processing header of the mail.

body(chunk)

This method is called when milter has received mail body. This method is called only once if mail body is small enough. This methos is called multiple times if mail body is large.

chunk is splitted body.

For examle, if the mail body includes "Hi!", this method is called only once.

chunk

"Hi!"

end_of_message

This method is called when SMTP client sends "<CR><LF>.<CR><LF>" that represents end of data.

abort(state)

This method is called when SMTP transaction is resetted. In particular, after end_of_message and SMTP client sends RSET.

state is a object represents timing of calling abort.

unknown(command)

This method is called when SMTP client sends unknown command in milter protocol.

command is command name.

reset

This method is called when initialize and finish mail transaction.

mail transaction has finished at:

  • Called abort callback.

  • Call reject in milter.

  • Call temporary_failure in milter.

  • Call discard in milter.

  • Call accept in milter.

finished

This method is called when completed milter protocol. TODO: write about timing

Using callbacks

We want to write the milter which will reject mails match against specified regular expression. The regular expression matches against subject and message body. It is necessary for us to select header callback and body callback. Template is as following.

require 'milter/client'

class MilterRegexp < Milter::ClientSession
  def initialize(context, regexp)
    super(context)
    @regexp = regexp
  end

  def header(name, value)
    # ... Check subject header
  end

  def body(chunk)
    # check chunk
  end
end

command_line = Milter::Client::CommandLine.new
command_line.run do |client, _options|
  # We want to reject mails include "viagra"
  client.register(MilterRegexp, /viagra/i)
end

Check subject header

First, let's check subject header.

class MilterRegexp < Milter::ClientSession
  # ...
  def header(name, value)
    case name
    when /\ASubject\z/i
      if @regexp =~ value
        reject
      end
    end
  end
  # ...
end

Reject mails if header name is matched "subject" and its value matches against specified regular expression.

Operation check

Let's try to execute this milter.

Now, your milter is as following.

require 'milter/client'

class MilterRegexp < Milter::ClientSession
  def initialize(context, regexp)
    super(context)
    @regexp = regexp
  end

  def header(name, value)
    case name
    when /\ASubject\z/i
      if @regexp =~ value
        reject
      end
    end
  end

  def body(chunk)
    # Check cunk
  end
end

command_line = Milter::Client::CommandLine.new
command_line.run do |client, _options|
  # We want to reject mails include "viagra"
  client.register(MilterRegexp, /viagra/i)
end

You can execute this file as milter below command if you save this file as "milter-regexp.rb". We add "-v" option because it is easy to check operation.

% ruby milter-regexp.rb -v

In this case (default), milter run in foreground. You can check operation via other terminal.

milter-test-server is very useful to test milter. Milter which is written in Ruby is launched on "inet:20025@localhost".

% milter-test-server -s inet:20025
status: pass
elapsed-time: 0.00254348 seconds

You can see "status: pass" in your terminal if you can connect properly. Let's check another terminal.

[2010-08-01T05:44:34.157419Z]: [client][accept] 10:inet:55651@127.0.0.1
[2010-08-01T05:44:34.157748Z]: [1] [client][start]
[2010-08-01T05:44:34.157812Z]: [1] [reader][watch] 4
[2010-08-01T05:44:34.157839Z]: [1] [writer][watch] 5
[2010-08-01T05:44:34.158050Z]: [1] [reader] reading from io channel...
[2010-08-01T05:44:34.158140Z]: [1] [command-decoder][negotiate]
[2010-08-01T05:44:34.158485Z]: [1] [client][reply][negotiate] #<MilterOption version=<6> action=<add-headers|change-body|add-envelope-recipient|delete-envelope-recipient|change-headers|quarantine|change-envelope-from|add-envelope-recipient-with-parameters|set-symbol-list> step=<no-connect|no-helo|no-envelope-from|no-envelope-recipient|no-end-of-header|no-unknown|no-data|skip|envelope-recipient-rejected>>
[2010-08-01T05:44:34.158605Z]: [1] [client][reply][negotiate][continue]
[2010-08-01T05:44:34.158895Z]: [1] [reader] reading from io channel...
[2010-08-01T05:44:34.158970Z]: [1] [command-decoder][header] <From>=<<kou+send@example.com>>
[2010-08-01T05:44:34.159092Z]: [1] [client][reply][header][continue]
[2010-08-01T05:44:34.159207Z]: [1] [reader] reading from io channel...
[2010-08-01T05:44:34.159269Z]: [1] [command-decoder][header] <To>=<<kou+receive@example.com>>
[2010-08-01T05:44:34.159373Z]: [1] [client][reply][header][continue]
[2010-08-01T05:44:34.159485Z]: [1] [reader] reading from io channel...
[2010-08-01T05:44:34.159544Z]: [1] [command-decoder][body] <71>
[2010-08-01T05:44:34.159656Z]: [1] [client][reply][body][continue]
[2010-08-01T05:44:34.159774Z]: [1] [reader] reading from io channel...
[2010-08-01T05:44:34.159842Z]: [1] [command-decoder][define-macro] <E>
[2010-08-01T05:44:34.159882Z]: [1] [command-decoder][end-of-message] <0>
[2010-08-01T05:44:34.159941Z]: [1] [client][reply][end-of-message][continue]
[2010-08-01T05:44:34.160034Z]: [1] [command-decoder][quit]
[2010-08-01T05:44:34.160081Z]: [1] [agent][shutdown]
[2010-08-01T05:44:34.160118Z]: [1] [agent][shutdown][reader]
[2010-08-01T05:44:34.160162Z]: [1] [reader][eof]
[2010-08-01T05:44:34.160199Z]: [1] [reader] shutdown requested.
[2010-08-01T05:44:34.160231Z]: [1] [reader] removing reader watcher.
[2010-08-01T05:44:34.160299Z]: [1] [writer][shutdown]
[2010-08-01T05:44:34.160393Z]: [0] [reader][dispose]
[2010-08-01T05:44:34.160452Z]: [client][finisher][run]
[2010-08-01T05:44:34.160492Z]: [1] [client][finish]
[2010-08-01T05:44:34.160536Z]: [1] [client][rest] []
[2010-08-01T05:44:34.160578Z]: [sessions][finished] 1(+1) 0

You cannot connect to milter if you can see nothing in this terminal. Please check to launch milter or to specify correct address to milter-test-server.

Let's check operation when milter process a mail included "viagra" in subject. You can reproduce the mail as following command.

% milter-test-server -s inet:20025 --header 'Subject:Buy viagra!!!'
status: reject
elapsed-time: 0.00144477 seconds

You can check expected result because you can see "status: reject" in you terminal.

In another terminal, you can see log as followings.

...
[2010-08-01T05:49:49.275257Z]: [2] [command-decoder][header] <Subject>=<Buy viagra!!!>
[2010-08-01T05:49:49.275405Z]: [2] [client][reply][header][reject]
...

The mitler reject the mail when process subject header.

milter manager provides usuful tools and libraries.

Check message body

Let's check message body.

class MilterRegexp < Milter::ClientSession
  def body(chunk)
    if @regexp =~ chunk
      reject
    end
  end
end

Reject mails if message body chunk matches against specified regular expression.

Let's try. milter-test-server can specify message body via "--body" option.

% tool/milter-test-server -s inet:20025 --body 'Buy viagra!!!'
status: reject
elapsed-time: 0.00195496 seconds

It is expected result because you can see "status: reject" in your terminal.

Problems

There are some problems in this milter because this milter simplify for this tutorial.

  1. Include MIME encoded header. Decoded "=?ISO-2022-JP?B?GyRCJVAlJCUiJTAlaRsoQnZpYWdyYQ==?=" includes "viagra", but original header value does not match against specified regular expression. And milter does not reject this mail

  2. The word splitted by chunk in message body. For exapmle, Specified regular expression does not match if first chunk has "via" and second one has "gra". And milter does not reject this mail.

You can solve problems about header if you use NKF library as following.

require 'nkf'

class MilterRegexp < Milter::ClientSession
  # ...
  def header(name, value)
    case name
    when /\ASubject\z/i
      if @regexp =~ NKF.nkf("-w", value)
        reject
      end
    end
  end
  # ...
end

You can solve problems about message body if milter check message body when milter receives all chunks.

class MilterRegexp < Milter::ClientSession
  ...
  def initialize(context, regexp)
    super(context)
    @regexp = regexp
    @body = ""
  end

  def body(chunk)
    if @regexp =~ chunk
      reject
    end
    @body << chunk
  end

  def end_of_mesasge
    if @regexp =~ @body
      reject
    end
  end
  ...
end

You can test multiple chunks as following.

% milter-test-server -s inet:20025 --body 'Buy via' --body 'gra!!!'
status: reject
elapsed-time: 0.00379063 seconds

Mails are rejected if it includes multiple chunks.

However, in this case, all messages place on memory. This is performance problem. In addition, this milter does not work if message body is BASE64 encoded.

Mail is useful library that handle mails.

Conclusion

This document describes how to write milter in Ruby. It is easy to write milter in Ruby. Let's try to write milter in Ruby!!