Sendmail | ClamAV | DSPAM | Dovecot | Sieve

Mail LogosAfter a considerable amount of research and time I have finally managed to put a number of packages together and set up a working email server. Before we start let me first apologize for the length of this post. Getting all the components installed and configured takes a bit of work.

Now, let's begin with a brief overview of each of the key components. I'll expand more on the specifics as we progress.

Software Package Description
Sendmail 8.14 pkg://solaris/service/network/smtp/sendmail Likely already installed on your system, Sendmail is a general purpose internetwork email routing facility that supports many kinds of mail-transfer and delivery methods, including the Simple Mail Transfer Protocol (SMTP) used for email transport over the Internet.
ClamAV 0.98.4 pkg://solarismmedia/antivirus/clamav
Clam AntiVirus (ClamAV) is a free and open-source, cross-platform antivirus software tool-kit able to detect many types of malicious software, including viruses.
DSPAM 3.10.2 pkg://solarismmedia/mail/dspam DSPAM is a scalable and open-source content-based spam filter designed for multi-user enterprise systems. On a properly configured system, many users experience results between 99.5% - 99.95%, or one error for every 200 to 2000 messages.
Dovecot 2.2.13 pkg://solarismmedia/mail/dovecot Dovecot is an open source IMAP and POP3 email server for Linux/UNIX-like systems, written with security primarily in mind. Dovecot is an excellent choice for both small and large installations. It's fast, simple to set up, requires no special administration and it uses very little memory.
Sieve 0.4.3 pkg://solarismmedia/mail/dovecot-pigeonhole The Pigeonhole project provides Sieve support as a plugin for Dovecot's Local Delivery Agent (LDA) and also for its LMTP service. The plugin implements a Sieve interpreter, which filters incoming messages using a script specified in the Sieve language (RFC 5228). The Sieve script is provided by the user and, using that Sieve script, the user can customize how incoming messages are handled. Messages can be delivered to specific folders, forwarded, rejected, discarded, etc.


Before we continue please note the following:

  • Embarking on this task was a training exercise for me.
  • Prior to beginning I had limited knowledge of how email systems functioned in conjunction with spam filters and anti-virus programs.
  • I still have a lot to learn and the configuration detailed below should not be considered expert advice.
  • The steps below simply outline the process I used to setup and test each of the elements to ensure the packages I created were functional and worked together.
  • Before setting up your own Solaris 11 based mail server you need to seek advice from a trained professional.
  • If you identify an issue or concern please email/comment.
  • The web interface for DSPAM is not covered below but may be documented in the future.
  • All feedback is welcome and encouraged.


  • DNS and MX records have already been setup appropriately.
  • A PostgreSQL instance is already running in your environment. It can be a dedicated server or localhost.
  • Your Solaris 11 installation is a fresh install with no existing customizations, especially related to email services.


Before beginning, let's paint a brief picture of the setup. The table below details each of the services and how they will be configured.

Software Service(s) Home/Data Directory Port
Sendmail 8.14 svc:/network/sendmail-client:default
/etc/mail 25
ClamAV 0.98.4 svc:/application/clamd:default
/test/clamav 3310
DSPAM 3.10.2 svc:/application/dspam:default /test/dspam 26
Dovecot 2.2.13 svc:/application/dovecot:default /test/dovecot 24
Sieve 0.4.3 N/A /test/dovecot N/A


Sendmail Server Certificate

The first thing we'll do is set up some certificates to use with Sendmail. For the purpose of this example we'll use a "self-signed" certificate. The Sendmail certificate will be used for SMTP based communication.

Sendmail Server Certificate
svcadm disable -t network/smtp:sendmail
cd /etc/mail
mkdir -p certs/CA
cd certs/CA
mkdir certs crl newcerts private
echo "01" > serial
cp /dev/null index.txt
sed 's|/etc/openssl|/etc/mail/certs/CA|g' /etc/sfw/openssl/openssl.cnf > openssl.cnf
# --- Create CA certificate
openssl req -new -x509 -keyout private/cakey.pem -out cacert.pem -days 365 -config openssl.cnf
# --- Create unsigned certificate and private key
openssl req -nodes -new -x509 -keyout newreq.pem -out newreq.pem -days 365 -config openssl.cnf
# --- Generate the certificate request
openssl x509 -x509toreq -in newreq.pem -signkey newreq.pem -out tmp.pem
# --- Using the certificate request, sign the certificate with the CA authority/certificate
openssl ca -config openssl.cnf -policy policy_anything -out newcert.pem -infiles tmp.pem
# --- Remove the temporary certificate request
rm -f tmp.pem
cd /etc/mail/certs
# --- Create the symbolic links that Sendmail will use
ln -s CA/cacert.pem CAcert.pem
ln -s CA/newcert.pem cert.pem
ln -s CA/newreq.pem key.pem
# --- Secure the key file
chmod go-r key.pem
ln -s $C `openssl x509 -noout -hash < $C`.0

SMTP Client Certificate

The Sendmail that ships with Solaris 11 doesn't have password authentication capabilities (from what I have read online anyway). Relaying will be limited to the local server and our internal subnet only. If we want roaming clients to use our SMTP/Sendmail server we can issue client certificates and allow clients with certificates to relay as well.

Let's create a client certificate for our roaming user, "John Doe".

SMTP Client Certificate
cd /etc/mail/certs/CA
mkdir ../${USER}
# --- Create unsigned certificate and private key
openssl req -nodes -new -x509 -keyout ../${USER}/unsigned.pem -out ../${USER}/unsigned.pem -days 365 -config openssl.cnf
# --- Generate the certificate request
openssl x509 -x509toreq -in ../${USER}/unsigned.pem -signkey ../${USER}/unsigned.pem -out ../${USER}/request.pem
# --- Using the certificate request, sign the certificate with the CA authority/certificate
openssl ca -config openssl.cnf -policy policy_anything -out ../${USER}/cert.pem -infiles ../${USER}/request.pem
# --- Remove the unsigned certificate from the key file
sed '/-----BEGIN CERTIFICATE-----/,$d' ../${USER}/unsigned.pem > ../${USER}/key.pem
# --- Concatenate the required files into one certificate file
cat cacert.pem ../${USER}/cert.pem ../${USER}/key.pem > ../${USER}/concat.pem
# --- Create the p12 file to pass onto John Doe
openssl pkcs12 -export -in ../${USER}/concat.pem -out ../${USER}/${USER}.p12 -name "${USER} Personal Certificate"
# --- Secure the key file
chmod go-r ../${USER}/key.pem
# --- Remove the files we no longer need
rm ../${USER}/request.pem
rm ../${USER}/unsigned.pem

A file called /etc/mail/certs/johnd/johnd.p12 has been created and can be given to John. He can import that file on his client system which will allow him to send email via our server while roaming. For additional security make the password for John Doe's certificate complex so it's impossible to guess/crack.

Sendmail Access Database

Now we need to setup the access database to specify what systems (and certificates) can relay via our server.

Sendmail Access DB
echo "10.1.1           RELAY" > /etc/mail/access
echo "        RELAY" >> /etc/mail/access
openssl x509 -issuer -noout < /etc/mail/certs/CA/cacert.pem \
    | perl -ple 's/^issuer= /CERTIssuer:/' \
    | perl -ple 's/([[:^print:]<>()"+ ])/sprintf "+%02x", ord $1/ge' \
    | perl -ple 's/$/\tRELAY/' >> /etc/mail/access
# --- Rebuild access.db
makemap hash /etc/mail/access.db < /etc/mail/access

Final Sendmail Configuration

The Sendmail configuration now needs to be changed to secure and setup the environment. We'll start by removing the help file (hide our Sendmail info and version). Then we will copy the existing configuration file and modify the copy.

mv /etc/mail/helpfile /etc/mail/
touch /etc/mail/helpfile
cd /etc/mail/cf/cf

Modify as follows.

define(`confFALLBACK_SMARTHOST', `mailhost$?m.$m$.')dnl
dnl---------BELOW ADDED BY YOU------------dnl
define(`confDOMAIN_NAME', `')dnl
define(`confCACERT_PATH', `/etc/mail/certs')dnl
define(`confCACERT', `/etc/mail/certs/CAcert.pem')dnl
define(`confSERVER_CERT', `/etc/mail/certs/cert.pem')dnl
define(`confSERVER_KEY', `/etc/mail/certs/key.pem')dnl
define(`confCLIENT_CERT', `/etc/mail/certs/cert.pem')dnl
define(`confCLIENT_KEY', `/etc/mail/certs/key.pem')dnl
define(`confSMTP_LOGIN_MSG',`$j MTA ready and waiting ; $b')dnl
define(`confPRIVACY_FLAGS', `authwarnings,noexpn,novrfy')dnl
dnl---------ABOVE ADDED BY YOU------------dnl

Rebuild and install a new file.

Update Configuration
/usr/lib/sendmail -C -v root </etc/hosts   # test to make sure your new file works
cp /etc/mail/ /etc/mail/
cp /etc/mail/
# --- Configure Sendmail for open mode but we restrict using access.db
svccfg -s smtp:sendmail setprop config/local_only = boolean: false
svcadm refresh sendmail
# --- Enable Sendmail
svcadm enable sendmail

Check /var/log/syslog and dmesg for any issues or errors.


Install ClamAV

ClamAV is the anti-virus program we will use to scan emails as they arrive. We'll set it up to run as a daemon. It will accept email/streams for scanning via port 3310 on the localhost.

pkg install gcc-49 clamav freshclam-service
groupadd clamav
useradd -g clamav -d /test/clamav -s /bin/false -c "Clam AntiVirus" clamav
cp /etc/opt/SMM/clamd.conf.sample /etc/opt/SMM/clamd.conf
chmod 660 /etc/opt/SMM/clamd.conf
chown root:clamav /etc/opt/SMM/clamd.conf
cp /etc/opt/SMM/freshclam.conf.sample /etc/opt/SMM/freshclam.conf
chmod 660 /etc/opt/SMM/freshclam.conf
chown root:clamav /etc/opt/SMM/freshclam.conf

Configure ClamAV

vi /etc/opt/SMM/clamd.conf

Below is the full content of my test clamd.conf. This represents the minimum settings required.

LogFile /test/clamav/clamd.log
LogFileMaxSize 2M
LogClean yes
LogSyslog yes
PidFile /test/clamav/
DatabaseDirectory /test/clamav
TCPSocket 3310
StreamMaxLength 20M

Configure FreshClam

Now we need to configure FreshClam to automatically update the virus definitions. Again, the full content of my test freshclam.conf is below. This shows the minimum settings required. When an update is found, FreshClam downloads the files and notifies ClamAV (clamd).

vi /etc/opt/SMM/freshclam.conf

DatabaseDirectory /test/clamav
UpdateLogFile /test/clamav/freshclam.log
LogFileMaxSize 2M
LogSyslog yes
PidFile /test/clamav/
NotifyClamd /etc/opt/SMM/clamd.conf

Now, setup all the required folders, log files and service properties.

mkdir -p /test/clamav
chown clamav:clamav /test/clamav
chmod 770 /test/clamav
touch /test/clamav/freshclam.log
chmod 600 /test/clamav/freshclam.log
chown clamav /test/clamav/freshclam.log
svccfg -s clamd:default setprop clamd/home = astring: "/test/clamav"
svcadm refresh clamd
svccfg -s freshclamd:default setprop freshclamd/home = astring: "/test/clamav"
svcadm refresh freshclamd

Before we can start any services we need to download the file definitions interactively/manually. You may receive an error when it tries to notify ClamAV. This can be ignored. ClamAV isn't running yet.

/opt/SMM/clamav/bin/freshclam   #get definitions

Start the services

svcadm enable clamd
svcadm enable freshclamd

Check /test/clamav/clamd.log and /test/clamav/freshclam.log for any issues or errors.


As emails are delivered Sendmail will forward them to DSPAM for processing. DSPAM will also pass them to ClamAV to be checked for viruses. In my test environment, emails with viruses or identified as spam will have the [SPAM] tag added to the subject line and delivered to the users Junk folder.

We'll be using a PostgreSQL database to store all DSPAM metadata so the DSPAM "StorageDriver" will be set to

DSPAM will be setup to run in client/server mode. The DSPAM server will accept emails on port 26 and, once processed, they will be passed into Dovecot via LMTP. Dovecot will be running on port 24. More on that later.

Install DSPAM

pkg install dspam
mkdir /test/dspam
chmod 0770 /test/dspam

Configure DSPAM

Edit dspam.conf.

chmod o-rwx /etc/opt/SMM/dspam.conf
vi /etc/opt/SMM/dspam.conf

It's important to read and gain a good understanding all of the options in the configuration file. Some of the key settings are shown below.

Home /test/dspam
StorageDriver /opt/SMM/dspam/lib/dspam/

# Where is Dovecot
DeliveryPort 24
DeliveryIdent localhost
DeliveryProto LMTP

Preference "spamAction=tag" # { quarantine | tag | deliver } -> default:quarantine
Preference "spamSubject=[SPAM]" # { string } -> default:[SPAM]

# The PostgreSQL Server
PgSQLPort 5432
PgSQLUser dbUser
PgSQLPass dbPassword
PgSQLDb dbName
PgSQLConnectionCache 3
PgSQLUIDInSignature on # PgSQLUIDInSignature is required for the
# DSPAM aliases for retraining to work correctly.

# Where is ClamAV
ClamAVPort 3310
ClamAVResponse spam

# Configuration for Client/Server Mode
ServerPort 26
ServerQueueSize 32
ServerPID /test/dspam/

ServerMode dspam
ServerPass.Relay1 "secret"

ClientPort 26
ClientIdent "secret@Relay1"

GroupConfig /test/dspam/group

DSPAM Aliases

Now we need to setup spam and notspam aliases. These mailboxes can be used by users to retrain DSPAM. Emails that are incorrectly categorised as spam can be sent to for retraining. Spam that has failed to be identified can be sent to for retraining.

vi /etc/mail/aliases

Append the following

spam: "|/opt/SMM/dspam/bin/dspam --client --user root --class=spam --source=error"
notspam: "|/opt/SMM/dspam/bin/dspam --client --user root --class=innocent --source=error"

Update the alias database


PostgreSQL for DSPAM

A single PostgreSQL database will be used to store the DSPAM data as well as the Dovecot user information. Let's create the role (user) and the database.

Note: Substitute dbUser, dbPassword and dbName with something more appropriate. Keep in mind that these need to match the database details in the dspam.conf file.

On the database server:

su - postgres

ALTER ROLE dbUser PASSWORD 'dbPassword';

Before importing the DSPAM database schemas, we are going to create a procedural language in the DSPAM database. This is done using the command below:

/path/to/postgresql/createlang plpgsql dbName 

Now we need to update pg_hba.conf to allow connectivity from the mail server.

vi /path/to/pg_hba.conf

Add the following. Substitute with your mail server details. If you are running PostgreSQL on the same host as your mail system you should be able to use

host dbName dbUser md5

Now we need to reload the PostgreSQL configuration for the changes to take effect. On the database server, reload the configuration and go back into PostgreSQL but this time we'll connect to the DSPAM database with dbUser.

su - postgres
/path/to/postgresql/pg_ctl -D {DATA DIRECTORY} reload
/path/to/postgresql/psql -d dnName -U dbUser

Now, import the schemes to create the DSPAM tables.

\i /opt/SMM/dspam/share/tools/tools.pgsql_drv/pgsql_objects.sql
\i /opt/SMM/dspam/share/tools/tools.pgsql_drv/virtual_users.sql

You may receive some warnings when the import scripts try to perform an 'analyze'. You can safely ignore these.


Configure the DSPAM Service and start DSPAM

On the mail server:

svccfg -s dspam:default setprop dspam/home = astring: "/test/dspam"
svcadm refresh dspam
svcadm enable dspam

DSPAM and Sendmail

Sendmail needs to be configured to redirect incoming email to DSPAM.

svcadm disable sendmail
cd /etc/mail/cf/cf

Make the following changes.

define(`confPRIVACY_FLAGS', `authwarnings,noexpn,novrfy')dnl
dnl---------LINES BELOW FOR DSPAM------------dnl
define(`LOCAL_MAILER_PATH', `/opt/SMM/dspam/bin/dspam')dnl
define(`LOCAL_MAILER_ARGS', `dspam "--client --lmtp-recipient=${recipient} --deliver=innocent" --user $u -d %u')dnl
dnl---------LINES ABOVE FOR DSPAM------------dnl

Rebuild and install your new file.

/usr/lib/sendmail -C -v root </etc/hosts
cp /etc/mail/ /etc/mail/
cp /etc/mail/
svcadm enable sendmail

Training DSPAM

Before we can benefit from DSPAM it has to be trained. The following procedure provides a significant number of sample "Spam" and "Ham" emails and tells DSPAM what each one is. This allows DSPAM to analyse each sample and learn common patterns. The sample files used are quite dated but they do provide a reasonable starting point.

First we need to create a global user that we'll use for the training. Other users benefit from what is learnt through global. It will be used as a first point of reference for their own emails.

useradd -g users -d /dev/null -s /bin/false global
echo "globalgroup:classification:*global" > /test/dspam/groups
/opt/SMM/dspam/bin/dspam_admin add pref global trainingMode TOE
svcadm restart dspam

Now for the training.

mkdir ~/train
cd ~/train
wget ""
wget ""
wget ""
wget ""
wget ""
wget ""
wget ""
wget ""
for i in $( ls *.tar.bz2 ); do gtar jxf $i; done
#The following can take a while
/opt/SMM/dspam/bin/dspam_train global spam easy_ham
/opt/SMM/dspam/bin/dspam_train global spam_2 hard_ham

Dovecot | Sieve

Install Dovecot and Sieve

Reading through the Dovecot wiki may seem a little overwhelming at first but getting a basic Dovecot instance up and running is pretty painless.

Firstly, let's install Dovecot and Sieve for Dovecot (dovecot-pidgeonhole).

pkg install dovecot dovecot-pideonhole

Create the home directories and the Dovecot users.

mkdir /test/dovecot
mkdir /test/dovecot/users
mkdir /test/dovecot/home
chmod a+rwX /test/dovecot/users
chmod a+rwX /test/dovecot/home
useradd -d /test/dovecot -s /bin/false dovenull
useradd -d /test/dovecot -s /bin/false dovecot

Dovecot Certificate

Just like Sendmail, if you want to use a secure connection via IMAP or POP3 we need to create the appropriate certificates and keys for Dovecot to use.

cd /etc/mail/certs/CA
mkdir ../${USER}
openssl req -nodes -new -x509 -keyout ../${USER}/unsigned.pem -out ../${USER}/unsigned.pem -days 365 -config openssl.cnf
openssl x509 -x509toreq -in ../${USER}/unsigned.pem -signkey ../${USER}/unsigned.pem -out ../${USER}/request.pem
openssl ca -config openssl.cnf -policy policy_anything -out ../${USER}/cert.pem -infiles ../${USER}/request.pem
sed '/-----BEGIN CERTIFICATE-----/,$d' ../${USER}/unsigned.pem > ../${USER}/key.pem
chmod go-r ../${USER}/key.pem
rm ../${USER}/request.pem
rm ../${USER}/unsigned.pem

Configure Dovecot

Create and secure the configuration files.

cd /etc/opt/SMM/dovecot
/opt/SMM/dovecot/bin/doveconf -n > dovecot.conf
touch dovecot-sql.conf
chmod 660 dovecot.conf
chmod 660 dovecot-sql.conf

Modify dovecot.conf for your environment.

vi dovecot.conf

An example configuration is shown below.

# OS: SunOS 5.11 i86pc
base_dir = /test/dovecot
protocols = imap pop3 lmtp
mail_home = /test/dovecot/home/%n
mail_location = mdbox:/test/dovecot/users/%n
namespace inbox {
inbox = yes
separator = /
location =
mailbox Drafts {
special_use = \Drafts
auto = subscribe
mailbox Junk {
special_use = \Junk
auto = create
mailbox Sent {
special_use = \Sent
auto = subscribe
mailbox Trash {
special_use = \Trash
auto = subscribe
prefix =
passdb {
driver = sql
args = /etc/opt/SMM/dovecot/dovecot-sql.conf
userdb {
driver = sql
args = /etc/opt/SMM/dovecot/dovecot-sql.conf
service lmtp {
inet_listener lmtp {
address = ::1
port = 24
protocol lmtp {
mail_plugins = $mail_plugins sieve
plugin {
sieve = /test/dovecot/home/%n/.dovecot.sieve
sieve_dir = /test/dovecot/home/%n/sieve
sieve_before = /test/dovecot/sieve.before
postmaster_address =
sendmail_path = /usr/lib/sendmail
rejection_subject = Rejected: %s
rejection_reason = Your message to <%t> was automatically rejected:%n%r
log_path = /test/dovecot/dovecot.log
ssl_cert = ssl_key =

Now modify dovecot-sql.conf for your PostgreSQL connection and settings.

vi dovecot-sql.conf

driver = pgsql
connect = dbname=dbName user=dbUser password=dbPassword
default_pass_scheme = BLF-CRYPT
password_query = SELECT username AS user, password FROM dovecot_users \
WHERE username = '%n' AND active = 'Y'
user_query = SELECT uid, gid FROM dovecot_users \
WHERE username = '%n' AND active = 'Y'
iterate_query = SELECT username AS user FROM dovecot_users

PostgreSQL for Dovecot

On the database server, go back into PostgreSQL. We'll use the same database that was setup for DSPAM.

/path/to/postgresql/psql -d dnName -U dbUser

Create the user table for Dovecot.

CREATE TABLE dovecot_users (
username VARCHAR(128) NOT NULL,
password VARCHAR(128) NOT NULL,

Create Mail User(s)

Let's create our account for John Doe.

On the mail server:

useradd -uid 2000 -g users -d /dev/null -s /bin/false johnd
# --- Generate the BCRYPT hash
/opt/SMM/dovecot/bin/doveadm pw -s BLF-CRYPT

Go back into PostgreSQL on the database server and add the user to the Dovecot table.

On the database server:

/path/to/postgresql/psql -d dnName -U dbUser

INSERT INTO dovecot_users (
username, uid, gid, password)

Sieve Junk

Now we need to create the Sieve rule to move Spam into the Junk folder of users.

We'll place this rule in the sieve.before file.

On the mail server:

vi /test/dovecot/sieve.before

require ["fileinto","envelope"];

if anyof (header :contains "X-DSPAM-Result" "Spam", header :contains "X-DSPAM-Result" "Virus") {
fileinto "Junk";

Compile the Sieve script.

/opt/SMM/dovecot-pigeonhole/bin/sievec /test/dovecot/sieve.before

Start Dovecot

Configure the Dovecot Service and start Dovecot

svccfg -s dovecot:default setprop dovecot/home = astring: "/test/dovecot"
svccfg -s dovecot:default setprop dovecot/ulimit = astring: "2000"
svcadm refresh dovecot
svcadm enable dovecot

Final Cleanup

rm /etc/opt/SMM/clamav-milter.conf.sample
rm /etc/opt/SMM/clamd.conf.sample
rm /etc/opt/SMM/freshclam.conf.sample
rm /etc/opt/SMM/dovecot/README


You should now have an operational SMTP, POP3 and IMAP server with SMARTTLS (SSL) support.

Murray Blakeman