Mail on your own domain
Hosted mail or your own server
Mail is the hardest part of running a website, and the part most people should not run themselves. A mailbox is easy to create; getting a message into a stranger inbox is the hard problem, because every large provider decides by reputation, not by your good intentions.
| Hosted mail service | Your own mail server | |
|---|---|---|
| Setup | Change MX records and create mailboxes | Install, configure and maintain two services |
| Deliverability | Handled by a provider with a reputation | Yours to build from zero, one IP at a time |
| Cost | Per user, per month | The server you already pay for |
| Control | Limited to what the provider offers | Every setting, every log line |
| Maintenance | The provider patches it | You patch Postfix, Dovecot and the filters |
| Good for | A business, a real mailbox, a shared team | Learning, many addresses, a mail gateway you understand |
A small site with one mailbox is better served by a hosted service or by the mail of the hosting provider from lesson 4. Build your own when you want to learn how mail works, when you need many addresses, or when you already run a server and want the logs in your own hands. The rest of this lesson does exactly that, on the same Ubuntu machine.
MX, SPF, DKIM, DMARC
Four records decide whether your mail is delivered, and each answers a different question.
- MX says where mail for the domain goes.
- SPF lists the servers allowed to send mail for the domain.
- DKIM signs each message with a key, so a receiver can check it was not changed.
- DMARC tells the receiver what to do when SPF and DKIM both fail to align with the domain in the From header, and where to send reports. It passes when SPF or DKIM gives an aligned pass.
With a hosted service, the provider gives you the exact values and you paste them into the DNS panel. With your own server, they look like this:
example.com. 3600 IN MX 10 mail.example.com. mail.example.com. 3600 IN A 203.0.113.10 example.com. 3600 IN TXT "v=spf1 mx -all" mail._domainkey.example.com. 3600 IN TXT "v=DKIM1; k=rsa; p=MIIBIjANBg..." _dmarc.example.com. 3600 IN TXT "v=DMARC1; p=none; rua=mailto:dmarc@example.com"
The SPF record means: the servers in the MX records may send, and everything else is a failure. There must be exactly one SPF record per name, and the record may cause at most ten DNS lookups. The DKIM value is long and is copied exactly, without line breaks. DMARC passes when SPF or DKIM passes and is aligned with the domain in the From header; it fails only when neither gives an aligned pass. SPF is checked against the envelope sender, which a relay or a mailing tool may rewrite, so an SPF pass is not automatically aligned; DKIM signs the From domain itself. DMARC starts at p=none, which only reports; after a month of clean reports, raise it to quarantine and then to reject.
Reverse DNS
Forward DNS turns a name into an address. Reverse DNS does the opposite, and mail servers check it: a message from an IP address whose reverse name is missing, or does not match the name in the greeting, is treated as suspicious. You cannot create it yourself; the owner of the address must set it.
In the panel of the VPS provider, find the reverse DNS or PTR setting for your IPv4 address and set it to mail.example.com. The IPv6 address needs its own PTR record. Then check from outside:
dig +short -x 203.0.113.10
mail.example.com.
The name in the answer must match the name your mail server announces in its greeting, which is myhostname in Postfix. A mismatch is the single most common reason a new mail server lands in spam.
Port 25 is often closed
Many VPS providers block outgoing port 25 by default, because it is the port spam uses. Mail still arrives, since incoming port 25 is usually open, but nothing you send leaves the machine. Test it before you configure anything:
sudo apt install netcat-openbsd
nc -vz alt1.gmail-smtp-in.l.google.com 25
Connection to alt1.gmail-smtp-in.l.google.com 25 port [tcp/smtp] succeeded!
A timeout instead of a success means the provider blocks the port. Ask support to open it, which many providers do for a verified account, or send through a relay: a smarthost that accepts your mail on port 587 with authentication and delivers it for you. A relay is also the practical answer when your IP has a poor reputation and you do not want to wait weeks for it to improve.
Postfix and Dovecot
Postfix moves mail between servers; Dovecot lets a mail client read the mailbox over IMAP. The firewall from lesson 8 allows only SSH and the web ports, so open the mail ports first, or the services will run and stay unreachable:
sudo ufw allow 25/tcp
sudo ufw allow 587/tcp
sudo ufw allow 993/tcp
Then install both services, and answer Internet Site with example.com as the system mail name when the installer asks:
sudo apt install postfix dovecot-core dovecot-imapd
The main configuration holds the identity of the server, the domains it accepts mail for, and the TLS certificate. Ubuntu's main.cf already contains most of these keys, so editing the file by hand means finding each old line and removing it. postconf -e replaces the value in place, one setting per command:
sudo postconf -e 'myhostname = mail.example.com'
sudo postconf -e 'mydomain = example.com'
sudo postconf -e 'myorigin = $mydomain'
sudo postconf -e 'inet_interfaces = all'
sudo postconf -e 'inet_protocols = ipv4'
sudo postconf -e 'mydestination = $myhostname, $mydomain, localhost.$mydomain, localhost'
sudo postconf -e 'home_mailbox = Maildir/'
sudo postconf -e 'mynetworks = 127.0.0.0/8'
sudo postconf -e 'relay_domains ='
sudo postconf -e 'smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated defer_unauth_destination'
sudo postconf -e 'smtpd_tls_cert_file = /etc/letsencrypt/live/example.com/fullchain.pem'
sudo postconf -e 'smtpd_tls_key_file = /etc/letsencrypt/live/example.com/privkey.pem'
sudo postconf -e 'smtpd_tls_security_level = may'
sudo postconf -e 'smtpd_sasl_type = dovecot'
sudo postconf -e 'smtpd_sasl_path = private/auth'
sudo postconf -e 'smtpd_sasl_auth_enable = yes'
sudo postconf -e 'smtpd_tls_auth_only = yes'
sudo systemctl reload postfix
mydestination lists the domains this server
considers its own; relay_domains is empty, so it
forwards nothing for anyone else. home_mailbox
makes Postfix write each message into the Maildir folder of the user,
which is where Dovecot reads it. The relay restrictions allow mail
from the machine itself and from authenticated users, and refuse
everything else for outside destinations. That single line is what
makes an open relay impossible.
smtpd_tls_auth_only = yes is the other half of
the lock: with authentication enabled globally, it makes Postfix refuse
a password until TLS is up. Without it, port 25 would accept a password
in clear text before the encrypted session starts.
The submission port, 587, is where mail clients send. The client connects in clear text, switches to TLS with STARTTLS, and only then may authenticate. Port 465 is the other arrangement, called implicit TLS: there the TLS session starts with the connection, from the first byte.
sudo vi /etc/postfix/master.cf
submission inet n - y - - smtpd -o syslog_name=postfix/submission -o smtpd_tls_security_level=encrypt -o smtpd_sasl_auth_enable=yes -o smtpd_relay_restrictions=permit_sasl_authenticated,reject
Dovecot needs a mailbox location, TLS, and a socket that Postfix uses to check passwords. Three files hold them:
sudo vi /etc/dovecot/conf.d/10-mail.conf
mail_location = maildir:~/Maildir
sudo vi /etc/dovecot/conf.d/10-ssl.conf
ssl = required ssl_cert = </etc/letsencrypt/live/example.com/fullchain.pem ssl_key = </etc/letsencrypt/live/example.com/privkey.pem
sudo vi /etc/dovecot/conf.d/10-master.conf
service auth { unix_listener /var/spool/postfix/private/auth { mode = 0660 user = postfix group = postfix } }
The block goes inside the existing service auth { } section, where it replaces the commented-out listener of the same name that the package already put there.
Restart both and create the first mailbox as a system user, which is the simplest setup for a handful of addresses:
sudo systemctl restart postfix dovecot
sudo adduser you
The account logs in to IMAP as you, with its system password, on port 993 with TLS, and sends on port 587 with STARTTLS. A larger setup stores users in a database, which is the same idea with more moving parts.
DKIM signing
DKIM signs every outgoing message with a private key, and the public key sits in a TXT record so receivers can verify the signature. Generate a 2048-bit key for the selector mail:
sudo apt install opendkim opendkim-tools
sudo mkdir -p /etc/dkimkeys
sudo opendkim-genkey -b 2048 -v -d example.com -s mail -D /etc/dkimkeys
opendkim-genkey: generating private key opendkim-genkey: private key written to mail.private opendkim-genkey: extracting public key opendkim-genkey: DNS TXT record written to mail.txt
opendkim-genkey is silent unless -v is given, which is why the command above has it. The command runs as root, so the private key belongs to root, and the OpenDKIM service, which runs as opendkim, cannot read it and would sign nothing. Give the key to that user:
sudo chown opendkim:opendkim /etc/dkimkeys/mail.private
The file /etc/dkimkeys/mail.txt holds the DNS record; publish its content as a TXT record at mail._domainkey.example.com. The file splits the long key over several quoted strings for readability; join them into one value, without spaces or line breaks.
Then tell OpenDKIM which key signs for which domain. Ubuntu's /etc/opendkim.conf already has an active Socket line, local:/run/opendkim/opendkim.sock: replace that line with the inet socket below, do not add a second one:
sudo vi /etc/opendkim.conf
Domain example.com Selector mail KeyFile /etc/dkimkeys/mail.private Socket inet:12301@localhost
Connect Postfix to the milter, then restart both services:
sudo postconf -e 'milter_default_action = accept'
sudo postconf -e 'smtpd_milters = inet:localhost:12301'
sudo postconf -e 'non_smtpd_milters = inet:localhost:12301'
sudo systemctl restart opendkim postfix
dig +short TXT mail._domainkey.example.com
"v=DKIM1; k=rsa; p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA..."
A signature that verifies at the receiver is the goal, not a record that exists. Send a message to a mailbox at a large provider and read its headers, as the testing section shows.
TLS for mail, no open relay
The certificate for mail.example.com comes from certbot like any other. Add the name to the server block of the site, so that the HTTP check can reach it:
sudo vi /etc/nginx/sites-available/example.com
server_name example.com www.example.com mail.example.com;
sudo nginx -t
sudo systemctl reload nginx
Then issue one certificate for all three names:
sudo certbot --nginx -d example.com -d www.example.com -d mail.example.com
Postfix and Dovecot now read the files under /etc/letsencrypt/live/example.com/: certbot names the folder after the first domain in the command, and the certificate itself covers all three. The nginx plugin of certbot reloads nginx only, so Postfix and Dovecot need a deploy hook of their own; every renewal runs it:
sudo vi /etc/letsencrypt/renewal-hooks/deploy/reload-mail.sh
#!/bin/sh systemctl reload postfix dovecot
sudo chmod 755 /etc/letsencrypt/renewal-hooks/deploy/reload-mail.sh
Now prove that the server does not relay. From another machine, connect to port 25 and try to send a message to an address that is not yours. The server must refuse with a relay error:
swaks --to someone@example.net --server 203.0.113.10:25
<** 554 5.7.1 <someone@example.net>: Relay access denied
A refusal is the correct answer. If the server accepts the message, the relay restrictions are wrong, and you have an open relay: stop and fix it before the address is used to send spam. The same test against port 587 without a password must also fail, with an authentication error.
Sending from the website
A contact form should not use the mail() function of PHP through the local queue if you can avoid it: a message sent that way has no authentication and is easy to mark as spam. Send through the submission port with authentication instead. msmtp is a small SMTP client that PHP can call in place of sendmail:
sudo apt install msmtp
Install msmtp alone, not the msmtp-mta package: that one replaces Postfix as the local mail transport agent, and this server needs Postfix.
sudo mkdir -p /etc/msmtp
sudo vi /etc/msmtp/example.conf
defaults auth on tls on tls_trust_file /etc/ssl/certs/ca-certificates.crt account example host mail.example.com port 587 user you password a-long-random-password from you@example.com account default : example
sudo chown root:example /etc/msmtp/example.conf
sudo chmod 640 /etc/msmtp/example.conf
The login is you, the system user from the mailbox step, not the full address: Dovecot authenticates system users by user name. The sender address stays you@example.com.
The file belongs to root and to the group of the site, so the PHP process can read the password and other users cannot. Then point the pool of the site at it. sendmail_path is a system setting, so it goes into the pool with php_admin_value:
sudo vi /etc/php/8.3/fpm/pool.d/example.com.conf
php_admin_value[sendmail_path] = /usr/bin/msmtp -t --file=/etc/msmtp/example.conf
sudo systemctl reload php8.3-fpm
A form can now send through your own server, with TLS and a real sender address. Never put the SMTP password inside the web root; a leaked .env file is one of the threats from lesson 7.
Testing with swaks and headers
Bash and a mail client are not enough to debug mail. swaks is a single command that speaks SMTP and prints the whole conversation:
sudo apt install swaks
swaks --to you@gmail.com --from you@example.com --server mail.example.com:587 --auth LOGIN --auth-user you --auth-password "your-password" --tls
<- 250 2.0.0 Ok: queued as 4Xy2Z -> QUIT <- 221 2.0.0 Bye === Connection closed with remote host.
A queued message is only half the proof. At the receiving mailbox, open the full headers and look for the authentication results:
Authentication-Results: mx.google.com; spf=pass (google.com: domain of you@example.com designates 203.0.113.10 as permitted sender) smtp.mailfrom=you@example.com; dkim=pass header.i=@example.com header.s=mail header.b=AbCdEfGh; dmarc=pass (p=NONE sp=NONE) header.from=example.com
Three passes mean the records are right. spf=fail
means an address that is not in the record sends your mail;
dkim=fail means the selector, the key or the
signature is wrong. dmarc=fail appears only when
neither SPF nor DKIM gives an aligned pass: both failed, or both passed
for a domain other than the one in the From header, which is what a
relay that rewrites the envelope sender produces. Read the first
failure, not the last.
If the message lands in spam with three passes, the IP reputation is the
usual cause. Check the address against the public blocklists at
check.spamhaus.org
and at
mxtoolbox.com.
A listed address must be cleaned up and then requested for removal;
while it is listed, most receivers will refuse or quarantine your mail.
MTA-STS and reports
Two optional records make mail more trustworthy, and both are worth adding once the basics pass.
MTA-STS publishes a policy that says mail to your domain must use TLS. It needs a TXT record and a small HTTPS page:
_mta-sts.example.com. 3600 IN TXT "v=STSv1; id=20261106T000000"
The policy itself is served at https://mta-sts.example.com/.well-known/mta-sts.txt:
version: STSv1 mode: testing mx: mail.example.com max_age: 86400
Start with mode: testing and read the reports; switch to enforce when every sender you care about still delivers. TLS-RPT is the report channel for TLS failures:
_smtp._tls.example.com. 3600 IN TXT "v=TLSRPTv1; rua=mailto:tlsrpt@example.com"
Both records are easy to add and easy to forget. Put them in the notes with the MX and SPF values, and check them again after any change to the mail server.
Quiz
1 What does the MX record of a domain say?
Answer
b) MX names the destination mail servers, with a priority. SPF is the list of allowed senders, DKIM the signature, DMARC the policy.
2 Your SPF record is v=spf1 mx -all. What does -all mean?
Answer
b) The minus sign is a hard fail. Use ~all during a migration, when some senders are not in the list yet.
3 Why does a new mail server need a matching PTR record?
Answer
a) A missing or mismatched reverse name is a strong spam signal. The PTR is set by the address owner, which is the VPS provider, not by your DNS panel.
4 nc -vz alt1.gmail-smtp-in.l.google.com 25 times out. What does it mean?
Answer
b) A timeout on outbound 25 is the classic block. Ask the provider to open it, or send through a relay on port 587.
5 Which Postfix setting stops the server from being an open relay?
Answer
b) The relay restrictions decide who may send mail to outside domains. The test from another machine must answer Relay access denied.
6 A message arrives with spf=pass, dkim=fail and dmarc=fail. Where do you look first?
Answer
b) SPF passed, but its pass was not aligned with the From domain, so DMARC still failed: the envelope sender belongs to another domain, for example a relay or a mailing tool, while the From header says your domain. DKIM signs the From domain itself, so it is the check to fix: compare the selector in the signature with the TXT record and the key on disk.
7 Why send website mail through authenticated SMTP instead of the local mail() queue?
Answer
b) An authenticated submission on port 587 with TLS is what the receiving side expects from a real mail domain. It also keeps the SMTP password out of the web application code.
Article author: Arthur Isaev