E-Mail Done My Way, Part 4 - The final stuff

After we learned about my general approach and setup in part 0, the details of my postfix configuration in part 1, how I use dovecot in part 2, all about DKIM/DMARC/SPF in part 4, we will close this little series with the mundane little things we need to take care of on a daily basis to make sure we and our mail server are happy.

Fail2ban

We have already done a lot to make sure our mailserver refuses connections from spammers, but we also have “hackers” out there trying to bombard every mail server with login and relay attempts. Fail2ban is a proven way to deal with those folks.

The TL;DR: fail2ban checks log files for specific entries and if it finds enough of them related to one IP address, it adds a block for that IP address to the firewall. After a defined time the block is removed again. This can go on forever, but it all happens fully automated and helps a lot to keep your server open for real connections.

So. Again. Let’s go through the config file /etc/fail2ban/jail.local to explain:

[DEFAULT]

# Ban IP/hosts for 4 days ( 1 day = 24h*3600s = 86400s):
bantime = 345600

# An ip address/host is banned if it has generated "maxretry" during the last "findtime" seconds.
findtime = 43200
maxretry = 2

#  sshd protection
[sshd]
enabled = true

# email protection
[postfix]
enabled = true
port = smtp,465,submission

[dovecot]
enabled = true
port = pop3,pop3s,imap,imaps,submission,465,sieve

First we define that offending IP addresses get banned for 4 full days. In my experience, they will time their attacks according to the default settings of fail2ban, so let’s tweak those ;).

Next we tell fail2ban to watch the log entries of the past 12 hours with findtime = 43200 and that the block kicks in when they try the same thing for 2 times in that period - maxretry = 2.

Fail2ban works with the concept of “jails”, which are technically a collection of regular expressions, grouped by service. So we tell it which services we want to be protected.

The first entry, [sshd] is simple. We enable it and done. Now if someone tries to log in using ssh unsuccessfully more than twice in 12 hours, they are blocked for 4 days.

So be careful - you could lock yourself out too this way - it just needs 2 times a typo in the username or password ;) You HAVE switched to using ssh keys for login a long time ago, HAVE YOU? ;)

Next we enable the [postfix] jail. Postfix does the smtp stuff, so we make sure fail2ban watches the respective ports: smtp(25), port 465 and submission(587). This activates a bunch of regexes that catch those bots/scripts that try to find out if we are running an open relay.

And now [dovecot]. As dovecot does the authentication and imap stuff, we activate the jail for pop3(110), pop3s(995), imap(143), imaps(993), sieve(4190) and again 465 and submission(587).

Again, make sure fail2ban is started as a service automatically and done.

Our little mailserver is now reasonably well protected against a lot of the usual attacks. So well protected, actually, that I don’t feel the need to install spam checkers like SpamAssassin or spamd. With this setup of postfix, dovecot, fail2ban, DKIM, DMARC, SPF my server has proven to be quite spam-resistent. A lean and mean setup - I like!

The daily stuff

With all that in place, the mail server is humming along nicely, with a typical RAM usage of below 700 MB and negligible CPU load. NICE! But still we need to look after it a bit every now and then (I do this mostly during lunch break or in the evening).

So I login with ssh to my mailserver and first check what IPs fail2ban has caught:

fail2ban-client status | sed -n 's/,//g;s/.*Jail list://p' | xargs -n1 fail2ban-client status

This command iterates over all activated jails and shows me the status:

Status for the jail: dovecot
|- Filter
|  |- Currently failed: 1
|  |- Total failed:     5
|  `- Journal matches:  _SYSTEMD_UNIT=dovecot.service
`- Actions
   |- Currently banned: 2
   |- Total banned:     5
   `- Banned IP list:   91.103.252.248 91.103.252.243
Status for the jail: postfix
|- Filter
|  |- Currently failed: 7
|  |- Total failed:     69
|  `- Journal matches:  _SYSTEMD_UNIT=postfix.service
`- Actions
   |- Currently banned: 26
   |- Total banned:     37
   `- Banned IP list:   182.101.41.184 182.103.27.185 59.63.151.180
      223.241.55.91 220.177.147.38 157.52.232.70 157.52.232.71
      157.52.232.72 157.52.232.73 157.52.232.74 157.52.232.75
      157.52.232.76 182.103.27.155 157.52.232.77 134.73.151.227
      134.73.151.228 157.52.232.78 134.73.151.229 157.52.232.79
      157.52.232.80 157.52.232.81 157.52.232.82 157.52.232.83
      220.177.146.252 111.76.18.143 182.103.26.59
Status for the jail: sshd
|- Filter
|  |- Currently failed: 0
|  |- Total failed:     0
|  `- Journal matches:  _SYSTEMD_UNIT=sshd.service + _COMM=sshd
`- Actions
   |- Currently banned: 0
   |- Total banned:     0
   `- Banned IP list:

Ah yes. A bunch of IP addresses trying to get into my postfix mostly, some trying dovecot and no one bothering with ssh (it runs on a non-standard port ;).

Let’s pick one of those IP addresses (182.103.26.59) to see what they are trying:

# cat /var/log/maillog | grep 182.103.26.59
Sep  4 13:39:18 mailhub postfix/smtpd[84235]: connect from unknown[182.103.26.59]
Sep  4 13:39:19 mailhub postfix/smtpd[84235]: NOQUEUE: reject: RCPT from unknown[182.103.26.59]: 450 4.7.25 Client host rejected: cannot find your hostname, [182.103.26.59]; from=<GA40n@8SPQ.com> to=<asdwjaswdojiwadwfwac11290192@tcpid.net> proto=ESMTP helo=<tcpid.net>
Sep  4 13:39:20 mailhub postfix/smtpd[84235]: NOQUEUE: reject: RCPT from unknown[182.103.26.59]: 450 4.7.25 Client host rejected: cannot find your hostname, [182.103.26.59]; from=<GA40n@8SPQ.com> to=<gaofen@tcpid.net> proto=ESMTP helo=<tcpid.net>
Sep  4 13:39:20 mailhub postfix/smtpd[84235]: lost connection after RCPT from unknown[182.103.26.59]

Ah yes, one of those ;) Trying to dump spam mail via SMTP but their IP address doesn’t resolve to a hostname, so postfix has decided nope. This IP address tried a second time to do the same thing and fail2ban kicked in and blocked the IP address on the firewall level for 4 days.

So, postfix is doing the right thing and fail2ban too. Nice! Lets’ do another check. This one is slightly more complicated but very helpful:

tail -n 3000 /var/log/maillog | grep -E "reject|fail|UGFzc3dvcmQ6" | grep -oE "(([0-9]|[0-9]{2}|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[0-9]{2}|1[0-9]{2}|2[0-4][0-9]|25[0-5])" | sort | uniq -c -d

Oh wow! That’s quite a mouthful :) Let me explain. We take the last 3000 lines of /var/log/maillog and look for lines with either reject, fail or UGFzc3dvcmQ6 in them. Then we boil that down to the IPv4 addresses in those lines (99.9% of all attacks are IPv4). After that we sort them by IP and reduce them to the unique ones with a count that is at least 2 using uniq -c -d

We get:

      4 111.76.18.143
      2 115.44.99.198
      2 116.23.16.233
      4 157.52.232.82
      4 157.52.232.83
      4 182.103.26.59
      2 185.220.101.184
      3 185.220.101.60
      2 186.236.230.168
      2 2.198.133.160
      2 212.193.29.42
      4 220.177.146.252
      2 31.210.22.190
      2 37.139.129.243
      5 51.38.32.100
      3 91.103.252.243

Nothing exceptional. No counts above 10, so we don’t need to take further action.

(You might wonder what that weird UGFzc3dvcmQ6 is - quite simple, actually. My mail server prefers TLS connections, so the hacker/script will do that to try to login. It typically will use a random username and password. UGFzc3dvcmQ6 is base64 encodced and means password ;)

That’s mostly it. If I do find a spam mail in my InBox or Junk folder, I will check the headers, grep the IP address from the mail server that dropped it and maybe block it with a swift:

firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='134.73.151.244' reject"

forever. My server, my rules :)

And that finally concludes my 2022 E-Mail server setup series. I hope you enjoyed it and maybe even learned enough to set up your own and free yourself from the gmail/microsoft stranglehold on e-mail. E-Mail is such a great thing, it should belong to all of us!

As always, feel free to comment using mastodon, contact me directly and maybe even share this series with your friends. It was a pleasure to finally get all of this written down so I can point to it in the future :)