Dealing with Massive Spam Levels In Postfix
Recently I encountered a situation where a vanilla install of Postfix had been targeted by what appeared to be a Spam Botnet. Bizarrely this server was what you could call an ‘Edge Server’ but it didn’t have any AntiVirus or AntiSpam to be honest it was pretty useless.
There were three issues to clear up. First there were 2,435,239 emails from the Botnet destined for a domain that was no longer hosted on the ‘backend’ server. Secondly none of this email was being Spam checked and finally the massive queues meant that the 568,372 legitimate emails stuck in the queue were being delayed by upto 8 hours.
Preventing Spammers getting to the queues
As it was evident that a Botnet was sending the spam the first thing to do was to enforce some of the settings that Postfix is capable of doing ‘out of the box’ to try and stop these hosts bombarding the server. [More information about these settings here: Leveraging Virtualisation to Fight Spam]
#Stop people hammering the hell out of us
smtpd_data_restrictions = reject_unauth_pipelining
#Start off with some blacklisting
# Check our black / white list | RealtimeBlacklist | Realtime blacklist
smtpd_client_restrictions = permit_sasl_authenticated, check_client_access hash:/etc/postfix/client_access, reject_rbl_client sbl.spamhaus.org, reject_rbl_client dul.dnsbl.sorbs.net
#Make clients use EHLO / HELO verbs
smtpd_helo_required = yes
#If we are being REALLY strict make them conform to the RFC
#strict_rfc821_envelopes = yes
#Now we make them say hello politely
# Crap remote hostname remote host not FQDN
smtpd_helo_restrictions = reject_invalid_hostname, reject_non_fqdn_hostname
# Non FQDN target email Allow inside Only relay our domains
smtpd_recipient_restrictions = permit_sasl_authenticated, reject_non_fqdn_recipient, permit_mynetworks, reject_unauth_destination
Spam Checking
An article on FreeSpamFilter.org shows you how to add Amavisd-new, SpamAssassin, Pyzor, Razor, DCC, and ClamAV to a Postfix setup. The article is geared towards Fedora 4 but works equally well on CentOS (the server in question) and RHEL etc. The article is very indepth and doesn’t require any further explaination.
Clearing Out the Queues
The final issue (and the bit of this Blog that is different to the Virtualisation one linked to earlier) is that I had to clear out these queues no matter what!
Postfix have some detailed explainations of some of the troubleshooting tools available and advice on what to do in this situation.
I however decided to write some ‘Sledgehammer to open a nut’ type scripts to fix the issue.
First of all with the number of emails in the queue qshape was unusable and the majority of the mails were deferred. So this script looks in the deferred queue directory, matches the domain name in question and then calls postsuper to delete the mail from the queue:
#!/bin/bash
lastid=0
cd /var/spool/postfix/defer
grep -R “DOMAINNAME” * | awk ‘{print $1}’ | while read line; do
x=${line%:*}
mailid=${x#*/}
if [ $lastid = $mailid ]; then
echo Deleting MailID $mailid
postsuper -d ${mailid%:*}
lastid=${mailid%:*}
else
lastid=${mailid%:*}
fi
done
This second script is for handling the massive amount of bouncebacks that had been generated and were sitting there trying to be delivered to either some poor persons back scatter targetted address or non existant domain. Something to note is that this would obviously have deleted all legitimate bouncebacks etc.
#!/bin/bash
# Delete all bounce backs from MAILER-DAEMON
# Blah blah rfc compliance blah blah
postqueue -p | grep MAILER-DAEMON | awk '{print $1}' | while read line; do
echo Deleting MailID $line
postsuper -d ${line%\**}
done;
I’m pretty sure that some people will object to these scripts but they did the job and the server is now swatting these botnet spam surges with ease.