NetworksAreMadeOfString

A few weeks ago I ‘inherited’ a frontend anti-spam solution that was getting crushed under the weight of the Christmas run-up. Emails were taking up to a week to arrive at the back-end system and a lot of spam was getting through too.

The solution consisted of two machines each with Dual Core 2.13Ghz Intel’s, 2Gb of RAM & SATA disks behind a hardware load balancer. Unfortunately the combination of having to use swap due to the volume of incoming mail coupled with BackScatter attacks [the BackScatter attacks would of course require the original mail and the NDA to be written to disk for later delivery] was killing the disks and the sheer volume of mail was overwhelming the I/O available to just two servers (in terms of Hardware and Postfix postdrop etc).

Whilst scaling out was considered, space and power consumption would have been an issue. I made the decision to go with two machines but this time one would be a virtualisation power-house and virtualise the anti-spam ‘appliances’.

The Machines:
Machine 1:
2x 2.4Ghz Quad Core XEON’s
8Gb ECC RAM
80Gb RAID5 SATA (Host OS)
300Gb RAID1 15k SCSI (Guest Stores)
Gigabit uplink to Load Balancer

Machine 2:
2.13Ghz Core2 Duo
2Gb RAM
80Gb RAID1 SATA
Gigabit uplink to Load Balancer

Machine 1 hosts 6 virtual servers which run the usual combination of Postfix, SpamAssassin & Amavis. These are Load Balanced via a hardware load balancer. Machine 2 runs the same software but with an extremely strict set of rules, the idea is that if all 6 of the Virtual Machines fail the Load Balancers fail over to Machine 2 but this will only talk to the most well behaved clients and only let the cleanest of mail into the mail queue to be processed by the Anti-Spam processes.

To ensure that these servers aren’t the source of BackScatter attacks the servers only accept truly legitimate mail (i.e it doesn’t accept mail for any destination, attempt to deliver it to the backend server and then bounce an undeliverable to the [usually faked] from address.)

Postfix Config:

#-------------------------------------------------------------
# Additional non-standard bits
#-------------------------------------------------------------
#Max message of 15mb
message_size_limit = 15728640
#
#Transport maps - i.e where are we sending this stuff
transport_maps = hash:/etc/postfix/transport
#
#Stop people trying to farm addresses
disable_vrfy_command = yes
#
#Stop people hammering the hell out of us
smtpd_data_restrictions = reject_unauth_pipelining
#
#Pass off content checking to amavis
content_filter = smtp-amavis:[127.0.0.1]:10024
#
#Fed up of mailer-daemon fucking up my queue
#fallback_relay = xx.xx.xx.xx
#
#If the above doesn’t work we can always make sure the
#mails don’t live for more than one bounce
bounce_queue_lifetime = 0
#
#Spam still went crazy - lets speed up delivery at the cost of resources
maximal_backoff_time = 120s
minimal_backoff_time = 10s
#
#This might adversely affect legitimate mail
maximal_queue_lifetime = 900s
#
#————————————————————-
# These things are the bits that can really cause us some issues
# with false positives etc etc
#
# Consult Pages 139 - 142 of postfix book for more info
#————————————————————-
#
#Start off with some blacklisting
# Check our black / white list | RealtimeBlacklist | Realtime blacklist
smtpd_client_restrictions = permit_sasl_authenticated, check_client_access hash:/etc/postfix/client_access, reject_rbl_client sbl.spamhaus.org, reject_rbl_client dul.dnsbl.sorbs.net
#
#Make clients use EHLO / HELO verbs
smtpd_helo_required = yes
#
#If we are being REALLY strict make them conform to the RFC
#strict_rfc821_envelopes = yes
#
#Now we make them say hello politely
# Crap remote hostname remote host not FQDN
smtpd_helo_restrictions = reject_invalid_hostname, reject_non_fqdn_hostname
#
#When a user says who they are we had better check what they are telling us
# Sender address not FQ | Might want to remove this
smtpd_sender_restrictions = reject_non_fqdn_sender, reject_unknown_sender_domain
#
#Rejects crap / faked / bad recipiants - This might be overkill
# Non FQDN target email Allow inside Only relay our domains
smtpd_recipient_restrictions = permit_sasl_authenticated, reject_non_fqdn_recipient, permit_mynetworks, reject_unauth_destination


The VM’s
With 8Gb of RAM and only 6 machines we can ’scale out’ if more machines are needed or leverage that ’spare’ RAM to improve I/O.

Anything that needs to be written to disk is written to a RAM disk which will prevent the disk wear and will improve I/O. The VM’s are restricted to the Physical RAM of the Host machine so there will be no swapping to the Host disk which once again improves I/O and reduces mechanical wear and tear.

The Results
As mentioned at the beginning the servers were seeing a massive spike in mail, mails were taking a week to arrive and many of them were spam. As soon as the Virtualised Solutuion went live the emails hitting the back-end servers dropped from almost 14,000 (per 5 minute MRTG interval) to around 800 and with zero false positives! Emails were making it through in less than 2 seconds despite the spam attacks becoming increasingly aggressive!

Links:
www.DNSStuff.com
http://spamlinks.net/prevent-secure-backscatter.htm
http://www.postfix.org/docs.html
http://www.vmware.com/

This entry was posted on Sunday, December 23rd, 2007 at 12:00 am and is filed under Datacenter, Email, Systems, Virtualization. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.