I moved to a shiny ‘new build’ flat a year or so ago and unfortunately my old Compaq 9000 Rack Cabinet wouldn’t have fitted up the stair case :/ so I bought a nice flat pack 28u Rack Cabinet and set about virtualising all my legacy stuff.

VM Hosts

Using the 2 S411 cases I already had I installed;
2x ASUS P5B-VM
2x Intel Core 2 Quad Q6600
8x Corsair 4GB DDR2 800MHz/PC2-6400 XMS2
2x Hiper 880W 85% Efficient PSU
4x Adaptec 1430SA 4Port PCI Express SATA II RAID Card
16x Western Digital Caviar Blue WD5000AAKS 500Gb SATA II Disk Drives
2x Plexus MV 1200VA UPS

Because I wanted performance at cheaply as possible I had to use HyperV instead of ESXi because all the hardware I had chosen worked flawlessly in Windows Server 2008 but is obviously unsupported in ESXi.

I upgraded my workstation at the same time but that was basically the same other than the motherboard ( ASUS P5N-D ) and two XFX 9800GTX+ 765MHz Edition graphics cards.

I didn’t bother with any P2V stuff as the old VM’s / physicals were a mix of Windows Server 2003 and CentOS 5.2.

Networks

I’d recently moved to Be* Internet and had ordered 16 IP’s (to go with my /48 IPv6 subnet) I had to split the Network off using 2 physical firewalls (a Cisco Pix 501 and a Netscreen NS5GT). Eventually I’ll probably replace one of them with a Cisco ASA 5505 Security Pack to do the multiple subinterface VLAN stuff but at £600 still I can live without it!.

The 3Com SuperStack 3 4400 proved itself a good investment yet again allowing me to VLAN off the internet facing VM’s from my internal ones using HyperV’s VLAN tagging config.

The Cisco 2600 router coupled with a Windows 2008 VM sorted out the IPv6 Network again.

All in all it was a pretty painless process and to the point of this post, pictures;

The Old Kit

All the old kit was either FreeCycled or otherwise donated to those who would benefit from DL380′s, switches, Fibre/Ethernet converters etc etc.

The saddest thing was smashing 5Tb of 300Gb / 250Gb / 160Gb / 80Gb / 40Gb disks into powder.

Cross Forest trusts are a possibility and merging one with the other (i.e having the Hosted Exchange solution bound to the existing domain) is another but there are many issues with that (mostly political).

What I intend to do is utilise the ‘Branch office’ concept that Read Only Domain Controllers were designed for to mock up a solution for Hosting the entire AD infrastructure remotely and having R/O DC’s on the customer premises.

What now?

For no other reason than that of satisifying my curiosity I built an entire AD infrastructure hosted at the data center and then had a remote ‘office’ running for a day without a local DC and then the following day with a Read Only Domain Controller sitting there.

There’s nothing new or crazy here other than maybe the fact that most people move bits of their AD infrastructure to the DC when its bandwidth requirements overwhelm their resources. What I’m playing with is the idea of having everything remote and only putting the stuff you need (NAS etc) in the office.

The Test

In the Red Corner we have a full Active Directory and Exchange infrastructure at the DC and then the ‘offices’ were built using a few Terminal Services servers running a respective amounts of users. The idea is to monitor traffic before dropping in a RO DC and then again afterwards.

The Infrastructure

Hosted Infrastructure

The Hosted infrastructure consists of a relatively standard Exchange 2007 deployment (if you follow the guidelines) visible to the world (selected ports only) is an Edge Transport server for handling the initial mail connections and the Client Access Server. Behind those is the Mailbox and Hub Transport (in reality these were on the same box but the diagram wasn’t as symmetrical then!).


The Domain controller is a special case because whilst we have no reason for the Internet at large to talk to it we need the read only Domain Controller at the client site to be able to communicate with it so an IPSEC LAN to LAN VPN was required.

The Results

AD Traffic From the TS to the Remote DC No Local DC

AD Traffic to the Remote DC with Local RODC

OWA Traffic During the Tests

Scripted behavior – so it was the same(ish) on both days

Conclusion

Well it did exactly what I expected it to do so nothing ground breaking there. It was interesting to see the spike just after I logged all the fake users off the Terminal Servers.

R/O DC’s were used because in an ideal world customers shouldn’t have write access to an AD infrastructure that a SysAdmin has an SLA to honor!

With the release of the HyperV Linux Componants I’m going to start moving some workloads away from my old ESX 2.5 servers and see if HyperV Synthetic devices work as well as ESX devices do.

Not that I knew it at the time but Microsoft have also updated their Support policies to include Exchange Server SP1 when virtualized.

The other good news is that there are reports that we’ll have System Center Virtual Machine Manager 2008 (final release) available within a couple of weeks!

Since this is about HyperV I found a nice PowerShell script for use with HyperV to create a new VM, my mention of PowerShell will become more apparant in the next post.

Once we’ve crossed a particular hurdle then we need to go about disseminating this new found information to the other SysAdmins.

So far the best way I’ve found of doing this is to take a VM image, break it in the same way that we encountered and then make it available for deployment to the VM infrastructure (minus of course the malicious payload if it was part of a botnet). These VM’s are then made available to the teams to deploy at their will or used as part of the weekly inter-team training sessions. The VM’s are ‘tagged’ and a description of the issue with the fixes found to date are attached.

This allows people to either:

a) Simply follow what was done so they know how to go about it themselves.

b) Find new ways of fixing the issue

c) Use it as an example of a particular technologies issues and the diagnostic methods used to ascertain what is wrong and fix it in classroom style training.

There is no additional hardware cost and with the use of snapshots people can experiment with various methodologies other than the ones that we initially used. If a quicker, cleaner or better way of fixing the issue is discovered this can be put on the wiki and tagged to the VM image. This practice really starts to pay off once you’ve got the ball rolling and people are experimenting, contributing or using it as teaching material and all without any further downtime or additional hardware cost.

At the recent IIS7 for Managed Hosting seminar in London we saw Microsoft make heavy use of ‘model’ VM Machines in conjunction with projected slides and Step-by-Step walkthroughs to show us the features and improvements of IIS7 (I still prefer apache if anyone is wondering) which just goes to prove that this is an effective and cost-effective method of hands on training.

This might seem like common sense to some people and may already be in place for some companies, but if its not then why not try it out and let me know how it works out for you?

As always I use VMWare technology.

The solution consisted of two machines each with Dual Core 2.13Ghz Intel’s, 2Gb of RAM & SATA disks behind a hardware load balancer. Unfortunately the combination of having to use swap due to the volume of incoming mail coupled with BackScatter attacks [the BackScatter attacks would of course require the original mail and the NDA to be written to disk for later delivery] was killing the disks and the sheer volume of mail was overwhelming the I/O available to just two servers (in terms of Hardware and Postfix postdrop etc).

Whilst scaling out was considered, space and power consumption would have been an issue. I made the decision to go with two machines but this time one would be a virtualisation power-house and virtualise the anti-spam ‘appliances’.

The Machines:
Machine 1:
2x 2.4Ghz Quad Core XEON’s
8Gb ECC RAM
80Gb RAID5 SATA (Host OS)
300Gb RAID1 15k SCSI (Guest Stores)
Gigabit uplink to Load Balancer

Machine 2:
2.13Ghz Core2 Duo
2Gb RAM
80Gb RAID1 SATA
Gigabit uplink to Load Balancer

Machine 1 hosts 6 virtual servers which run the usual combination of Postfix, SpamAssassin & Amavis. These are Load Balanced via a hardware load balancer. Machine 2 runs the same software but with an extremely strict set of rules, the idea is that if all 6 of the Virtual Machines fail the Load Balancers fail over to Machine 2 but this will only talk to the most well behaved clients and only let the cleanest of mail into the mail queue to be processed by the Anti-Spam processes.

To ensure that these servers aren’t the source of BackScatter attacks the servers only accept truly legitimate mail (i.e it doesn’t accept mail for any destination, attempt to deliver it to the backend server and then bounce an undeliverable to the [usually faked] from address.)

Postfix Config:

#-------------------------------------------------------------
# Additional non-standard bits
#-------------------------------------------------------------
#Max message of 15mb
message_size_limit = 15728640
#
#Transport maps - i.e where are we sending this stuff
transport_maps = hash:/etc/postfix/transport
#
#Stop people trying to farm addresses
disable_vrfy_command = yes
#
#Stop people hammering the hell out of us
smtpd_data_restrictions = reject_unauth_pipelining
#
#Pass off content checking to amavis
content_filter = smtp-amavis:[127.0.0.1]:10024
#
#Fed up of mailer-daemon fucking up my queue
#fallback_relay = xx.xx.xx.xx
#
#If the above doesn't work we can always make sure the
#mails don't live for more than one bounce
bounce_queue_lifetime = 0
#
#Spam still went crazy - lets speed up delivery at the cost of resources
maximal_backoff_time = 120s
minimal_backoff_time = 10s
#
#This might adversely affect legitimate mail
maximal_queue_lifetime = 900s
#
#-------------------------------------------------------------
# These things are the bits that can really cause us some issues
# with false positives etc etc
#
# Consult Pages 139 - 142 of postfix book for more info
#-------------------------------------------------------------
#
#Start off with some blacklisting
# Check our black / white list | RealtimeBlacklist | Realtime blacklist
smtpd_client_restrictions = permit_sasl_authenticated, check_client_access hash:/etc/postfix/client_access, reject_rbl_client sbl.spamhaus.org, reject_rbl_client dul.dnsbl.sorbs.net
#
#Make clients use EHLO / HELO verbs
smtpd_helo_required = yes
#
#If we are being REALLY strict make them conform to the RFC
#strict_rfc821_envelopes = yes
#
#Now we make them say hello politely
# Crap remote hostname remote host not FQDN
smtpd_helo_restrictions = reject_invalid_hostname, reject_non_fqdn_hostname
#
#When a user says who they are we had better check what they are telling us
# Sender address not FQ | Might want to remove this
smtpd_sender_restrictions = reject_non_fqdn_sender, reject_unknown_sender_domain
#
#Rejects crap / faked / bad recipiants - This might be overkill
# Non FQDN target email Allow inside Only relay our domains
smtpd_recipient_restrictions = permit_sasl_authenticated, reject_non_fqdn_recipient, permit_mynetworks, reject_unauth_destination


The VM’s
With 8Gb of RAM and only 6 machines we can ‘scale out’ if more machines are needed or leverage that ‘spare’ RAM to improve I/O.

Anything that needs to be written to disk is written to a RAM disk which will prevent the disk wear and will improve I/O. The VM’s are restricted to the Physical RAM of the Host machine so there will be no swapping to the Host disk which once again improves I/O and reduces mechanical wear and tear.

The Results
As mentioned at the beginning the servers were seeing a massive spike in mail, mails were taking a week to arrive and many of them were spam. As soon as the Virtualised Solutuion went live the emails hitting the back-end servers dropped from almost 14,000 (per 5 minute MRTG interval) to around 800 and with zero false positives! Emails were making it through in less than 2 seconds despite the spam attacks becoming increasingly aggressive!

Links:
www.DNSStuff.com
http://spamlinks.net/prevent-secure-backscatter.htm
http://www.postfix.org/docs.html
http://www.vmware.com/

Windows didn’t like that and hung, VMware Virtual Center disliked it even more. Once I recovered the machine (ohhh AD was in a mess!) VMVC refused to launch and spewed some messages to the console. The sole purpose for this entry is to list the event log errors and list the solution so hopefully some one else out there won’t be in the dark.

The errors always happen in batches as follows:

[2376] [VpxdMutex] Locking InvtLock (5) conflicts with InvtHostBarrier (0)

[3372] (HY000) – [Microsoft][ODBC Microsoft Access Driver] Not a valid bookmark.

[3372] [ARCHIVER] Q: SELECT COUNT(*) FROM STATS_HOST_DATA WHERE HOST_ID=? AND HIST_ID=? AND COL_ID=? AND ROW_ID=?

[3372] [ARCHIVER] Failed to count stats. ODBC error=-1

[3372] [Vpxd] Initiating shutting down

[3352] [Vpxd] Shutting down…

[3352] [Vpxd] Shutting down now

The description for Event ID ( 1 ) in Source ( VMware Virtual Mount Service Extended ) cannot be found. The local computer may not have the necessary registry information or message DLL files to display messages from a remote computer. You may be able to use the /AUXSOURCE= flag to retrieve this description; see Help and Support for details. The following information is part of the event: vmount2 service stopped 1.5.2 build-32167.
Then the VMVC process hangs, it shows as ‘Started’ in the Services MMC but any connection attempts are met with an active refusal.

Taking the hint that a database was corrupted I made a copy of the Template / Uploads directory and bit the bullet; initiated the dreaded MSI Repair mode.

Answering ‘Yes’ to keeping the existing database followed by a swift reboot and all was working again.

One of the VMWare servers had hung (it was deploying from a template when this happened!) and had to be kill -9′d but after that all came up fine again.

Fun eh?