The first thing we need to do is install the pre-requisites via a privileged PowerShell;

Import-Module ServerManager
Add-WindowsFeature NET-Framework,RSAT-ADDS,ADLDS -Restart
















Once the machine has restarted you will need to ensure you set the machine name and a DNS suffix.

Assuming the checks all work you can click install and off you go;















Once all is installed the next step is to prepare a config bundle / auth package (I can’t quite remember what Microsoft call it) by issuing the following command in the EMS on the Edge server;
New-EdgeSubscription -FileName "C:\EdgeSubscriptionInfo.xml"

With that done you can either attempt to import the file via the EMS on a Hub Transport server or utilise the EMC. I chose the EMC as I kept on running into syntax errors and I was being impatient;















Unfortunately I hit an issue which was probably due to a very poor choice of mine in transfering the file. A quick rethink of moving the file and the Edge Synchronisation was complete;















The dialog shows a warning claiming the port 50636 needs to be available and the host be contactable, I checked with telnet and all was OK. So maybe the warning icon was just to draw my attention to it.

Once that is done simply issue the command;

Start-EdgeSynchronization

Unfortunately within minutes of me adding this server as a secondary MX I received a spam message which wasn’t very fun;

Return-Path: XXXXXXXXXX@yahoo.com
X-MS-Exchange-Organization-PRD: yahoo.com
Received-SPF: None (XXX-XXXX-XX.networksaremadeofstring.co.uk: XXXXXXXXXX@yahoo.com does not designate permitted sender hosts)
X-MS-Exchange-Organization-PCL: 2
X-MS-Exchange-Organization-Antispam-Report: DV:3.3.5705.600;SID:SenderIDStatus None;OrigIP:195.XXX.XXX.122
X-MS-Exchange-Organization-SCL: 5
X-MS-Exchange-Organization-SenderIdResult: NONE
X-MS-Exchange-Organization-AuthSource: XXX-XXXX-XX.networksaremadeofstring.co.uk
X-MS-Exchange-Organization-AuthAs: Anonymous

The core of this setup was a series of 3G Modems linked up with 3G to ethernet devices such as the Solwise NET-3G-3GWIFIMRW.

These were backed off onto a server running Squid and BIND with DHCP containing all the relevant proxy auto config data (plus some IPTables magic for those that didn’t play nice).

Couple this with three Access points with 8db omni’s spread about to acheive maximum coverage I was pleased to see that at one point the network was sustaining over 8mbit/s of throughput!

There was a lot of web browsing, I was streaming spotify, people were blogging and tweeting and despite the heat it all stayed alive.

Next time I’ll be doing it without mains power either so mount up the UPS’s and gas the generators!

This is extremely easy to do but I haven’t posted anything in a while so I thought I’d document the process.

To start just load up the Programs and Features section of Control Panel, choose Exchange and select Uninstall.















The first screen is utterly pointless

At the next screen you need to deselect the options you no longer require. Ideally as I’m Uninstalling and not Modifying the installation I’d have thought you’d select that which you want to remove.






























Clicking next will start the uninstall process

Unfortunately even though this machine wasn’t actually doing anything there was still a connector that used this server as a source transport server so the uninstall of the Hub Transport Role failed.















To resolve this all you need to do is load up the Exchange 2010 Management console, navigate to the Hub Transport role within the Organisation Configuration and in the Send Connectors tab remove the Connector that uses the server as a source.















Running the uninstaller again allows all the Readiness Checks to pass















Nearly 2 hours later (This machine had virtually no RAM as I’d allocated it elsewhere) the uninstall completed















Easy as anything.

A base copy of Windows Server 2008 (non R2) will have the following pre-requisites;

.Net 3.5 (SP1)
PowerShell 2.0
Office 2007 System Converter Filter Pack

Once these are installed you will need to wield ServerManagerCmd and get some other things installed;

ServerManagerCmd -i RSAT-ADDS
ServerManagerCmd -i Web-Server
ServerManagerCmd -i Web-ISAPI-Ext
ServerManagerCmd -i Web-Metabase
ServerManagerCmd -i Web-Lgcy-Mgmt-Console
ServerManagerCmd -i Web-Basic-Auth
ServerManagerCmd -i Web-Digest-Auth
ServerManagerCmd -i Web-Windows-Auth
ServerManagerCmd -i Web-Dyn-Compression
ServerManagerCmd -i Web-Net-Ext
ServerManagerCmd -i NET-WIN-CFAC

This went exceptionally smoothly and then it was time to run the Exchange 2010 setup.

Unfortunately it threw an error regarding the Net.Tcp Port Sharing service needing to be set to automatic;



















To fix this just set the service to auto start;
sc config NetTcpPortSharing start= auto

A quick retry of the Readiness checks came back with a clean bill of health and we are good to go!

Just over half an hour later Exchange 2010 is installed and ready to go!



















Since my last post regarding Exchange I’ve been using the Active Directory Certification Services to handle all my SSL stuff which worked lovely with IIS7 and Exchange 2010 so there was no need to do any of the crazy private key recovery stuff as we did with Exchange 2007.

A full install and even a test mailbox migration in just over an hour, not bad.

I moved to a shiny ‘new build’ flat a year or so ago and unfortunately my old Compaq 9000 Rack Cabinet wouldn’t have fitted up the stair case :/ so I bought a nice flat pack 28u Rack Cabinet and set about virtualising all my legacy stuff.

VM Hosts

Using the 2 S411 cases I already had I installed;
2x ASUS P5B-VM
2x Intel Core 2 Quad Q6600
8x Corsair 4GB DDR2 800MHz/PC2-6400 XMS2
2x Hiper 880W 85% Efficient PSU
4x Adaptec 1430SA 4Port PCI Express SATA II RAID Card
16x Western Digital Caviar Blue WD5000AAKS 500Gb SATA II Disk Drives
2x Plexus MV 1200VA UPS

Because I wanted performance at cheaply as possible I had to use HyperV instead of ESXi because all the hardware I had chosen worked flawlessly in Windows Server 2008 but is obviously unsupported in ESXi.

I upgraded my workstation at the same time but that was basically the same other than the motherboard ( ASUS P5N-D ) and two XFX 9800GTX+ 765MHz Edition graphics cards.

I didn’t bother with any P2V stuff as the old VM’s / physicals were a mix of Windows Server 2003 and CentOS 5.2.

Networks

I’d recently moved to Be* Internet and had ordered 16 IP’s (to go with my /48 IPv6 subnet) I had to split the Network off using 2 physical firewalls (a Cisco Pix 501 and a Netscreen NS5GT). Eventually I’ll probably replace one of them with a Cisco ASA 5505 Security Pack to do the multiple subinterface VLAN stuff but at £600 still I can live without it!.

The 3Com SuperStack 3 4400 proved itself a good investment yet again allowing me to VLAN off the internet facing VM’s from my internal ones using HyperV’s VLAN tagging config.

The Cisco 2600 router coupled with a Windows 2008 VM sorted out the IPv6 Network again.

All in all it was a pretty painless process and to the point of this post, pictures;

The Old Kit

All the old kit was either FreeCycled or otherwise donated to those who would benefit from DL380′s, switches, Fibre/Ethernet converters etc etc.

The saddest thing was smashing 5Tb of 300Gb / 250Gb / 160Gb / 80Gb / 40Gb disks into powder.

Cross Forest trusts are a possibility and merging one with the other (i.e having the Hosted Exchange solution bound to the existing domain) is another but there are many issues with that (mostly political).

What I intend to do is utilise the ‘Branch office’ concept that Read Only Domain Controllers were designed for to mock up a solution for Hosting the entire AD infrastructure remotely and having R/O DC’s on the customer premises.

What now?

For no other reason than that of satisifying my curiosity I built an entire AD infrastructure hosted at the data center and then had a remote ‘office’ running for a day without a local DC and then the following day with a Read Only Domain Controller sitting there.

There’s nothing new or crazy here other than maybe the fact that most people move bits of their AD infrastructure to the DC when its bandwidth requirements overwhelm their resources. What I’m playing with is the idea of having everything remote and only putting the stuff you need (NAS etc) in the office.

The Test

In the Red Corner we have a full Active Directory and Exchange infrastructure at the DC and then the ‘offices’ were built using a few Terminal Services servers running a respective amounts of users. The idea is to monitor traffic before dropping in a RO DC and then again afterwards.

The Infrastructure

Hosted Infrastructure

The Hosted infrastructure consists of a relatively standard Exchange 2007 deployment (if you follow the guidelines) visible to the world (selected ports only) is an Edge Transport server for handling the initial mail connections and the Client Access Server. Behind those is the Mailbox and Hub Transport (in reality these were on the same box but the diagram wasn’t as symmetrical then!).


The Domain controller is a special case because whilst we have no reason for the Internet at large to talk to it we need the read only Domain Controller at the client site to be able to communicate with it so an IPSEC LAN to LAN VPN was required.

The Results

AD Traffic From the TS to the Remote DC No Local DC

AD Traffic to the Remote DC with Local RODC

OWA Traffic During the Tests

Scripted behavior – so it was the same(ish) on both days

Conclusion

Well it did exactly what I expected it to do so nothing ground breaking there. It was interesting to see the spike just after I logged all the fake users off the Terminal Servers.

R/O DC’s were used because in an ideal world customers shouldn’t have write access to an AD infrastructure that a SysAdmin has an SLA to honor!

After some initial experimentation with their frontend app and the API I had moved all the content for one of my project sites to the ‘Cloud’ and (I hope) through to the CDN.

The Setup:

The configuration for the experiment is as follows:
2x LAMP Server [ Core2 1.86GHz / 4Gb RAM / RAID 1 SATA]
1x Static Content Server [Quad / 4Gb RAM / RAID 1 SATA / Server 2008]
1x Mosso CloudFiles account

The first LAMP server is configured with mod_deflate & mod_expires and does the PHP processing / general page stuff, the second is configured the same and used solely to pull email addresses from the database and displays them as a JPEG via PHPGD.

The static content server serves all the PNG’s, Javascript and CSS and is configured with ETAGs and Expires settings.

The Mosso account is set to whatever defaults are configured.

The Test:

Utilising a machine in a remote Datacenter I used ApacheBench with the following settings to hammer the server:
ab -n 10000 -c 100 http://the.testurl.com

The website (including all the dynamically generated images) weighs in at 830Kb of which 332Kb is static content and another 336Kb is text generated by the LAMP server.

The Results:

Unfortunately it seems that the Mosso CDN not only increased the total document size (granted only by 11Kb) but it also suffered from 836Kb/s less throughput.

Round 2

Just to make sure it wasn’t the DB stuff causing an issue (the MySQL process didn’t appear to be a bottleneck but it can’t hurt to check) I wrote a quick page that dumped some of the static images out to a page with a bit of formating. The new page weighs in at 160kb, 4Kb of PHP outputted text and 156Kb of static images.

This time around the CDN beat the throughput of the other tests by 26Kb/s which isn’t exactly ground breaking, the timings are pretty much the same and once again the document size was slightly bigger when using the CDN.

Conclusions?

Mosso has the advantage of being a ‘cloud’ and should therefore never break whereas having a (single?) seperate server for handling static content could result in a very ugly looking website if it failed.

Cost wise Mosso would win as Dedicated servers (such as the Static Content Server) are quite expensive compared to the cost of Mosso at 22¢ a Gb for transfer and 15¢ a Gb of storage on a Pay as you Use basis. Don’t forget that even with Dedicated servers if you go over your transfer allowance then you’ll be paying per Gb too and it probably won’t be 22¢ a throw!

Would I utilise cloud storage? Maybe, but as it stands at the moment using the same Apache instance that processes the PHP configured correctly with mod_deflate and mod_expires seems to do the job just as well.

Criticisms / Notes

Mosso provides you with a URL that cannot be pointed to with a CNAME as it contains a folder (some form of hash) as part of the path. This results in the Mosso URL ( http://cdn.cloudfiles.mosso.com/hash/file.ext ) showing up not only in the status bar but also throughout your source code.

If you happen to change your ‘Container’ from Public to Private and then back again you’ll have a new hash meaning your source code and clients browser caches are out of date.

The mod-rewrite script used for redirecting to Mosso:

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteRule ^(.*)(file|exts)$ http://cdn.cloudfiles.mosso.com/hash/$0 [NC,L]
</IfModule>


I took a selection of decommissioned mid-range [Pentium Dual Core / SATA I Disks etc] Workstations to which I installed CentOS 5.0 and the CleverSafe software. 4 IDE SliceStores and an Accesser provides enough IOP throughput for around 50 mailboxes. Ramping this up to 6 SATA SliceStores and there is enough IOP throughput for 70 mailboxes.

We’re obviously not talking about large scale Enterprise / Data Centre solutions here but for a small scale Exchange install [I'm thinking Test Lab setups maybe at a push a SOHO install] that needs reliable disk storage this seems to work quite well especially if used as the target storage for LCR [Consider that powering up a couple of old workstations is probably cheaper than the purchase of a decent SATA II hardware RAID card and extra disks.]

Utilising a 1Gb/s Network would improve the throughput considerably but for this test I was only utilising 100Mb/s between the Accesser and the Exchange server with a latency of around 40m/s so a constant Read Throughput of 2.8Mb/s will have to do.
 
 
DSNet offers access to the Storage ‘Vault’ via an iSCSI interface so there isn’t any extra work involved in preparing the Vault for use with Exchange 2007. Since this isn’t a normal SCSI device you have to disable “Synchronous Transfer” and “Tagged Queuing”. Their purposes are detailed here. Since these are enhancements to the operation of SCSI it shouldn’t adversely affect the operation of Exchange.

With the disks mounted and formatted it was a simple matter of changing the Queue and DB locations with PowerShell:
Move-StorageGroupPath -Identity "Second Storage Group" -LogFolderPath:"D:\Mailbox\Second Storage Group" -SystemFolderPath:"D:\Mailbox\Second Storage Group"

Running the Exchange server for a couple of days shows no degradation in usability even when a DSNet storage Vault was used for the Mailbox stores as well as the LCR.

Looking at other DSNets that are out there and looking at the potential for improvements in this DSNet then it may be possible to scale a DSNet out to production grade usability, if I have enough time and can get hold of some newer SATA II disks, better machines and a gigabit LAN infrastructure I’ll do a proper experiment and see what can be acheived.

Turns out that there has been a massive drop in Spam levels coming through some of the filters I have dotted around the place.

C&C or Spam Hosts?

I wrote a little script to see what percentages of machines that had made incoming connections that were then classified as spam were still alive.

The majority of hosts were still contactable in some way shape or form but weren’t listening on known SMTP ports, some were alive and actively listening on SMTP ports. A small percentage were totally offline.

This basic test appears to indicate that the Datacenter that went offline was merely a Command & Control hub rather than actual spamming hosts.

Elsewhere

Just to make sure that there were no anomolies with my graphs I checked a few other places and it does indeed appear that there is a worldwide drop in spam:


 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
It probably won’t be long until the levels are back where they used to be but for now our AntiSpam servers can rest for a while.

After applying for a tunnel from SixXS it was time to set it up. Unfortunately none of my JUNOS or Cisco IOS images have IPv6 support so rather than buying another 2600XM I decided to use the Windows Server 2008 server that performs IPv6 DHCP as the router.

The advice for setting up a tunnel on the Wiki only covers up to Windows Server 2003 and is below:

netsh interface ipv6 install
netsh interface ipv6 add v6v4tunnel SixXS [Your IPv4 Endpoint] [PoP IPv4 Endpoint]
netsh interface ipv6 add address SixXS [Your IPv6 Endpoint]
netsh interface ipv6 add route [Tunnel Prefix]/[Prefix Length] SixXS
netsh interface ipv6 add route 0::/0 SixXS publish=yes


The first line is redundant as IPv6 is already installed on 2008, the second to last command results in a warning that the object already exists and the last command needs to be:

netsh interface ipv6 add route ::/0 interface=SixXS nexthop=[PoP IPv6 Endpoint] publish=yes


In order to ‘prove’ your tunnel is alive it has to be pingable the advice on the SixXS site is to run the following command:

netsh firewall set icmpsetting SixXS enable all


Unfortunately Windows Server 2008 now has the ‘Windows Firewall with Advanced Security’. In order to allow pings you need to set the ‘Public’ profile to allow “File and Printer Sharing (Echo Request – ICMPv6-In).
You could add your own rule for ICMPv6 (Protocol type 58) but this was the easiest option at the time.
 
 
 
 
 
 
 
 
 
 
 
 
With all that done we now have an IPv6 (in IPv4 tunnel) up and running on Server 2008:

 
 
 
 
 
 

Windows Server 2008 – IPv6 Routing

To get packets moving through the network you need to configure forwarding on both interfaces. Then on the internal interfaces (the SixXS side of the Network has a static route) enable advertising which will help IPv6 enabled hosts to configure their routes. Once this is done the interfaces should look like this:
WAN [Tunnel] Interface

 
 
 
 
 
 
 
LAN [Internal] Interfaces
 
 
 
 
 
 
 

Its now time to see if all this is working, a quick renew on a machine on the LAN and we see this:

 
 
 
 
 
 
 
 
The first thing that struck me about that output is that the Default Gateway is a Link Local address. It turns out that for indirect delivery of packets (in which the destination is not on a local link) the next-hop address is typically the link-local address of the neighboring router.

Trusting that the Autoconfiguration has done its thing I fired off a traceroute and it works!

 
 
 
 
 
 
 
 
As a quick check I disabled the Firewall on the test box and it was publically available (which is good) but then leaving the local firewall disabled I added a rule on the Router to block ALL IPv6 packets but it carried on pinging which is because the Windows Firewall with Advanced Security is only for the Host itself not forwarded interfaces.

So how do I protect my precious IPv6 beer fridge from attackers?

Windows Server 2008 IPv6 Tunnel Security

I went through the Microsoft Press Understanding IPv6 book, hammered the hell out of my ? key in the netsh environment and then just as I started to read the Technet netsh command reference (is stubborness a virtue?) I remembered that you can add filters to any interface within netsh once the RRAS role feature has been installed!

Unfortunately even with the RRAS role features installed Windows still couldn’t address the SixXS tunnel. So, I bit the bullet and decided to secure the Network a different way.

Even though I can’t stop packets coming in from the SixXS tunnel I can prevent them getting forwarded to interfaces. The following netsh commands block all packets except pings, those that originate from the LAN or are for port 80:

set filter name="LAN Zone" filtertype=OUTPUT action=DROP
add filter name="LAN Zone" filtertype=OUTPUT srcaddr=:: srcprefixlen=0 dstaddr=:: dstprefixlen=0 proto=ICMP type=255 code=255
add filter name="LAN Zone" filtertype=OUTPUT srcaddr=:: srcprefixlen=0 dstaddr=:: dstprefixlen=0 proto=TCP srcport=0 dstport=80
add filter name="LAN Zone" filtertype=OUTPUT srcaddr=2a01:348:18e:1:: srcprefixlen=64 dstaddr=:: dstprefixlen=0 proto=ANY
set filter name="LAN Zone" fragcheck=disable


This is by no means perfect so I’ve subsequently added a lot more rules to the router. (No port scanning please, the text message sound for netflow alerts is rather jarring!)

Conclusions

IPv6 is an exciting new area to explore, the IPv6 Internet isn’t quite there yet (IPv6 sites are still few & far between) but it is nice to see applications out there and its a refreshing change to not have to worry about NAT.

With Christmas fast approaching I think I’ll reward myself with a shiny ‘new’ 2600XM with the IPv6 stack to handle the routing between my various zones and an ASA or two for the LAN segment. If I do then I’ll probably do another quick follow up regarding IPv6 subnetting, experiences with the Cisco IPv6 stack and whatever else I’ve stumbled upon in the mean time.

Interesting Notes

During the course of this little experiment I found a few random quirks that might amuse:

• The Windows Server 2008 DHCP Server can bind to a 6 in 4 tunnel but the DNS Server cannot!
• A very petty observation but theres a typo in the Interface Properties!