The core of this setup was a series of 3G Modems linked up with 3G to ethernet devices such as the Solwise NET-3G-3GWIFIMRW.

These were backed off onto a server running Squid and BIND with DHCP containing all the relevant proxy auto config data (plus some IPTables magic for those that didn’t play nice).

Couple this with three Access points with 8db omni’s spread about to acheive maximum coverage I was pleased to see that at one point the network was sustaining over 8mbit/s of throughput!

There was a lot of web browsing, I was streaming spotify, people were blogging and tweeting and despite the heat it all stayed alive.

Next time I’ll be doing it without mains power either so mount up the UPS’s and gas the generators!

Cross Forest trusts are a possibility and merging one with the other (i.e having the Hosted Exchange solution bound to the existing domain) is another but there are many issues with that (mostly political).

What I intend to do is utilise the ‘Branch office’ concept that Read Only Domain Controllers were designed for to mock up a solution for Hosting the entire AD infrastructure remotely and having R/O DC’s on the customer premises.

What now?

For no other reason than that of satisifying my curiosity I built an entire AD infrastructure hosted at the data center and then had a remote ‘office’ running for a day without a local DC and then the following day with a Read Only Domain Controller sitting there.

There’s nothing new or crazy here other than maybe the fact that most people move bits of their AD infrastructure to the DC when its bandwidth requirements overwhelm their resources. What I’m playing with is the idea of having everything remote and only putting the stuff you need (NAS etc) in the office.

The Test

In the Red Corner we have a full Active Directory and Exchange infrastructure at the DC and then the ‘offices’ were built using a few Terminal Services servers running a respective amounts of users. The idea is to monitor traffic before dropping in a RO DC and then again afterwards.

The Infrastructure

Hosted Infrastructure

The Hosted infrastructure consists of a relatively standard Exchange 2007 deployment (if you follow the guidelines) visible to the world (selected ports only) is an Edge Transport server for handling the initial mail connections and the Client Access Server. Behind those is the Mailbox and Hub Transport (in reality these were on the same box but the diagram wasn’t as symmetrical then!).


The Domain controller is a special case because whilst we have no reason for the Internet at large to talk to it we need the read only Domain Controller at the client site to be able to communicate with it so an IPSEC LAN to LAN VPN was required.

The Results

AD Traffic From the TS to the Remote DC No Local DC

AD Traffic to the Remote DC with Local RODC

OWA Traffic During the Tests

Scripted behavior – so it was the same(ish) on both days

Conclusion

Well it did exactly what I expected it to do so nothing ground breaking there. It was interesting to see the spike just after I logged all the fake users off the Terminal Servers.

R/O DC’s were used because in an ideal world customers shouldn’t have write access to an AD infrastructure that a SysAdmin has an SLA to honor!

After applying for a tunnel from SixXS it was time to set it up. Unfortunately none of my JUNOS or Cisco IOS images have IPv6 support so rather than buying another 2600XM I decided to use the Windows Server 2008 server that performs IPv6 DHCP as the router.

The advice for setting up a tunnel on the Wiki only covers up to Windows Server 2003 and is below:

netsh interface ipv6 install
netsh interface ipv6 add v6v4tunnel SixXS [Your IPv4 Endpoint] [PoP IPv4 Endpoint]
netsh interface ipv6 add address SixXS [Your IPv6 Endpoint]
netsh interface ipv6 add route [Tunnel Prefix]/[Prefix Length] SixXS
netsh interface ipv6 add route 0::/0 SixXS publish=yes


The first line is redundant as IPv6 is already installed on 2008, the second to last command results in a warning that the object already exists and the last command needs to be:

netsh interface ipv6 add route ::/0 interface=SixXS nexthop=[PoP IPv6 Endpoint] publish=yes


In order to ‘prove’ your tunnel is alive it has to be pingable the advice on the SixXS site is to run the following command:

netsh firewall set icmpsetting SixXS enable all


Unfortunately Windows Server 2008 now has the ‘Windows Firewall with Advanced Security’. In order to allow pings you need to set the ‘Public’ profile to allow “File and Printer Sharing (Echo Request – ICMPv6-In).
You could add your own rule for ICMPv6 (Protocol type 58) but this was the easiest option at the time.
 
 
 
 
 
 
 
 
 
 
 
 
With all that done we now have an IPv6 (in IPv4 tunnel) up and running on Server 2008:

 
 
 
 
 
 

Windows Server 2008 – IPv6 Routing

To get packets moving through the network you need to configure forwarding on both interfaces. Then on the internal interfaces (the SixXS side of the Network has a static route) enable advertising which will help IPv6 enabled hosts to configure their routes. Once this is done the interfaces should look like this:
WAN [Tunnel] Interface

 
 
 
 
 
 
 
LAN [Internal] Interfaces
 
 
 
 
 
 
 

Its now time to see if all this is working, a quick renew on a machine on the LAN and we see this:

 
 
 
 
 
 
 
 
The first thing that struck me about that output is that the Default Gateway is a Link Local address. It turns out that for indirect delivery of packets (in which the destination is not on a local link) the next-hop address is typically the link-local address of the neighboring router.

Trusting that the Autoconfiguration has done its thing I fired off a traceroute and it works!

 
 
 
 
 
 
 
 
As a quick check I disabled the Firewall on the test box and it was publically available (which is good) but then leaving the local firewall disabled I added a rule on the Router to block ALL IPv6 packets but it carried on pinging which is because the Windows Firewall with Advanced Security is only for the Host itself not forwarded interfaces.

So how do I protect my precious IPv6 beer fridge from attackers?

Windows Server 2008 IPv6 Tunnel Security

I went through the Microsoft Press Understanding IPv6 book, hammered the hell out of my ? key in the netsh environment and then just as I started to read the Technet netsh command reference (is stubborness a virtue?) I remembered that you can add filters to any interface within netsh once the RRAS role feature has been installed!

Unfortunately even with the RRAS role features installed Windows still couldn’t address the SixXS tunnel. So, I bit the bullet and decided to secure the Network a different way.

Even though I can’t stop packets coming in from the SixXS tunnel I can prevent them getting forwarded to interfaces. The following netsh commands block all packets except pings, those that originate from the LAN or are for port 80:

set filter name="LAN Zone" filtertype=OUTPUT action=DROP
add filter name="LAN Zone" filtertype=OUTPUT srcaddr=:: srcprefixlen=0 dstaddr=:: dstprefixlen=0 proto=ICMP type=255 code=255
add filter name="LAN Zone" filtertype=OUTPUT srcaddr=:: srcprefixlen=0 dstaddr=:: dstprefixlen=0 proto=TCP srcport=0 dstport=80
add filter name="LAN Zone" filtertype=OUTPUT srcaddr=2a01:348:18e:1:: srcprefixlen=64 dstaddr=:: dstprefixlen=0 proto=ANY
set filter name="LAN Zone" fragcheck=disable


This is by no means perfect so I’ve subsequently added a lot more rules to the router. (No port scanning please, the text message sound for netflow alerts is rather jarring!)

Conclusions

IPv6 is an exciting new area to explore, the IPv6 Internet isn’t quite there yet (IPv6 sites are still few & far between) but it is nice to see applications out there and its a refreshing change to not have to worry about NAT.

With Christmas fast approaching I think I’ll reward myself with a shiny ‘new’ 2600XM with the IPv6 stack to handle the routing between my various zones and an ASA or two for the LAN segment. If I do then I’ll probably do another quick follow up regarding IPv6 subnetting, experiences with the Cisco IPv6 stack and whatever else I’ve stumbled upon in the mean time.

Interesting Notes

During the course of this little experiment I found a few random quirks that might amuse:

• The Windows Server 2008 DHCP Server can bind to a 6 in 4 tunnel but the DNS Server cannot!
• A very petty observation but theres a typo in the Interface Properties!

IPv6 Only Domain Controllers:

The installation didn’t cause any trouble but after initially logging on and running a DCDiag we see this:

It turns out that the install of the DNS Server had set the NIC’s properties to be ::1 but the DNS Server was only listening on the Static Site Local address FEC0::2 and its self assigned Link Local. This of course caused all DNS reliant checks to fail and cause a whole world of pain.
 
 
 
 
 

Changing the NIC DNS properties to FEC0::2 resulted in a different set of errors:

There were actually 10′s of the System Log errors which made me panic for a bit till I remembered that this is expected behaviour. The expected behaviour is that if there are any Warnings or other bad apples in the Event logs DCDiag will throw some unfriendly errors.

These errors appear to be even more unfriendly owing to a bug in how the Event Viewer copes with IPv6 address.
 
 
 
 
 
 

A quick purge or the Event logs later and we are back in business:

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Preparing the Exchange Server

Exchange 2007 has several prerequisites that need to be installed before Setup can be successfully launched. First I’ll add the prerequisites that don’t affect / require a machine to be bound to a domain, if they install correctly then I’ll bind it to the domain (and see how that goes) and finish off with the prerequisites that require a machine to be bound to the domain.

Installing the non-domain related roles / features went smoothly:

 
 
 
 
 
 
 
 
 
 
 
 
Unfortunately binding to the domain was not as smooth:

The first issue to resolve is whether Windows Server 2008 actually wants a A record or whether someone just hasn’t got round to updating the error dialog.
 
 
 
 
 
 
 
Firing up the NSLookup tool we see that [despite some timeouts (??)] DNS is working as expected. Although this was evident because the SRV lookups for the domain resulted in the name of the Domain Controller.

 
 
 
 
 
 
 
 
So I did what every curious Windows Admin knows might fix the problem, do it again. And who’d have guessed it?

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Hmm, well with that sorted and following a quick reboot its time to see if the final prerequisite piece of the puzzle works:

Excellent!

There are a few errors but these may not be show stoppers.

Onto the next stage!
 
 
 
 
 
 
 
 
 
 

Installing Exchange 2007 on an IPv6 Only 2008 Server

Well unfortunately we’ve stumbled at the first hurdle:

 
 
 
 
 
 
 
 
 
 
 
 
 
The link the error mentions tells us the following:

IPv6 is only supported in Exchange 2007 SP1 when it is installed on a Windows Server 2008 computer that has both IPv4 and IPv6 enabled. If you disable the IPv4 protocol, Exchange 2007 SP1 can't support IPv6.

Well screw that, the Install button hasn’t greyed out so onwards to Victory!

 
 
 
 
 
 
 
 
 
 
 
 
 
Setup claims to have installed, lets see if the SMTP element is contactable:

 
 
 
 
 
 
 
 
 
 
 
 

Using Exchange 2007 in a Native IPv6 Environment

OWA worked without any major hassles and sending internal email worked fine. Unfortunately trying to send email to an ‘external’ host resulted in a “451 4.4.0 DNS Query Failed”.

Looking through the Event Logs I found this:

 
 
 
 
 
 
 
 
 
 
 
Running the suggested powershell command did indeed show that no DNS Servers were set:

 
 
 
 
 
 
 
 
No matter what I did using netsh or the NIC GUI the error remained, I resorted to setting the DNS entries in the Hub Transport server properties:

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
We now get another error but hey at least its progress:
451 4.4.0 primary target IP address responded with "421.4.4.2 unable to connect."attempted failover to alternate host, but that did not succeed.Either there are no alternate hosts, or delivery failed to all alternate hosts.

Turns out I forgot to make the Postfix server listen on its IPv6 address :/ A few quick config changes later and Victory is mine!

 
 
 
 
 
 
 
 
 
 
 

Conclusion

Well its a bit painful to get going but it appears to work once setup. I’ll keep this going with some scheduled in and out email to see if it dies after a week or so.

Armed with a [now depreciated] netblock I set about the practise run for the Plan O’ Doom.

Configuring the scopes is easy and so doesn’t really need to be covered but in order to split my /48 [SixXS subnets are /48] into more usable subnets I cheated on the calculations and used Rhys Koedijk’s IPv6 Subnet calculator.

A word of caution, despite setting up the Scopes and binding static IPv6 addresses on the relevant interfaces the clients were not getting any leases because the Interfaces on the relevant subnets were not configured correctly. You will need to run the following command to get leases working:
netsh interface ipv6 set interface advertise=enabled manageAddress=enabled












Unfortunately a day later I had a quick look at the leases on my ‘test’ subnetwork and was greeted by the following:

Thinking that this could be due to my previous experiments I deleted the leases and then logged onto each machine that should have IPv6 enabled and did a release6 & renew6.

Unfortunately there was still one left:















A.J. Anto has an old post which details why these BAD_ADDRESS’s should be showing up but that doesn’t seem to be the case here.

A bit more research found a Hyper V’d Vista machine that I’d forgotten about which had a 12 day lease.

Interestingly even if you assign a static IPv6 address to an interface Windows Server 2008 (and probably Vista too) still request an IP from DHCP:










The next IPv6 experiment is a SixXS tunnel to a Juniper Netscreen 5GT with the LAN and Wireless segments having ‘real’ IPv6 addresses. – Keep an eye out for the “NetworksAreMadeOfString IPv6″ SSID.

With the most tracks HOPE has ever had I was truly spoilt for choice but I spent most of my time [when I wasn't showing our US friends how drinking should be done] visiting talks that had potential datacenter impact.

Kevin Figueroa, Marco Figueroa and Anthony L. Williams reminded me that VLAN’s and other layer 2 stuff is still vulnerable to many attacks. Most are just Denial of Service stuff that would be detected almost instantly and very easily fixed (although not easily preventable) but the cross VLAN packet injection / snooping made me rethink some of my installations that packet injection / snooping would not be a critical issue but might not be desirable. Bare in mind though that in order to attack layer 2 someone needs to own a box on my layer 2 infrastructure which is much more of an issue!

The demonstration that stuck with me the most was Jacob Appelbaum, Dino Dai Zovi and Karsten Nohl’s talk regarding the Debian OpenSSL catastrophe. When I first heard about it there was much smugness (being as I only use Red Hat Enterprise / CentOS and FreeBSD) My fellow SysAdmins and I had many hours of fun reminding our collegues who had chosen Debian as their Distro of choice of this ‘little’ bug.

Unfortunately our smugness was shortlived as these Debian keys can end up in a RedHat servers authorised_keys file which results in that server being vulnerable to Brute Forcing. Knowing how lazy people are and how widespread the careless use of root is there was likely to be a few machines out there just waiting to get rooted.

The majority of tools (asides from the official Debian tools) are designed for discovering vulnerable servers that don’t belong to you! The Debian tool is great unless for one reason or another you don’t / can’t have Perl installed.

Therefore I present to you another hacked together bash script that anyone could have put together in 5 minutes but maybe this’ll save someone the hassle.


#!/bin/bash
#--------------------------------------
# Looks for fscking debian client keys
# Gareth#NetworksAreMadeOfString.co.uk
#--------------------------------------

for KeysFile in `locate authorized_keys`; do
  echo
  echo Testing $KeysFile for weak keys
  echo -------------------------------------------------

  cat $KeysFile | while read line; do

   echo $line > pubkey.tmp

   RawFP=`ssh-keygen -l -f pubkey.tmp | awk '{print $2}'`

   FP=${RawFP//:/}

   MatchCount=`grep -c $FP FingerPrints.db`

   if [ $MatchCount -gt 0 ]; then
   echo -e '\E[40;31m'"\033[1m!!!!! WEAK KEY FOUND!!!!!!\033[0m"
   echo $line
   fi
   done
  done
 echo
 echo ------------- DONE -------------


The FingerPrints.db file can be found here

Hopefully you’ll find vulnerable user accounts before someone else does!

Feature List
Where to get it

Update – 22/11/2007
In little under 2 weeks I’ve seeded nearly 40Gb! Ubuntu 7.10 has only seeded 33Gb and has been out for twice as long. Read into that what you will!

For those SysAdmins with hundreds of servers in a datacenter and can’t afford to saturate their links downloading all of these packages each and every time RedHat offer some products one of which is the RHN Satellite entitlement. Now I thought to myself that if RedHat can provide what they call RedHat Proxy for RHN Satellites then it is possible that I can do something similar for Yum. (I’m letting the RHEL servers update directly because they have plenty of bandwidth available and more importantly I don’t have any Satellite entitlements!).

The first thing I considered was utilising Squid on my trusty router o’ doom and then I realised that it is running RedHat 9, only has a 4Gb HDD and 128Mb of RAM. That idea was quickly scrapped. Then I realised that the CacheFlow 600 was DualNIC enabled and even with the neighbourhood trying to hammer my bandwidth it was still sitting quite happily at only 13% capacity.

After configuring the CacheFlow (since this article is more about the benefits of caching, coupled with the fact that I guess I’m the only person who has a CacheFlow at home I’ll skip that bit) it was a simple case of editing the yum.conf of each machine.

I didn’t want all traffic to be going through the CacheFlow (not sure why) and since the CacheFlow is not the default gateway for their side of the network it would have required a static route on the Router O’ Doom to send all traffic from the CentOS machines for the repo’s back through to the CacheFlow. Obviously it wouldn’t be wise to forget to filter for the source of the requests otherwise the CacheFlow’s requests would get sent back to itself and then I would probably end up in a whole new world of pain.

Adding a proxy for Yum to use is simple:

[main]
cachedir=/var/cache/yum
debuglevel=2
logfile=/var/log/yum.log
pkgpolicy=newest
distroverpkg=centos-release
tolerant=1
exactarch=1
retries=20
obsoletes=1
gpgcheck=1
proxy=http://172.16.0.7:8080


With that done I ran the first CentOS machine off on a little jaunt to do a full package update. With the CentOS Plus repository enabled the package list came to 178Mb, multiple that by 5 machines and thats 890Mb excluding overheads of course. Ok so maybe all of this is a bit overkill but hey its something to do and after all electricity is free isn’t it………

Initial Run:
Time: 26mins
Average Throughput: 146.4 kB/s

4 Remaining Servers:
Time: 9mins
Average Throughput: 580.2kB/s

Whilst the times and average speeds speak for themselves the MRTG graphs of what happened are even better:

CacheFlow Client Throughput (Cacheflow to Clients)


 
CacheFlow Server Throughput (Cacheflow to Internet)


 
The small ‘blip’ in traffic towards the internet is the CacheFlow checking whether anything had changed since it was last cached.

The benefits here are apparant even if at this scale it isn’t really worth the effort. However NAMOS isn’t about that, hell it’ll probably never be about that.

The Network isn’t totally insecure, the Access Points sit behind a Juniper NS5GT and all web / IM / mail is piped through a BlueCoat Security Appliance. I noticed that the client stats were increasing and I was wondering what people were doing with the system.

As indicated earlier a Firewall and Security appliance sits between the wireless and my Internet router, the Firewall only allows Web, IM and IMAP/POP3 through. So I was thinking what would these people would do if I let them do what they want. An investigation was called for!

So cue sticking NTOP on a spare machine and a quick mirrored port later means I can watch what happens. 24 hours later I had some interesting results.

Basic Traffic Breakdown:

Packet Size Breakdown:

Distance of Hosts

Network Throughput
Actual 681.8 Kbps
Last Minute 624.4 Kbps
Last 5 Minutes 644.0 Kbps
Peak 7.7 Mbps
Average 617.7 Kbps

So I think we can safely say that people like Torrents and Usenet.

A note on Privacy: The BlueCoat device displays a splash screen when a user first attempts to connect to a website and also inserts a message to both parties at the start of an IM conversation. So people knew that there was the possibility that what they were doing was going to be recorded.