blog.richardmcdougall.com

The Solaris Internals Virtual Shell Game

In a stark contrast to recent past when Solaris Internals was hosted on physical machines, we’ve just experienced one of the core values of virtualization — encapsulated, portable state allowing geographic mobility. Solaris Internals has over the last two weeks been teleported out of my datacenter (As Jim notes, the “RMCplex”, which refers to my rack of managed servers), to the VMware’s CTO’s iMac, and then to my new physical location — keeping the the solarisinternals.com site live the entire duration. Yes, I just moved house, with zero downtime!

The Physical RMCplex

It wasn’t long ago that solarisinternals.com was hosted on a triad of physical machines – a SPARCstation 5 as the router, a dual-socket Opteron machine as the ZFS NAS server, and a dual core AMD server.

The RMCplex is hosting many sites today:

• solarisdatabases.com
• solarisinternals.com
• pronk*.com
• joost*.com
• zygan*.com
• mail server
• dns server
• shared shell environment server

The router provides the perimeter security and QoS rules for the public traffic. I have four DMZ networks — management consoles, public Zone/VM traffic, a public wireless DMZ, and a private NFS-Only gigabit DMZ. All the data is held on a ZFS server, which is replicated to a remote location. The total power consumption of the rmcplex was about 650w — which incidently, in California costs about $1,300 a year.

In the past, upgrading or fiddling with versions of Solaris was a very hands-on operation — interacting directly with the physical console of the machine, while the site was momentarily down. I often upgrade to later builds of ZFS or Solaris (I’ve been hosting on Solaris Nevada since snv-21). We’d incur downtime when I had to upgrade or do maintenance on any one of the three machines.

The Virtual RMCplex

Last year, I moved the entire setup to VMware ESX. I’m using ESX 3.5, managed with Virtual Center. There were multiple motivations for this, which I’ll summarize later, but one of the most compelling was the ability to stage new/old versions of each now-virtual server concurrently, to eliminate the downtime during upgrades. I can easily bring up a new version of solarisinternals.com on snv101 for example, while the older one is still running. When I’m happy with the new app/OS provisioning, I just flip over to the new VM.

The new setup looks logically similar, but it’s appearance in the physical world is stunningly different. It went from three physical machines down to one, and from four to one ethernet switches.

To move the images from physical to virtual, I simply copied the raw disk image using dd, and encapsulated the raw disk image into a vmdk-format virtual disk. A small change to bootenv.rc and the image boots fine as a virtual machine, the rest of the OS environment is unmodified.

I’ve virtualized everything — including the router/firewall, all of the physical LAN segments (using Vlans and Vlan capable switch), and the Solaris ZFS server. The DMZ networks the link between the router and the hosted site virtual machines are now completely virtual — i.e., they are virtual ethernet networks with TCP traffic flowing between virtual machines, but never connect to anything physical. Consequently, I was able to eliminate the switch hardware for those segments of the configuration, too.

The ZFS server is interesting — it’s the NAS server for the virtual disks off all the virtual machines hosted on ESX, except for itself. This allows me to fully utilize ZFS’s hourly snapshot capability to roll forward, backward, and leverage compression. The ZFS virtual appliance is about 8TB of storage using 10 SATA disks, in a single virtual machine. I’m using ZFS with VMware ESX’s raw disk mode — so ZFS still see’s a whole physical disk and is able to operate just as it was before on physical — managing RAID-Z across the disks, and performing fault management isolation etc,… Incidently, I benchmarked the ZFS appliance just after converting it, and noted that it streams from a 5 disk SATA based zpool at about 350MB/sec sustained — likely enough for what we need That’s pretty close to the raw transfer speed of the 5 SATA disks. I’ve also enabled the ZIL on a NVRAM backed device, and used a flash ARC cache, which greatly helped performance. I’ll do a separate post sometime on the ZFS virtual appliance.

Teleportation

Back to the story of teleporting the running instance. Since I’m using just a single node of ESX, with fairly simple management tools around it, I was looking for a quick way of keeping solarisinternals.com up and running while I moved house. I was considering signing up for a virtual-hoster company to move the solarisinternals.com virtual machine to temporarily, but there was even a simpler way. Our CTO run’s VMware fusion at home on his iMac, and is close by. So we copied the solarisinternals.com virtual machine on to a 8GB compact flash card, and did a file->load onto VMware-fusion at the remote end.

Within a few minutes, we had a fully running replica of solarisinternals.com hosted at a different physical location. We overlapped the move by disabling updates at the original location, changing DNS, and enabling updates at the new location. That way, any requests with the stale IP would get read only access for the 5 minute TTL configured in DNS. We did this 10 days ago, and apart from the small hint “hosted on VMware Fusion” logo on the front page, it should have been completely transparent.

I monitor the site internally with the soon to be released “AppSpeed” product that is developed from the BHive aquisition by VMware last year, and externally using mon.itor.us. Over this weekend I did the reverse move, back onto the rmcplex. You can see from the monitoring data that we maintained similar performance levels before and after the move.

solarisinternals.com latency on VMware Fusion

And just after the move back to the RMCplex:

solarisinternals.com latency on VMware ESX

Out with all that Physical Stuff!

After this mini-datacenter consolidation,I ended up being able to throw out several old ethernet switches, about 50 cables, and three servers:

I’m saving quite a bit of power — going from 650 watts of power down to 214w (a saving of $880 per year in energy costs).

And it looks quite a bit simpler:

Summary

The new setup is much easier to manage, I get remote consoles from any IP connected device, it has the ablity to stage/roll forward/backward, and has geographic mobility when it’s required. Patching and upgrading is much easier now too.

That’s it for now — I have some interesting stories about the SQL and HTML performance data we’ve been able to collect with the Appspeed setup on the site, and I’ll share that soon in a subsequent post…