This week, I attended talks at the international high performance transaction workshop. Following are my rough notes from the Flash Memory discussions.

First up Steve Kleiman from Network Appliance spoke about their intent to move  flash into the clients that access NAS, so that it can intelligently cache and interact with the backend storage:

• Netapp is building host-side caches for VMware and HyperV
• The cache will be a block based read cache 
• Cache is write through

Andy Bechtolshiem, cofounder of Sun and now at Arista Networks talked about trends affecting system design and how Flash could be leveraged.

• 3d chip packaging will become common, to solve power and latency fundamentals around higher clock rate memory systems. 
• This will mean that systems will become very NUMA, since the memory is directly attached to the cpu core.
• This is the decade where we switch to optical in the datacenter. Cutover is 10gbit (copper), 20 requires optical and will move to volume commodity optical interconnects for distances greater than 1m.
• Flash densities to grow 100x between 2010 and 2022
• Processors will have 1024 cores in 2022
• NOR flash is not scaling as well as NAND flash, in terms of cost and density
• Major barrier to non-voltaile main memory is the single host failure issue.
• Current controllers (flash on sata) are not up to the speed
• Speed of flash increases from 100us down to 10us from 2009 to 2012
• ASP per $1GB of device flash $100 in 2004 to $1/gb in 2010 and 0.25c in 2012 (Halves every year)
• Controllers improving rapily to keep up with flash throughput (they are deficient now)
• Flash Interface moving from sata to pci-e in the near term

Microsoft Research Database - Phil Bernstein, Microsoft

• DB designed specifically for flash
• no-partitioning
• Taking advantage of native-flash chip semantics :append-only, db using log-appends, Updates the flash in append-only pages
• Database is ZFS-like: root block, copy on write binary tree 
• Uses a shared storage layer
• MySql “rethinkdb?” is doing something similar

Bradley Kurszmaul, Tokutek

• MySQL on SSD btree performance severely lacking
• Solution: build a MySQL storage engine suited for flash
• Observation: change read/write sizes and concurrency to suite: for today’s flash: ~20k reads, ~10k writes.
• Significant improvement in transaction throughput observed using the above.

IBM on Phase Change Memory

• Next generation after flash: Phase Change Memory
• Prediction is that SCM/PCM will flip the size relationship between RAM and 2nd level memory
• Much more 2nd level, small fast on-chip RAM
• PCM as fast as 100-1000ns
• Discussion centered around Flash being a short term trend, and due to limitations (wear leveling etc) PCM is more likely to be part of the systems architecture long term.

Recently, we were able to run a full TPC-C like workload inside a virtual machine, showing that we can run at over 90% efficiency, even given the brutal nature of this workload (it’s I/O rate is actually about 6 times higher than what you would typically see in a customer workload).

The methodology is that green is considered for the easy databases — typically 2 CPUs or less and less than 500 disk I/O’s per second. These virtualize easy, with minimal best practices required. Yellow is for larger databases, where the I/O rate is higher — they typically need more care and attention to the way the storage is provisioned underneath the virtual machine. For example, if the database needs 1,000 IOPS, then it should be on a disk or LUN with at least 10 disks backing it, to provide the necessary I/O throughput.

Red is typically when you have a database which is much larger than the limits provided for by a virtual machine — e.g. the maximum amount of memory or number of virtual CPUs possible.

That got me thinking — now that vSphere 4.0 (the new name for VMware’s full virtualization offering) we have three things happening at once — greatly improved database efficiency, a now-wider SMP virtual machine (SMP up to 8 virtual CPUs), and the increased throughput of the new Intel Nehalem systems. That combined, we have enough throughput to run some of the larger databases — often hosted on larger RISC machines.

Since it’s a database system, it’s relatively easy to migrate from SPARC to x86, given that there is an x86 equivalent for the OS (e.g. Solaris or Linux) and the database (e.g. Oracle).

By taking a composite metric of several different benchmarks, we can create a capability mapping chart between different RISC CPUs and the performance that we see inside a virtual machine. Some of the data may be surprising — some of those large older SPARC machines can now fit into a virtual machine on the newer x86 systems.

The first version of the table is over at the VMware performance blog.

The Physical RMCplex

It wasn’t long ago that solarisinternals.com was hosted on a triad of physical machines – a SPARCstation 5 as the router, a dual-socket Opteron machine as the ZFS NAS server, and a dual core AMD server.

The RMCplex is hosting many sites today:

• solarisdatabases.com
• solarisinternals.com
• pronk*.com
• joost*.com
• zygan*.com
• mail server
• dns server
• shared shell environment server

The router provides the perimeter security and QoS rules for the public traffic. I have four DMZ networks — management consoles, public Zone/VM traffic, a public wireless DMZ, and a private NFS-Only gigabit DMZ. All the data is held on a ZFS server, which is replicated to a remote location. The total power consumption of the rmcplex was about 650w — which incidently, in California costs about $1,300 a year.

In the past, upgrading or fiddling with versions of Solaris was a very hands-on operation — interacting directly with the physical console of the machine, while the site was momentarily down. I often upgrade to later builds of ZFS or Solaris (I’ve been hosting on Solaris Nevada since snv-21). We’d incur downtime when I had to upgrade or do maintenance on any one of the three machines.

The Virtual RMCplex

Last year, I moved the entire setup to VMware ESX. I’m using ESX 3.5, managed with Virtual Center. There were multiple motivations for this, which I’ll summarize later, but one of the most compelling was the ability to stage new/old versions of each now-virtual server concurrently, to eliminate the downtime during upgrades. I can easily bring up a new version of solarisinternals.com on snv101 for example, while the older one is still running. When I’m happy with the new app/OS provisioning, I just flip over to the new VM.

The new setup looks logically similar, but it’s appearance in the physical world is stunningly different. It went from three physical machines down to one, and from four to one ethernet switches.

To move the images from physical to virtual, I simply copied the raw disk image using dd, and encapsulated the raw disk image into a vmdk-format virtual disk. A small change to bootenv.rc and the image boots fine as a virtual machine, the rest of the OS environment is unmodified.

I’ve virtualized everything — including the router/firewall, all of the physical LAN segments (using Vlans and Vlan capable switch), and the Solaris ZFS server. The DMZ networks the link between the router and the hosted site virtual machines are now completely virtual — i.e., they are virtual ethernet networks with TCP traffic flowing between virtual machines, but never connect to anything physical. Consequently, I was able to eliminate the switch hardware for those segments of the configuration, too.

The ZFS server is interesting — it’s the NAS server for the virtual disks off all the virtual machines hosted on ESX, except for itself. This allows me to fully utilize ZFS’s hourly snapshot capability to roll forward, backward, and leverage compression. The ZFS virtual appliance is about 8TB of storage using 10 SATA disks, in a single virtual machine. I’m using ZFS with VMware ESX’s raw disk mode — so ZFS still see’s a whole physical disk and is able to operate just as it was before on physical — managing RAID-Z across the disks, and performing fault management isolation etc,… Incidently, I benchmarked the ZFS appliance just after converting it, and noted that it streams from a 5 disk SATA based zpool at about 350MB/sec sustained — likely enough for what we need That’s pretty close to the raw transfer speed of the 5 SATA disks. I’ve also enabled the ZIL on a NVRAM backed device, and used a flash ARC cache, which greatly helped performance. I’ll do a separate post sometime on the ZFS virtual appliance.

Teleportation

Back to the story of teleporting the running instance. Since I’m using just a single node of ESX, with fairly simple management tools around it, I was looking for a quick way of keeping solarisinternals.com up and running while I moved house. I was considering signing up for a virtual-hoster company to move the solarisinternals.com virtual machine to temporarily, but there was even a simpler way. Our CTO run’s VMware fusion at home on his iMac, and is close by. So we copied the solarisinternals.com virtual machine on to a 8GB compact flash card, and did a file->load onto VMware-fusion at the remote end.

Within a few minutes, we had a fully running replica of solarisinternals.com hosted at a different physical location. We overlapped the move by disabling updates at the original location, changing DNS, and enabling updates at the new location. That way, any requests with the stale IP would get read only access for the 5 minute TTL configured in DNS. We did this 10 days ago, and apart from the small hint “hosted on VMware Fusion” logo on the front page, it should have been completely transparent.

I monitor the site internally with the soon to be released “AppSpeed” product that is developed from the BHive aquisition by VMware last year, and externally using mon.itor.us. Over this weekend I did the reverse move, back onto the rmcplex. You can see from the monitoring data that we maintained similar performance levels before and after the move.

solarisinternals.com latency on VMware Fusion

And just after the move back to the RMCplex:

solarisinternals.com latency on VMware ESX

Out with all that Physical Stuff!

After this mini-datacenter consolidation,I ended up being able to throw out several old ethernet switches, about 50 cables, and three servers:

I’m saving quite a bit of power — going from 650 watts of power down to 214w (a saving of $880 per year in energy costs).

And it looks quite a bit simpler:

Summary

The new setup is much easier to manage, I get remote consoles from any IP connected device, it has the ablity to stage/roll forward/backward, and has geographic mobility when it’s required. Patching and upgrading is much easier now too.

That’s it for now — I have some interesting stories about the SQL and HTML performance data we’ve been able to collect with the Appspeed setup on the site, and I’ll share that soon in a subsequent post…

It's been well talked up that consolidating physical systems can save a lot of datacenter engery -- for example, consolidating 20 systems, all with 3-5% utilization can reduce power by up to 10-15x -- or considered in more tractable terms, each server removed is equivalent to taking four cars off the road or planting 55 trees in a year.

Taking this concept further, the distributed power management feature in Virtual Infrastructure allows us to further reduce power by another 50%. Since load levels of typical applications aren’t constant, there are periods of the day where less servers are required — and in those periods, power can be saved by moving the remaining work onto fewer servers and powering down the unneeded capacity

A colleague put together this entertaining video, in which we measured the performance and watts consumed during typical peaky workload. The scenario is based on the email load curve typically observed — that peaks at 9am in the morning, and is at a minimum during the evening hours.

Given our obsession with things that go fast, the track code “F1” is most appropriate

http://blogs.vmware.com/performance/2008/10/one-day-tutoria.html

Taking this concept further, the distributed power management feature in Virtual Infrastructure allows us to further reduce power by another 50%. Since load levels of typical applications aren’t constant, there are periods of the day where less servers are required — and in those periods, power can be saved by moving the remaining work onto fewer servers and powering down the unneeded capacity

A colleague put together this entertaining video, in which we measured the performance and watts consumed during typical peaky workload. The scenario is based on the email load curve typically observed — that peaks at 9am in the morning, and is at a minimum during the evening hours.

http://teslafounders.wordpress.com/2008/07/19/tesla-founders-car/

Great to see the Telsa is finally shipping!

This was done using 77TB of storage, adding a 3rd rack totalling 495 disks — enough storage to hold the entire printed library of congress, and all being driven from a single ESX server!

Chethan’s just written a detailed report over on VROOM…

The interesting points about this effort to me are:

• There was almost no tuning required
• There is no visible I/O throughput ceiling for ESX — the throughput is roughly that of 50 4-cpu databases…
• Even at 100,000 IOPS, the virtualization layer only added 0.1ms of additional latency, which amounts to 1.4% virtualization overhead

That’s 44TB of disk, and enough spindles to drive over 100,000 exchange users…

More disks are on the way…

We’re at Santa Cruz this week at the workshop for file system benchmarking. , chaired by Erez Zadok of StonyBrook University.

The objective of this week is to converge on a set of strategies and recommendations towards more accurate and representative benchmarking of file systems and storage.

Many of the File Systems and I/O benchmarks are represented:

• iozone
• FileBench
• SFS
• ZFS
• NFS
• Microsoft Clients

Devaki, Irfan and I will be presenting on the methodologies and strategies for measuring I/O performance with virtualized platforms. Some of the topics we’ll raise are

• Virtualization brings a diverse sets of applications: No single workload footprint on the system
• Workload characteristics are combined when workloads are consolidated onto a single system, resulting in stochastic workload patterns
• Different IO patterns to the same volume, or IO from one app split to different volumes
• Provisioning operations should be considered separate from applications run-time footprint (e.g. Create VM, Power On VM
• Virtual machines are stored in a clustered file system, need to consider locking
• CPU and virtual device emulation can impact storage performance
• System setup can affect performance
• Partition alignment can affect performance
• File system based or raw device mapping possible
• New hardware assist technology should be factored and/or considered
• Proxied I/O protocols are common: e.g. SCSI proxied over NFS within hypervisor

I’ll try to post an update of the workshop outcomes…