Virtually There: Steve Herrod's Blog

Mon, 30 Oct 2006

As mentioned in the previous entry, we now support 64-bit virtual machines on VMware ESX Server and across our whole product line. It has taken substantial effort across our engineering and QA team to reach the level of stability and performance that we’re delivering in these guests, and I thought a peek behind the curtains would be interesting.

We started this effort back in March 2004, expecting that the x86-compatible approach to 64-bit computing would eventually be in big demand by our customers. VMware software was completely built for 32-bit systems at that point, so the initial work was around collecting the appropriate compilers, debuggers, and general code framework for supporting these longer data types. We then began the process of converting appropriate 32-bit data types to be 64-bit ready. And here’s one of the earliest comments that came from our standard developer code review process:

I did want to point out that I find it really, really difficult to read hex 64-bit values. Is there any kind of standard for formatting these things in a more readable way?

The developers among you probably recognize this type of response! Well, we got used to those 0xf581818debaba080 values eventually. By June we had nailed down the basic concepts of 32 and 64 bit peers to enable us to leverage the 32 bit code as much as possible without cluttering the shiny new longmode (64 bit) world with any “legacy” code. In the following months, the notion of “48-sharing” code became omnipresent (48 being the halfway point between 32 and 64 bits).

A motivated team pushed forward, culminating in a major milestone on November 11, 2004… our first 64-bit Windows Blue Screen of Death (BSOD). Oh the joy! We were clearly cruising now and, as you can imagine, this time we were rather excited to see those long hex numbers.

The next milestone came less than 3 weeks later when, on December 1st, we completed a boot of 64-bit Windows XP, making it to a command prompt! For posterity’s sake, we saved this screenshot:

The careful reader will notice that this screenshot reporting a CPU speed of 595 MHz in the status bar. Yes, we were quite slow then with huge amounts of debugging code and completely untuned algorithms… getting to this point in the boot process required running for the better part of the night. Nowadays we can whip through a full installation and boot of Windows XP 64 in just 10 minutes! This is faster than native hardware as a virtual device such as the CD-ROM can actually be substantially faster than its physical incarnation.

In parallel to adding the core CPU and memory support to our virtual machine monitor, we launched efforts to update our networking virtual device. Our trusty AMD Vlance virtual device was fairly old and not supported by the latest 64-bit operating systems. We selected the Intel E1000 as our new 64-bit virtual networking device and moved forward on the implementation and performance tuning effort. Not long after the first set of boots, we were able to surf the net, mount network shares, transfer data, and serve web pages all at very high speeds.

From here it was on to development of the appropriate UIs, VMware tools for each new guest operating system, and delivery of the other pieces of VMware’s full guest OS support:

In parallel, we were of course going through substantial testing and additional performance tuning. Our earliest testing focused on building and executing large suites of instruction-level “unit tests”, ensuring test coverage of our core CPU and memory virtualization. From this point, we moved on to testing the guest operating systems themselves, installing and running substantial workloads in almost every incantation of 64-bit operating systems available. And from there it was on to system-level stress- and boundary-condition testing. This is where we really put our code through the ringer, running several stressful workloads in parallel, and using internal settings to regularly drive our code into conditions that would normally by quite infrequently reached.

Now on to shipping the products…

We first shipped VMware Workstation 5.5 with full support for 64 bit guests in October of 2005. We often ship core technologies such as new virtual machine capabilities on VMware Workstation first as it has hundreds of thousands of users who help us shake out any issues that we may have missed. We followed by including support in the new, royalty-free VMware Server 1.0 product, and then enabled experimental support in ESX Server 3.0 in June of this year. The effort culminated in ESX Server 3.0.1 and Virtual Center 2.0.1 as we finalized our tuning and testing and added complete support to Virtual Center (not the least of which was VMotion capabilities). And finally, our partner engineering team continues to work with our partners to certify ESX Server and these 64-bit virtual machines across the wide variety of Intel- and AMD-based servers now available.

Whew! Perhaps a bit more information than you wanted, but it’s been quite a journey and we’re very proud of the result.

posted at: 21:27 | | | permanent link

Mon, 02 Oct 2006

We’re ready to go with ESX Server 3.0.1, and I’m particularly excited with its full support for 64-bit guest operating systems. As you'll read in this blog, the transition to 64-bit computing is well underway, and VMware virtualization is a great way to easy some of the transition pain. Now for a little background...

With the launch of ESX Server 3.0, we offered “experimental” support for these guests, but we’ve now completed the testing, performance measurement and tuning, tools support, Virtual Center support, VMware DRS support, and all of the other pieces that go into VMware declaring a guest OS ready for production usage.

More specifically, we’re now fully supporting guests that utilize the x86-64 architecture (not Itanium or other 64-bit CPUs at this point). As of this blog, we’re supporting:

• Microsoft Windows Server 2003 (Standard and Enterprise Server R2)
• Red Hat Enterprise Linux 3 64-bit (UP7, UP8)
• Red Hat Enterprise Linux 4 64-bit (UP2, UP3)
• SuSE Linux Server (SLES) 10 64-bit
• Sun Solaris 10 (U2)

The Systems Compatibility Guide for ESX Server 3.0 will note when we add other guests. You can keep an eye on changes through a news feed: just go to our unified subscription page, select "Hardware Compatibility List", and add the resulting URL to your favorite newsreader.

There are specific hardware requirements for 64-bit guest operating system support. For AMD-based systems, the processors must be Athlon64 or Opteron Rev E or later. For Intel-based systems, the processors must include support for Intel’s Virtualization Technology (VT). Note that servers that include CPUs with VT support might ship with it disabled by default. You’ll have to enable it in the BIOS setup screen (or possible need a new BIOS version). We have a CPU compatibility tool included on the product CD-ROM to check this for you.

What is 64-bit computing?

Similarly to the transition from 16-bit to 32-bit computing back in 1985, Intel and AMD have extended the x86 architecture again, this time to 64-bits. For the most part, this means that memory addresses (pointers) are 64-bits and that all of the registers are 64-bits wide. AMD and Intel’s implementations are quite similar, though not exactly so. In case you’re wondering, AMD calls the resulting architecture  “AMD64”, Intel calls it “EM64T”, Microsoft calls it “x64”, and various others call it “x86-64”. If you’re interested in a lightish version of the details, Wikipedia has some pretty good articles on AMD64 and EM64T.

Why does 64-bit computing matter?

Our customers are beginning (or are well into) the process of migrating to 64-bit computing. Almost all of the major hardware and software vendors have multiple Web sites pointing out the importance of this technology. Here are just a few quick pointers:

AMD’s site:

Why is 64-bit technology important?
A: The need for 64-bit technology is driven by applications that require high performance and address large amounts of physical and virtual memory...

AMD’s evolutionary approach to 64-bit technology enables a gradual transition from 32-bit to 64-bit software and hardware for end users who need greater computing functionality for some or all of their applications. AMD64 is designed to provide superior performance for applications requiring multi-processor scalability, larger addressable memory, video/audio/3-D processing or high computational accuracy.

Intel’s site:

Transcending the 4GB memory limit of 32-bit processing, platforms with 64-bit capabilities can access virtually unlimited physical memory. In addition, the 64-bit processor can manipulate data and execute instructions in chunks that are twice as large (64-bits versus 32-bits).

Microsoft’s site:

The 64-bit versions of the Windows Server 2003 family are designed to run on high-performance 64-bit processors, which improves performance and scalability—often with dramatic benefits. This lowers your total cost of ownership by delivering superior price performance and reducing administration costs.

Sun’s site:

The ever-increasing complexity of applications makes 64-bit computing more compelling than ever. A 64-bit computing environment provides an address space more than four billion times the size of the current 32-bit environment. This affords many advantages to the developer creating high-end applications. Among these advantages are:

• Faster operations. With wider CPU operations, one instruction can handle operations that would otherwise require multiple instructions.
• More addressable memory. A 64-bit machine can address more memory directly, enabling the CPU to bring more data into memory for faster operations. The Solaris operating environment with 64 bits supports 18 Exabytes of virtual address space per application.
• Very Large file sizes. In a 64-bit environment, applications can work on files well in excess of the two-gigabyte maximum of the 32-bit environment. The Solaris operating environment with 64 bits supports file sizes up to one terabyte.
• Improved disk management. The ability to handle large files means that larger amounts of data can be accessed as a single file, rather than needing to break the file down into two-gigabyte pieces.

And the list goes on…

ISVs, particularly for the higher-end applications, are taking advantage of these benefits. In fact, we’re seeing several ISVs begin the transition to only supporting 64-bit versions of their applications. This includes key products from Microsoft, SAP, Oracle, and several other vendors. We expect this transition to continue to ramp, and think virtualization can play a key role in helping manage it.

How does VMware virtualization fit in?

Of course customers can realize the benefits and solutions that they’re used to with our products such as server consolidation, provisioning speed, dynamic load-balancing utilizing VMotion, higher availability, and a rich set of disaster recovery solutions. We also support these 64-bit guests on our VMware Workstation and VMware Server products, allowing you to share the same 64-bit image across various products, depending on your specific needs.

However, there is an extra important set of benefits that VMware virtualization provides during the migration to 64-bit computing. In his SearchServerVirtualization article entitled “How virtualization can simplify your transition to 64-bit computing”, Alessandro Perilli does a good job of describing the benefits of virtualization to the migration process:

Depending on your business needs, you could approach the migration with two opposite strategies.

The first one consists of moving existing 32-bit applications into virtual machines, while modernizing the hardware population and testing new products on the new servers. In this way, virtual machines will act as proven solutions on standby when we decide to adopt 64-bit applications, and they will do so without requiring significant extra efforts in maintaining two different machine sets. And, as a side benefit, dismantling 32-bit servers will proceed much faster.

This strategy is the most aggressive and is probably better for companies that wish to gain benefits from new architecture as soon as possible.

The opposite approach is to maintain the existing 32-bit hardware population and carefully introduce 64-bit new applications in virtual machines in which new products can be tested and verified. The huge lack of 64-bit drivers for many devices makes virtual environments a more stable platform in which to work.

This second strategy fits better in mission-critical scenarios, but it's likely to force virtualization adoption. If at a certain point the company decides to move applications back onto physical hardware, the whole process could be painful. For this reason, it's best to take this path only if server consolidation was already part of the plans.

So as you can see, the transition to 64-bit computing is underway, and there are compelling reasons to use virtualization during this process. We hope this ESX Server release goes a long way in helping ease this transition!

This is turning into quite a long entry, so I’ll close it off now. Stay tuned for a follow-up entry that provides details into how we went about the engineering work to enable this milestone. It’s been a long journey, and I think you’ll find it interesting.

posted at: 00:00 | | | permanent link