Hyper-V FAQs, Tips, and Tricks – Checkpoints and Snapshots

One of the topics at our Hyper-V FAQs, Tips, and Tricks discussion at TechEd was that of checkpoints and snapshots, and whether they should be used in production or not, as well as the gotchas that one may come across when using them.

Snapshots versus Checkpoints

First, I have to mention that snapshots and checkpoints are the same thing. Hyper-V calls them snapshots, and System Center Virtual Machine Manager calls them checkpoints. When you make a checkpoint in VMM, it actually creates a snapshot in Hyper-V. When you create a snapshot from Hyper-V, you’ll see it in the VMM console as a checkpoint. With that out of the way, let’s discuss what they are for, and when you night use one.

What are snapshots and when/why should I use them?

Snapshots are representations of a virtual machine at the point of time from which the snapshot was taken. They are most useful when you’re about to make a change which you might want to quickly roll back. You can take a snapshot, make your change, and then revert back to your previous state. One of the most common scenarios is in doing software sequencing with something like App-V. You can take a snapshot of your VM, sequence the installation of your software, copy the results off of the machine, and then roll back. However, snapshots are not to be used for backup purposes. Snapshots don’t behave the same way a backup would behave, and traditional backups may likely need to be used, depending on your scenario (either SAN level or from something like System Center Data Protection Manager).

When/why should I not use them? What are the “gotchas”?

Though snapshots are a valuable tool, there a few things to be aware before you ever start using snapshots. Knowing these few things will save you a lot of grief and wasted time in the future:

  • Machine rollbacks can cause Active Directory connection problems
  • Leaving machines running for an extended period of time after a snapshot can use up significant disk space
  • Merging snapshots into existing VM can take many hours, during which the VM has to remain offline to complete the merge

Rollback to previous snapshot can cause Active Directory Password problems

The first thing to be aware of is that computers have a unique machine account password within Active Directory that by default changes every 30 days for security reasons. If the password changes between the time you take the snapshot and the time you want to rollback, you’ll break the computer’s access to Active Directory when you revert to the snapshot. This will in turn cause you to lose the ability to login to your VM with any domain credentials unless they were previously cached. Once you find yourself in this state, you need to login via a local account or account with cached credentials and unjoin/rejoin the computer from the domain.

To prevent this automatic password change from occurring, you can disable the automatic password reset by following the steps within Microsoft KB 154501 (thanks to Aaron Sudduth for the live link during the session). Once you flip the registry key, you can safely revert without having to worry about whether the machine is in sync with AD, since it will permanently keep the same machine account password. (This would apply to a traditional backup as well – anything which brings the VM back to a previous point in time.)

Snapshots, AVHD, and disk space

Once you take a snapshot of your VM, all of the VMs future writes will be written to a differencing disk (AVHD) rather than your original VHD. This means that over time, the AVHD will continue to grow indefinitely, even if space inside the VM isn’t growing, but is merely changing alot. If you’re leaving this machine up for years, it’s only a matter of time before you eventually run out disk space. At that point, you have to make the decision to power down the VM and merge the snapshot into the current VM, which brings me to my final point.

Merging snapshots can mean hours of downtime

In the current version of Hyper-V, the only way you can merge the AVHD back into a VHD is by deleting your original snapshot and powering off the VM, and then waiting for Hyper-V to merge the files. Depending on the speed of your storage, and size of the AVHD, this can take minutes, hours, or days. In general, you don’t have days of time that you can wait for AVHDs to merge, which means you don’t want to get yourself into the situation to begin with.

Conculsion

So the moral of the story is: feel free to use snapshots and checkpoints, but only for VMs that need to be rolled back regularly, and which won’t be left on for really long periods of time after the snapshot is taken. If you actually want to keep a pristine copy of your VM for months or years, use a traditional backup instead. You can do a SAN-level snapshot, a DPM backup, or a simple file copy of your VHD to external storage.

Good luck, and happy virtualizing!

Rapid Provisioning with System Center Virtual Machine Manager

During my presentations at TechEd this year, I spoke about a somewhat rarely used and somewhat unknown feature of System Center Virtual Machine Manager known as Rapid Provisioning. The feature isn’t well known for a couple of reasons:

  • It’s not available in the GUI – you can only access it from PowerShell.
  • It’s not referred to as Rapid Provisioning from within PowerShell.

As such, generally the only people using Rapid Provisioning are those who had the feature demo’d to them by someone else, or who stumbled upon the feature while reading a blog post or doing some thorough documentation reading.

So what is it?

In short, Rapid Provisioning allows you to deploy a virtual machine using Virtual Machine Manager with all of the goodness that VMM provides (Hardware Profile, Guest OS Profile, templates), without the need of waiting for Virtual Machine Manager to do a BITS copy of the Virtual Machine from the VMM Library to the Hyper-V host, which can take awhile, and put unnecessary IO loads on your storage and network.

So how does it work?

Basically, you’d do everything identically to the way you normally do when you create a new VM, except you add one cmdlet: Move-VirtualHardDisk. This cmdlet tells VMM to not actually do the file copy, but to instead use the existing destination VHD already on the Hyper-V host. You might be asking, “Why is the cmdlet called Move-VirtualHardDisk, when it’s not moving anything?” That’s a good question, and one that I can’t actually answer. Not everyone names their cmdlets intuitively. A better answer would probably be that it’s meant to signify that you’re moving a new Virtual Hard Disk into your deployment job in place of the blank VHD which was specified in the template. So you aren’t actually moving a VHD across storage, but within the VMM job space.

So how does the VHD actually get to the destination?

That’s up to you. There are a few ways you can accomplish the task. You can:

  • Use a SAN snapshot or similar hardware solutoin to present the VHD to Hyper-V
  • Use something like Virsto to present the VHD to Hyper-V.
  • Manually copy the VHD from local storage to your destination location on Hyper-V (this still has the added benefit of keeping traffic off of the network).

Here’s an example of a deployment script end to end.

function create-newdemoVM
{
param($vmname, $csv, $vmnumber, $CPU, $RAM)
$VMMServer = Get-VMMServer "MYVMMSERVER"
$JobGroupID = [Guid]::NewGuid().ToString()
$Template = Get-Template | where {$_.Name -eq "2K8R2_SP1_GOLD"}
$VMHost = Get-VMHost | where {$_.ComputerName -eq "MY_HYPER-V_SERVER"}
Move-VirtualHardDisk -IDE -BUS 0 -LUN 0 -Path "c:\clusterstorage\$CSV\$vmnumber\AIT_R2_SP1_ENT.vhd" -JobGroup $JobGroupID
New-VirtualNetworkAdapter  -VirtualNetwork "Guest" -VLanEnabled $false -Synthetic -JobGroup $JobGroupID
$VM = New-VM -Name $VMName -Path "c:\clusterstorage\$CSV\$vmnumber" -CPUCount $CPU -MemoryMB $RAM  -Template $Template -VMHost $VMHost -ComputerName $VMName -JobGroup $JobGroupID -UseLocalVirtualHardDisks -RunAsynchronously -JobVariable "NewVMJob" -SkipInstallVirtualizationGuestServices -FullName "Joe User" -OrgName "Acme"
}

Let’s walk this little function one line at a time. First:

param($vmname, $csv, $vmnumber, $CPU, $RAM)

Here, we’re simply telling the function that we’ll be calling five parameters:
* VMname – the name of the VM, both at the Hyper-V/VMM level, and inside the guest OS
* CSV – in this case, we’re using a Cluster Shared Volume, and parameters specifies the name of the CSV from the C:\ClusterStorage path
* VMNumber – in this example, we have a numbered folder within the CSV – for example C:\ClusterStorage\MYCSV1\3\
* CPU – Simply passing the number of CPUs we’d like the VM to have (overriding the template setting)
* RAM – Passing the number of RAM we’d like the VM to have (overriding the temlpate setting)

Next, we set the Virtual Machine Manager server:

$VMMServer = Get-VMMServer "MYVMMSERVER"

Here, we create a new randomly generated GUID with which we can group all of the tasks together.

$JobGroupID = [Guid]::NewGuid().ToString()

The job grabs our existing template from the library where we store our W2K8R2SP1 gold image settings, inluding the Unattend file for the Sysprepped image.

$Template = Get-Template | where {$_.Name -eq "2K8R2_SP1_GOLD"}

In this step, I’m calling a specific server where I want to place the VM. Theoretically, I could use VMM Intelligent Placement here, but for simplicity’s sake, I’m hard coding the destination server.

$VMHost = Get-VMHost | where {$_.ComputerName -eq "MY_HYPER-V_SERVER"}

This line is where the magic happens:

Move-VirtualHardDisk -IDE -BUS 0 -LUN 0 -Path "c:\clusterstorage\$CSV\$vmnumber\AIT_R2_SP1_ENT.vhd" -JobGroup $JobGroupID

I’m specifying that the VHD located on BUS 0 LUN 0 should be mapped the path on the destination server instead of the VHD from the VMM Library.

One last housekeeping item – I need to attach a NIC from my Guest network:

New-VirtualNetworkAdapter  -VirtualNetwork "Guest" -VLanEnabled $false -Synthetic -JobGroup $JobGroupID

And finally, the one line to create the VM itself:

$VM = New-VM -Name $VMName -Path "c:\clusterstorage\$CSV\$vmnumber" -CPUCount $CPU -MemoryMB $RAM  -Template $Template -VMHost $VMHost -ComputerName $VMName -JobGroup $JobGroupID -UseLocalVirtualHardDisks -RunAsynchronously -JobVariable "NewVMJob" -SkipInstallVirtualizationGuestServices -FullName "Joe User" -OrgName "Acme"

So that’s rapid provisioning in a nutshell. If you’re interested in learning more, here’s some more information from TechNet.

Feel free to leave a comment below or hit me via Twitter if you have any questions.

Good luck, and happy virtualizing!

Changing your password in a “Cloud and Devices” world

We have a policy within our department that system adminstrators need to change their password every 90 days. It’s always painful and people avoid it because we have an account lockout policy in our environment where once bad creds are attempted 25 times, AD locks the account, and it invariably ends up in someone getting their account locked repeatedly for a day or two until all of the last places are found with a stale password. I went through the process this week, and wanted to document it for the others in my department, as well as myself the next time I change it.

  1. Close Outlook anywhere it’s open
  2. Close Communicator anywhere it’s open
  3. Unplug power from both my work and home Communicator phones
  4. Log out of all IU RDP connections
  5. Put iPhone, iPad, and other smartphones/devices in Airplane Mode
  6. Verify that any Scheduled Tasks on any PCs using domain creds are disabled
  7. Change Passphrase
  8. Update Mail Password on devices while still in airplane mode (iOS will tell you it can’t verify, but say OK twice and it will let you change it)
  9. Take devices out of airplane mode and verify it’s working
  10. Relaunch Outlook
  11. Relaunch Communicator and login
  12. Plug the Communicator phone power back in and authenticate via PC

The WiFi login will also need updated across all devices, but unfortunately, at least on iOS, those can’t be changed ahead of time that I can tell, so you just allow auth to fail once, and then re-enter creds on iPhone, iPad, et cetera. The key is trying to keep the number of failed logins to a minimum in the process. I still missed a couple things this time around and suffered two to three lockouts before all was said and done, but the process has gotten a bit easier each time I’ve made the update.

My Favorite Things – Capresso CoffeeTEAM Therm

I’ve been meaning for some time to do a regular series of posts about all the things I use regularly, and which I think have been life-changing in some form or another. Today, I begin this series with the all-important way to start the day.

capresso

Capresso CoffeeTEAM Therm Stainless Coffeemaker

Let me start by saying this: I love coffee, and I love gadgets. The Capresso 455 CoffeeTEAM Therm Stainless Coffeemaker combines those two loves to earn one of the top spots on my all-time list of favorite things.

What makes the Capresso different than any other cofeemaker?

The Capresso 455 has few things going for it that make it stand above and beyond all of the cofeemakers I’ve tried historically:

  • A built in bean grinder

  • A stainless steel thermal carafe, so there’s no hot plate required to keeep the coffee hot

  • A programmable interface to allow scheduled grinding/brewing

Conical Burr Grinder

The primary feature that drew me to this coffeemaker was the out of band grinder. With other grind and brew coffeemakers I had seen/tried, the grinder sits directly above the carafe, meaning it’s constantly getting steamed and gummed up, making the whole grind and brew process more trouble than it’s worth. However, the grinder on the Capresso 455 sits off to the side, with a filter bucket on a spring-loaded hinge. With this setup, the grind mechanism doesn’t get steamed, and pretty much never needs cleaned (though I double-check it every few months just to make sure all is well). I’ve done a few quick calculations, and based on the date I bought this unit (January, 2009) and the number of cups I make per day (around 10), I’ve ground about 8,500 cups worth of beans, without a single failure. This is a very set it and forget it device, and that’s refreshing in this world of oft-failing high-tech gadgetry. You simply pour in beans when the beans are getting low, and empty out and rinse the filter basket after brewing.

Stainless Steel Thermal Carafe

I wouldn’t have thought that this was a big deal when I was purchasing it, but having a portable coffee pot has proven to be a very handy feature with this coffeemaker. I usuaully work from home a few days a week, and my home office is a small jaunt from the kitchen, so if I want to bring the carafe with me to the office, I can just carry it along and refill as I get low. Even if you don’t have any need to move the carafe around, having the ability to keep coffee hot all day without even having a hotplate is a great feature. On other coffeemakers, I’ve had people empty the pot and forget to turn the hotplate off, which results in a scorched glass (and sometimes damaged pot). This unit doesn’t even have a hotplate, as it’s not needed. The cofee stays hot in the thermal carafe for many hours.

Programmable interface

The other main feature that sets the Capresso 455 apart from other cofeemakers is it’s easy to use interface. You can program 4, 6, 8, or 10 cups at a time, and you can keep a separate setting for “Grind Now” versus a scheduled grind time. Once you have that set, you simply pour in your water as you’re going to sleep, and press the big “A” (for Auto) button, and then you have a pot of freshly ground and brewed coffee at the ready in the morning.

The one (easily fixable) flaw – a super small bean basket

The one major issue I had with this coffeemaker when I got it is that the bean basket is very small, so if you drink as much coffee as I do, you end up having to fill the basket pretty much every other day. Capresso sells a larger basket (Part 4455.7) , and I’d recommend it to anyone that bought this coffeemaker. Once I put the larger bean container on, I’ve never had an issue with the coffeemaker, other than ID10T errors.

It’s not totally idiot proof

There are a couple gotchas with the device which can still result in a big mess if you’re careless: leaving the lid up, and not having the carafe seated. If you leave the lid up, it will spray water all over the room, and if you don’t have the carafe seated, the auto-stop pour-while-brewing mechanism will cause the water to back up and overflow, resulting in a huge coffee mess all over the counter. However, those two things exist on pretty much every coffeemaker ever built, so I’m not counting that as a flaw of the product.

All in all, I feel this is probably one of the best investments I’ve made in kitchen. Sure, $168 (my original refurb unit’s purchase price) is a lot to spend on a coffeemaker, but it’s paid for itself several times over in the amount of use it’s received.

So where can I get one?

For some reason, Capresso doesn’t actively advertise this coffeemaker any more, so it’s hard to find on their website without a direct link, athough I’m not sure why. The direct link on their website has a purchase link at the bottom for $200, but you can also find refurb units quite a bit cheaper at Amazon ($119 when I last checked).

If you decide to buy one, or already have one, let me know what you think!

Janssen

Hyper-V FAQs, Tips, and Tricks – C-States

During my discussion sessions at TechEd 2011 this year (VIR471-INT – Hyper-V FAQs, Tips, and Tricks), one of the first things we discussed was the topic of C-States. I’ve been recommending people disable C-States ever since we started seeing the technology on our systems (at both the desktop and server level), but it’s taken some time for a lot of information to be publicly available outside of the hearsay realm. Over the past year or two, some information has finally begun to accumulate on the web, much of which now shows up in KB format from Microsoft. Before we go any further, let’s first stop and discuss the first question:

So what are C-States?

In short, C-States are power saving states that your CPU can enter into to save electricity (and hopefully some money, if you pay for your own electricity).

Hey, that sounds great! I love money! Why would I disable them then?

In short, because while C-States sound great in theory, they don’t always work as great in practice. On almost any system we’ve every deployed, both at the desktop and server level, it’s just a matter of time before a system with C-States begins to manifest problems, ranging from performance issues to bug checks. Without going into too much more detail here, the gist of it is that once the processor starts to enter deeper states of sleep (like C3), it doesn’t wake up as quickly as it should, and then things start to go wrong. For more information about C-States in general, you can check out this article from Hardware Secrets, but if you just want to know where the problems lie, you can skip that and read on in this post.

OK, you’ve got my attention. So tell me about what can go wrong.

The first major sign of something going wrong with C-States appeared right after Windows 2008 R2 shipped in September 2009, when lots of people started seeing their Hyper-V servers crash after enabling the Hyper-V role when C-States were enabled. Microsoft quickly released KB974598 – “You receive a “Stop 0x0000007E” error on the first restart after you enable Hyper-V on a Windows Server 2008 R2-based computer” for this particular issue.

However, within a couple days, a bigger issue began to emerge with early adopters, which was that Hyper-V systems seemed to randomly, intermittently, but somewhat regularly crash on any system with a Nehalem Processor. Microsoft responded with a patch for this particular issue by mid-October 2009: KB975530 – Stop error message on a Windows Server 2008 R2-based computer that has the Hyper-V role installed and that uses one or more Intel CPUs that are code-named Nehalem: “0x00000101 – CLOCK_WATCHDOG_TIMEOUT”

At this point, many had already decided to just leave C-States disabled, but for those that decided to keep them enabled with the patches, people still noticed intermittent performance issues, such as the two KBs below:

KB2532917 – Hyper-V Virtual Machines Exhibit Slow Startup and Shutdown

KB2000977 – Hyper-V: Performance decrease in VMs on Intel Xeon 5500 (Nehalem) systems

OK, OK, I get your point. So is this issue unique to Hyper-V?

C-States problems are definitely not limited to Hyper-V. This just happened to be a Hyper-V centered discussion. Do a couple web searches on C-States, and you’ll see that people have reported performance problems with C-States using VMWare and OpenSolaris (search on C-States, and you’ll find where they finally pointed their fingers at C-States being the root cause), and IP over Infiniband.

So… Now that you’re armed with the links, you can make the call on your servers, but we’re going to continue disabling C-States for the foreseeable future.

Good luck, and happy virtualizing.

I’m a Mac. And a PC. And a Kindle. And a Chromebook. And a…

I’ve used a PC since Windows 3.0, and up until 3 years ago, the only non-Windows computer I ever owned was a Commodore 64.  In the past 15 years, I’ve probably been through 15 PCs. From a Packard Bell 486DX250 (which I mistakenly put Windows 95 on) up to the Core i7 machines I have now, I’ve had plenty of PCs.

That being said, we live in a new world, and Microsoft doesn’t dominate the industry like it once did. Over the past 10 years, they’ve pretty much missed the boat on home entertainment and other consumer devices.  For me, it all started with a long series of sub-par Windows Mobile devices, which were painful to use and which generally left me pretty frustrated.  

Enter the iPhone. After ridiculing this product for a year or two without understanding it, I decided to give it a try to see what the fuss was about.  Holy cow.  I could do things on this phone I never dreamed possible on a Windows Mobile Phone.  Then came the iPad.  Then the AppleTV2.  Then the Macbook Air. Let’s face it.  Apple has been hitting these out of the park from a hardware and software perspective for the past five years in a way that Microsoft has not been able to do (partly because they don’t control or ship the hardware, and partly because they’ve done a few resets in the past few years).

Fast forward to TechEd 2011.  I was previously using a 7 pound Dell Latitude when travelling.  The laptop is a solid laptop (in more ways than one), but after carrying it on your back for several days straight at a conference, you’re left begging for mercy.  I decided to opt for the Macbook Air on this trip instead.  It weighs under 3 pounds and has all-day battery life.  In general, it’s a better “conference laptop” all the way around.

Taking a Macbook to a Microsoft event generally causes grief.  People come up to you and and deride your decision to bring an Apple product to a Microsoft event.  Even though it can run Windows, and even though Microsoft doesn’t make hardware, people still complain.  I think it’s primarily a deep festering wound left over from all those years of the “I’m a Mac.  I’m a PC.” commercials.  Those commercials always upset me when I saw them because they were very misleading, and often untrue.  In fact, it was probably those commercials that kept me from buying an iPhone for as long as I did.  

That being said, Apple realized at one point that there’s a lot of uncharted territory in the devices market, and it’s time was not best spent by fighting with Microsoft, but by innovating in totally different spaces.  They let the Windows bashing commercials cease, and they focused on shipping products that delight customers.  On that front, they’ve succeeded.

Technology is a very large space to play in, from the cloud, to business devices, to gaming, to phones, to tablets.  In this space, there are a lot of big players, from Google to Amazon to Sony to Apple.  I believe that Microsoft needs to work on how they can best integrate with all of these companies, and leave the partisan bickering of yesteryear behind.  To that end, they’ve made some strides.  One can now find Bing and Messenger on the iPhone, and Microsoft has publicly stated that there will be more on the way. All I can say is “bring on the cooperation”.  

I use a lot of tech.  I use Macs, PCs, iPhones, iPads, Kindles, ChromeOS netbooks, and more.  All I generally  want is to get a job done (read an article, edit a video, play a game, listen to a song).  The more these companies can work together on open standards, cross-platform support, compatible media, et cetera, the more customers, and ultimately the companies themselves, will succeed.

Here’s hoping that 2011 will see more Microsoft apps come to the iPhone, more Apple apps (Facetime?) come to Windows, and more cross-platform support of all software.  We’re all growing up now.  Let’s all leave the fanboy bickering behind and focus on shipping products that delight consumers.

TechEd North America 2011

I recently returned from TechEd North America 2011, where I had the pleasure of presenting three sessions this year – two on Virtualization FAQs, Tips, and Tricks, and one on Fluid Data Management at Indiana University. Unfortunately, the two Virtualizaiton FAQs sessions weren’t recorded, but luckily, we took notes during the sessions, and will be posting up many of the questions and answers over the coming week or so.  I was fortunate to have three other Microsoft MVPs join me on stage for the Q&A:

Nathan Lasnoski (Virtual Machine MVP)
William Bressette (Clustering MVP)
Annur Sumar  (Clustering MVP)

Nathan posted up a round of the Q&A from the first session here (Thanks Nathan!):
http://blog.concurrency.com/infrastructure/virtualization-faq-1-presenting-at-vir471-int/

I’ll follow up with another round from the Q&A, as well as post some more deep dives, in the near future.