Virtualization FAQs, Tips, and Tricks - Virtualizing USB dongles

If you've been visiting my site lately, you'll notice that I've been regularly posting a series of Virtualization FAQs, Tips, and Tricks as a follow-up to the TechEd 2011 VIR471-INT session.

Today's installment deals with how you can virtualize a workload which requires access to a USB key, such as a software licensing dongle.

Thanks go to Aaron Sudduth for discovering this one. Aaron is the sysadmin on my team that discovered this solution, and he's the one that did our initial installation and troubleshooting.

As discussed in the session, using software that requires a specialized USB license dongle isn't actually a blocker for a workload that you'd like to virtualize. We've spent the last couple years virtualizing such workloads without issue.

The trick to making this work is to use USB over IP with a network-attached USB hub like the ones found here. The specific model we've been using is the LTB AnywhereUSB/5 Gen 1. It appears that this product is being discontinued and replaced with a much more expensive model for about twice the price (though it doesn't appear to do anything differently that I can see), so we're considering buying a couple to stock up before they totally disappear.

There are a lot of other models out there, but since we haven't tested them, I can only say that I know this one works for anything we've tested.

How it works

The process is actually pretty straightforward. You hook this box up to your network and assign it an IP, and then you plug in your USB key. At that point, you just have to install a piece of software inside your VM, and it will connect to this box over the network and bring the USB key into Device Manager just like it was running locally. So long as you have network connectivity, you'll continue to have access to the dongle.

Some gotchas to be aware of

First, it's important to note that not all devices will work this way, but we've not found that didn't. It's probably worth testing if you're not sure.

Second, this device comes up a little later in the stack, once network connectivity has been established. It's possible that you have a service starting up which requires this dongle that may try to start before this device comes online. In that case, you need to setup your service to have a delayed start in order for this to all fall into place.

Third, Windows VMs don't have the necessary drivers to use this out of the box (since VMs don't have USB controllers) unless the VM was originally P2V'd, so you actually have to add those drivers before this will work.

Here are the notes from a Digi guide on how to make it work

The following workaround is required for installing an AnywhereUSB in a Virtual Machine:

1. Make sure Windows is configured to show file extensions. In My Computer, click "Tools / Folder Options" then click the "View" tab. Scroll down and uncheck "Hide extensions for known file types" (unless it's already unchecked) then click OK.

2. Search the Windows CD (that matches the OS of the Host PC) for a file called "usbd.sy_" (it should be in the i386 folder). For example, if the Host PC is running Windows 2003, then use a Windows 2003 CD.

3. Copy "usbd.sys_" and paste it in "c:\windows\system32\drivers" (or c:\winnt\system32\drivers for Windows 2000), then rename the file to "usbd.sys". Be sure to paste it in the "drivers" folder, not just "system32".

4. Open the AnywhereUSB Configuration Utility, click "View / Driver Information / Uninstall" and reboot the Virtual Machine when prompted.

5. After Windows loads, re-install the AnywhereUSB software and follow the normal installation procedure - you should go through a total of 5 found new hardware wizards, and the 5 port LEDs should be solid green once the AnywhereUSB is installed and connected to the Host PC. 

Some additional support documentation from Digi can be found here

In summary, if you come across a workload that you'd like to virtualize, but don't think you can because of a software dongle, check out this solution as a way to get that workload off of metal and into a VM!

If you have any questions, feel free to contact me via the links to the right.

Good luck, and happy virtualizing!

Hyper-V FAQs, Tips, and Tricks - Post SP1 hotfixes

As promised, here's the next post in the Hyper-V FAQs, Tips, and Tricks Series, following up from the TechEd 2011 VIR471-INT session. Today's post discusses the most important hotfixes you need to make sure you have installed in your Hyper-V environment.

If you're running W2K8R2SP1, there aren't too many fixes to be aware of just yet, but there are a few fairly important ones:

The critical ones (in my opinion)

• KB2263829 - Guest Networking lost under load
• KB2531907 - The Validate hotfix

The important ones

• KB2494016 - Stop error 0x0000007a in guest on failed CSV Redirection
• KB2496089 - VMMS stops responding intermittently on service restart
• KB2521348 - VM online backup fails when SAN policy is set to "Offline All"
• KB2519946 - Timeout Detection and Recovery (TDR) randomly occurs in a virtual machine that uses the RemoteFX feature in Windows Server 2008 R2

Time is short today, so I won't go into details on the bugs, but you can follow the links and find out if they apply to you.

Good luck, and happy virtualizing.

Hyper-V FAQs, Tips, and Tricks - Checkpoints and Snapshots

One of the topics at our Hyper-V FAQs, Tips, and Tricks discussion at TechEd was that of checkpoints and snapshots, and whether they should be used in production or not, as well as the gotchas that one may come across when using them.

Snapshots versus Checkpoints

First, I have to mention that snapshots and checkpoints are the same thing. Hyper-V calls them snapshots, and System Center Virtual Machine Manager calls them checkpoints. When you make a checkpoint in VMM, it actually creates a snapshot in Hyper-V. When you create a snapshot from Hyper-V, you'll see it in the VMM console as a checkpoint. With that out of the way, let's discuss what they are for, and when you night use one.

What are snapshots and when/why should I use them?

Snapshots are representations of a virtual machine at the point of time from which the snapshot was taken. They are most useful when you're about to make a change which you might want to quickly roll back. You can take a snapshot, make your change, and then revert back to your previous state. One of the most common scenarios is in doing software sequencing with something like App-V. You can take a snapshot of your VM, sequence the installation of your software, copy the results off of the machine, and then roll back. However, snapshots are not to be used for backup purposes. Snapshots don't behave the same way a backup would behave, and traditional backups may likely need to be used, depending on your scenario (either SAN level or from something like System Center Data Protection Manager).

When/why should I not use them? What are the "gotchas"?

Though snapshots are a valuable tool, there a few things to be aware before you ever start using snapshots. Knowing these few things will save you a lot of grief and wasted time in the future:

• Machine rollbacks can cause Active Directory connection problems
• Leaving machines running for an extended period of time after a snapshot can use up significant disk space
• Merging snapshots into existing VM can take many hours, during which the VM has to remain offline to complete the merge

Rollback to previous snapshot can cause Active Directory Password problems

The first thing to be aware of is that computers have a unique machine account password within Active Directory that by default changes every 30 days for security reasons. If the password changes between the time you take the snapshot and the time you want to rollback, you'll break the computer's access to Active Directory when you revert to the snapshot. This will in turn cause you to lose the ability to login to your VM with any domain credentials unless they were previously cached. Once you find yourself in this state, you need to login via a local account or account with cached credentials and unjoin/rejoin the computer from the domain.

To prevent this automatic password change from occurring, you can disable the automatic password reset by following the steps within Microsoft KB 154501 (thanks to Aaron Sudduth for the live link during the session). Once you flip the registry key, you can safely revert without having to worry about whether the machine is in sync with AD, since it will permanently keep the same machine account password. (This would apply to a traditional backup as well - anything which brings the VM back to a previous point in time.)

Snapshots, AVHD, and disk space

Once you take a snapshot of your VM, all of the VMs future writes will be written to a differencing disk (AVHD) rather than your original VHD. This means that over time, the AVHD will continue to grow indefinitely, even if space inside the VM isn't growing, but is merely changing alot. If you're leaving this machine up for years, it's only a matter of time before you eventually run out disk space. At that point, you have to make the decision to power down the VM and merge the snapshot into the current VM, which brings me to my final point.

Merging snapshots can mean hours of downtime

In the current version of Hyper-V, the only way you can merge the AVHD back into a VHD is by deleting your original snapshot and powering off the VM, and then waiting for Hyper-V to merge the files. Depending on the speed of your storage, and size of the AVHD, this can take minutes, hours, or days. In general, you don't have days of time that you can wait for AVHDs to merge, which means you don't want to get yourself into the situation to begin with.

Conculsion

So the moral of the story is: feel free to use snapshots and checkpoints, but only for VMs that need to be rolled back regularly, and which won't be left on for really long periods of time after the snapshot is taken. If you actually want to keep a pristine copy of your VM for months or years, use a traditional backup instead. You can do a SAN-level snapshot, a DPM backup, or a simple file copy of your VHD to external storage.

Good luck, and happy virtualizing!

Rapid Provisioning with System Center Virtual Machine Manager

During my presentations at TechEd this year, I spoke about a somewhat rarely used and somewhat unknown feature of System Center Virtual Machine Manager known as Rapid Provisioning. The feature isn't well known for a couple of reasons:

• It’s not available in the GUI – you can only access it from PowerShell.
• It’s not referred to as Rapid Provisioning from within PowerShell.

As such, generally the only people using Rapid Provisioning are those who had the feature demo’d to them by someone else, or who stumbled upon the feature while reading a blog post or doing some thorough documentation reading.

So what is it?

In short, Rapid Provisioning allows you to deploy a virtual machine using Virtual Machine Manager with all of the goodness that VMM provides (Hardware Profile, Guest OS Profile, templates), without the need of waiting for Virtual Machine Manager to do a BITS copy of the Virtual Machine from the VMM Library to the Hyper-V host, which can take awhile, and put unnecessary IO loads on your storage and network.

So how does it work?

Basically, you'd do everything identically to the way you normally do when you create a new VM, except you add one cmdlet: Move-VirtualHardDisk. This cmdlet tells VMM to not actually do the file copy, but to instead use the existing destination VHD already on the Hyper-V host. You might be asking, "Why is the cmdlet called Move-VirtualHardDisk, when it's not moving anything?" That's a good question, and one that I can't actually answer. Not everyone names their cmdlets intuitively. A better answer would probably be that it's meant to signify that you're moving a new Virtual Hard Disk into your deployment job in place of the blank VHD which was specified in the template. So you aren't actually moving a VHD across storage, but within the VMM job space.

So how does the VHD actually get to the destination?

That's up to you. There are a few ways you can accomplish the task. You can:

• Use a SAN snapshot or similar hardware solutoin to present the VHD to Hyper-V
• Use something like Virsto to present the VHD to Hyper-V.
• Manually copy the VHD from local storage to your destination location on Hyper-V (this still has the added benefit of keeping traffic off of the network).

Here's an example of a deployment script end to end.

function create-newdemoVM
{
param($vmname, $csv, $vmnumber, $CPU, $RAM)
$VMMServer = Get-VMMServer "MYVMMSERVER"
$JobGroupID = [Guid]::NewGuid().ToString()
$Template = Get-Template | where {$_.Name -eq "2K8R2_SP1_GOLD"}
$VMHost = Get-VMHost | where {$_.ComputerName -eq "MY_HYPER-V_SERVER"}
Move-VirtualHardDisk -IDE -BUS 0 -LUN 0 -Path "c:\clusterstorage\$CSV\$vmnumber\AIT_R2_SP1_ENT.vhd" -JobGroup $JobGroupID
New-VirtualNetworkAdapter  -VirtualNetwork "Guest" -VLanEnabled $false -Synthetic -JobGroup $JobGroupID
$VM = New-VM -Name $VMName -Path "c:\clusterstorage\$CSV\$vmnumber" -CPUCount $CPU -MemoryMB $RAM  -Template $Template -VMHost $VMHost -ComputerName $VMName -JobGroup $JobGroupID -UseLocalVirtualHardDisks -RunAsynchronously -JobVariable "NewVMJob" -SkipInstallVirtualizationGuestServices -FullName "Joe User" -OrgName "Acme" 
}

Let's walk this little function one line at a time. First:

param($vmname, $csv, $vmnumber, $CPU, $RAM)

Here, we're simply telling the function that we'll be calling five parameters: * VMname - the name of the VM, both at the Hyper-V/VMM level, and inside the guest OS * CSV - in this case, we're using a Cluster Shared Volume, and parameters specifies the name of the CSV from the C:\ClusterStorage path * VMNumber - in this example, we have a numbered folder within the CSV - for example C:\ClusterStorage\MYCSV1\3\ * CPU - Simply passing the number of CPUs we'd like the VM to have (overriding the template setting) * RAM - Passing the number of RAM we'd like the VM to have (overriding the temlpate setting)

Next, we set the Virtual Machine Manager server:

$VMMServer = Get-VMMServer "MYVMMSERVER"

Here, we create a new randomly generated GUID with which we can group all of the tasks together.

$JobGroupID = [Guid]::NewGuid().ToString()

The job grabs our existing template from the library where we store our W2K8R2SP1 gold image settings, inluding the Unattend file for the Sysprepped image.

$Template = Get-Template | where {$_.Name -eq "2K8R2_SP1_GOLD"}

In this step, I'm calling a specific server where I want to place the VM. Theoretically, I could use VMM Intelligent Placement here, but for simplicity's sake, I'm hard coding the destination server.

$VMHost = Get-VMHost | where {$_.ComputerName -eq "MY_HYPER-V_SERVER"}

This line is where the magic happens:

Move-VirtualHardDisk -IDE -BUS 0 -LUN 0 -Path "c:\clusterstorage\$CSV\$vmnumber\AIT_R2_SP1_ENT.vhd" -JobGroup $JobGroupID

I'm specifying that the VHD located on BUS 0 LUN 0 should be mapped the path on the destination server instead of the VHD from the VMM Library.

One last housekeeping item - I need to attach a NIC from my Guest network:

New-VirtualNetworkAdapter  -VirtualNetwork "Guest" -VLanEnabled $false -Synthetic -JobGroup $JobGroupID

And finally, the one line to create the VM itself:

$VM = New-VM -Name $VMName -Path "c:\clusterstorage\$CSV\$vmnumber" -CPUCount $CPU -MemoryMB $RAM  -Template $Template -VMHost $VMHost -ComputerName $VMName -JobGroup $JobGroupID -UseLocalVirtualHardDisks -RunAsynchronously -JobVariable "NewVMJob" -SkipInstallVirtualizationGuestServices -FullName "Joe User" -OrgName "Acme" 

So that's rapid provisioning in a nutshell. If you're interested in learning more, here's some more information from TechNet.

Feel free to leave a comment below or hit me via Twitter if you have any questions.

Good luck, and happy virtualizing!

Hyper-V FAQs, Tips, and Tricks - C-States

During my discussion sessions at TechEd 2011 this year (VIR471-INT - Hyper-V FAQs, Tips, and Tricks), one of the first things we discussed was the topic of C-States. I've been recommending people disable C-States ever since we started seeing the technology on our systems (at both the desktop and server level), but it's taken some time for a lot of information to be publicly available outside of the hearsay realm. Over the past year or two, some information has finally begun to accumulate on the web, much of which now shows up in KB format from Microsoft. Before we go any further, let's first stop and discuss the first question:

So what are C-States?

In short, C-States are power saving states that your CPU can enter into to save electricity (and hopefully some money, if you pay for your own electricity).

Hey, that sounds great! I love money! Why would I disable them then?

In short, because while C-States sound great in theory, they don't always work as great in practice. On almost any system we've every deployed, both at the desktop and server level, it's just a matter of time before a system with C-States begins to manifest problems, ranging from performance issues to bug checks. Without going into too much more detail here, the gist of it is that once the processor starts to enter deeper states of sleep (like C3), it doesn't wake up as quickly as it should, and then things start to go wrong. For more information about C-States in general, you can check out this article from Hardware Secrets, but if you just want to know where the problems lie, you can skip that and read on in this post.

OK, you've got my attention. So tell me about what can go wrong.

The first major sign of something going wrong with C-States appeared right after Windows 2008 R2 shipped in September 2009, when lots of people started seeing their Hyper-V servers crash after enabling the Hyper-V role when C-States were enabled. Microsoft quickly released KB974598 - “You receive a "Stop 0x0000007E" error on the first restart after you enable Hyper-V on a Windows Server 2008 R2-based computer” for this particular issue.

However, within a couple days, a bigger issue began to emerge with early adopters, which was that Hyper-V systems seemed to randomly, intermittently, but somewhat regularly crash on any system with a Nehalem Processor. Microsoft responded with a patch for this particular issue by mid-October 2009: KB975530 - Stop error message on a Windows Server 2008 R2-based computer that has the Hyper-V role installed and that uses one or more Intel CPUs that are code-named Nehalem: "0x00000101 - CLOCK_WATCHDOG_TIMEOUT"

At this point, many had already decided to just leave C-States disabled, but for those that decided to keep them enabled with the patches, people still noticed intermittent performance issues, such as the two KBs below:

KB2532917 - Hyper-V Virtual Machines Exhibit Slow Startup and Shutdown

KB2000977 - Hyper-V: Performance decrease in VMs on Intel Xeon 5500 (Nehalem) systems

OK, OK, I get your point. So is this issue unique to Hyper-V?

C-States problems are definitely not limited to Hyper-V. This just happened to be a Hyper-V centered discussion. Do a couple web searches on C-States, and you'll see that people have reported performance problems with C-States using VMWare and OpenSolaris (search on C-States, and you'll find where they finally pointed their fingers at C-States being the root cause), and IP over Infiniband.

So... Now that you're armed with the links, you can make the call on your servers, but we're going to continue disabling C-States for the foreseeable future.

Good luck, and happy virtualizing.