Powershell scripts for working with NTFS virtual disks

Accessing data within VMDK files can be challenging from a Windows system especially when you consider a VMDK on a remote datastore.  Luckily there are some real gurus out there that have written code that we can leverage to get that job done.  This means no mounting of a separate virtual disc to an existing VM, only simple commands from a Windows system necessary.  This post is going to demonstrate some custom scripts leveraging existing open source dotNet assemblies as well as use cases for when they can be used.  A unique distinction here is that we are working from the bottom up, ie. NFS/local -> VMDK -> NTFS -> C:\ instead of top down OS -> C:\ which means that we can do these processes without the virtual machine powered on.

The Underlying dotNet Assemblies and Executables (Open Source MIT License)

DiscUtils http://discutils.codeplex.com/

It is important to mention that there are executable tools that do some of this magic.  The scripts are augmentations to the existing capabilities as listed below.  An unstable branch of the code is leveraged here which has updates for NFS paths and the handling of exports.  The stable version will be replaced once released from DiscUtils project.

Accessing VMDKs

A great feature is that we can run the scripts against a remote NFS datastore and preservation of sparsenesss (thin from vmdk to vmdk).  See below examples of an NFS uri “nfs://10.10.10.10/share1//winxp/winxp.vmdk”.  See how “share1//winxp” the double slashes represent the mount point and starting directory for NFS.  An alternative to this would be to copy the vmdk to the local system and then run the tool against a dense VMDK (much longer than doing it against an NFS datastore via this tool).

Extracting and Viewing Information from NTFS VMDK files

ps_discutils_vmdk_ntfs_get_files.ps1

This script will walk a VMDK file formatted as NTFS and return all directories, their accumulated sizes, detailed file information, and allow for the return of said files.  There are three commands “showdirs,showfiles,copyfile”.  A future command to be included is something like injectfile hint hint.

NTFS Partition Alignment

ps_discutils_vmdk_ntfs_check_alignment.ps1

This script will attempt to open a specified VMDK file and return the NTFS alignment characteristics.  The VMDK can either be local or on an NFS share that the system running the script has root/rw access to.

ps_discutils_vmdk_ntfs_fix_alignment.ps1 (very experimental)

This script will create a new VMDK in specified location and copy contents of original VMDK while aligning it to a proper partition start point (2MB default).  When the script is done you will have a new VMDK (.vmdk and –flat.vmdk).  The current version of the script is expecting at this point that you edit the virtual machine settings and add the realigned virtual disk (destvmdk) and remove the (srcvmdk).

Partition alignment has always been a critical issue and the effects of misalignment can differ among vendors.  As a best practice it is always recommended that you properly align partitions per your vendors specifications as part of a gold image that gets redeployed.  This can be done in many different ways, however it is common that you find partitions that are misaligned.  This is where these scripts come into play.  Traditionally you would mount a virtual disk to another host, or other methods.  The scripts outlined below will show how we can leverage a remote NFS datastore to do the offline checks of disk alignment and realignment operations.

See screenshots attached below.

Overall Summary

My approach here is to leverage Powershell to reproduce and enhance some really cool tools from the dotNet community (DiscUtils) for the Posh community which can manage partitions and file systems within virtual disks.  There are examples of the DiscUtils programs that are runnable out of the box that allow you to do certain things like file extraction, online operating system cloning, disk cloning, etc.

Now the cool stuff begins.  I took the executable DiscUtils examples and rewrote some of the functionality in Powershell and leveraged the DiscUtils assemblies to get the job done.  So for example, there are tools for DiscUtils for OScloning, however it does not include a realignment portion or choice of buffer size for doing a copy.  The scripts below will do most of what OSclone does (create a new vmdk and copy all files inside existing to new vmdk) as well as align it properly to specified partition start point (2MB).

The scripts are absolutely the first run and should serve as an instigator for leveraging Posh for a bottom up approach to managing VMDK and NTFS partitions.  Please do not use them in anything but a LAB!

A big bonus of rewriting in Powershell is the ability to tweak and modify per your needs.  I hope the community takes the work done here to the next level!

Existing Online Methods

There are existing methods for checking alignment using PowerCLI and powershell.  However, these methods expect that the VM is running and queries WMI and other methods to get this information.  These scripts and DiscUtils is different as it actually goes from the bottom up and opens the VMDK file to get the appropriate information.  See below for existing methods for checking alignment.

http://ict-freak.nl/2009/12/15/powercli-check-partition-alignment-windows-vms-only/

The Underlying dotNet Assemblies

DiscUtils http://discutils.codeplex.com/

Copyright (c) 2008 Kenneth Bell

(see LICENSE.txt) in zip

We are currently using a non-stable customized build of the DiscUtils binaries.  There were modifications to make NFS work without authentication and other minor changes.

Questions

Why does it take longer than a standard disk clone?

In order to re-align the partition we need to create a new virtual disk per the geometry needs and then do a file system copy of all files one at a time to the new virtual disk.  You can increase the buffer size (- bytesize 65536, default is 4096) which might speed the operation.

MD5: EE7E776048E06BBA4352B54F82DF44C7