As a Gold Partner, the first commercial defragmenter designed for WHS, and even one of the first 8 products to declare support for WHS when Bill Gates first announced it at CES several years back, we’ve been real close with the development team at MS.
The MS team’s insight into the needs and capabilities of defrag has led to significant cooperation as they complete the new Drive Extender. We’ve had many calls over the past months and are very happy to say that many of our suggestions and requests have been accommodated. The functionality that has been added from those meetings will result in well integrated and functional third party software solutions.
Supporting Drive Extender 2.0…
Vail, which is in public beta, has already been generating a great deal of buzz on WHS forums. Drive Extender (DE) is a storage subsystem that extends storage functionality above and beyond what a typical Windows NTFS volume offers. Key purposes of DE are to offer fully automated and easy to use storage. All the complexities associated with fault tolerance solutions like RAID to provide drive reliability, expanding storage over time, and even solving data reliability concerns of commodity drives.
Drive Extender in Windows Home Server today implements drives independently and pools them into a common volume. This pool of storage is then presented as a single volume to the user (i.e. D:\). And, just off the root of this pool (D:\shares) you had all your WHS shared folders; e.g. Users, Photos, Videos, etc…
The user experience of WHS today is already such that you don’t need to care or interact with the volumes, you could even argue that its discouraged.
What is unique in DE 2.0 is that this paradigm has kind of been flip flopped. While it all still looks like a common repository the delineation of storage now begins with those shared folders. So, as an example, let’s take the shared folder “Photos”. In DE 2.0 that folder now becomes a dedicated NTFS volume presented out of the shared storage pool. The folder “Videos” becomes its own NTFS volume, and so on. This design was introduced to support features like real time folder level duplication, etc.. The only minor side effect is that because there are only 26 letters in the English Alphabet, there will be a limitation of the shared folders you can create in this location on WHS. Not a big deal, given the value of the features that this new design offers.
If you’ve read this blog and the comments, you’ll pick up that DE is extending a volume (i.e. a shared folder), in 1 GB chunks, taking those chunks from the total available storage pool.
What you effectively have with Drive Extender, then and now, is storage virtualization. Any time you pool storage and then divvy it up exclusively to requestors (in this case the shared folders that become lettered volumes) you need some form of logic for allocating data from the pool. SAN and virtualization administrators already understand this concept, including related technologies such as Thin Provisioning.
DE 2.0 now adds this to their storage virtualization solution. As you add more data to a shared folder, DE 2.0 will allocate, in 1 GB chunks, more space to the shared folder/volume from the common storage pool. And, should you delete ALL the data in a 1 GB chunk, the 1 GB chunk will dynamically return to the available storage pool to be allocated to any other shared folder that may need the space. DE is well designed to fill up 1 GB chunk before requesting to use more. Very cool stuff!
Here’s a demo of how the provisioning works. Assuming 20GB of space (divided into those 1GB chunks)
You now start to fill up storage adding a little over 4GB of photos and a little over 2GB of music files. That has now pulled eight 1GB chunks from the common pool and these volumes have dynamically expanded to hold up to 5 GB and 3 GB respectively. Keep in mind that files place in the Photos folder will NOT reside on the same 1 GB chunk that contains Music files. In this case, under the DE “covers” they are on completely separate Windows volumes.
And, as those eight 1 GB chunks are provisioned to shared folder volumes, the storage pool shrinks by 8GB.
Now… If you delete all the MP3 music files that reside in one of those 1GB chunks…
DE can return that chunk back to the storage pool for re-provisioning re-use with any other shared folder.
Subsequently shrinking the Music folder/volume to two 1 GB chunks:
However, there are some conditions in which this provisioning technique can use some assistance, and Diskeeper will be helping out (per the request of the WHS/DE team at MS). Should you delete some of the data from 1 GB chunks, but not all of the data within a 1 GB chunk, you can have a lot of 1 GB chunks allocated to a shared folder/volume, but not actually using all of the space it is taking from the common storage pool. Diskeeper will be helping in these cases to group together all the data spread across sparsely filled 1 GB chunks. We’ll effectively be squishing the data together aligning it along 1 GB boundaries. The benefit of this is that some 1 GB chunks may then be freed up and returned to the storage pool to be assigned to your other shared folders.
Here’s a quick graphic to explain the process. Five 1 GB chunks are taken up by the Photos folder/volume. Over time, unwanted photos may be deleted, but the space they were taking up is not made available to any folder other than Photos. In order to make the space (3GB in this example) available for Videos or Music, you would need to move the data out of the sparsely filled chunks. Once done, those now empty 1 GB chunks can be used elsewhere.
Is this an issue you’ll come across? Maybe. If you do, it’ll likely be a bit of time and a lot of file deletions down the road. The Microsoft provisioning design is well suited to most users who mainly add and retain data. Those who do housecleaning or more involved data management can benefit from the upcoming Diskeeper solution – stay tuned.
While we’d like to take credit for this new feature in a future release of Diskeeper HomeServer, it was quite frankly MS directly asking us, as a partner, to add this into our product. Perhaps they’ll add this data squishing into WHS down the road? In the mean time, you can look to Diskeeper to help.
PS: thanks to the Microsoft WHS team for reviewing and approving this blog post.