I recently came across an older article about defragmenting SANs (read it here). It includes interviews with analysts, SAN vendors (some pro-defrag, some against), and an employee from Diskeeper Corporation.

I was particulary impressed with the EMC'ers response:

"The SAN can't do anything about the fact that Windows sees the file in 30 bits," said Wambach. "That's really something that is happening outside of the storage realm."

He highlights the abstraction perfectly.  SAN vendors claim that a defragmenter cannot correct fragmentation due to the fact it is abstracted from the physical blocks. We absolutely agree with this statement. And for that same reason, SANs cannot fix fragmentation in the NTFS file system, which causes excess and unnecessary overhead on the OS.

 

I had a test run here internally in order to make a point about what does, or more specifically "does not", happen when you VMotion/SVMotion a Windows Guest OS (GOS). We wanted to demonstrate that, while VMware is copying the VM to another host/storage, it does nothing about the internal fragmentation of files in Windows.

We felt this was a valuable demonstration as one of the old (1980s) ways to "fix" fragmentation was to copy off the files/backup, reformat the volume, and then copy back/restore. This offered a degree of success, but required taking the data offline in order to get rid of most of the fragmentation. On a side note, backing up/copying fragmented files takes a lot longer than it would on contiguous and ordered files.

Anyway, S/VMotion is such a cool feature because it works on live VMs. So, if the VMDK movement somehow did align/reorder files in Windows, it could be a great solution to Windows file system fragmentation! So here's how we tested...

1. Setup 2 ESX 4.1 Servers with iSCSI storage and vCenter with SVMotion capability.

2. Create a VM with Windows 7 in one of the ESX Server storage (Ex: Storage1) and a 20 GB Thin virtual disk.

3. Using an internal tool, create moderate fragmentation on the virtual disk (80k fragments, average fragments per file around 3.0, around 50% free space).

4. Install V-locity with all features (e.g. defrag, IntelliWrite, etc...) disabled. This is just so we can run a fragmentation analysis and save the reports.

5. Save the "Before SVMotion" analysis report, and then stop V-locity Windows Service (to make sure it is entirely inactive).

6. Using SVMotion move the live VM to the other ESX Server storage (Ex: Storage2).

7. Once the move is completed, restart the V-locity Windows Service and perform a post "After SVMotion" analysis.

8. Save this job report.

We saw what we expected, given VMotion leverages Changed Block Tracking (CBT) technology and is block, not file based. I attached the report, so you can see the side-by-side analysis data, files in Windows are not defragmented in an SVMotion. Now, that's not to say possible fragmentation of the VMDK files themsleves (on VMFS datastores) was not affected, but that's a topic for another post. 

A new white paper, The Importance of Defragmentation in Virtual Environments, co-authored by Osterman Research and Diskeeper Corporation, demonstrates that virtual environments require defragmenting even more than physical environments. This is due to the fact that virtual environments support multiple operating systems and create a higher intensity of disk activity. 

“The need for defragmentation is even more acute in virtual environments,” the white paper states. “This is because physical hardware in a virtualized storage environment must support more operating systems and so can undergo even more disk access and more stress than in a non-virtualized environment. Further, disk I/O in one virtual machine has a cascading effect on disk I/O in other virtual machines, and so the problem of excessive disk I/O in virtual machines is, in fact, even worse than what would be experienced in a physical disk environment.” 

The white paper indicates that fragmentation, which reduces system performance in a physical storage infrastructure, can even create more of a performance loss in a virtual storage infrastructure. Virtual disks can become fragmented over time just like the physical disk or disks on which they reside. The result is a fragmented virtual disk on a fragmented physical disk—or fragmentation within fragmentation.  This data is especially important in light of the rapid growth of virtual environments. Organizations are particularly interested in virtualization due to its many benefits, which include reduced hardware costs, ease of adding additional capacity to existing infrastructure, ease of administration and maintenance, and simplified migration from one server to another.  Because of the complexity of I/O traffic in virtual environments, simple defragmentation is not enough to fully address the fragmentation issue. For that reason, Diskeeper Corporation has developed new technology for virtual environments, found in their V-locity™ 2.0 virtual platform disk optimizer. A recent product release for VMware and Hyper-V, V-locity 2.0 is the first optimizer that truly eliminates the barriers to full virtual efficiency. V-locity 2.0  employs IntelliWrite™ and InvisiTasking® technologies to both prevent a majority of fragmentation in the first place and to efficiently coordinate VM resources when defrag is running invisibly in the background. The complete white paper is located here.

IT Pro, Brad McGehee knows SQL, and writes about it frequently. He's even started several websites and blogs dedicated to SQL. A few years ago, he co-authored a paper with me where I discussed fragmentation of the SQL database files (.mdf, .ldf) commonly referred to as external fragmentation, and Brad discussed using DBCC statements to address what is commonly referred to as internal fragmentation.  You can read that article here.

On his current SQL blog, SQL Aloha, he ran a survey asking how DBAs handled internal fragmentation. Here are the results: 

You can read his full post here.

In addition to the script he links to, there are other good third party options out there to address that internal fragmentation including Idera's SQL defrag manager and Norb Technologies SQL Defrag Studio.

PS: while on the subject, here's another recent article on the subject 

 

BURBANK, CA -- 07/20/10 -- Diskeeper Corporation has officially shipped V-locity™ 2.0, a new virtual platform disk optimizer designed to deliver invisible background optimization of all Windows® Guest operating systems running on the VMware ESX and Microsoft Hyper-V platforms. New to V-locity 2.0, is the addition of the breakthrough IntelliWrite™ fragmentation prevention technology originally introduced with Diskeeper 2010. Utilizing IntelliWrite technology, V-locity writes files to the disk to prevent up to 85 percent of fragmentation from occurring before it even happens.

Already a VMware and Microsoft partner for its Diskeeper® performance software, V-locity was made to create a virtual-specific product that not only performs defragmentation functions, but also synchronizes the complex and ongoing activity between host and multiple guest operating systems in a virtualized environment.

As virtualization platforms begin to age, fragmentation of files in Windows host and guest operating systems generates more disk I/O than should be necessary. Fragmentation creates more overhead on the OS and file system. While CPU, Network, and memory resources may allow for greater VM density, the disk subsystem can become a virtualization "high" hurdle.

"Fragmentation clogged disk subsystems can lead to an inability to run more VMs on given hardware infrastructure, and lead to disk performance bottlenecks for VMs that share a common storage subsystem," notes Diskeeper Product Manager, Michael Materie. "V-locity is designed to alleviate the 'virtual' disk bottleneck for VMs and provide a faster and more efficient computing platform for new consolidation and provisioning initiatives, without having to add more hardware."

"First of all, when building out my environment I make sure to have fast hard drives, controllers, etc., in order to make sure not to suffer bottlenecks. In addition, I am very careful about not over-committing resources. However, I was shocked to see, after deploying V-locity, how much performance increased on the servers I had hosting disk-intensive operations. This includes our database server, our Exchange server, and to a certain extent, two terminal servers," stated Bill Philpot Manager of Information Technology at Mesa Industries, Inc.

Diskeeper Corporation's proprietary technology, IntelliWrite writes files in a non-fragmented condition. Copy on write solutions (e.g. as used by Snapshots) take action on changes to data at a block level. Moving data, as is done in a defragmentation job to consolidate file fragments in a logical file system, can trigger copy on write solutions to take extra actions such as using more storage capacity, unnecessarily. Writing a file contiguously eliminates the need to defragment it after it has been created.

InvisiTasking® technology, another Diskeeper Corporation proprietary technology, is specifically engineered to allow "background" applications to operate with zero impact/overhead on a system. With V-locity 2, the InvisiTasking technology has been "enlightened" to operate across a virtual platform. Even as more VMs are added to a host platform or dynamically migrated to new hosts (e.g. vMotion, Live Migration), the enlightened InvisiTasking will continue to dynamically adjust to changing environments, providing V-locity 2.0 users with "Set It and Forget It" ® optimization of their virtual disk platforms.

V-locity also frees up vital storage resources by eliminating virtual disk "bloat." This is the wasted disk space that takes place when virtual disks are set to dynamically grow but don't then shrink when users or applications remove data. V-locity actually compacts the virtual disk, thereby preventing waste and allowing IT Managers to better allocate their virtual storage resources.

V-locity has four unique solution sets:

• InvisiTasking - coordinates resource usage to ensure defrag is 100% invisible
  
  
• IntelliWrite - prevent fragmentation and offer 100% compatibility in all environments
  
  
• Virtual Disk Intelligence - automatically detects and configures based on virtual disk type
  
  
• Virtual Disk Compaction - shrinks/compacts virtual disks that are set to grow-as-needed

V-locity consists of three components:

• V-locity Host installed in the VMware ESX Host or Windows Server 2008/R2 operating system running Hyper-V.
  
  
• V-locity Guest installed in all Windows virtual machines.
  
  
• For VMware ESX platforms: V-locity includes a small application that allows you to remotely connect from your Windows desktop to the V-locity Host component on an ESX system.

Note: On Windows platforms, each component will optimize its respective OS; performing defragmentation of files and consolidation of free space. This minimizes unnecessary I/O passed from the OS to the disk subsystem and aligns data on the drives for optimal access.

Our Executive Vice President had a 3-year old Dell computer that had probably never been defragmented.  It had only 512MB of RAM and when I worked on it, it was EXTREMELY slow.  To open up MS Word had to take 10-15 seconds or more.   I installed Diskeeper Professional and let him know what I did to speed up his PC.  Much to my surprise he called me back a day later and thanked me for speeding up his PC.  He said it was quicker and the applications he opened were more responsive.   I hung up the phone in amazement. Usually the only time a user calls is when they have a problem, not to tell you how well something is working.  Obviously Diskeeper made quite an impact on him and his PC! 

Tony Laird, United Fire GroupCedar Rapids, IA