Microsoft officially released the next version of Windows last week – Windows 8. While this new release contains various technological advancements, issues with I/O performance and its effect on Windows systems still remains.

Every I/O operation that occurs takes a measureable amount of time. There’s no such thing as an instant I/O request, and simply put, the more I/Os necessary, the longer it will take for Windows to complete a particular task. 

To understand why this is still an issue on Windows 8 and even Windows Server 2012, let’s explore a bit deeper. When data is written within the Windows file system, it is naturally written in a non-optimized way. Thus when an application requests the data, the initial I/O request generally gets broken down and  splits into many additional requests (called split I/Os), and thus increases the time necessary to retrieve the information. So, as this activity naturally occurs on a daily basis, it takes more and more I/O requests and increasingly impacts the performance of your servers and workstations. 

The Windows built-in optimization tool, which is set to run on a weekly basis, attempts to handle the mounting I/O traffic, but that’s after you’ve already experienced the performance impact in the first place. For example, say I’m working on a project on a Tuesday afternoon – how is running the built-in optimization utility on Wednesday going to address this concern?

Proactive Windows I/O acceleration is the key to successful operations and improved response time to users and this is why Condusiv Technologies created our Diskeeper product. Diskeeper’s InvisiTasking and IntelliWrite technologies helps prevent the vast majority of extra I/O requests from occurring and does so without taking any additional resources from the system or other applications. This ensures that you get the least number of I/Os to go to the storage and allows your applications to run that much faster. 

 In fact, recent independent testing by openBench labs shows up to 98% few I/O requests, server throughput increased by 130% and data throughput up to 5X faster on workstations. You can read more of this report here.

We recently responded to a forum post on our YouTube channel regarding SSDs and Defragmentation - you can view the video here: http://www.youtube.com/watch?v=hznCSqb4Mzg

Below are some "before and after" graphs that provide proof that fragmentation affects SSDs:

 

In today’s day and age, many SMBs and enterprise-level businesses are “taking to the skies” with cloud computing. These companies realize that working in the cloud comes with many benefits – including reduced cost of in-house hardware, ease of implementation and seamless scalability. However, as you will read on and discover - performance-impacting file fragmentation and the need for defragmentation still exists and is actually amplified in these environments. Based on these factors, it must now be addressed with a two-fold proactive and preventative solution.

Let’s face it – we generate a tremendous amount of data and it’s only the beginning. In fact, findings included in a recent study by IDC titled “Extracting Value from Chaos” predict that in the next ten years we will create 50 times more information and 75 times more files. Now regardless of destination, most of this data is generated on Windows-based computers, which are known to fragment files. Therefore, when files are manipulated they become fragmented before even reaching the cloud. This occurs because as they are worked with, they get broken up into various pieces and scattered to numerous locations across the hard disk. The result is increased time necessary to access these files and affects system performance.

So how does the above scenario affect the big picture? To understand this, let’s take a closer look at your cloud environment. Your data, and in many cases, much of your infrastructure, has “gone virtual”. Users are able to access applications and work with their data basically anywhere in the world. In such an atmosphere, where the amount of RAM and CPU power available is dramatically increased and files are no longer stored locally, how can the need for defragmentation still be an issue?

Well, what do you think happens when all this fragmented data comes together? The answer is an alarming amount of fragmented Big Data that’s now sitting on the hard drives of your cloud solution. This causes bottlenecks that can severely impact your mission-critical applications due to the large-scale unnecessary I/O cycles needed to process the broken up information.

At the end of the day, traditional approaches to defragmentation just aren’t going to cut it anymore and it’s going take the latest software technology implemented on both sides of the cloud to get these issues resolved. It starts with software, such as Diskeeper 12, installed on every local workstation and server, to prevent fragmentation at its core. Added to this is deploying V-locity software across your virtualized network. This one-two punch defragmentation software solution addresses I/O performance concerns, optimizes productivity and will push cloud computing further than you ever thought possible. In these exciting times of emerging new technologies, Cloud computing can send your business soaring or keep it grounded - the choice is up to you.

IntelliWrite technology has been around for about two years now, optimizing literally millions of systems worldwide. It seamlessly integrates with Windows, delivering optimized writes upon initial I/O (no need for additional, after-the-fact file movement). What does that translate to? Actual fragmentation prevention.

Interestingly, we do occasionally get asked how it bears up against modern storage technologies:

“Don’t the latest SANs optimize themselves?”

“Do I really need this on my VMs? They aren’t physical hard drives, you realize…”

Or even…

“I don’t need to defragment my SAN-hosted VMs.”

Now, there are some factors which must be considered when you’re looking at optimizing I/O in your infrastructure:

• I/O from Windows is just abstracted Reads and Writes from a higher layer, even directly over a bare metal disk.
• Due to the way current Windows file systems are structured, I/O can be greatly constrained by file fragmentation—no matter what storage lies underneath it.
• Fragmentation in Windows means more I/O requests from Windows—even if files are stored perfectly contiguously at the SAN level, Windows still has to send X amount of requests because of the fragmentation that it sees within its top level.
• File fragmentation is not the same as block-level (read: SAN-level) fragmentation. Many SAN utilities resolve issues of block-level fragmentation admirably; these do not address file fragmentation.
• Finally, and as noted above, IntelliWrite prevents fragmentation in real time by improving Windows “Best Fit” file write logic. This means solving file fragmentation issues with no additional writes which could create issues with SAN de-dup or various copy-on-write data redundancy measures.

We performed testing with a customer recently in order to validate the benefits of IntelliWrite over cutting-edge storage. This customer’s SAN array is less than a year old, and while we don’t want to go into specifics in order to avoid seeming partial, it’s from one of today’s leading SAN vendors.

Testing involved apples to apples comparison on a production VM hosted over the SAN. A non-random workload was generated 3 times, recording Windows-level file fragmentation, several PerfMon metrics, and time to complete the workload. The test was then repeated 3 times, now with IntelliWrite enabled on the same VM’s test volume.

Here were the results:

 

Diskeeper Corporation recently delivered a live webinar hosted by Ziff Davis Enterprise. The principle topics covered were:

• Measuring performance loss in Windows over SAN
• Identifying client-side performance bottlenecks in private clouds
• Expanding performance awareness to the client level
• The greatest and often-overlooked performance issue in a virtual ecosystem

The webinar was co-hosted by:

• Stephen Deming, Microsoft Partner Solution Advisor
• Damian Giannunzio, Diskeeper Corporation Field Sales & Application Engineer

Don't miss out on this critical data! If you missed the webinar, you can view the recorded version online here.

Here are some additional, relevant resources:

White Paper: Diskeeper 2011: Improving the Performance of SAN Storage

White Paper: Increasing Efficiency in the IT Environment

White Paper: Inside Diskeeper 2011 with IntelliWrite

White Paper: Running Diskeeper and V-locity on SAN Devices 

In today’s well connected world of electronics and instant communications I received a text from a friend asking if I had seen the recent PC World magazine (February, 2012).  He said it had some tidbit of information concerning one of my favorite subjects; system performance, defragmentation, and SSDs.  I located a copy here at the office and found the article. As I read the first line I realized the debate on the virtues of defragmentation especially on SSDs will be one that goes on indefinitely as no one really talks about the issue with supporting hard facts and numbers.  Most articles are rehashing ideas and opinions long since debunked.  They continue to surface because very few truly understand the intricacies of the Windows NTFS file system and that of the storage media, whether it is rotating magnetic hard disks or electronic solid state disks.

So let’s set the record straight… Fragmentation is exponentially more of a problem with today’s data explosion. Defragmenting once a week will still cause the user to experience slowdowns from the degradation effects and doesn’t address the issue when files are initially being written.  And yes, never do a traditional defrag on SSDs.

NTFS file and free space fragmentation happens far more frequently than you might guess.  It has the potential to happen as soon as you install the operating system.  It can happen when you install applications or system updates, access the internet, download and save photos, create e-mail, office documents, etc…  It is a normal occurrence and behavior of the computer system, but does have a negative effect on over all application and system performance.  As fragmentation happens the computer system and underlying storage is performing more work than necessary.  Each I/O request takes a measurable amount of time.  Even in SSD environments there is no such thing as an “instant” I/O request.  Any time an application requests to read or write data and that request is split into additional I/O requests it causes more work to be done.   This extra work causes a delay right at that very moment in time.  Whoever thought that defragmenting once a month or weekly was good enough, simply didn’t understand fragmentation.

Disk drives have gotten faster over the years, but so have CPUs.  In fact, the gap between the difference in speed between hard disks and CPU has actually widened.  This means that applications can get plenty of CPU cycles, but they are still starving to get the data from the storage.  What’s more, the amount of data that is being stored has increased dramatically.  Just think of all those digital photos taken and shared over the holidays.  Each photo use to be approximately 1MB in size, now they are exceeding 15MB per photo and some go way beyond that.  Video editing and rendering and storage of digital movies have also become quite popular and as a result applications are manipulating hundreds of Gigabytes of data.  With typical disk cluster sizes of 4k, a 15MB size file could potentially be fragmented into nearly 4,000 extents.  This means an extra 4,000 disk I/O requests are required to read or write the file.  No matter what type of storage, it will simply take longer to complete the operation.

Suppose I chose to do some editing of my family videos on Tuesday evening.  Even the built-in defragmentation tool in Windows 7 doesn’t do me much good because it isn’t schedule to run until Wednesday morning at 1:00am.  This also means that quite a bit of fragmentation has built up since the previous week when it last ran.  Maybe I’ll manually run it, but that can take quite a while and I’ve wasted time that I would have rather spent on my project.  Unfortunately, the Windows built-in defragmentation utility doesn’t prevent fragmentation so even after running it manually; I still will wind up with fragmentation and slow access speed of my newly created files. 

I’ve often thought about why Wednesday at 1:00am was chosen as the time to schedule defragmentation.  Why isn’t it scheduled all the time?   It is because there could be system resource conflicts that either interfere with getting the task done or the defragmentation process has difficulty throttling back under a variety of conditions.  Regardless, this wait a week to clean up fragmentation doesn’t really help me when I need it most.

As pointed out in the article, the built-in defragmenter does not have the technology advancement to properly deal with fragmentation and SSDs. The physical placement of data on an SSD doesn’t really matter like it does on regular magnetic HDDs.  With an SSD there is no rotational latency or seek time to contend with.  Many experts assume that fragmentation is no longer a problem, but the application data access speed isn’t just defined in those terms.  Each and every I/O request performed takes a measurable amount of time.  SSD’s are fast, but they are not instantaneous.  Windows NTFS file system does not behave any differently because the underlying storage is an SSD vs. HDD and therefore fragmentation still occurs.  Reducing the unnecessary I/O’s by preventing and eradicating the fragmentation reduces the number of I/O requests and as a result speeds up application data response time and improve the overall lifespan of the SSD.  In essence, this makes for more sequential I/O operations which is generally faster and outperforms random writes.

In addition, SSD’s require that old data be erased before new data is written over it, rather than just writing over the old information as with HDDs.  This doubles the wear and tear and can cause major issues with the speed performance and lifespan of the SSD.  Most SSD manufactures have very sophisticated wear-leveling technologies to help with this. The principle issue is write speed degradation due to free space fragmentation.  Small free spaces scattered across the SSD causes the NTFS file system to write a file in fragmented pieces to those small available free spaces.  This has the effect of causing more random I/O traffic that is slower than sequential operations.

I think I have clearly made my point….  The built-in defragmenter in Windows 7 is not a solution for neither the consumer/home user, nor the enterprise business user.  Data access speeds are far more critical in the business world where time is money.  In the enterprise environment there are generally many more files that are used by higher number of users that are accessing data across shared type of storage such as SAN.  Even virtual platforms benefit from the same points covered.  This opens the door and is the reason why robust solutions such as Diskeeper exist.  More data about Diskeeper and the superior technology it offers can be found at http://www.diskeeper.com.

IT professional and author, Justin James of TechRepublic published a top 10 list of ways to speed up your PC. Number 9 was one very familar to us:

"9: Defrag. Defragging your hard drives is a great way to get some more performance. While modern Windows systems automatically defrag on a regular basis, I’ve found that the Windows defragging is fairly unaggressive. We’ve reviewed a lot of different defrag apps here at TechRepublic. I suggest that you check out your alternatives and find one that does a better job for you."

Their findings mimic what we see with many of our business customer seeking to maximize Windows 7 performance. The built in defragmenter sounds like an attractive option at first, but closer inspection and testing clearly demonstrates significant value (better ROI) in advanced third party optimization technology. 

Read the whole article here: http://www.techrepublic.com/blog/10things/10-things-you-can-do-to-boost-pc-performance/2712?tag=nl.e101 

Diskeeper has long been a popular compliment for design and engineering software systems, including by the the most popular design and enginerring software company itself:

“Diskeeper indeed provided the ROI we expected. We are very satisfied. I have been using your product on my personal desktop and at work for the past few versions. It always proved to be a valuable product for transparent defrag and overall performance. 

Prior to installing Diskeeper our systems were almost unresponsive. This easy deployment was done right away as Diskeeper was much needed. It is a fine product that we rely upon.  It was reported to me that the automatic defrag did not take any visible resources or produce any slower performance which was really appreciated. 

The automatic defrag also helped overall performance greatly on the files sytem access. One of the folders has millions of files in it with no subfolders and it was almost inaccessible before the Diskeeper implementation due to the fragmentation level being very high” 

- Marc-Andre, Autodesk   

Autodesk, Inc., is a leader in 3D design, engineering and entertainment software. Customers across the manufacturing, architecture, building, construction, and media & entertainment industries—including the last 16 Academy Award winners for Best Visual Effects—use Autodesk software to design, visualize, and simulate their ideas before they’re ever built or created. From blockbuster visual effects and buildings that create their own energy to electric cars and the batteries that power them, the work of our 3D software customers is everywhere you look. Since its introduction of AutoCAD software in 1982, Autodesk continues to develop the broadest portfolio of state-of-the-art 3D software for global markets.

Diskeeper Corporation announced today the launch of a PC Speed Contest where contestants are asked to Show – via a picture or short video and Tell – the remarkable gains that they have received in accelerating PC performance using Diskeeper® 2011 performance software.

Diskeeper increases the speed and reliability of all PCs by improving system performance, faster boot-up times and increased speeds on everything from internet browsing to antivirus scans. Diskeeper 2011 gives PCs faster-than-new speed without any effort from the user.

The grand prize winner will receive a new Laptop computer and a copy of Diskeeper 2011 Pro-Premier software.

To enter the contest: http://tinyurl.com/3b37ayq

Dawn Richcreek, Vice President of Marketing has been recognized by Everything Channel’s CRN Magazine as one of the top Women of the Channel in 2011. This annual list recognizes female executives for their accomplishments over the past year, based on their achievements as executives and the amount of influence they wield over the technology channel.  This year’s Women of the Channel were chosen by the editors of CRN Magazine from a field of vendor channel organizations, distributors and solution providers. 

Over the past year, Richcreek successfully developed the Diskeeper Channel Awareness Campaign with Tech Data.   Additionally, she has launched two new products, Diskeeper^® 2011 data performance software and V-locity^®  2.0  virtual platform disk optimizer, that make the day-to-day lives of reseller customers easier and more productive.  Richcreek’s previous accomplishments include implementation of the Diskeeper Corporation Channel Marketing program where she provided profit generating training and sales tools for thousands of resellers on the Diskeeper Corporation suite of products enabling them to better service and provide solutions to their customers. She is a marketing veteran with over 25 years of experience in the industry. The complete article is located here:  http://tinyurl.com/3dakwmy
 

Improve Data Performance 

For years, we have loved Diskeeper and the way it works in the background to keep your hard drives defragmented. Defragmentation programs have been available for decades, but Diskeeper is among the true innovators.With the release of 2011, the tool has gotten even better. 

Basic Bits 

Drive defragmenters aggregate the data—which drives normally break into chunks and store wherever necessary— into contiguous blocks. This speeds access time (and therefore system performance), and can literally prevent a drive from crashing.  Diskeeper takes a multipronged approach to defragmentation that eases its system impact and helps keep your drives humming along smoothly. For one thing, its IntelliWrite technology prevents fragmentation before it happens. Its new Efficient Mode feature improves the defragmenting process by identifying problem fragmentation. The Instant Defrag feature tidies up any leftover fragments it cannot process during the initial file save. Both of these happen on the fly, in the background,and with little to no impact on system resources.  

The full review is located here.

Windows 8 Coming in 2012. Microsoft chief Steve Ballmer on Monday confirmed that Windows 8 will be available in 2012. 

Microsoft denies Intel statements about Windows 8 on ARM‎. Microsoft last week denied statements made by Intel executives about Microsoft's plans for Windows 8 on ARM processors, calling them “factually inaccurate.” 

Rackspace offers hosted virtual desktops from Citrix‎. Rackspace announced Monday that it is offering hosted Citrix XenDesktop and XenApp, hoping to make using virtual desktops easier. 

Jive buys OffiSync to deepen links with Microsoft apps‎. Jive Software has acquired OffiSync, whose technology links Microsoft desktop applications with cloud-based enterprise social collaboration suites. 

ARM chases server opportunities for low-power processors. ARM Holdings is seeking the support of software makers for its plan to put its low-power processors in servers. 

Fusion-io is going public. Fusion-io, the company that makes SSDs for companies like Facebook, is planning a bigger-sized initial public offering. 

Twitter acquires popular client TweetDeck‎. Twitter has reached an agreement to acquire TweetDeck, a Twitter desktop client preferred by many of the social-media service's most active users, for $40 million in cash and stock. 

Amazon adds electronics to Trade-Ins‎. Amazon last Thursday expanded its Trade-In program to include electronics, allowing consumers to ship gadgets to the online retailer for free in exchange for a gift card. 

Microsoft blasts EU antitrust fine in appeals hearing‎. Calling it "excessive," Microsoft this week criticized in an appeals hearing the 899 million euro fine imposed three years ago by EU antitrust regulators on Microsoft for failing to comply with a court order. 

ISuppli warns explosion at China plant may slow iPad production. iPad manufacturer Foxconn last week suspended operations at its facility in Chengdu, China after a fatal explosion, which could result in a production loss of 500,000 Apple’s iPad 2 tablets in the second quarter.

“We are now running Diskeeper 2011 EnterpriseServer on our a DFS server which houses all our corporate documents. This server has about 500GB of files and was heavily fragmented prior to installing Diskeeper. 

“Users were experiencing latency issues, login delays (roaming profiles) and sometimes apps would freeze upon saving to that server. After running Diskeeper for a few weeks, there was a major performance improvement. We went from getting several calls a week prior to the Diskeeper install, down to a couple of calls a month. 

“The other server is a backup repository that houses our SQL backups, BackupExec files and other archived info. This server has over 10TB of storage. Since running Diskeeper, the overall performance of the server and backup times improved greatly.” 

Sandy Hyde, Senior Systems Analyst, Business Support Services, Family Insurance Solutions

Diskeeper 2011 was covered on Wugnet.  Howard Sobel stated, “They introduced technology that slowed down and prevented fragmentation in Diskeeper 2010 so I thought it was impossible to improve on the concept of defrag much more. Not so! By increasing the efficiency of their algorithms, they have decreased the wear and tear on your hard disk and your computing performance while it works in the background. Decreasing the overall disk activity also decreases your electrical consumption. This may not be a huge savings on "your" electric bill but consider how much savings this amounts to in datacenters where thousands of hard disks are used. So being GREEN doesn't mean paying a penalty. In fact, the opposite is true for Diskeeper. I would go as far as declaring this the software utility "Product of the Year" if we didn't have 8 more months to go in 2011.” The full article is located here: http://www.wugnet.com/tips/this_week.asp  

 

When I asked Patrick Blanche, of Hobbs+Black Architects, whether he saw good results from his use of Diskeeper, this is what he said: 

“Good results? … you mean GREAT right!  Allow me to explain,

We are an architectural business and work with many small interrelated files that can get highly fragmented, so for us using Diskeeper Server is a must to maintain our performance. For the past few years Diskeeper has been working full time on all file servers in Michigan, Arizona even Utah, and the result has been significant improvement in the user experience.  Back before Diskeeper, (a time I would rather forget) we suffered from literally days of waiting for Windows built-in defrag to try and cope with our TB size volumes.  Finally Diskeeper came along and we have seen those issues vanish.”

Then we worked up a fine case study about the performance benefits to his company and now they are deploying Diskeeper on all of their servers and workstations company-wide!  

The complete case study is located here: http://downloads.diskeeper.com/pdf/HobbsBlackSANCaseStudy.pdf

 

Michael Vizard, Industry leader and IT Editor wrote an article on Diskeeper 2011 and stated,

"But as more applications begin to share the same IT infrastructure thanks to the advent of virtualization and cloud computing, the more fragmentation becomes an I/O performance optimization issue."

 The full article is located here: http://www.ctoedge.com/content/intelligent-disk-defragmentation

Daiwa Odakyu Construction Co., Ltd.  

Problem 

Daiwa Odakyu Construction Co., Ltd did back up of 15 Servers every day using Symantec Backup Exec. But the problem of latency of backup appeared. Initially the system was designed to finish backup within 3 hours at most. But the time grew to 8 hours. They thought additional hardware was needed but had no additional budget. At that time they discovered the cause of latency was fragmentation. First they tried to solve fragments with the built-in defrag. But it solved only 20%. Next a free defrag software was tried. It proceeded to 35% and shortened backup time by more than 1 hour. But 2 days after, the backup time got back to 8 hours. 

Solution 

Mr. Kawata, manager of the Information System Group of Daiwa Odakyu, found Diskeeper® via the internet and downloaded trialware from SOHEI. Diskeeper solved all fragmentation and shortened the backup times down to 1.3 hours. What’s more, IntelliWrite® prevented fragmentation up to 99%! In accordance with their trial, the system without Diskeeper added 1 hour a day to their backup times. 

	Back up Time(hour)	Number of Fragments	Number of fragments prevented
Initial design	3.0	-	-
Fragmented	8.5	10,000,000以上	-
After defrag by Diskeeper	1.3	250	3,000,000～7,200,000
Improvement	85%	99.9975%	（断片化防止率99％）

 “I couldn’t estimate such a big influence of fragmentation in the 50% used volume.  The incredible power of Diskeeper is surprising me. The massive fragmentation I gave up trying to solve is now easily defragmented and prevented by Diskeeper, and then this leads to prevent latency of back up completely.  

“The trial without Diskeeper, which resulted in adding 1 hour a day to backup time, scared me so much. With increasing numbers of files, the backup caused massive fragmentation and gets into a dangerous situation. Less than $1,000 investment in Diskeeper is equal to more than $10,000 worth of Servers. I think Diskeeper is vital for file servers and backup servers."  

“I’m thankful to the developers of this surprising software.” 

System: 

HP DL380G4 Xeon E5430 4GB Memory 410GB SAS RAID10(146GB SA*6）

HP StorageWorks MSA60 3.41TB SATA RAID10(750GB SATA*10）  

 

Diskeeper (data performance for physical systems) and V-locity (optimization for virtual systems) are designed to deliver performance, reliability, longer life and energy savings. Increased performance and saved energy from our software are relatively easy to empirically test and validate. Longer life is a matter of minimizing wear and tear on hard drives (MTTF) and providing an all around better experience for users so they can continue to be productive with aging equipment (rather than frequent hardware refreshes).

Reliability is far more difficult to pinpoint as the variables involved are difficult, if not impossible, to isolate in test cases. We have overwhelming anecdotal evidence from customers in surveys, studies, and success stories that application hangs, freezes, crashes, and the sort are all remedied or reduced with Diskeeper and/or V-locity.

However, there is a reliability "hard ceiling" in the NTFS file system; a point in which fragmentation/file attributes become so numerous reliability is jeopardized. In NTFS, files that hit the proverbial "fan", and spray out into hundreds of thousands and millions of fragments, result in a mess that is well... stinky.

In short, fragmentation can become so severe that it ultimately ends up in data loss/corruption. A Microsoft Knowledge Base article describes this phenomenon. I've posted it below for reference:

A heavily fragmented file in an NTFS file system volume may not grow beyond a certain size caused by an implementation limit in structures that are used to describe the allocations.

In this scenario, you may experience one of the following issues:

When you try to copy a file to a new location, you receive the following error message:
In Windows Vista or in later versions of Windows
The requested operation could not be completed due to a file system limitation
In versions of Windows that are earlier than Windows Vista
insufficient system resources exist to complete the requested service
When you try to write to a sparse file from the Application log, Microsoft SQL Server may log an event that resembles the following:
In Windows Vista or in later versions of Windows
Event Type: Information
Event Source: MSSQLSERVER

Description: ...
665(The requested operation could not be completed due to a file system limitation.) to SQL Server during write at 0x000024c8190000, in filename...
In versions of Windows that are earlier than Windows Vista
Event Type: Information
Event Source: MSSQLSERVER

Description: ...
1450(Insufficient system resources exist to complete the requested service.) to SQL Server during write at 0x000024c8190000, in file with handle 0000000000000FE8 ...
When a file is very fragmented, NTFS uses more space to save the description of the allocations that is associated with the fragments. The allocation information is stored in one or more file records. When the allocation information is stored in multiple file records, another structure, known as the ATTRIBUTE_LIST, stores information about those file records. The number of ATTRIBUTE_LIST_ENTRY structures that the file can have is limited.

We cannot give an exact file size limit for a compressed or a highly fragmented file. An estimate would depend on using certain average sizes to describe the structures. These, in turn, determine how many structures fit in other structures. If the level of fragmentation is high, the limit is reached earlier. When this limit is reached, you receive the following error message:

Windows Vista or later versions of Windows:
STATUS_FILE_SYSTEM_LIMITATION The requested operation could not be completed due to a file system limitation

Versions of Windows that are earlier than Windows Vista:
STATUS_INSUFFICIENT_RESOURCES insufficient system resources exist to complete the requested service

Compressed files are more likely to reach the limit because of the way the files are stored on disk. Compressed files require more extents to describe their layout. Also, decompressing and compressing a file increases fragmentation significantly. The limit can be reached when write operations occur to an already compressed chunk location. The limit can also be reached by a sparse file. This size limit is usually between 40 gigabytes (GB) and 90 GB for a very fragmented file.  

WORKAROUND
For files that are not compressed or sparse, the problem can be lessened by running Disk Defragmenter. Running Disk Defragmenter will not resolve this problem for compressed or sparse files.

Overview:

As high performing storage solutions based on block protocols (e.g. iSCSI, FC), SANs excel at optimizing block access. SANs work at a storage layer underneath the operating systems file system; usually NTFS when discussing Microsoft Windows®. That dictates that a SAN is unaware of “file” fragmentation and unable to solve this issue.

Fig 1.0: Diagram of Disk I/O as it travels from Operating System to SAN LUN.

With file fragmentation causing the host operating system to generate additional unnecessary disk I/Os (more overhead on CPU and RAM) performance suffers. In most cases the randomness of I/O requests, due to fragmentation and concurrent data requests, the blocks that make up the file will be physically scattered in uneven stripes across a SAN LUN/aggregate. This causes even greater degradation in performance.

Fig 1.1: Sample Windows Performance Monitor Report from fragmented SAN-attached NTFS volume.

Fortunately there are simple solutions to NTFS file system fragmentation; fragmentation prevention and defragmentation. Both approaches solve file fragmentation at the source, the local disk file system.

IntelliWrite® “The only way to prevent fragmentation before it happens™”

IntelliWrite is an advanced file system driver that leverages and improves upon modern Windows’ file system “Best Fit” file write design in order to write a file in a non-fragmented state on the initial write. Intelligently writing contiguous files to the disk provides four principal benefits above and beyond defragmentation, including:

• Prevents most fragmentation before it happens
• Better file write performance
• An energy friendly approach to improving performance, as defragmentation is not required for files handled by IntelliWrite
• 100% compatibility with copy-on-write technologies used in advanced storage management solutions (e.g. snapshots)

While eliminating fragmentation improves performance. it is important to properly configure and account for advanced SAN features.

With the increasing popularity of SANs, we've included instructions in the Diskeeper installation to ensure users properly configure Diskeeper:

We suggest reading this full document before executing any of the recommended configurations. These instructions apply to V-locity (used on VMs as well).

Best Practices:

Highlights:

Implementing Diskeeper on a SAN is simple and straightforward. There are two principal concepts to ensuring proper configuration and optimal results:

• Ensure IntelliWrite is enabled for all volumes.
• Find a time to schedule Automatic Defragmentation (more details below)

Details:

If you are implementing any of the following SAN based technologies such as Thin Provisioning, Replication, Snapshots, Continuous Data Protection (CDP) or Deduplication, it is recommended to follow these guidelines.

Defragmentation can cause unwanted side effects when any of the above referenced technologies are employed. These side effects include:

This is why it is important to enable the fragmentation prevention (IntelliWrite) and change the Automatic Defragmentation to occur during non-production periods to address the pre-existing fragmentation:

During Installation, disable Automatic Defragmentation;

Uncheck the “Enable Automatic Defragmentation” option during installation.

Upon installation ensure IntelliWrite is enabled on all volumes (default). IntelliWrite was specifically designed to be 100% compatible with all advanced SAN features, and should be enabled on all SAN LUNs. IntelliWrite configuration is enabled or disabled per volume, and can be used in conjunction with Automatic Defragmentation, or exclusively.

To ensure IntelliWrite is enabled, right click a volume(s) and select the feature.


Then confirm “Prevent Fragmentation on this volume” is selected, and click “OK” to complete.

Once installed, enable Automatic Defragmentation for any volumes that are not mapped to a SAN LUN. This may include the System Partition (e.g. C:\).

To enable Automatic Defragmentation, right click a volume(s) and select the feature.


Then check “Enable Automatic Defragmentation on the selected volumes” and click “OK” to complete.

If you are not using any advanced SAN features, it is recommended to enable Automatic Defragmentation for all days/times. However, note that pre-existing fragmentation will require significant effort from Diskeeper to clean up. This effort will generate disk I/O activity within the SAN.

Therefore, if existing fragmentation is significant, initially schedule Diskeeper to run during off-peak hours. As Diskeeper has robust scheduling capability, this is easily configured.

To enable Automatic Defragmentation during non-production periods, right click a volume(s) and select the feature.


Then check “Enable Automatic Defragmentation on the selected volumes”. Diskeeper is then scheduled by using your mouse to highlight over the 30 minute blocks in the interactive weekly calendar.

The above example disables defragmentation Monday through Friday. It also disables defragmentation Saturdays and Sundays except between 7pm until 3:30am the following morning. This would afford 17 hours of defragmentation availability per week. Immediately following these scheduled defragmentation periods is when SAN maintenance for advanced features should be addressed (e.g. thin reclamation, deduplication).

Should accommodating SAN maintenance be difficult (e.g. limited maintenance windows)using a weekly optimization process, very granular scheduling is also available with Diskeeper. Note, maintenance windows are not required in order to implement and benefit from IntelliWrite.

To schedule for specific non-reoccurring dates and times in the future, select the “Turn Automatic Defragmentation on or off based on specific dates” option. Click any multitude of dates and times using Shift-Select or Ctrl-Select. Once done, click OK to complete.

If you are implementing the above mentioned advanced technologies and your SAN provides hot block optimization / data tiering, it is also recommended to disable I-FAAST® (Intelligent File Access Acceleration Sequencing technology). I-FAAST sequences hot “files” (not blocks) in a Windows volume, after determining hardware performance characteristics. The sequencing process creates additional movement of data for those advanced SAN features, and is therefore generally recommended to disable when similar SAN solutions are in place.

To disable I-FAAST, right click a volume(s) and select the feature.

Note, I-FAAST requires Automatic Defragmentation be enabled. Also note that I-FAAST is disabled by default in Diskeeper 2011 in certain cases. Also note that I-FAAST generates additional disk I/Os and will therefore cause an increase in the aforementioned Automatic Defragmentation side effects.

Once pre-existing fragmentation has been removed, increase the periods in which Diskeeper actively optimizes the Windows file systems. With real-time defragmentation and InvisiTasking® technology, Diskeeper immediately cleans up fragmentation (that is not prevented by IntelliWrite). This minimal ongoing optimization generates only invisible, negligible I/O activity.

New features in Diskeeper 2011 to improve SAN performance:

Diskeeper 2011 introduces SAN specific solutions. These default solutions automate many of the configurations required for SAN-attached servers.

Diskeeper 2011’s new Instant Defrag™ technology dramatically minimizes I/O activity, and exponentially speeds up defragmentation. The Instant Defrag engine is provided fragmentation information, in real-time, by the IntelliWrite file system filter driver (those fragments that it does not prevent). Without the traditional need to run a time and resource intensive whole-volume fragmentation analysis, Instant Defrag can address the recently fragmented files as they occur. This dynamic approach prevents a buildup of fragmentation, which could incur additional I/O overhead to solve at a later date/time.

Diskeeper 2011’s new Efficiency Mode (default) maximizes performance, while minimizing disk I/O activity. By focusing on efficiency and performance and not on presenting a “pretty disk” visual display, Diskeeper 2011 minimizes negative side effects (e.g. reduce snapshot storage requirements or thin LUN growth, etc..) while maximizing performance benefits. It is a SAN-optimized defrag mode and our recommended solution for SAN-attached Windows volumes.

By default, Efficiency Mode also disables proprietary file placement features such as I-FAAST.

Also, by default, Diskeeper 2010/2011 moves data to lower NTFS clusters, and hence generally “forward” on SAN LUNs.

Best Practices Summary:

• Ensure IntelliWrite is enabled for all volumes.
• Automatic Defragmentation should be enabled at all times for all direct attached storage volumes.
• Use Efficiency Mode of Diskeeper 2011.
• Schedule Automatic Defragmentation on SAN LUNs, based on use of advanced SAN features.
• Run SAN processes such as space reclamation and/or deduplication on recently defragmented LUNs using advanced SAN features.

Want this in PDF form. Get it here: Best Practices for using Diskeeper on Storage Area Networks.pdf (3.00 mb)

When Windows NT 4.0 was released, Diskeeper 2.0 hit the market. NT 4.0 had limitations about the type of data that could be safely moved online. So, a market-first innovation that Diskeeper brought to market with Diskeeper 3.0 was what we called Boot Time Defragmentation. Boot Time Defragmentation addressed these special data types during the computer boot process, when it was safe to do so. The files that Diskeeper optimized included metadata (data which "lives above" your data), directories (folders), and the paging file. 

Metadata are special files that the NTFS file system driver uses to manage an NTFS volume. The most famous piece of metadata is the MFT (Master File Table), which is a special file typically consisting of 1024-byte records. Each file or directory on the volume is described by at least one of these MFT records. It may take several MFT records to fully describe a file... especially if it is badly fragmented or compressed; A 271MB compressed file can require over 450 MFT records!

Defragmenting the MFT, data files, and folders are all vital for optimal performance. The example below of what occurs when NTFS goes to read in the 1-cluster file \Flintstone\Barney.txt, makes that case.
 
1. The volume's boot record is read to get the cluster address of the first cluster of the MFT.
2. The first cluster of the MFT is read, which is used to locate all of the pieces of the MFT.
3. MFT record 5 is read as it is predefined to be the MFT record of the root directory.
4. Data cluster 0 of the root directory is read in and searched for "Flintstone".
5. If "Flintstone" is not found, then at least one other data cluster of the root directory needs to be
read to find it.
6. The MFT record for the "Flintstone" directory is read in.
7. Data cluster 0 of the "Flintstone" directory is read in and searched for "Barney.txt".
8. If "barney.txt" is not found, then at least one other data cluster of the "Flintstone" directory needs.
to be read to find it.
9. The MFT record for the "Barney.txt" file is read in
10. Data cluster 0 of the "Barney.txt" file is read in.
 
This is a worst-case scenario. It presumes the volume is not yet mounted, so the NTFS cache is empty at step 1, and the MFT itself needs to be located.  But it shows how many I/Os are required to get at a file that is only one level removed from the root directory: 10. Each one of those 10 I/Os requires a head movement. Any fragmentation along that path only increases the amount of disk I/Os required to access the data - slowing the whole process down.

And, if you follow the step-by-step I/O sequence outlined above, you'll see that every time a new directory is encountered in the path is an additional two or three I/Os. For obvious performance reasons it is beneficial to keep the depth of your directory structure at a minimum.  It also makes the importance of defragmenting these special file types quite obvious. 

As Windows progressed with newer iterations, many of the files that required offline defragmentation, were supported in online defragmentation (including the MFT and directories), so while Boot Time defrag still exists today, the need to run it has diminished. As a great deal of metadata is typically cached from boot to shutdown, perhaps the last remaining system file that is vital to defragment "offline" is the paging file. We've heard arguments over the years that due to the random nature of the data in the paging file that defrag was not valuable, but anyone who has cleaned up a badly shredded paging file will tell you otherwise.