We have been running Diskeeper 2010 EnterpriseServer for two months on an HP EVA SAN 4000 and 4400, with 4 1TB volumes each.  

Diskeeper removed over 45 million fragments in the last two months on a specific volume that had only 15% free space, and IntelliWrite prevented 24,000 fragments. I believe that will be even better as soon as we can extend this volume to two TB. 

We see a big improvement on the backup time which came down from 48 hours to 32 now, and it’s still going down. 

I believe Diskeeper worth the price and I never had any trouble with software from Diskeeper Corporation, so that alone narrowed the field of choices. 

Jean-François Poirier
Technicien Telecommunication
Spectra Premium Industries Inc.

Later this month (Thursday, May 27, 2010 at 12:00pm Eastern / 9:00am Pacific), IT Analyst David Chernicoff and I will co-host a webinar covering the benefits and caveats of defragmenting SAN attached servers.

Here is the abstract: 

As storage technologies have become more advanced there is a tendency for storage administrators to believe that the hardware is handling all of their data maintenance needs, keeping their files optimized in the best possible way for top performance and availability. The reality is that hardware solutions alone aren't the most efficient way to keep your critical data stored in an optimal fashion. With this webinar we will give you the information you need to understand how your data is actually being handled and what you can do to improve the performance and optimization of your Windows Server storage.

You can register here: www.windowsitpro.com/go/seminars/diskeeper/windows_san 

We also have two webinars planned for June on the topic of virtualization. One jointly with Redmond Mag, and the other with Microsoft. I'll post registration links on this blog when they are available. Lastly, we'll have a reprise of the SSD webinar we did with Microsoft sometime in July - this time we'll host and Microsoft will be our guest.

If you live in on the west coast (USA), you may have seen an ad campaign about happy cows coming from California. Given we're a CA-based software company we concur.

OK wait a minute. Why are we talking about cows? What does this have to do with defrag? Maybe you're thinking "Michael's talking bovine, he's had a few too many".

Well maybe yes, but... I do have a point.  

CoW, or "copy on write" refers to a low level technology that looks at block data changes and then takes a particular action. Every time there is a write, a CoW technology would seek to make a copy of just the changes - not the entire new file. That action may be part of inline data dedupe, may be a snapshot, etc. The basic point is that CoW is used to isolate changes to data at a granular level rather than at a file level.

The consideration is that most CoW technologies are unable to distinguish between changes to data or movement of pieces of data, due to a defragmentation job. In other words, if you run a defrag, CoW may think there are actual data changes to files, and then take action to make block-change copies, run a dedupe, etc...

Holy CoW you say!

So, it is possible that a defrag, any kind of defrag, causes a CoW based technology to go into hyperactive mode. That may mean a CDP (continuous data protection)/snapshot program may be triggered in to keeping/taking far more copies that it needs to. It is basically fooled into thinking that a defrag represents actual changes to data, and that they need to process those changes. Now this is a false processing but, because most CoW solutions function underneath the file system layer, they simply cannot differentiate that actions that take place at the file system are not changes to data, but rather data-movement-for-optimization reasons.

So, if we're talking about a CoW solution that takes snapshots, we could see an increase in the amount of copies that a snapshot solution would have to take. That would make for a fairly fat CoW (a lot of extra data storage demands created by not understanding defrag).

CoW based solutions that work at the file system level (NTFS), have the possibility to recognize changes due to defrag and differentiate them from changes due to new actual data to a file. One such CoW solution is Microsft's VSS (Volume Shadow Copy Service). However, as noted, most CoW technologies operate at storage layers that are abstracted from the local disk file system - they operate at levels underneath the file system. Again, that means defrag jobs in the file system will not be recognized by these solutions, and a defrag job of any kind, will cause unnecessary copies.

In comes IntelliWrite. Now, it makes sense to think that IntelliWrite was invented as a faster and, perhaps cooler, solution to fragmentation. But, the truth is that solving fragmentation at the source (when it is created) is vital to ensure compatibility with many of the CoW solutions that may be implemented at virtualization host or SAN layer (i.e. underneath and hence, unaware of the file system).

So, in a nutshell, IntelliWrite was very much designed to ensure that defrag offered full compatibility with modern storage technologies. That, we feel, makes for happy CoWs.

Storage Area Networks (SANs) are becoming increasingly more common. A few years ago "SAN" was an acronym that rarely made it out of the lexicon of IT Storage Admins at 1000s+ employee multinationals. In more recent years the SAN IHV/ISVs have greatly simplified and reduced the installation, maintenance, technical effort, and acquisition costs. It's increasingly more common to see SANs in medium sized businesses. Many SAN providers have even offered targeted "mid-range" solutions often for those mid-sized organizations. EMC is one such vendor that targets just such a solution with their Clariion product line.

Microsoft has also been at the forefront of advancements in data storage centralization. Technologies like Storport (introduced in Windows Server 2003), iSCSI software initiator, multipath I/O storage stack, and more.

A great deal of innovative software from Microsoft and SAN vendors make the whole system work.

An important point to be aware of is where in the whole computer system, the SAN "plugs" in. A SAN is, in essence, a replacement for a single disk. In the Windows I/O storage stack a SAN solution replaces the disk driver (disk.sys), with its own drivers. Eventually data must reside on a physical storage device of some kind, so any request to read or write data will have to go through this disk driver, or SAN replacement thereof. However, before the I/O request gets to this lower level it goes through a local disk file system. When talking about Windows in a SAN, that local disk file system is pretty much always going to be NTFS. Fragmentation as Windows sees it and cares (same for Diskeeper), is at NTFS. So, if files are fragmented as NTFS sees it, the local disk file system has to send a great deal more I/O traffic into the SAN, causing the SAN to do more work that it should.

We have a very thorough white paper that covers defragmenting SAN. It also includes Best Practices for Diskeeper. Check it out here.

Even SAN vendors recommend defragmenting Windows. EMC includes a paragraph about the need for their Clariion family of products in a white paper here. In it (pg. 5) it says:

The interesting part is they also note the SAN file system solutions they offer are NOT designed to handle NTFS fragmentation, and that they recommend to defragment that "local disk" file system. When we here at Diskeeper talk about the need to defrag SAN attached systems, were talking about doing what we always have done - defragging NTFS in Windows (from Windows). That is an important point as SANs also use a file system to organize data. Diskeeper, nor Windows defrag addresses this. "If" defrag of some kind is needed in this SAN file system it is handled by the SAN vendor - you can check with their support staff on that subject. Defragmenting NTFS and defrag of SAN file systems are two completely different subjects and should not be confused.

Even more reading:

Ziff Davis Enterprise (from the same parent company of eWeek) just released a paper on defragmenting SANs, including benefits and covering some considerations as well. You can read that here.

In summary, even with the tremendous amount of technology that has gone into SANs over the past decade, defragmenting SANs is still just as vital as defragmenting DAS (direct attached storage).