As a follow-up to my post addressing migrating from Novell, I wanted to share the results of a few simple network packet captures.  Before doing so, let me quickly revisit the issues leading to the tests:

• Our Windows Vista x64 clients were experiencing 15-20 second delays, intermittently, when trying to access Novell shares
• Our Windows XP x86 clients (regardless of Outlook versions) were experiencing 15-20 second delays, intermittently, when trying to interact with email or access a Microsoft file share

As a corollary, we introduced a new Exchange server (2007 SP1) and a new archival tool from C2C Systems.

As the calls to HelpDesk began to add up, I attempted to record as much information about the user, though my efforts were almost pointless as no one else thought to do this — even after my explicit instruction.  At any rate, given the data I collected, I could not find any correlation between cached/non-cached mode or version and the intermittent slowdowns.  My next logical leap was to blame our new archival process for slowing everyone down.  I then disovered that the archival procedure — removing literally thousands of emails from Exchange databases — left holes in the database, prompting me to take each database offline and defragment.  This improved performance overall, but the intermittent slowdowns continued and the support calls rapidly increased.

Practically at my wits end, I received an interesting demand from an employee — an upgrade to Vista.  They claimed that every XP user in their department experienced slowdowns in Outlook AND in accessing our newly-created Microsoft file share (the first share completely migrated from Novell).  I quickly put out that fire by asserting that it couldn’t possibly be an operating system-specific issue.  I was only halfway right, as you’ll see.

I had little to no experience using packet capturing software, I’m sad to say, so this was really my first chance to dig in deep.  I installed Microsoft Network Monitor on the Vista lover’s PC (a normal machine by our standards, running Office 2003 and XP SP2 x86) and proceeded to capture traffic.  I kept the user logged in and checked traffic over a period of time to see if I could glean anything useful.  It wasn’t until the user could predictably create the 15-20 second delay that I learned anything useful — the client was attempting to connect to a share on the Exchange server during the slowdown.

I quickly set up a number of test PCs to try and reproduce the slowdown in both Outlook and accessing the Microsoft share.  The machines had the following in common with the user’s PC:

• Outlook 2003 in cached mode
• C2C client installed
• Novell Client 4.91 SP4
• Tested under an account not licensed for C2C
• Tested under an account that was a power user on the PC

As it turns out, I was able to reproduce the issue described by the user at start-up.  In other words, when I booted the machine and attempted to access the Microsoft share, there was an excruiciating 15 second delay before the entire contents of the share were displayed.  Feeling quite good at my ability to reproduce the issue (hah!), I installed WireShark and took the following steps to capture whatever network traffic was generated during the slowdown:

1. Logged in as a power user that wasn’t licensed for C2C
2. Opened Outlook
3. Opened WireShark
4. Started a WireShark capture
5. Attempted to open an unread email that was not displayed in the preview pane
6. Stopped WireShark once the message opened (I double-clicked the message to open it)

The results:

Look at all of those attempts to connect to the Exchange share!  To the developer in me, this SCREAMS bad coding.  Why would a non-licensed C2C user need to access the C2C share?  Beats me.  Shame on you, C2C developers, for your inefficient coding.  To work around this problem, I decided to un-install the C2C client, reboot, and repeat the test.  Here are the results:

Wow — what a reduction in SMB chatter — not to mention there was absolutely no delay in viewing any message.  I reproduced the same results, both with the C2C client and without, when attempting to access the Microsoft share.  Removing the C2C client seemed like a good idea, but ultimately this wasn’t the solution I was looking for.  All of our licensed users need the client to search through their old mail.

Elated at finding the source of the problem (SMB chatter), I went to the oracle (aka Google) and started a search for SMB performance problems and found a few notable recommendations.  An awesome TechRepublic post detailed how to modify a few settings to allow for quicker SMB performance.  I tested this out with a few users and the first thing they noticed was a huge performance hit when accessing Novell shares.  While Outlook performance was stellar, these users would have rather had slow Outlook performance and speedy Novell access.

My next promsing lead was a Novell TID written in an extremely vague fashion.  It discusses creating a registry entry and listing out the servernames and IP addresses hosting SMB shares.  It was immediately obvious where to put this new registry entry, so I had to search deeper and found a much better explanation on the Novell Forums.  Here is a screenshot of the registry fix:

I tested this registry hack on select machines throughout the company with a 100% success rate.  Using Configuration Manager 2007, I pushed out a VB script to make the appropriate changes on every Windows XP machine in the company.

Another step away from Novell and another step closer to Microsoft.  /sigh…

For the past few weeks, during the transition from Exchange 2000 to Exchange 2007, we’ve experienced a number of complaints of slowness in Outlook.  Although the symptoms could be attributed to literally almost anything, we evaluated every possible option:

1. Exchange 2007 server issues: does it have adequate RAM, hard drive space, processing power, etc.?
2. Client issues: what version of Outlook are they using?  Is an add-in causing the perceived slowness?
3. Is it a result of a recent update to a file server, which was made public, and thus the slowness is not real but just a result of information the end user is applying to a different situation?

Today we learned something about our friend, C2C, which has been actively archiving nightly.  As it turns out, C2C is eating gigantic holes in our mailbox databases which cannot be fixed with online defragmentation.  It would have been nice to know this going into the whole upgrade, alas, the information was not available to me.

I moved myself to a mail storage group that contained only one mailbox database.  Its function is to hold accounts until we have backed up all of their personal files to portable media.  At any rate, it once contained mailboxes but they have since been purged from Exchange.  After I added my little mailbox (~1.3GB) the total size of the mailbox database was 20.4 GB (22,004,252,672 bytes).  I made a quick backup using NTBackup and then dismounted the database.  Using the Exchange Shell I ran an offline defrag (eseutil /d %database_name%) and it took a little over two mins to complete.  It’s recommended to perform a backup immediately following defragmentation, so I performed another NTBackup.  After the process was complete, I found that the newly defragmented database was only 1.32 GB (1,423,548,416 bytes)!  That’s a 65% reduction (if my math is correct, haha)!

I plan on performing the same defragmentation on all of our mailbox databases in an attempt to eliminate all complaints of Outlook slowness.

I know I’m just a simple computer science major, but does every database seriously operate in such an inefficient manner?  Why couldn’t online defragmentation take care of the 19GB of whitespace?

At any rate, we’re almost there.  I’m still migrating public folders nightly but their numbers are dwindling.

My first attempt at using the Exchange Power Shell scripts for migrating public folders did not go so well.  Thankfully, this was performed on a secondary Exchange server and not the primary legacy server.  For reasons still unknown, public folder instances on the legacy Exchange server persisted even after running the somewhat scary ReplaceReplicaOnPFRecursive.ps1 power shell script for all of the public folder hierarchy.  To work around this problem, I simply migrated using the old-fashioned (and tedious) method of adding the 2007 server as a replica to every public folder listed on the legacy server’s public folder database.  After my end-users were satisfied that all of their data had replicated (again, this required an “update” for every public folder my 2007 server could see in the SP1 public folder tool), I removed the legacy server from the replication list using PFDAVAdmin.

This evening, I discovered an undocumented quirk in the RepliceReplica script that may have led to the problems I experienced with the first legacy server — though it requires a path for the top-level public folder, if the path has a space in it, you must use an additional set of apostrophes.  I don’t know why I didn’t think of this the first time around, but the process seems to be running smoothly at the moment.  I’m still very skeptical of running the script against the entire public folders folder, so I’m running it against each subfolder individually so I can literally watch the public folder instance disappear from the legacy server.

So it’s clear, to move the top-level folder entitled “A/P Staff” from my legacy server to the 2007 server, I had to issue the command as follows: 

[PS] C:\Program Files\Microsoft\Exchange Server\Scripts>.\ReplaceReplicaOnPFRecursive.ps1 -TopPublicFolder “‘\A/P Staff’” -ServerToAdd %2007Server% -ServerToRemove %LegacyServer%

Luckily the Event Log, with almost verbose logging enabled for the Exchange 2007 server, still reports Replication Incoming/Outgoing 3028/3027 messages.  The legacy server eventually removes the public folder instances once the replication is complete.

With one legacy server completely off and zero complaints thus far, I will be removing it from Active Directory tomorrow, fingers crossed!