I ran into this today and thought I would document it somewhere.  In trying to save space on an ESX host, I wiped out (almost) everything in the tools-isoimages directory.  I didn’t realize what I had done until I tried to install VMWare Tools on a few new machines I created.  It wouldn’t be such a big deal if I had only deleted the iso on one of the two ESX hosts I maintain, but I managed to wipe both — yay!  In order to replace the ISO file, it’s necessary to extract the file(s) from the RPM directory on the ESX installation media. 

1. Mount the ESX ISO file (or insert the CD) into a machine that has file system access to the ESX host.  (I use Veeam Backup and FastSCP for transferring files)
2. Navigate to the the VMWare/RPMS directory
3. Transfer the VMWare-esx-tools-%version%.rpm file to the ESX host
4. SSH into the host and navigate to the RPM location.  I stuck mine in the tools-isoimages directory just to make it easier on myself
5. Replace the tools package by running the command rpm -i %file.rpm% –replacepkgs where %file.rpm% is the name of the RPM file.
6. If everything works, you will be put back at the command prompt and a few new .iso files will be present in your tools-isoimages directory
7. Attempt to install VMWare Tools on your host and all should be well
8. If you were as adept as me, copy the Windows.iso and Windows.iso.sig to the other host you hosed!

I hope this helps someone out there!

Following up on my question to the wiki-dev list, I received a response with some unattributed code for generating a table of contents for a wiki page:

// set aside the old prepare method

if (window.prepare) window.beforeTOCPrepare = window.prepare;

// now add our own, which calls the old one

window.prepare = function(inAlwaysRun) {

if (window.beforeTOCPrepare) beforeTOCPrepare(inAlwaysRun);

if ($(‘editable_content’)) gTOCGenerator = new TOCGenerator();

// also fix up the WLTEditor class so that it redraws the TOC when saving

if (window.gEditor) {

gEditor.beforeTOCEditorCleanElementForEditing = gEditor.cleanElementForEditing;

gEditor.cleanElementForEditing = function(inElement) {

gEditor.beforeTOCEditorCleanElementForEditing(inElement);

gTOCGenerator.remove();

}

gEditor.beforeTOCCleanElementAfterEditing = gEditor.cleanElementAfterEditing;

gEditor.cleanElementAfterEditing = function(inElement, inResponseDict) {

gEditor.beforeTOCCleanElementAfterEditing(inElement, inResponseDict);

gTOCGenerator.draw();

}

}

}

Simply copy and paste this code into a .js (JavaScript) file and save it to your theme’s folder structure: /Library/Application Support/Apple/WikiServer/Themes/[your_theme_name]/

I sincerely apologize for the formatting but I’m sick of fighting with WordPress to get it right!  I’m no JavaScript expert so I do not know how the code works, but this can server as a starting point for future ToC development.  For now, when the title of an article with multiple headings is clicked, a menu of the headings appears:

Apple Wiki Table of Contents using JavaScript

With more time and a little Headfirst: JavaScript, I might just have a solution that makes a dynamic, permanent Table of Contents at the top of each article.

The code has not been attributed to any one person but I will gladly cite the author if someone claims to be he or she.

I’ve been a faithful Wiki Server use since it launched with Leopard Server.  I’ve converted my company’s two major MediaWiki installations with a reasonable amount of success and we now have 11 wikis spanning 9 departments.

Discovering the limitations of the software has been extremely disheartening.  I’ve posted a few times on the support forums with little interaction with the wiki dev team.  I even attended WWDC ‘08 in hopes of learning how I could improve our wiki implementation.  In more words, I was told to learn JavaScript and/or XSL to modify the wiki.  While this is great for someone who’s job is to maintain the wiki and nothing more, this is not advice for a network administrator.  Maintaining the wiki occupies such a small portion of my responsibilities that it is simply not economical for me to learn a new language just to make the tool more user-friendly and more applicable to the other departments in the company.

What we, Apple Wiki devs/users, need is an API.  Apple: give us an API.

Seriously, narrow themes should be a thing of the past at this point.  This should make images and articles much easier to read.  If only I had enough time to develop my own theme.

Bah, it’d probably suck anyway!

As a follow-up to my post addressing migrating from Novell, I wanted to share the results of a few simple network packet captures.  Before doing so, let me quickly revisit the issues leading to the tests:

• Our Windows Vista x64 clients were experiencing 15-20 second delays, intermittently, when trying to access Novell shares
• Our Windows XP x86 clients (regardless of Outlook versions) were experiencing 15-20 second delays, intermittently, when trying to interact with email or access a Microsoft file share

As a corollary, we introduced a new Exchange server (2007 SP1) and a new archival tool from C2C Systems.

As the calls to HelpDesk began to add up, I attempted to record as much information about the user, though my efforts were almost pointless as no one else thought to do this — even after my explicit instruction.  At any rate, given the data I collected, I could not find any correlation between cached/non-cached mode or version and the intermittent slowdowns.  My next logical leap was to blame our new archival process for slowing everyone down.  I then disovered that the archival procedure — removing literally thousands of emails from Exchange databases — left holes in the database, prompting me to take each database offline and defragment.  This improved performance overall, but the intermittent slowdowns continued and the support calls rapidly increased.

Practically at my wits end, I received an interesting demand from an employee — an upgrade to Vista.  They claimed that every XP user in their department experienced slowdowns in Outlook AND in accessing our newly-created Microsoft file share (the first share completely migrated from Novell).  I quickly put out that fire by asserting that it couldn’t possibly be an operating system-specific issue.  I was only halfway right, as you’ll see.

I had little to no experience using packet capturing software, I’m sad to say, so this was really my first chance to dig in deep.  I installed Microsoft Network Monitor on the Vista lover’s PC (a normal machine by our standards, running Office 2003 and XP SP2 x86) and proceeded to capture traffic.  I kept the user logged in and checked traffic over a period of time to see if I could glean anything useful.  It wasn’t until the user could predictably create the 15-20 second delay that I learned anything useful — the client was attempting to connect to a share on the Exchange server during the slowdown.

I quickly set up a number of test PCs to try and reproduce the slowdown in both Outlook and accessing the Microsoft share.  The machines had the following in common with the user’s PC:

• Outlook 2003 in cached mode
• C2C client installed
• Novell Client 4.91 SP4
• Tested under an account not licensed for C2C
• Tested under an account that was a power user on the PC

As it turns out, I was able to reproduce the issue described by the user at start-up.  In other words, when I booted the machine and attempted to access the Microsoft share, there was an excruiciating 15 second delay before the entire contents of the share were displayed.  Feeling quite good at my ability to reproduce the issue (hah!), I installed WireShark and took the following steps to capture whatever network traffic was generated during the slowdown:

1. Logged in as a power user that wasn’t licensed for C2C
2. Opened Outlook
3. Opened WireShark
4. Started a WireShark capture
5. Attempted to open an unread email that was not displayed in the preview pane
6. Stopped WireShark once the message opened (I double-clicked the message to open it)

The results:

Look at all of those attempts to connect to the Exchange share!  To the developer in me, this SCREAMS bad coding.  Why would a non-licensed C2C user need to access the C2C share?  Beats me.  Shame on you, C2C developers, for your inefficient coding.  To work around this problem, I decided to un-install the C2C client, reboot, and repeat the test.  Here are the results:

Wow — what a reduction in SMB chatter — not to mention there was absolutely no delay in viewing any message.  I reproduced the same results, both with the C2C client and without, when attempting to access the Microsoft share.  Removing the C2C client seemed like a good idea, but ultimately this wasn’t the solution I was looking for.  All of our licensed users need the client to search through their old mail.

Elated at finding the source of the problem (SMB chatter), I went to the oracle (aka Google) and started a search for SMB performance problems and found a few notable recommendations.  An awesome TechRepublic post detailed how to modify a few settings to allow for quicker SMB performance.  I tested this out with a few users and the first thing they noticed was a huge performance hit when accessing Novell shares.  While Outlook performance was stellar, these users would have rather had slow Outlook performance and speedy Novell access.

My next promsing lead was a Novell TID written in an extremely vague fashion.  It discusses creating a registry entry and listing out the servernames and IP addresses hosting SMB shares.  It was immediately obvious where to put this new registry entry, so I had to search deeper and found a much better explanation on the Novell Forums.  Here is a screenshot of the registry fix:

I tested this registry hack on select machines throughout the company with a 100% success rate.  Using Configuration Manager 2007, I pushed out a VB script to make the appropriate changes on every Windows XP machine in the company.

Another step away from Novell and another step closer to Microsoft.  /sigh…

It’s an unfortunate event to watch a once-stable environment slowly fall apart.  We’ve been using Novell for as long as we’ve had file servers and only with the introduction of a formidable combination of events have we arrived at the decision to part ways with NDPS and NetWare for good.  The combination of AutoDesk AutoCAD, Microsoft Exchange 2007, and Windows Vista rendered Novell incapable of providing the file server performance we have come to know and love.  

To address how the introduction of newer version of AutoDesk software have driven us away from Novell, AutoDesk now explicitly states that AutoCAD is designed to work ONLY with Microsoft networks.  Any other network-type is considered 3rd party and thus is not supported by AutoDesk.  Judging from the list given by AutoDesk, this specific shift in support occurred exactly with the introduction AutoCAD 2009 products.  

Exchange 2007, an absolutely enormous undertaking, allowed us to implement a new backup technology to relieve stress on our SAN and reduce the raw data size of most users’ inboxes.  The introduction of iPhone 2.0 firmware also gave us an opportunity to explore viable alternatives to BlackBerry handhelds.  To make matters more interesting (or complicated) we decided to pursue an advanced Exchange backup system to allow us to migrate old emails to another device and off our SAN.  The archiving policy, a C2C Systems product, requires that users who wish to search their archived emails have a specific client installed.  The client, as I’ll show in detail later, maps a share on the Exchange server.  In order for the archival policy to work, the Exchange environment had to be 2007-native; in other words, no hybrid 2000/2007 environment would work.

With the demise of a certain in-house software package and the healthy circulation of a buzzword, or really an acronym, “BIM”, the need for a “more advanced” mechanical systems software was pushed through without any sort of IT involvement.  This strategic shift (and subsequent purchase of $50,000+ in software) required Windows Vista and an enormous 16GB of RAM to function with any sort of usability.  Fortunately for us, Novell had gotten around to writing that 64-bit client many, including myself, had complained about.  The main problem?  It causes share access performance to absolutely plummet.  The client made life a living hell for any engineer routinely accessing files on a Novell share from Vista.  After talks with Novell engineers, a fix wouldn’t be available until the next major release of the client, which could be as long as a year from now.  

Having sacrificed many man-hours in the investigation of the Novell share performance, $100K+ in new software and hardware, and who knows how many bottles of headache medicine, I was given the word to set up a Microsoft file server (Windows 2003) to experiment with for CAD file storage.  The results were phenomenal.  As soon as the rest of the engineers got wind of the share, EVERYONE wanted access as soon as humanly possible.  A massive migration occurred as a result and now upwards of 75 employees interact with the Microsoft share on a daily basis.  We even managed to teach users how to take advantage of VSS to restore files — a relief after losing Novell’s salvage ability.

The timing couldn’t be worse for such a transition.  We’ve had no time to research alternatives to throwing all of our eggs in the proverbial Microsoft basket and my apprehension grows every day.  The most daunting task of this migration (the entire migration process will be a part of my blog) is restructuring, or at a minimum mirroring, the complex permissions in Novell NDS.  I do believe that the easiest task will be setting up printing.  NDPS has not been fun to maintain, especially relying on ConsoleOne and NWAdmin in lieu of iPrint, and even more cumbersome to configure for users (administrators are the only users who can see the NDPS tree and subsequently add printers to a workstation).  Although I really wish we had time to investigate alternatives, I’m not really sure there are any worth pursuing.  I don’t trust any Linux distribution without enterprise support and while Apple has served us well as Wiki software and as a Splunk server, their support for non-creative professionals is really non-existent (c’mon Steve, I know you have ONE MORE THING for us).  

RIP Novell.

Just trying to pass the message along.  Apple’s recent move to remove all matte displays from its notebook line has left me with a sour taste.  I was looking forward to updating my notebook but I guess I’ll be waiting for a new Mac Pro instead.  If you’re interested in signing a petition for Apple to offer matte displays, check this one out (6000 strong so far…).

For the past few weeks, during the transition from Exchange 2000 to Exchange 2007, we’ve experienced a number of complaints of slowness in Outlook.  Although the symptoms could be attributed to literally almost anything, we evaluated every possible option:

1. Exchange 2007 server issues: does it have adequate RAM, hard drive space, processing power, etc.?
2. Client issues: what version of Outlook are they using?  Is an add-in causing the perceived slowness?
3. Is it a result of a recent update to a file server, which was made public, and thus the slowness is not real but just a result of information the end user is applying to a different situation?

Today we learned something about our friend, C2C, which has been actively archiving nightly.  As it turns out, C2C is eating gigantic holes in our mailbox databases which cannot be fixed with online defragmentation.  It would have been nice to know this going into the whole upgrade, alas, the information was not available to me.

I moved myself to a mail storage group that contained only one mailbox database.  Its function is to hold accounts until we have backed up all of their personal files to portable media.  At any rate, it once contained mailboxes but they have since been purged from Exchange.  After I added my little mailbox (~1.3GB) the total size of the mailbox database was 20.4 GB (22,004,252,672 bytes).  I made a quick backup using NTBackup and then dismounted the database.  Using the Exchange Shell I ran an offline defrag (eseutil /d %database_name%) and it took a little over two mins to complete.  It’s recommended to perform a backup immediately following defragmentation, so I performed another NTBackup.  After the process was complete, I found that the newly defragmented database was only 1.32 GB (1,423,548,416 bytes)!  That’s a 65% reduction (if my math is correct, haha)!

I plan on performing the same defragmentation on all of our mailbox databases in an attempt to eliminate all complaints of Outlook slowness.

I know I’m just a simple computer science major, but does every database seriously operate in such an inefficient manner?  Why couldn’t online defragmentation take care of the 19GB of whitespace?

At any rate, we’re almost there.  I’m still migrating public folders nightly but their numbers are dwindling.

My first attempt at using the Exchange Power Shell scripts for migrating public folders did not go so well.  Thankfully, this was performed on a secondary Exchange server and not the primary legacy server.  For reasons still unknown, public folder instances on the legacy Exchange server persisted even after running the somewhat scary ReplaceReplicaOnPFRecursive.ps1 power shell script for all of the public folder hierarchy.  To work around this problem, I simply migrated using the old-fashioned (and tedious) method of adding the 2007 server as a replica to every public folder listed on the legacy server’s public folder database.  After my end-users were satisfied that all of their data had replicated (again, this required an “update” for every public folder my 2007 server could see in the SP1 public folder tool), I removed the legacy server from the replication list using PFDAVAdmin.

This evening, I discovered an undocumented quirk in the RepliceReplica script that may have led to the problems I experienced with the first legacy server — though it requires a path for the top-level public folder, if the path has a space in it, you must use an additional set of apostrophes.  I don’t know why I didn’t think of this the first time around, but the process seems to be running smoothly at the moment.  I’m still very skeptical of running the script against the entire public folders folder, so I’m running it against each subfolder individually so I can literally watch the public folder instance disappear from the legacy server.

So it’s clear, to move the top-level folder entitled “A/P Staff” from my legacy server to the 2007 server, I had to issue the command as follows: 

[PS] C:\Program Files\Microsoft\Exchange Server\Scripts>.\ReplaceReplicaOnPFRecursive.ps1 -TopPublicFolder “‘\A/P Staff’” -ServerToAdd %2007Server% -ServerToRemove %LegacyServer%

Luckily the Event Log, with almost verbose logging enabled for the Exchange 2007 server, still reports Replication Incoming/Outgoing 3028/3027 messages.  The legacy server eventually removes the public folder instances once the replication is complete.

With one legacy server completely off and zero complaints thus far, I will be removing it from Active Directory tomorrow, fingers crossed!

Of the 2 legacy Exhange (2000) servers in my organization, I believe one is ready to be completely removed. Public Folder migration was completed and verified earlier this week after refreshing (updating as the 2007 Console refers to it) the public folder hierarchy on the new 2007 server. Luckily, this particular legacy server did not host an offline address book or any other auxilary services.

As a test run, I dismounted the public folder store and the lone mailbox store and so far haven’t received any complaints. The next step will be powering it off before completely uninstalling Exchange 2000. Wish me luck!