Your basic ITPro blog... What's going on at work, what I'm interested in.

Wednesday, June 17, 2009

File Store saga

So, we had an issue with a Dynamic disk in a VM. This disk would not be active after a VM restart. I had to manually reactivate it. In doing this, my shares and ABE settings were lost and had to be reset.

After some research, I found that there were two options to fix this that did not include backup-rebuild-restore. These options are:

  • Attach disk to IDE rather than SCSI.
  • In Registry, change the 'START' value in “HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\storvsc” from 3 to 0.

The first option seemed the ‘better’ choice, as it is just using the tools/software, rather than reg-hacking. So, that is the option I went with first. But, our disk size ended up causing problems with this. We experienced some file corruption (thank God for backups) and ended up having to move the disk back to SCSI. So, it was off to Plan B…

… which worked perfectly. After changing the registry value and doing a couple of test reboots, everything looked good and stable. Then, we just had to ‘clean up’ our corrupted files. Users a re still trickling in with files that can’t be opened. But, a quick restore from our pre-problem backups is fixing things in most cases.

WHAT I LEARNED:

  1. Verify and Clarify! Do your research and develop a plan. Then, verify that plan, not just the steps/technologies/ideas, but the actual plan! Run through it one more time. Get one more pair of eyes on it. Verify that the actual steps you are planning on taking are solid. My discussions on this topic lead me to believe that moving from SCSI to IDE was the best approach, but I didn’t run my actual plan by other engineers. I am confident my flaw would have been caught had I done so.
  2. Take precautions! I could have/should have taken extra precautions before executing my plan. I had recent backups, but not up-to-the-minute backups. Should have done that. Is it too much to ‘expect’ failure and prepare accordingly? Maybe not…
  3. Be thankful for the Grace of God found in His people! My co-workers were/are awesome! I am humbled and grateful for their understanding and grace during this ordeal.
  4. Don’t rush. I has anxious to get this fixed. And, because of that, I rushed things. Oh, I didn’t feel like I was rushing things at the time. But looking back (isn’t hindsight great?!) I see now that I should have taken more time to contemplate this issue. Overconfidence? Perhaps…

Also, as a result of this, we made some changes to our DR plans… Specifically, we increased the frequency of our file store backups… from once a day to every six hours.

No comments:

Additional Info

My photo
email: support (AT) mangrumtech (DOT) com
mobile: 480-270-4332