I’m part of a small I.t. dept for an engineering firm. While I’m mostly a software developer I wear many other hats due to the small size of the company. Today I’m wearing the hat of I.T. manager stressing out over backup restoration.
The environment we have here in the office is this:
– 2x fairly new physical servers with VMWare ESX (free version) installed
– 3 main windows server 2008 r2 VM’s on those machines
– the VM data is stored on a Netgear Readynas and opened on the physical machines
– we use Shadow Protect server to backup these windows vms onto a second readynas which simply holds the backup images.
– gigabit network connection between all this
We originally had 6x 500 gb drives in X-Raid2 config on the readynas, but we rand out of space recently so we purchased 1tb of the same model. We’d been slowly pulling 1 drive out, putting another in and letting it resync. We had done this for 3 of the drives, without issues, and then on friday we added the 4th. It resynced for 30 mins then we lost all connection to the NAS. On rebooting we couldnt get acess to the volumes even though teh ‘recovery’ process seemed to run ok. after much time on the phone with netgear support, they basically told us to get our data back from backups.
These VMs are as follows:
1. Rafter – File Server – 1.7tb in size
2. Venus – IIS/Domain controller – 100gb in size
3. Serena – Exchange – 400gb in size
In order to recover the data, I’ve picked the best machine in the office, put 3x2tb brand new drives in RAID3 inside it, and installed FreeNAS. Created a NFS share and connected to it from our ESX installation on our physical servers. I create a new VM on that share, boot into recovery environment and connect to our backup NAS to restore the image (this is a shadow proteect SPF image, and in the recovery environment I select to install it to the virtual partition). This worked fine for Venus, which restored in an hour and a half. It looks like it will be ok for Serena, which while taking nearly 12 hours, should still work, however the 1.7tb from Rafter is the trouble, its a huge amount of data, and it goes very slow.
The speed for restoring these images, for all the ones I’ve tried, is around 5-7MB/s. it starts off slower but builds up to this. I’m not sure why its this slow, but I think it must be shadow protect thats the bottleneck, because if I run 2 restores at once, they both go at 7MB/s instead of halving?
As the rafter restore will take another day and 12 hours, meaning our company will be down for a whole day of work, I decided to try recreating the Vm on our other server from scratch, installing windows server 2008 r2 and booting into it, then connecting a USB drive to the server with the shadow protect file,mounting it using shadow protects windows based mounting features and robocopying the contents. This was just as slow if not worse! After 5 hours it has done barely any data.
So my question to you, is how can I get this data restored faster? Even if it’s just selected folders from the backup (ie I can pick the folders that are our active jobs, which will significantly reduce the size, but at current speeds with the USB method its still not going to make it)
Ps I realize there are probably steps that could have been done to predict this, but that’s too late now, it’s not my field of expertise and we setup this system with the help of a 3rd party firm, I just need your assistance getting this at least partially running asap.