Here we go again!

ive been asked to put together a document outlining possible solutions and improvements to my companies backup system...

Currently we run mainly use Veritas Netbackup 6mp4 hooked up to a HP MSA6000 tape library with 58 slots and 4 ultrium1 drives.

the problem is we are currently using 12 tapes a night, to back up about 80 servers, exchange, sql and *ix boxes and are at the absolute maximum backup window we can afford to have, so money needs to be spent to win us back some time and storage capacity!

i think at a given we will upgrade the library drives to ultrium 3 which will give us a big capacity and tape speed boost.. but there have been mentions of utilising disc based backups etc to speed things up further and give us some flexibility to when backup jobs can run... Netapp has been mentioned..

I know theres alot of IT bods on here, so what kind of backup strategy/hardware do you use? straight to tape or disc backup first?

i also found out today that we have a DR trial in 2 weeks where we go offsite and try to recover a bunch of servers :rofl: its going to be a double edged sword really, we will get it in the neck because our backups are crap (although we can argue the case), and also itll be highlighted how outdated our backup systems are and hopefully pursuade them to invest in an upgrade

You need something like what we are looking at implementing,

A system that basically backs up every server to a big server with lots of hard disk space, which is then in turn backed up to tape during the day. This has a number of advantages, the overnight backup is very quick, especially if you have everything connected via gigabit lan, and you can keep several backups on disk at the same time which makes restoring from the last couple of backups quicker because no tape needs to be put in. I think the system we are looking at is provided by CMS peripherals but cant remember what it consists of exactly. As a guestimate we currently backup 36 servers, and it was estimated that the whole lot would fit on 2 LTO3 tapes.

edit. your avatar is vey confusing. I thought I had an insect on the monitor, until I tried to remove it, then thought it was stuck in the panel until I moved the IE window and it moved to.

I'm in the process of looking for something like this myself.

I'll let you know what I come up with

What i can say is that Netbackup natively supports backups to disk, and also using disk as a staging area... (with either automagic or manual backing up of the staging area)

It also supports NDMP hosts (licence required of course ) which would allow it to utilise a Netapp filer amongst other things (which we already have, but another disk tray would cost 15k+ so is probably going to be thrown out the window pdq)

the problem here is going to be the classic "dumb management" and the IT director is notoriusly snide at spending money.. sobecause the backups seem to be working fine according to the reports it will be difficult to make him see the light..

at the moment the easiest (and prob cheapest) way ive found would be to get an entry level MSA SAS/SATA storage array and hang it off the back of the backup server for disk backups

Manny: yup... i found it on another site in someones signature which thew me for a while too - you should save it and stick it on someones active desktop and see how long it is before they notice

To be fair backups are only as good as the tape changer / Iron Mountain / system administrator / new guy , who has been given the duty of managing the backups. Take the human being out of the equation and you are on to a winner. Think about backing up to remote onsite storage. There are plenty of companies out there that offer this service, they do the planning and decide which software to use. The big costing would be a link to there datacenter and of course the fee. Hopefully should free you and your colleagues up and let you sleep at night.

Best of luck!

In answer to your question. Hardware is always HP/ Compaq / Dell / Sun all the big names so that you have someone to fall back on should you have problems. Software backup strategy at most places is tape man running around ejecting and loading tapes late in the day for overnight backup. Most big corporates nowadays prefer the solution above although they would backup directly to there own data center offsite.

Don't bother with LTO3, go straight to LTO4. It'll increase your capacity eight-fold.

We're currently using LTO3 and shifting about 8TB (compressed) a week through a 2-drive jukebox.

Pretty much at convenience/time capacity now, so looking at LTO4 and will be investigating disk based solutions too. We've toyed with disk solutions in the past, but it was a few years ago with cheap PATA disk systems. Performance wasn't great. Should be a lot better with SAS.

Don't bother with LTO3, go straight to LTO4. It'll increase your capacity eight-fold.

We're currently using LTO3 and shifting about 8TB (compressed) a week through a 2-drive jukebox.

Pretty much at convenience/time capacity now, so looking at LTO4 and will be investigating disk based solutions too. We've toyed with disk solutions in the past, but it was a few years ago with cheap PATA disk systems. Performance wasn't great. Should be a lot better with SAS.

I think LTO4 is still a bit expensive and probably overkill for our enviroment... 4.5k for the drive and £80 per tape compared to 1.5k and £25 for LTO3 - plus as it stands currently all our backups would just overrun 1 LTO4 tape a night (assuming 2:1 compression) making the MSA library redundant. plus with the speed of a LTO4 it would be pretty difficult to keep the drive streaming on our infrastructure. we have quite a few "less than ideal" G2 and G3 too servers that struggle to get more than 6mb/s across the lan.

Are you thinking of just swapping the drives in the jukebox? How does that work out cost-wise?

I ask because a few years back we had a small fire and our LTO2 jukebox was written off with smoke damage(*). We wanted to move to LTO3 but there were only standalone drives on the market at the time. So we tried to get quotes for an LTO2 jukebox with upgrade to LTO3 when available. None of the suppliers would do it. They all said it would only be pennies more to get a brand new jukebox when the time came. And they were right.

(*) As our tape drives were all fecked we had to invoke our rapid response disaster recovery contract to bring us replacement kit... which duly arrived in a fast red Octy 1 RS

As a bit of an update on this... ive not got very far

I think we are swaying towards some kind of Virtual Tape Library (VTL) solution, then it will intergrate with our current netbackup enviroment.

the things im still trying to find out are details about offsite mirroring, recovery and other options... its all very complicated

We use Netbackup (still ver 5.1 at the moment) and back up to tape, but also use disk staging. At our main site we have 2 media servers, one with a 4 drive LTO2 tape library, the other a 2 drive LTO2 tape lib. This worked well, but the amount of data and servers to be backed up just keeps growing. We had some spare SATA disks on the SAN, so we started using 4 TB on each media server as disk staging areas, alongside the tape drives, to do the backups with scheduled jobs to relocate the disk backups to tape in the morning. This made a big difference to the throughput we got, and made restores much easier for those on disk. Ideally we'd want to disk stage everything, but that would use a lot of space:eek:. We thought about a VTL, but at the time - a few years ago - it made more sense to use the SATA disks because we had them and they were cheaper per Gb. Netbackup licencing was an issue as well. Disk staging was free. Licences for VTLs depend on their capacity. We also considered LAN-free backups, but Netbackup licencing was going to cost too much.

For offsite backups, we simply send the tapes offsite to another office and vice versa. I think the netbackup vault option will help with this (at a cost), but we just wrote our own scripts. Do we sound like cheapskates yet?

If doing it again now, I think a VTL would be quite a likely option along with some Information Lifecycle Management capability. If only we could get users to delete those files from 10+ years ago that they never open, don't need and will never actually use again we could go back to using QIC tapes for the backups:rolleyes:

The next step will be replicating or mirroring data to a remote site. Tape shuffling can be a nightmare if it isn't carefully managed and carried out properly.

I'm told that with ver 6.5 licencing models have changed (but I don't know if that's good or bad) and they also seem to be moving away from tape.

I don't know if this helps, or just adds to the confusion.

Well im starting to make progress now..

VTL is definately the way, because of its ability to augment our existing enviroment yet can be scaled up to multi-site/DR replication. (part of our medium/long term plan)

Theres a wide range of products varying from Falconstor, which is software so can be flexible on the hardware configs and used as OEM by HP, IBM, EMC etc to Netapp which is as expected.. uber-expensive lol.

The one im looking into in more detail is the Quantum DXi series which seems to tick all the boxes we want.. (it also uses de-dupe technology which DRASTICALLY cuts down the actual storage needed for a backup.) think of it as a cross between a byte level incremental backup and single instance storage. it also intergrates well to netbackup, which is where some other products fall over when phyical tape backups/archiving are still required. the DXi allows tape-duplication to be handled by nbu so any physical tapes written will be in the nbu database. i know its a minor thing but it helps keep the audit trail nice and neat.

The scalability is ideal too, because we currently have 3 sites with totally separate NBU enviroments whereby manual tape-changing introduces a large factor of "inexperienced user error".. we would be able to install VTLs in all sites and have them replicate to a central/DR location

Another related question.. regarding backup/DR

What kind of DR do you use in your company? do you have any special DR backups/images you have in case of a DR scenario?.. eg things like Netbackup bare metal restore.. things that would allow full systems to be recovered onto different hardware etc?

As a bit of an update on this... ive not got very far

I think we are swaying towards some kind of Virtual Tape Library (VTL) solution, then it will intergrate with our current netbackup enviroment.

the things im still trying to find out are details about offsite mirroring, recovery and other options... its all very complicated

VTL you say... Can help you there Give me a kick on messenger.

Although I will say make sure that the VTL is in a secure off site location and still do backup to tape sent to another location every so often (IMHO of course)

You can always consider a PetaSite

I can help you on that front

Choice of S-AIT2 or LTO4 Ultrium Also has one of the highest densities of data. Is 2.4 PB large enough for your stash or pron?

2.4PB - Pahhh :P

For DR testing, I take the time consuming approach.

Install O/S and backup software, configure machine roughly the same as the original.

Perform a full restore, including registry

Reboot (usually blue screens due to different H/W)

Perform a repair install of the O/S to fix the drivers,

Voila, it usually boots fine after this.

For DR testing, I take the time consuming approach.

Install O/S and backup software, configure machine roughly the same as the original.

Perform a full restore, including registry

Reboot (usually blue screens due to different H/W)

Perform a repair install of the O/S to fix the drivers,

Voila, it usually boots fine after this.

Hmm.. the "proper" way

our current way is similar, except when we restore we skip a couple of files (NTLDR, hal.dll kernel etc etc) which means the machine will usually boot up without needing a repair install.

we are investigating quicker ways of doing it.. BMR, Acronis true-image etc and maybe even VMware images, so its a case of chucking image onto a box, and then just dropping the up-to-date data onto it.

but all this relys on good documentation and a watertight change-control framework in place, which we dont have...

Acronis stuff is good, but don't forget to run sysprep first on the image so that it will boot up as a first time install and get the correct drivers.

That means you don't have to restore onto the same hardware in the even of something fatal to the system such as fire, flooding or theft etc.