Preventing a Disaster: My Methodology Step 2

In my previous post I introduced My Methodology in Disaster Recovery along with Step 1: Integrity Checks.

Today, let’s talk about Step 2: Backups; but first we have to answer 2 questions.  How much data can are you willing to lose and how long are you willing to be down after a disaster?  In this line of business we call these Recovery Point Object and Recovery Time Object (RPO/RTO).  

Recovery Point Objective

RPO is the “line in the sand” where you say, I can’t lose XXXX amount of data.  Of course this line is determined by what type of data it is, how often the data is changed and is it OK to lose any of the data or can the data be rebuilt.

Example: A hospital would not want to lose any of their data.  I would like to believe my medical history is safe and I do not have to keep my own copies.  This type of data is crucial to keep, so basically acceptable loss of data is Zero to none! This type of RPO usually requires High Availability Disaster plans which is outside the scope of this article, but lets just say you will take many, many transactional log backups!

Recovery Time Objective

RTO is the “line in the sand” whey you say, “I can’t be down more than XXXX amount of time”.  How long does it take you to get a restorable server up and running. Again, the type of data determines this as much as anything else.

Using a previous example of a hospital:  electronic medical records could possibly be down a little while because there are such things a paper medical records that can be scanned/typed in later. It is not ideal situation; but hey following a disaster I would be willing nothing is working ideally!

Backups

Backing up your databases will (not may) save your job one day, it has mine! The most important thing to remember about backups is to ensure that they are done.  They second most important thing to remember about backups is to ensure that they are “verified”.  And finally ensure your backups are valid.

First let’s talk about methods of backing up databases. Then I will discuss my thoughts on where to put backup files.

There are two main ways to backup databases: Native and 3rd Party tools. I hope to break down the pros and cons of each.

Steps of the Backup Process

1.  Perform the Backup – using what ever method your company uses; make sure the backups are being done.

2. Verify the Backup — SQL provides a way to verify the file written to the drive is verified against the live data file. This process basically ensures what was read was truly written to the disk.  This doesn’t verify IF your data is valid; that is where Step 1 of Disaster Recovery Integrity Checks.

USE master
GO

RESTORE VERIFYONLY [MyDatabase]
	FROM  DISK = N'C:\Backups\>MyDatabase.bak' 
GO

BackupVerify

Backup Methods

Native SQL Backups

Native backups are built into SQL Server, i.e. no further costs.  SQL Server is expensive enough you shouldn’t have to spend more money to just backup your data.  Native Backups (NBs) have been around since at least SQL 2000, that’s the beginning of my SQL experience.  NBs functionality has grown up right along with SQL.  The ability to write to network shares, ability to compress, ability to verify are all options that make NBs an excellent choice.

If your SQL Server is experiencing CPU pressure, you may want to reconsider using compression.  The compression is actually done with CPU cycles before writing the data to a file; so using compression will increase your CPU usage during your backup window.

3rd Party Backup Tools

Prior to SQL Server 2008 R2, I could make an argument for using 3rd Party Tools because of the importance of saving space with backup compression.  However, since 2008 R2, backup compression is an available feature in all but Express editions.

Some 3rd Party Tools, are nothing more than a windows service that manages backups.  Some 3rd Party Tools are actually a windows service that will perform proprietary backups these are the ones that usually provide a variety compression levels to choose from.

If your SQL Server is experiencing CPU pressure, you may want to reconsider using compression.  The compression is actually done with CPU cycles before writing the data to a file; so using compression will increase your CPU usage during your backup window.

Your choice of methods is completely up to you and your business requirements.  I personally see no reason to spend more money on a product that another product already does. My method of choice is Native Backups.

Location of Backups

This is always a hotly contested arguments.  Do I backup my files to a local drive, attached SAN drive, network share or cloud?

To start with, I think the majority of the SQL Community will agree that the faster you get the BAK and TRN files off of the SQL Server the better off you will be.

Using a network share as your backup folder destination is just as valid as using a local drive or an attached SAN drive or heaven forbid an attached external USB drive. Remember, the destination should not be the same drive that your MDF and/or LDF files are on.

The real point of the discussion of Location of Backups should not be where my backups should be stored; but what is the final resting place of those backups?  Does my company have a plan to move the files off site to a secure location?  Is that plan automatic or manual?

Once your daily backup routines are complete, there should be a process in place to archive these flat files off of the SAN and relocate them off site. I have seen everything from an external USB drive carried home to SAN replication to a completely different data center on the other side of the country.  Either method is functional in getting the backups off site.

One very reasonable and possible location is cloud storage.  This concept, if enough space is purchased, kills the preverbal “two birds with one stone”. It not only backups up your files off of the same drive as your live data but it also by its very nature ensures that the backups are indeed offsite.

My Personal Ideal Scenario

My personal opinion is to use native SQL Backup with Compression and write the files to a network share, preferably a different SAN/NAS than your data. This does two things: 1) compression reduces the size of the backup files going across the network and 2) physically writes the files off of your SQL server as fast as possible and allows your immediate backups to be stored on a different physical drive than your live data.

Some will argue and say “it will blow up my network”; however, with the current specs of modern NIC cards and routers, it is highly unlikely that you will cause network issues.

Offsite Archiving

The most difficult part of this planning is how to get the backup files offsite.  Replicated SANs can be VERY expensive, archiving to Tape is an option, cloud storage is available as an option or just copying files to external drive. The thing to remember is the amount of time it takes you to get the files to their final resting place is the amount of time it takes to recover them (RTO).

I believe as cloud storage gets cheaper and cheaper (except for you Microsoft), the feasibility of backing up directly to the cloud will be more and more popular.

Bottom Line

Know your RPO/RTO of your company, have a plan to accomplish those goals and know how to make that happen in the cheapest, fastest, safest way possible.

Preventing a Disaster: My Methodology

I was hoping a career in consulting would possibly spark blog ideas! 

disaster-recoveryOne thing I am passionate about with SQL Server is Disaster Recovery.  Having worked for a hospital during hurricane season as a DBA, I truly had some sleepless night wondering if I could bring a 2nd system up successfully in the event of a total disaster. Medical data is of the utmost importance when it comes to the field of health care.  And I am sure, everyone has the stance that “their data is the most important data”!  That is why you must protect it at all costs.

Either way, as a DBA it is my job to be able to stand up a 2nd server as soon as possible in the Desktop-Disaster-Recoveryevent of a total disaster.  If it was only as easy as pushing a button, our job as DBAs would be much easier.  Unfortunately though, proper disaster recovery requires fore thought, planning and testing.

I have a 4-fold plan of Disaster Recovery and I would like to discuss my thoughts on this.  Part one will be discussed here with later parts in subsequent posts.

  1. Integrity Checks
  2. Backups
  3. Off-Site duplicates
  4. Recovery Testing

1. Integrity Checks

Most everyone is aware of DBCC CHECKDB commands, and they are vital to the stability of your database.  If you are not familiar with them, then please for the love of the SQL Gods learn about it!

It is of utmost importance to run these as often as feasible.  I typically run integrity checks once a day during non-business hours or downtime. This process will be resource intensive which is why it needs to be done during downtime.

Now some of you are going to say, I tried running DBCC CHECKDB on my 350 GB database and it brought my server to its knees so I stopped doing them. All I can say is I hope your data pages are not corrupt.

Per MSDN DBCC CHECKDB does the following:

  • Runs DBCC CHECKALLOC on the database.

  • Runs DBCC CHECKTABLE on every table and view in the database.

  • Runs DBCC CHECKCATALOG on the database.

  • Validates the contents of every indexed view in the database.

  • Validates link-level consistency between table metadata and file system directories and files when storing varbinary(max) data in the file system using FILESTREAM.

  • Validates the Service Broker data in the database.

Now that’s a whole lot of checking!  If CHECKDB command does all these commands, then possibly we can shorten the duration by manually executing CHECKALLOC one night, then CHECKTABLE another, and maybe CHECKCATALOG a 3rd night.  It is a thought.

There are many options to integrity checks that can shorten the execution time.  Many blogs posts by Paul Randal, Aaron Bertrand (to name a few) have written many articles about DBCC CHECKDB and how to effectively use the different options and actually the different commands of DBCC to shorten the duration of the integrity check.

The one thing to remember here is if you databases is corrupt, so are your backups!  SQL Backups are only copies of what is in the databases, if the data pages are corrupt so is your backup!  This is why it is impetrative to regularly perform integrity checks on your databases.

One of my favorite methods because I can script it out right after Step #4 of My Methodology is off-load integrity check.  If your databases are too large or too busy to do checks in production, after you have test restored your backups (hopefully you are doing this), that is an ideal time to run DBCC CHECKDB.  The restored database is ideally on another server that will not have any impact on users if you run the DBCC CHECKDB commands.

In my next post, I will discuss my thoughts on SQL Backups: discussing native vs. 3rd Party and local vs. network backups.

Oops, I did it again!

No, I am not talking about Brittany Spears; one blog post could not possibly define or explain that child!

I am talking about SQL User Groups!  I opened my mouth for a brief second and sure enough I was shocked and surprised by what came out of it.

“I will present at the November meeting.”

What the heck did I just say? What did I just volunteer my self to do? I really need to learn to take my wife’s advice.  “Just say NO”  LOL

Any way, on to picking a topic to present. One of the most intriguing things for me as a SQL DBA is performance tuning. The ability to push the limits and make queries go faster!  I get excited when I figure out how to shave milliseconds of off of a query.

For me the best way to approach performance is to try and develop your queries and indexes correctly from the start.  My presentation, “Indexes and Execution Plans”, will begin with a brief overview of indexes and execution plans; how to read them and to use them to your advantage in development. I will then give code examples on how to use execution plans to determine the best need (if any) for indexes.

See you all on November 11th at the Louisiana Tech Park on Florida. customLogo

My presentation and code will be available after the user group meeting.

It’s a new dawn, it’s a new day

On Monday, September 14th, I began a new chapter in my career as a SQL DBA Consultant for a local IT Service/Consultant firm here in Baton Rouge, LA.

After almost 10 years of being a “jack-of-all-trades/master-of-none”, I have decided to concentrate my efforts on DBA work and do my best to help others.  Over the last 10 years, I have been a SharePoint Admin, .NET web developer, .NET desktop developer, a BI Analysts, a SQL Developer and a SQL DBA. It was time to dedicate my career.

The company I am working for now believes in the concept of “transferring knowledge”; not just fixing their problems.  I like that approach and I like the idea of something different everyday!  Hopefully, this transferring of knowledge translates to more blog posts!

The concept of “billable time” is definitely something to get used to; but aren’t there new things about every new job?