Skip to Content
8-26-2011 @ 2:15PM
I definitely think the question re: catastrophic failure is a bit of a moot point like you mentioned, Mat. There's no way that Blizzard doesn't have an absurd amount of redundancy built-in so that even if a nuke hit a data center/server cluster they would still be able to repair the damage. Plus, armory data is probably on a separate storage center than the servers themselves, so they could probably scrape that to at least get a patchwork rebuild.
8-26-2011 @ 3:23PM
I agree, which is why I basically said "it can't happen." But! What if it did... and so forth.
8-26-2011 @ 3:50PM
The thing I'm more interested in is what they will do as they attempt to restore the data. I work at a casino and 6 years of its history takes quite a bit of time to gather and back up, much less restore from and that's just the gaming information. I can't even imagine how much goes into even one realm.If the data center in Chicago, for example, was destroyed, it would take days to get back up and running and that is based on an unrealistic backup plan. In truth, it would take weeks just to get sufficient amounts of sufficient hardware, not to mention contract agreements with another datacenter, etc.Then again, if any of the datacenters that host WoW are utterly destroyed, we would lose a lot more than just WoW. I think everyone would be surprised by just how much.Back on topic, what would Blizzard do to appease those people as they work on the problem?
8-29-2011 @ 8:30AM
I work in disaster recovery in IT so I can shed some light here on icepyro's points - and the reality is, you're just plain wrong :PYou don't restore servers from backups in high-availability and DR protected environments. The essential setup will be something like this: for each server in any WoW datacenter there is another server in the same center that is continuously synchronized with it (whether it's a database server or a whatever doesn't matter) - that's the HA (high availability server) - in wow's case there could even be three or four of them per server - at each site.These servers do not serve users, they just sit there, synchronizing everything that happens on the live server every second of every day. Other servers refer to the live server by a name, NOT an IP address, should that server go down for any reason whatsoever, an automated system immediately updates the IP that the name refers to - sending requests to the next server in the qeue. No backups, no restores, just instant fail-over to a new server that is already completely up to date.So that's HA. The next level is DR, in most setups DR is manual but it works in much the same way. Somewhere else - on a different continent usually (and we know WoW has datacenters in Europe as well) there is a clone of the entire setup as it stands at the main one - with a replica of every single server, including the live one - and these synchronize all the time as well. Now it's true that international transfers take longer, so the "live" DR server may be an hour or so behind, but not more than that - and the HA servers on that side sync to it again. In the event of a catastrophy like a natural disaster wiping out the chicago datacenter, a fairly simple process can be initiated that lets the European servers take over, you may lose an hour or so at worst (and in WoW's case I doubt it will be more than 10 minutes if anything at all).Far smaller companies have DR setups that can do a complete fail-over to another continent in under 6 minutes, I would be surprized if WoW can't beat that.I am NOT saying this is blizzard's setup - I'm explaining to you what is done as STANDARD with critical data in important datacenters everywhere in the world today. Blizzard undoubtedly employs something similiar but I would be surprised if they didn't have some additional levels of protection.Now it's true this isn't ideal, if a tornado knocks out the entire chicago DC and they have to let their American susbscribers fail-over to replica servers in the EU then there will be significant lag involved until the Chicago DC is back up and they can revert again - but that's better than a complete loss of data and an utter inability for anybody to access your servers (something you are PAID to provide them).Now I want to say to you - if the Casino you work for does NOT have complete off-site replication with the ability to fail-over all servers with under 10-minutes of downtime in a disaster without having to restore a single backup then either your system administrator or the people who write his budget is incompetent.
First time? A confirmation email will be sent to you after submitting.
Members enter your username and password.
Enter your AOL or AIM screenname and password.
Please keep your comments relevant to this blog entry. Email addresses are never displayed, but they are required to confirm your comments.
When you enter your name and email address, you'll be sent a link to confirm your comment, and a password. To leave another comment, just use that password.
To create a live link, simply type the URL (including http://) or email address and we will make it a live link for you. You can put up to 3 URLs in your comments. Line breaks and paragraphs are automatically converted — no need to use <p> or <br /> tags.