Maintenance extended: 5:00 p.m. PST
by Adam Holisky 
Nov 23rd 2010 at 2:36PM
So sit back, enter our contests today, and catch up on everything that was and will be.
This post will be updated and bumped up to the front of the site as the situation changes.
Updated 11:40a.m. PST: Maintenance has been extended until 5:00 p.m. PST.
Filed under: News items
Reader Comments (Page 1 of 17)
Paul S Nov 23rd 2010 1:16PM
Quelle surprise! :-p
Paulio Nov 23rd 2010 1:18PM
^ This.
mansa1975 Nov 29th 2010 4:05PM
.....
Tuttutt Nov 23rd 2010 2:49PM
They have moved it to 5pm pst.Sorry to hicjack a comment just thought i would send it along.
Aedilhild Nov 23rd 2010 2:46PM
Whenever I read the phrase "additional complications," I think of the Millennium Falcon setting off Echo Base's docking bay fire alarm. "Turn it off! Turn it off!"
Bart Nov 23rd 2010 2:58PM
THEY WERE NOT PREPARED!
bjoker1 Nov 23rd 2010 3:19PM
Well... east coast got screwed again
ENOUGH! Nov 23rd 2010 3:28PM
Every time they get into one of these 12 hour plus upgrades I find myself wondering what exactly they are doing. I have about 30 years IT experience and 15 years in large distributed systems using Oracle.
Patching / imaging the actual realm server CANNOT take 12 hours. If it does they are doing something very wrong.
So it comes down the the database server(s). It has been documented that WoW is running on Oracle databases.
I have also seen that there is one big ass database server per data center, not a database per realm, and common databases where you need cross realm support like BG and instance finders.
Since Oracle is a shared storage architecture, meaning that it does not matter how many CPUs you add, they are all looking at the same hard drives...storage IO bandwidth always ends up being the bottleneck. That's the communication medium between the database servers and the database hard drives.
If Blizzard had more database IO parallelism, which can probably only be achieved by building out databases systems that are as heavy duty as the existing ones on a realm by realm basis, we would be playing right now.
Prances in Underpantss Nov 23rd 2010 3:26PM
Maybe "additional complications" is referring to the millions of people trying to log in every 5 minutes, causing their systems to go haywire.
But, I'm sure it doesn't work like that.
thebl4ckd0g Nov 23rd 2010 3:38PM
@ENOUGH! Nov 23rd 2010 3:28PM
"Every time they get into one of these 12 hour plus upgrades I find myself wondering what exactly they are doing. I have about 30 years IT experience and 15 years in large distributed systems using Oracle.
Patching / imaging the actual realm server CANNOT take 12 hours. If it does they are doing something very wrong.
So it comes down the the database server(s). It has been documented that WoW is running on Oracle databases.
I have also seen that there is one big ass database server per data center, not a database per realm, and common databases where you need cross realm support like BG and instance finders.
Since Oracle is a shared storage architecture, meaning that it does not matter how many CPUs you add, they are all looking at the same hard drives...storage IO bandwidth always ends up being the bottleneck. That's the communication medium between the database servers and the database hard drives.
If Blizzard had more database IO parallelism, which can probably only be achieved by building out databases systems that are as heavy duty as the existing ones on a realm by realm basis, we would be playing right now."
If they use Oracle as their DB, that could definitely explain why they have so much downtime, b/c Oracle is a piece of crap DB and so is most of their software. I work in tech support too and whenever we see any problems with Oracle DB's or Oracle RAC, it's hilarious. Oracle Support is also a joke.
Dendaris Nov 23rd 2010 3:38PM
@bjoker1
[Edit: 3 paragraphs of snarky, sarcastic comments about how the eastern time zone is clearly superior to the rest of the world and thus more deserving of special attention have been deleted. Will leave it at this for the sake of decency.]
Blizzard began the update process at 3 *AM* Pacific time, where Blizzard is based. That's a pretty early start to a day. They had full expectations (although many of the rest of us didn't) that they would be completed by the afternoon, ensuring that all of the east coast people could enjoy WoW by the time they got off work or home from school or whatever.
Unfortunately, Murphy's law exists, and the servers are not going to go back up as quickly as previously thought. At this point, there's no way of knowing when they'll actually be back up. It was quite a bit later than 5 PM PST when 4.01 was released, effectively ensuring that all US time zones were in fact "screwed."
Find something else to do in the meantime, be thankful that Blizzard is working hard to give us such a huge, free patch at all today, and try to be a little more tolerant in the future.
Narshe Nov 23rd 2010 3:43PM
So much for "flipping a switch" haha. Oh well, we have years to play the new Azeroth so I can wait a bit for them to make everything work right.
dru.reuther Nov 24th 2010 12:28AM
@Enough I agree, Ive been doing large enterprise IT work (very large) for a over 10 years and IT in general for 30. Its like Blizz tests he functionality of the updates but not the delivery. Since I have to guess that they have tested the code update internally as well as on the PTR the only reason for these update delays is the lack of testing of the delivery process. Where I work, an omission like that would lead to the update not getting approved and everything would have to go back for additional testing for the update process itself, its length of time, change file records (before and after) along with any other required metric to determine if the update was successful, in the amount of time expected. Its not rocket science, its due diligence. Its a linear process that should be automated and only requiring a human to initiate the task script(s) when directed.
1) Bring system off-line (end connections and prevent new ones)
2) Backup data
3) Test back up
4) Apply update(s)
5) Test/Confirm update results
6) Bring system on-line
7) Enjoy.....
Whether for 1 host or a 1000 the process doesn't change, only the scope. At a guess it looks like Blizz does too much work manually allowing the gnomes and goblins too many opportunities to use their grubby fingers. Then again, maybe they employ Taurens to do the updates, that could explain the stunned servers in the data centers.
Scooter Nov 23rd 2010 4:03PM
#ENOUGH!
Really? 30/15 years of experience and you cant see any reason why something as large as WoW would have a reason to extend the maintenance? I wont rant over the possibilities because for all we know they are throwing a party and don't want to do any work for a few more hours. I mean come on man that gnome cake isn't going to eat itself.
primal.geek Nov 23rd 2010 4:13PM
"be thankful that Blizzard is working hard to give us such a huge, free patch at all today"
Free? Not to be one of "those" people, but last I checked over the years I've paid upwards of $150 for software on top of 14.99/month. Blizzard is a company, they're not giving us this "huge patch" as a favor, they're in it to make money, they make money by making a good product.
My point isn't that it should be working now, I'm actually fine with the wait, I think it will be worth it. It just strikes me as odd when people tell me to thankful for something I've paid for.
Protocollie Nov 23rd 2010 4:27PM
@Enough
As someone working in IT, I can't believe you'd make such a newbie mistake. IO parallelism is not the only factor involved in this sort of rollout, and you know it. You have to imagine that there's more to the process than 'image and go.' Wow's a complex system, the servers no doubt operate in clusters, the 'single database server' per battlegroup is almost guaranteed to be clustered. With the sheer number of points of failure that must be involved, not to mention the fact that there's not a 'standard' hardware configuration (newer realms have newer hardware) you can be damn sure there's most likely a tedious, thorough automated smoke test of every facet of connectivity and performance, not to mention multiple failures per rollout requiring engineering attention. This is completely ASIDE from the fact that they're generally not keen on bringing down servers for extended maintenance all that frequently and so more complex hardware upgrades, growing of RAID volumes and other longer-term tasks are PROBABLY on their schedule for these patch rollouts when they have more leeway (people don't expect the average tuesday downtime to run over, patch days it's a known fact of life.)
I mean, seriously. You're acting like this is a fire and forget operation.
ENOUGH! Nov 23rd 2010 4:30PM
@Scooter
The code has already been written, tested multiple times and been on the PTR. If the code isn't ready, there would be no patch happening.
Patching all the realms can occur at the same time, or pushing an image out to all the realm servers can happen at the same time.
And it does not take 12 hours.
What takes time is running updates on the database, like setting all players positions to their hearthstone locations, and whatever else the patch requires, like refunding talent points to some classes, etc.
How much time database operations take is variable. Now if you were running one database script updating the about 20,000 players per realm times the the number of realms in the data center...
If all those players were on one set of drives, the time it takes is limited to the speed of the IO channel between the database and the drive array.
If there was a separate database and drive array per realm, you could execute (number of realms) scripts in parallel and there would be (number of realms) different IO channels between (number of realms) databases and (number of realms) drive arrays. For a speed increase to do the same operation that is (number of realms) times faster.
Pyromelter Nov 23rd 2010 4:42PM
Would it help if I got out and pushed?
Aaron Nov 23rd 2010 4:46PM
@Pyromelter I almost fell out of my chair laughing. /cheer
eenersumbrella Nov 23rd 2010 4:48PM
@ Enough and dru.reuther
If you have the experience you say you do, then you would know that 12+hours can happen sometimes with the sheer amount of places things can go wrong with the extreme amount of data that is this game. I love the various people that boast about their IT knowledge while forgetting exactly how big this game has gotten since its release 6 years ago.
With all that noted, there hasn't been a patch like this in WoW history. Like many people, I hoped that we would get to play at the original time, but playing for 6 years has taught me that it isn't always going to be the case. Heck, I remember servers being offline for a few days in the beginning. While we pay to play, they clearly state that they don't guarantee that servers will always be up and running.