Ghostcrawler: "Balancing around recount stats is like using a thermometer to predict the weather. "
by Matthew Rossi 
Feb 3rd 2009 at 1:00PM
Ghostcrawler chimes in on game balance and nuance, and frankly makes some excellent points. The one that really surprised me (not in terms of its message, but that he was so upfront in admitting it) is that "Selfishly, we get a lot more out of player posts than we get from offering a Q&A. I don't mind a Q&A from time to time, but understand that 90% of my motivation to be here is selfish -- it helps us make the game better." It's a form of selfishness I can get behind.We also get some information on why downranking was removed ("The math problem was getting difficult and after many meetings we decided we could better spend our design time on more pressing issues"), discussion on what GC sees players concerned with ("Many players are worried about, e.g. whether their raid buff merits being invited to a raid, which has no PvP ramifications. Many players want to have more CC or CC breaks, which has very few PvE implications.") and there's a very interesting back and forth discussion of Recount and other damage/healing meters and their limitations as a predictative tool. All in all, it's a pretty illuminating thread on how Blizzard designs and interacts with the player base.
Filed under: Analysis / Opinion, Blizzard, News items, Forums
Reader Comments (Page 1 of 2)
Zelph Feb 3rd 2009 1:12PM
threat = treat?
Eddie Feb 3rd 2009 1:12PM
Totally agree. Thinking of a player only as the damage they can do is not the whole picture. Will they decurse? get out of the fire? Whine to everyone about everything?
Play to have fun and get through the content. :)
Bubsa Feb 3rd 2009 1:12PM
1. Day is hot. Hot thermometer says day is hot. Predictive value = just fine.
I lol'd
Jeffy Feb 3rd 2009 1:21PM
I'm going out on a limb here but I think he's saying recount stats do not reflect the same outcome tomorrow as they did right now. Much like it could be 80 degrees out and then suddenly the temperature drops in a matter of an hour. It's unpredictable.
Doc X Feb 3rd 2009 1:52PM
That's not doing anything predictive. Just because it's hot today doesn't mean it'll be hot tomorrow, next week or next month. Also, whether or not it's hot has no bearing on whether it's humid or dry. Thermometers are good for what they do, but there's a whole host of weather-related information it does not provide.
To make this more clear, WWS reports provide quite a bit of information, but they only give a snapshot of what is in place when the combat logs were generated. They make have very little predictive ability (what's going to happen with X game mechanic change, Y buff or Z nerf?) given the number of variables that go into WWS stats (player skill, gear/enchantment/gemming choices, individual latency, talent choices, raid composition, etc) that the comparability of WWS reports will vary wildly.
Hope this helps.
Eisengel Feb 3rd 2009 4:36PM
A thermometer is technically a 'transducer'. What that basically means is that it measures something, and transforms that information into something else... in this case, it measures radiant heat and transforms that into a number. However, a thermometer gives you one sample that tells you what is happening right now. That is not a prediction though.
Let's say we want a room to be 60 degrees. We look at the thermometer and see it's 80 degrees. No noes! Nerf the temp! We open a few windows. Check the thermometer... it's only 79! We turn on the air conditioning. Check the thermometer... it's still 77! We open the refrigerator, the freezer, scatter ice cubes around... finally.. the thermometer hits 60 degrees. Yay! Then it immediately starts to plummet. WTF?! Now temperature needs a buff!
The problem is we didn't relate the different readings from the thermometer. Each reading was a sample, a single piece of data. Without looking at many samples together you can't get an idea about what's going on. WWS and Recount are single samples. Posting them is useful, but that is only one event. You can't post a single trace and say 'hey, based on these numbers X and Y have to happen', because it is a single sample. You'll have the same problem I outlined above, you'll go way to far in one direction, then you'll have to overcorrect, over and over. Keep in mind, any group of numbers you see is one group.
Let's say that of the 11 million subscribers, all of them have a level 80, and that all of them can raid Naxx-10. Let's also say that all 10-man raids are unique, no player is in more than 1 raid. If that was the case, then there would be over 1 million different 10-man groups. A million. That's a thousand thousand different groups of 10 people... which is really a lot... with each person having a different class, spec, gear, computer, different ages, experience, good/bad day at work, different amounts of sleep, etc. If you could monitor the performance of all of them, it's likely you would get a very, very wide range of traces.
Reality certainly isn't that cut and dried. Let's be conservative and cut the figure from 11 million+ to 10 million to account for alternate accounts from the same user. Let's say 75% of those 10 million accounts have an 80, and let's say 75% of them can raid Naxx-10. Still, that's over half a million different Naxx-10 groups.
The point is the amount of variability is immense. Your numbers are always useful, because all data is useful, but just because your numbers show something, they only show it for your group makeup of those 10 people. For instance, I have a recount screenshot where my Spriest did 16,900 DPS. Does that mean Spriests need to be nerfed into the ground? No... just that at that one specific time, given the fight, the type of mobs, my buffs, my gear, my spec, I was able to get 16,900 DPS. It was on one double trash pull in Naxx against a bunch of weak mobs whlie part of the raid was drinking, so that I could do most of the damage, and I popped a trinket and a cooldown.
Numbers are just numbers. Don't lose sight of the fact that this game is huge, and while you may have a certain experience, not everyone may see that same thing for lots of different reasons.
Angus Feb 3rd 2009 1:13PM
"pretty illuminating threat on how Blizzard"
He's threatening us now?!?!?!
;)
If they can't trust recount to show how much dps people actually get, how the hell are they going to know if their models work?
Sorry, but even a good simulator fails to take into account human failure. Sure, affliction can do awesome DPS, but if it requires an extra 13 fingers all set with hot keys, it fails.
Chucker Feb 3rd 2009 1:22PM
I think they get a lot of from constructive feedback like this. If you were to raise a complaint that said "I can't react to raid situations because I'm watching 8 different cooldowns," I would imagine that getting more attention than "WAY TO GO BLIZZ MAKING LOCKS SO COMPLICATED CAN'T SEE QQQQQ!!!"
Recount is a great tool to help mentor a DPS/healer in identifying potential issues with gear/rotation, but even the highest DPS could also be the bum in your raid who doesn't pay attention, shift properly on Thaddius, or pulls aggro.
I don't care how "high" your damage is. If you're going to make my raids become too risky, you're not likely to join them. I think this was the point of GC's reponse to basing raid member value entirely off of WWS or Recount.
mirilene Feb 3rd 2009 1:25PM
Its not about 'trusting recount'. You did READ what ghostcrawler said, right? The point is that DPS and damage done are only one piece of the whole puzzle, one tool in the tool box if you will, whereas players tend to treat it as the be all end all of measurements.
If i stand still in sapphiron i can do a lot more dps than never running away from blizzards. my dps and damage will likely be higher than if im being careful.
If i have an amazing healer who can keep me up through standing in blizzards and you look at recount, is it telling you that i stood in blizzards and are thus, terrible at the fight (not to mention how freaking awesome one of your healers is) or do you just see "wow, that guy did a lot more damage than everyone else".
VSUReaper Feb 3rd 2009 1:41PM
I had a LONG conversation with an officer and a DPS'er in my guild about recount (and other various DPS/healing parsing mods) and tried to explain why they are not the determining factor of whos good and whos not.
DPS meters are !!GREAT!! when it comes down to saying I pull XXXX amount of dps on a fight like patchwork where I dont have to move or avoid stuff. Its a totally different thing when the DPS'er has to avoid fire, or maybe get caught in a ring of fire nowhere near the boss (thinking of Shade of Aran back in Kara). The DPS will go down the longer they are running or moving around.
I would rather, and the officer agreed with me, take the DPS'er that knows the fights, knows his/her rotations, and knows that they need to get out of the fire/poison/shadow voids/etc.... and in the event that they can toss heals (no matter how small they might be) I expect that person to pull oout of their tunnel vision and start popping the heals.
To many times, I see DPS'ers dead at the end of the fight, and when I ask them why they died (knowing full well why) I usually get this resonse: "I dunno, but did you see that 1337 dps I was pulling?!?!?"
BTW, the dps never moved out of the blizzard in DTK, stood still during the entire Keristraza fight in nexxus, and never ran from Loken's nova.
Angus Feb 3rd 2009 2:08PM
I think we're talking past one another.
"Sorry, but even a good simulator fails to take into account human failure. "
I meant that to take into account such things as stupid dps, and their potential dps.
Patchwerk is an easy fight for DPS. Stand and shoot. Gives you a pretty good show of potential DPS.
Other fights show the exact same player pulling horrible DPS. Is it because of the player or mechanics? If the player has 2 nukes and has both of them have cast times, his DPS is not going to be great in fights requiring heavy movement.
If a player had great DPS but had poor total damage because they couldn't step out of a circular graphic of death, that's not the classes fault.
If a class takes a LOT of damage, is it the player's fault when EVERY time you see a WWS that same spec is eating more damage than similar roles with other classes? Enhance shaman eat a lot of close range damage but cannot cloak, evade, or bubble to survive it. When I was enhance I was left off of certain bosses and on add duty because while rogues could sit comfortably in the rain of fire with a cloak on, I would cause the healers fits.
If a healer is having a LOT of overhealing constantly, but it is consistent for this spec/class, then it might mean their heal takes too long to cast and the plethora of instants or viable quick cast spells is causing them to be not as effective.
If a healer has a ton of overhealing and is running OOM fast, then it is likely the player. If EVERY healer is showing this sign, it might not be the player.
Recount/ WWS is great at showing trends when a lot of people use it. You will get an occasional glitch, but if 500 raids show the exact same thing whenever certain conditions are met, it isn't likely the players doing something stupid, it is their class/spec working a specific way.
Using the stats to get a feel for why things are working how they are is a great way to find a way to balance things so that either less encounters happen where specific specs are hosed, or the spec can actually do it's job in those roles.
FenSat Feb 3rd 2009 1:21PM
Honestly, I feel the meter threads should be required reading for all WoW players. I'm consistently at the top of the DPS meters, but too many people I've encountered use the meters as the primary means of determining the value of someone to a dungeon/encounter, all to the detriment of progress. I'd rather have that 2,500 dps player who does nearly everything right for the encounter/team, rather than that 3,000 dps player who doesn't shoot down the sparks when asked, doesn't focus fire on the Gargoyles, won't break of DPSing to drop his cloud in the right spot, etc.
Sadly, there is no "stupidity meter" in WoW, and even more sadly, it seems like a significant portion of the WoW community would still continue to rate player effectiveness solely/primarily based on a Recount readout even if there was. I call those people "Meter Monkeys." You know, the guy who is always linking recount meters...but only on encounters specifically favorable to his playstyle/class. The dude who sometimes even openly refuses to focus fire because it'll "hurt mah DPS." He's the guy who won't stop DPSing if he's having aggro issues, all because he's worried about where he'll end up on recount.
I hate Meter Monkeys.
Wasuremono Feb 3rd 2009 1:28PM
As a healer the focus on meters become even less enlightening. I can top the healing meter but it doesn't mean I am doing the right thing for the raid.
Viscero Feb 3rd 2009 1:31PM
I think GC's main point in all of that blueberry mush is that the player's don't always know as much as they think. They are biased in some cases, elitist in some cases and blinded by numbers in the rest.
Chalk it up to this, the devs know better because they look at everything not just one class in regards to the rest.
sephirah Feb 3rd 2009 1:39PM
A lot of people told GC that retradin and hunters were OP looking at their Recount but still he didn't believe them...
So sometime players thermometers are better than Blizzard's satellite imagery...
Matthew Rossi Feb 3rd 2009 1:45PM
I don't think he would have been right to believe them based on the recount data.
We have a shadow priest in our raids who is always #1 DPS. He can put out 5400 to 6000 DPS in some instances, and is almost never below 5000. Does that mean Shadow Priests are OP? I can show you lots of recount data of several spriests doing at least 4.8k DPS, doesn't mean the class is broken or OP.
Frankly, since GC has admitted they overnerfed hunters, perhaps the player thermometers were off kilter due to local conditions of envy.
sephirah Feb 4th 2009 11:44AM
Matthew, I'm not talking of one or two threads, I'm talking of the massive number of threads about both hunters and retradins, containing Recount data, WWS parses, formulas and so on.
I'm talking about HUGE OPness, not just "FFB mages are doing 5% more dps than any pure dps".
Obviously devs must keep in mind that players are always QQing about anything, but when even hunters and retradins wrote "hmm, maybe we are really doing too much damage", you could double check your internal data if they are telling the opposite.
Reyven Feb 3rd 2009 1:48PM
That is a pretty stupid statement, since in this case Blizzard is on control of the weather. Their balacing is actually what determines the weather tomorrow.
darian Feb 3rd 2009 2:27PM
Blizzard has the capacity for limitless, authoritarian control over the game, but to do so would quickly destroy balance. They're limited to careful, precise and minute changes.
They could force the temperature in Boston to 80 degrees *and* make it snow, but doing so would break down the rules governing snow with *global* consequences. That hamhanded governance is a quick ticket to destruction.
In the end, Blizzard is basically saying "there's more to a boss fight than the numerical value of one's DPS". Like many others have said, a DPS who doesn't stand in the fire is many, many times more valuable than one who does even if the latter does incredible DPS.
ME Feb 3rd 2009 1:54PM
I agree 100% that game balance shouldn't revolve around damage meters, but in the "REAL VIRTUAL" World of Warcraft, it does. Your gonna be very hard pressed to find a raid that isn't focused on damage meters.
Just look back at every post on WI that deals with nerfing a class, and see how many people post saying, "it's about time, is so OP".
I could agree if they were commenting about OP in a PVP context, but most are talking about OP in a PVE context.