The perils of progressive testing
by Mike Schramm 
Feb 28th 2008 at 7:04PM
After Skellum of Dalaran posts on the forums asking just where all these PTR changes are coming from, Nethaera sobers us all up about what the PTR is all about: progressive testing. She says that Blizzard has said from the beginning that there will be changes in the notes, and that things we thought were the end of the world in the beginning have changed or been completely removed. Welcome to the perilous world of progressive testing.And it occurs to me that I've broken my own rule about patience on the PTRs. When those Shaman notes dropped with only a Stormstrike icon, it didn't occur to me (or almost any other players) that Blizzard wasn't done yet. And while the latest changes still aren't done (we're still waiting for an Elemental buff), things have been fixed somewhat since then.
But is this really just players doubting Blizzard? They posted the patch notes with just one small disclaimer -- would they have been able to quell the furor a bit more if they'd made it more clear that what's posted on the PTR notes has almost no connection to what will show up on the live realms? But then again, Neth never really answered the question of where these changes are coming from -- if Shamans hadn't QQ'd so much over the Elemental Mastery and Nature's Swiftness nerf, would it ever have been reverted? Blizzard seems to be simultaneously telling players to be patient and also give feedback. If players had been patient when the EM and NS nerf came down, and it hadn't caused such an uproar, would it have been changed back at all?
Filed under: Patches, Analysis / Opinion, Blizzard
Reader Comments (Page 1 of 1)
Salacious Feb 28th 2008 7:27PM
Testing by its very nature is an organic process,give them time... And yes do assume that player feedback plays a role in everything,just dont assume that they will always listen.
Slayblaze Feb 28th 2008 7:29PM
This latest round of PTR tests and all the negativity that has seen the light of day really makes me think that the whole system of how Blizzard tests changes is just the wrong way to do it. They make some changes which pisses off the masses, and then later even if they change it back due to the outrage, the bitter taste is still there in our mouths. Even though we know that they are just testing out their latest rounds of tweaks, we all end up reading so much into it. At this stage, many of us know as much about the game mechanics as the devs do, and their intentions are obvious to us...we can see where this all is "going" even if they end up changing it later, we can still read their intent.
I really think it would be in their own best interest to stop the practice of bringing things online to the public in its own individual realm as is their current practice. It does have some positive elements, but the bad outweighs the good by a long shot. This 2.4 testing makes it even more obvious now than it has been in the past. They really need to keep all testing to wholly internal crew instead of using us as guinee pigs - even though we do seem to enjoy it, although I'm sure many would disagree with me. The whole theory of a Public Test Realm is arguably too flawed, and should be discontinued for future testing.
Matt Feb 28th 2008 10:52PM
They could take the middle ground and simply have closed test realms, where they do testing on an invite only basis.
Overall though it is a shame that the WoW community is so immature in their attitudes about changes. Even among some of the "Professionals" here, they can't restrain themselves from fussing about changes, or the lack thereof, despite Blizzard explicitly saying that the notes didn't include all the changes.
It shouldn't be a surprise to anyone when Blizzard doesn't openly communicate with a community that has deemed itself utterly incapable of being rational. Of course the easy response is that it is Blizzard's fault the community is like that, clearly Blizzard forced players to behave like children.
Lest anyone think I'm simply calling Shamans out, I think it applies to every class.
Thank goodness there are private forums that Blizzard can collect consistently reasonable feedback like EJ, otherwise the community may not have a voice at all.
ummyeah Feb 29th 2008 8:05AM
Well, i dont know where you are getting this "massive negative outcry" - i mean, your making it sound like everyone hates this patch or something. Sure the official forums might be the (very) vocal minority, but I think this patch is absolutely wonderful - and so doesnt almost everyone else i have spoken too. Some people just love to be negative i guess...
infection Feb 29th 2008 8:46AM
For the past few weeks I've been noticing more and more how the some bloggers here use this site to cry about their class.
This place is a wealth of information, but it seems like it is turning into the QQ corner.
Why do most of you have to throw in QQ's in your stories?
"And while the latest changes still aren't done (we're still waiting for an Elemental buff), things have been fixed somewhat since then."
Whether you are getting nerfed or not, why keep leaning this site towards a shaman and warlock QQ constantly?
Yes, I have a warlock. Yes, I am pissed about tap. No, I don't like seeing almost EVERY OTHER THREAD being a QQ about it, and each other thread about shamans.
Spoony Feb 28th 2008 7:37PM
I think they could have done this whole thing a little better had they have put the information on the notes from the get go and only implement them over time. This way everyone can see what is coming and discuss it, qq over it, etc before it goes on the ptr. Then when blizzard is ready to test it put it on the ptr and then edit the ptr notes as they change each thing on the list.
This would at least quell a lot of the complaints of the my class isn't getting any changes again posts.
dan Feb 29th 2008 9:38PM
That wouldn't accommodate changes along the way. That's what this is about. This is basically agile customer co-development: prototype something, get reaction, tweak, prototype again.
Kadaan Feb 28th 2008 7:59PM
I definitely think people should "QQ" about changes they feel are unfair or unneeded. The biggest problem is the way people go about it. Slinging accusations like "devs are obviously dumb" and "looks like I'll never play my again" comments doesn't really help devs at all.
What they really need is an "I support this post" button. That way threads can have only a few pages of constructive comments and not 25 pages of "I agree, **** bliz" posts.
Here's hoping the flametongue buff gets nerfed to self-only, adding it to the totem is insane! :)
Johan Feb 28th 2008 8:13PM
This is essentially why companies hardly ever do public betas of their products, and usually do closed group user testing. It's because when you open your product up to the public, they assume it's "final", when in fact we all know it's not.
When companies usually open their products up for public "beta" consumption, it's more about stress testing the product versus actual changes to the product. If they really want GOOD feedback vs. QQ, they try to have a closed group of diverse testers (aka Alpha).
What it boils down to, is a change of philosophy. Blizzard when from a waterfall type of development to an AGILE (or collaborative) type of development. The first acts exactly like it sounds, the changes happens at the top of the waterfall and you (users) receive the bottom. There isn't much change possible since sending changes back up the waterfall is very difficult. This is how Blizzard has usually run their PTRs.
Now they've changed to AGILE, which means, progressive development. You break up the larger product (in this case the patch) and you break it up into smaller chunks. You work on these chunks and go through a whole dev cycle for each of these chunks. So you do idea genesis, testing, QA, tweaking, etc, until you get that chunk right. When you do, you move on to the next chunk. It's a very beneficial method of development as it allows you to tackle smaller problems and hopefully make it a better product.
What makes it difficult is that people usually work in the waterfall method, which is the reason why people are having such big problems understanding progressive testing.
Basically, AGILE lives and dies on trust. The end users have to trust that their feedback will be incorporated and trust that devs will do what is correct. Right now, users don't trust Blizz for whatever reason, and it's making this cycle on the PTR more problematic. It'll all subdue soon and you'll realize all this fear you have right now will be over nothing much at all.
thebvp Feb 29th 2008 3:14AM
Maybe this is going to sound pessimistic of me, but I really think Blizzard ought to put something on the test realms along the lines of allowing priests to spec for "mortal wand" or possibly a dual wield wand spec "wands of fury." Maybe you could turn imp mana burn into "mortal burn," so that it comes with a debuff that increases mana cost for spells by 50%, but nerf it by giving it a 5 second cooldown. Buff destro locks by making death coil an aoe, spammable effect that doesn't suffer from diminishing returns and give warriors the ability to heal themselves with rage.
Why? To show that THESE ARE IN NO WAY FINAL.
draeth Feb 28th 2008 10:10PM
I pray every night for a rage to health ability....
Pappagallo Feb 28th 2008 10:22PM
What the process needs is greater transparency. Ideally a "testing timetable" for each patch. As you mentioned above, the QQing over Elemental nerfs would have been tempered if people had seen the buffs coming. If Blizz put up a rough timetable, maybe not tied to dates, but with a list of phases, like:
Phase 1
- Elemental shaman nerf
- Lifetap nerf
- Arena mana regen through drinking nerf
...
Phase 2
- Fine tuning of previous changes based on feedback and analysis
- Flametongue buff
- Toughness buff
...
People would freak out less if they saw that a) changes weren't set in stone and may be altered at a later date and b) there were some buffs in the pipeline to balance out the nerfs.
jrodman Feb 29th 2008 4:31AM
The reason there is *so much* wailing and gnashing of teeth is not because the changes aren't finely tuned, but because plenty of them are so amazingly stupid.
Life tap gets worse for warlocks the better they are geared? Restoration Druids (the lowest single target throughput healer) get their throughput reduced? These changes make NO SENSE AT ALL. If the changes were comprehensible but overshot the mark, or needed tuning, I could see a constructive dialogue occuring. When they are entirely laughable, and (eg druid lifebloom nerf) go without any form of response for weeks, then it's impossible. You cannot get a constructive dialogue by making nonsense changes and then refusing to communicate.
Dan Feb 29th 2008 5:22AM
At #8, showing a plan would make people expect the plan to be followed, and if it had the items you said it would make people even angrier if they move onto the next phase when the first phase (ie lifetap nerf) is still wrong (as in, stat breaking).
But having a progress on each item would be handy, having a designer give a bit more detail on the reasoning and aim would also be handy.
The problem is, we can assume that they have internal testing, we can assume that they think hard on these changes. So when a nerf appears that goes against the class design, or nerf an area that the class is already weak in, you can't help but wonder what the hell they are doing, since they obviously thought about it before putting it in, and did some internal testing first, shouldn't they already know it's a bad idea?
Not to mention that it gives the feeling that the PTR is the "final acceptance criteria", so if you don't complain loudly enough it *will* go live in that form.
These assumptions might be wrong, but then they should be honest and tell us that they are currently putting in new ideas on short notice and do their internal testing on the PTR so we know. Marking changes as "Nearly complete" would also help, then we know if they are still just playing or if they think it's good as is.
Dave Feb 29th 2008 11:21AM
I think you've inadvertently exposed and understood (just, not on purpose) the actual point of the PTR.
It's not to get fanboys excited about things. It's not to get people excited for a new patch, new content, a new raid, new non-combat pets or anything at all that %99 of the vocal minority goes on and on about.
It's to provide Blizzard with feedback based on the potential changes, balances and things that affect the game. They don't play the game nearly as much as the customers, and despite what the vocal minority likes to whine about, they do listen to the complaints for the most part. When they don't, it's probably because JUST the vocal whiny minority is complaining about it, but the rest of the playerbase doesn't really see the harm/benefit in something. Most people waste their time on the PTR, hoping for premade characters so they can have epic PVP battles that don't provide any quality testing for Blizzard. Recently I think it's been getting better and people are actually spending time and doing math on changes (thanks EJ mostly as far as I can tell) but mostly PTR's are just raid testing grounds and that's why historically when they've thrown in patch changes the horrible crap has stuck. People bitch and moan on blogs and forums but don't actually go and provide real feedback through 1st person experience and gameplay. Blizzard isn't going to fix anything because you blogged about it. The people responsible probably don't develop wow all day and then go home and read the fanboy pages. The CM's do, but at the end of the day they're really going to just send a laundry list to the developers and if 3 blogs and 5 players complained about something, that's not necessarily going to shake the developers nearly as much as if 300 or 3000 players logged onto the PTR and filled out the feedback forms and told the developers directly that this change sucks or that something else is needed to make it not suck.