Creating an open source WoW database
by Mike Schramm 
Dec 8th 2008 at 1:30PM
This is an interesting idea -- Daniel over at Marenkay.com is the creator of phpArmory, which is the closest thing we have to an official API for Blizzard's Armory site, and he's now turning his eye towards unofficial databases. Sites like Wowhead and Thottbot are extremely informative, but the one thing they don't allow is player access directly to their own data -- obviously they have a monetary interest in keeping their information on their site. But an open source site, as Daniel says, would allow players to get at that information whenever and for whatever purpose they wanted.Very interesting idea, and it sounds like he's got the coding chops to do it -- he's already got a working prototype together, apparently, and he's taking suggestions on where to go next. We'll keep a curious eye on this one. Competition is always good for customers, and while the current database sites might not be interested in an up-and-coming open source version of themselves (actually, the great WoWWiki is pretty open already, though they don't really collect as much numerical information), having widespread open data on drops, kills, and gear would be very beneficial for players. This could turn out to be a very important and helpful project.
Filed under: Items, Analysis / Opinion, News items
Reader Comments (Page 1 of 2)
Mecandes Dec 8th 2008 1:39PM
...and you know, sometimes you can take things too far and take all the fun out of a game. Unless you think statistics or cheating is "fun." ;)
Gnosh Dec 8th 2008 1:41PM
Explain "cheating"
Ryan Dec 8th 2008 1:49PM
@Mecandes, Why do you care if someone takes the fun out of their game?
@Author, WTF do you keep mentioning an "official API"? The Armory is XML, want Blizzard to duplicate what is available?
nbcaffeine Dec 8th 2008 1:48PM
Apparently, sharing what you've learned with others is "cheating"
wrshamilton Dec 8th 2008 1:56PM
It's kind of weird to suggest that the overlap of people who find RPGS and statistics, respectively, fun, is somehow marginal.
Jay in Oregon Dec 8th 2008 1:56PM
I know there's another attempt to do something similar, at least as far as achievements go, over at http://wow-achievements.com/
It's a bit rough around the edges right now, but I'm hoping to be able to discover fun things like "which member of my realm/guild was the first to get 'To All The Squirrels I've Loved Before' or 'Accomplished Angler'?"
pb Dec 8th 2008 2:02PM
only problem with an open DB is there will be people out there who will intentionally try to corrupt the data, I am sure this happens with wowhead, etc but with their client they can help weed it out.
not saying it isn't possible, I'd like to hear more
SimpleSurvival Dec 8th 2008 2:03PM
...
What sort of computer background does the author have... playing video games and writing for a corporate sponsored blog do not quite feel like the right criteria for being 'in the know' for this sort of thing...
I hate to sound elitist... but the armory is VERY simple to program for... if you have trouble consuming XML, maybe you should stick to static sites in dreamweaver...
The Wow Armory works very well from a SOA (Service Oriented Architecture) standpoint... it is the data in its most simple form, presented in a very universal format... it can be presented in whatever which way you'd like...
Programming a PHP wrapper that is more or less nothing but an XML parser isn't "the coding chops" necessary to accomplish what he plans...
He's completely overlooked the actual database, more concerned with the parsers necessary to populate the aforementioned database... He's also overlooked the infrastructure necessary to host such a thing...
Everyone's favorite wow db site would prolly be ad free if they could host their sites for nothing...
"But why are all of these databases closed, or commercial? Some even want you to buy premium subscriptions to access guides or more data."
Which database do i sign up to get the premium search results??? or is that just in his imagination?
His choice of API to handle client uploads is also chosen without much thought... He's using a reasonably bloated API to handle something that could be simply accomplished without any api at all... increasing the effort necessary server side, and complicating the previously mentioned difficult to figure out database problem...
anyways... vaporware, move on
Ekimus Dec 8th 2008 2:25PM
WoWHead does offer a bit of an API, much like the armory. You can access item data in XML by simply adding &xml to the end of an item url. Example:
http://www.wowhead.com/?item=amani+divining+staff&xml
or
http://www.wowhead.com/?item=33494&xml
Item IDs are easy to come by, or guess. You could write a loop from 1 to 1000000 and load every valid item...ignoring the errors.
Secondly, WoWHead seems to be pretty open to sharing data. Case in point, www.emmerald.net. He has been using WoWHead provided data to parse out the best possible upgrades for Feral Druids. I believe all he did was ask nicely.
Furthermore, the Rawr application uses data from the Armory and WoWHead to provide and parse upgrades for a wide array of classes/specs.
The driving force of these sites is the data, that is the draw. Thottbott and WoWhead were the same thing, WoWHead became more popular because the way that data was presented, the actual data remained consistent between the two. Then, WoWHead began to offer other features, like a talent calculator (I know, it came before the data...), item comparisons, etc..
The data is already out there, available in very digestible formats for any decent framework or programmer to consume.
I'm guessing he is specifically referring to a out of the box OSS LAMP solution to allow anybody to create a WoWHead clone. This is a fair idea, but like anything else, is it really necessary?
Software is created to, among other things, fill a gap. That gap could range anywhere from missing features, to the entire package itself. Would an OSS solution be providing anything different, besides an on-demand WoW database?
Kiaden Dec 8th 2008 2:29PM
A nice idea but the whole "whenever and however" is window dressing because you're still at the mercy of the actual Blizzard Armory for the data.
A lot of information isn't supplied by the game API and or would mean some very large LUA tables having to be constantly submitted.
So this means, when the Armory doesn't update NEITHER DO THEY.
It's a sad fact of life but there is no way around this, Blizzard is a gate keeper holding all the keys...
Arturis Dec 8th 2008 2:54PM
I have to agree - Ive done some XML parsing (using ColdFusion) of the Armory data to build myself a few nifty compare tools, but the biggest headache Ive ran into is the Armory's instability. Still, its the only way to get the character data I want, so I do my best to work around it. Ive found that caching the data locally to a temp database helps a lot, as long as you have a "force update" option for when you know the data is old. Simply timestamping the local data makes it easy to slip in some "if older then 1 hour" logic, which is often fine enough for my needs.
Ryan Dec 8th 2008 4:15PM
@Arturis You should always be caching your data when using 3rd party sources no matter the project. It is poor form to do live queries.
el Dec 8th 2008 2:49PM
The main value the large database sites add isn't the raw data. It's community, usability, and ability to make increasingly complex patterns "accessible".
And all this overlooks the irrational circus of Blizzard obfuscating information and third-parties reassembling it.
The rational conclusion from that is Blizzard simply open up its internal data, and let third-parties make it accessible to the player-base. Something they've consistently been better at doing that Blizzard. I could go on...
(As aside, Blizzard own all the raw data, and I expect would challenge anyone that declares anything derived directly from WoW to be "open source". So that term needs to be used with exceptional care.)
(Also worth mentioning Adys' http://www.wowwiki.com/OpenWDB .)
Mep Dec 8th 2008 3:22PM
By API, wowinsider is refering to an easier way to gather information from Blizzard's website. Currently, we can only ping their website about once a second without getting our IP's temp banned... if we were able to ping the system as fast as we wanted through an official API a lot of sites like http://www.wowjutsu.com and http://www.wow-achievements.com would have a much easier time gathering their data and would be able to provide richer experience for their users. I'm 100% for a Blizzard API, and as the owner of http://www.wow-achievements.com I realize that it would make it slightly easier for developers to copy what I worked hard to work around, but you know what? It would make my job easier and I could develop unique features with the time I'd save.
I applaud this open source version of the armory and wish the developer luck.
Swiss Dec 8th 2008 3:58PM
PHP. The scripting language of choice for the scrubs of the web development community.
Ryan Dec 8th 2008 4:18PM
Please explain oh master.
Swiss Dec 8th 2008 4:22PM
If you have ever seen the rat's nest that is a PHP script, you would understand.
Ryan Dec 8th 2008 5:31PM
I develop in PHP. Please explain further, surely such a ambiguous statement is not the best you can do.
Dez Dec 8th 2008 5:59PM
PHP is a nice enough language, but it isn't exactly going to turn you into a computer scientist overnight. Coding a website to rival Wowhead would require very high knowledge of programming in general, which most PHP programmers, unfortunately, lack.
As an aside, EVE Online provide database dumps to third-party websites (in CSV I believe), the result being you only need to Google EVE and you get about fifty billion item databases at your disposal.
It's no bad thing, but covering live data from the realms (which is what we're really all interested in; items and spells can be data-mined) takes much more effort, and Blizzard would probably need to dedicate a fair bit of time and effort into it (unlike in the EVE case, where the dumps are provided ad-hoc by their equivalent of MVP "green" posters).
Daniel Dec 8th 2008 6:17PM
Well, being the person mentioned in this blog post, I am surprised that people have an issue with the whole idea.
Let me clarify a few things:
1. My background does not matter for it does not change the quality of the idea.
2. Do not misunderstand Open Source here. The main idea is to make the site code and related tools open source. I am not talking about open sourcing the data, as all data is copyrighted by Blizzard and shall remain theirs.
3. I am only using C# / .NET 2.0 for a simple reason. It is nice if you want to do quick mockups and tests. Think of it as an implementation prototype. I can write the whole client in plain C++ with STL only, if that's what people want. I do use Windows, Linux, and Mac OSX and have no issue with making a Mac app.
4. I know very well about database design, and have worked with databases that outnumber the size and complexity of a WoW content database with ease. WoW content is not really big in terms of data.
5. Yes, there is PHP for the server-side application, but it could be Java, Ruby, or whatever you want. Even Python or Perl are no issue for me.
Please, do not misunderstand this. The post is about a development prototype. Neither are the chosen platforms final, nor is the whole idea final.
If there really is any developer among the people commenting, he / she should really know that ideas start with quick prototypes, that are iterated again and again until they reach something usable.
I am open for discussion, but please keep away the flames.
PS: Paid services and database... e.g. allakhazam -> http://www.allakhazam.com/Premium_Chart.html but hey, I am just making this up, do I?