Because AI is hard. Very, very, very hard. Essentially impossible for anything beyond a nearly trivial problem. You give Chess as an example: Chess has 6 total different unit types, each one of which only has about 4, at most, possible moves at any one time. Each side only has 16 pieces. Its very rare that there's ever a point at the game where there's more than 20 possible moves. Ultimately, its a pretty simple game. This problem took DECADES to solve. Massive supercomputers build that spent years at a time working out every possible move that could be made at every possibly game state.

WoW is a vastly, VASTLY more difficult problem. You have 10 different classes, each of which have dozens of different abilities. Moves aren't limited to simple 'can only move diagonally' either. You can run just about anywhere you want, in 3 dimensional space. Even in a one-on-one battle with a player, like it would be in chess, the options of magnitudes of order more complicated.

If Blizzard would be willing to dedicate a supercomputer to each and every player, and spend a few decades programming AI routines, what you propose MIGHT be possible. Realistically, its not going to be happening in our lifetimes.

*ahem*

Faction Champions.

*Faction Champions.*

Not really. The faction champions have SLIGHTLY better AI than the standard boss fight, but they're still extremely dumb. The only reason they're any challenge at all is because Blizzard gave them several million HP and more-or-less infinite mana. If they actually had realistic health, which would be something like 1/100th of what they actually do, they'd go down in seconds.

"Realistically, its not going to be happening in our lifetimes."

Might I point out that none of you have actually said what isn't going to happen in our lifetimes. I mean, "better" "smarter" and "more variables" doesn't really illustrate what you actually mean. If the "unrealistic" future you talk about has AI that kites, fakeheals, crowd controls, and reacts dynamically to your every move, a future where AI could realistically compete in arena combat, then that future seems pretty... annoying, anyways.
Essentially, the "better" part in "better AI" means better at killing you, the player. Yes, it could be toned down a little, but immagine going out to grind with your warrior, and suddenly you're kited, stunlocked, and when you finally think you've got a mob nailed down he CCs you and runs away to regain health. Is that fun? Is it really that fun to have an unpredictable and smart AI beat you into submission? Good AI wastly excels at anything it is designed for, and it is not hard to immagine that something that is 100% aware of everything happening around it, 100% aware of every possible application for it's abilities, and no reaction time to speak of would excel at this game. Having constant PvP like PvE is ultimately not that great, there are people who like one but not the other for a reason, and most people who enjoy PvP do so because they beat other people and not computers.

Ultimately, my point is that everything is fine as it is. Theres nothing wrong with having to go into very controlled environments (instances) and resort to very specific scripted behaviour (which is also "smart" AI, just not very dynamic) to experience anything different from the usual, run towards player - attack / cast spell behaviour. People fail enough at the memorisation-heavy encounters of today, and adding reaction time and the ability to dynamically react as a group to that equation just has epic fail written all over the place.

However, on the "not happening in our lifetimes" part, I'm not so sure if that's correct. Good AI (not just in games) is one of the things that modern society could benefit immensely from in so may ways, and seeing just the rate of advancement in computer technology of the last decade, it's not so hard to immagine that such an important field would see major breakthroughs in the foreseeable future.