The Lawbringer: Lagpocalypse 2010-2011

The Lawbringer is taking a little detour today to discuss one of the more esoteric issues (like I try to do here) that many players are facing today dealing with World of Warcraft. It is interesting that one of the biggest hurdles to playing an MMO is present outside the game rather than within. Today, I want to talk about Time Warner/Brighthouse and the intense lag of 2010 and 2011. In fact, we'll visit the past and see how these companies interacted with Blizzard, and then take a stroll into the present and try to understand what's going on right now.
Confused? You might be, especially if you aren't a Time Warner or Brighthouse customer. There are some other ISPs affected by all this mess, but for now, I'm sticking to the most complained-about. Here's a quick little rundown of what's been happening over the last few weeks (and for some people, months) due to issues with Time Warner and Brighthouse internet service.
The rundown
A few weeks ago, the lag started to become unbearable for me in raid instances during peak hours. I tried to fix everything -- and I mean everything -- which eventually led me to the support forums on the WoW community site. When I got to the support forums, it was like showing up to a protest already in full swing. Customers from Time Warner, Brighthouse, and the U.K. provider Virgin Media were up in arms over huge lag issues exactly like the one I was having. So I did what every red-blooded internet denizen would do: I read every single thread.
From what I could gather based on the Blizzard responses and user complaints, something happened after Blizzard changed the traffic patterns of World of Warcraft data. The resulting issues that have come about may have created service interference within certain geographic areas of Time Warner service. (I'll just be typing "Time Warner" from here on out, although your ISP might also be included, for the sake of not writing out all of the affected ISPs every time.) Blizzard seemed to confirm that it was something to do with the new traffic patterns:
Actually, Jjvalour, the issue is not that we changed the content of the packets, it is that we changed our traffic pattern, and this is what is triggering traffic management systems to throttle individual connections.
Our engineers are more than willing to work with the few ISPs affected by this change. A few ISPs have even taken the initiative to fix the issue on their end, already. Please see this post as evidence: http://community.virginmedia.com/t5/Broadband-down-your-phone-line/World-of-Warcraft-Latency-Issues/m-p/161857
We know this is extremely frustrating to people, and we are actively looking at resolutions from our end, as well. Thank you for your patience while we attack those gremlins from all angles. :)
Our engineers are more than willing to work with the few ISPs affected by this change. A few ISPs have even taken the initiative to fix the issue on their end, already. Please see this post as evidence: http://community.virginmedia.com/t5/Broadband-down-your-phone-line/World-of-Warcraft-Latency-Issues/m-p/161857
We know this is extremely frustrating to people, and we are actively looking at resolutions from our end, as well. Thank you for your patience while we attack those gremlins from all angles. :)
One of the more interesting things that this post illustrates is the relationship Blizzard support has with ISPs -- it's basically the same relationship we as consumers have. Sometimes I forget that these two companies are intrinsically on the same page.
So about this time in the story, I'm frantically calling my ISP and fighting my way to a senior technician while still browsing the forums, ripping into threads like children looking for a golden ticket in chocolate bars, begging the universe for a blue post that had my answer. Slowly but surely, the data begins to coalesce, and there is a common issue with players connecting to Chicago servers on Time Warner service.
Brianl, my personal hero through all of this, lets us know that there are fixes in the works:
I apologize for the lack of updates recently; I have been exceedingly busy working on these issues behind the scenes. I am in the process of gathering data to provide to our server engineers, as well as working with some of the more technically inclined people exploring further options. We are actively looking at these problems.
And then, two nights ago, things seemed to start getting better.
We have remained in contact with both the ISPs in question and the backbone providers and the issue seemed to disappear last night. I am going to leave this thread stickied for now just to keep an eye on things.
Thank you for your patience with this problem; we know it can be intensely frustrating.
Thank you for your patience with this problem; we know it can be intensely frustrating.
As a quick aside, Blizzard's support on the community site support forums is phenomenal. Blizzard, you guys are really knocking it out of the park by helping people with their issues and being communicative through this invaluable resource.

Time Warner and World of Warcraft have had a shaky relationship in the past, with an eerily similar issue at the time. Alex wrote about this particular issue back in 2008. The same east coast issues seem to be rearing their ugly heads again. The issue was so contentious that it actually made the New York Post. Something was going on between World of Warcraft internet traffic and user's huge pings and latencies. Some people have had that issue rectified since 2008, while others were still feeling the hit. But the common thread was Time Warner internet service, east coast customers, and servers near the Chicago data centers.
Tunneling for solutions
Another interesting development began to surface as we fought during the past few weeks about who, or what, was at fault for the Time Warner east coast peak server time lagfest -- WoW tunneling services were, for the most part, fixing the problem. What is SSH tunneling, you ask? SSH tunneling, or secure shell tunneling, transmits your data through an encrypted tunnel from one point to another. SSH tunneling services like SmoothPing can solve many people's lag issues with World of Warcraft because the connection to Blizzard's servers are made through their data centers, not yours, and the connection between you and the service is through an SSH tunnel.
It was interesting, then, when users began to use tunneling services to hopefully bypass the Time Warner problem and were met with success. Changing the way the packets were dealt with were affecting people's success with lag. Blizzard responded thusly:
I had noticed that and forwarded the same info regarding SSH tunneling. Without pointing fingers, it's a very interesting piece of information that gives us some clues as to possible root causes.
SSH tunneling services will do a few things:
1) raise the priority of traffic
2) encapsulate packets
3) encrypt packets (SSH)
4) force a port (SSH uses port 22)
5) change the route (you're going to the tunneling service instead of straight to Blizzard through your normal route)
6) send smaller data packets due to the technology involved
as well as a few small other changes. Why, exactly, it is having this effect is currently unknown. This is something being actively investigated at Blizzard currently.
Brighthouse has been very forthcoming and helpful. According to the information they passed to us, they do not employ any sort of Deep Packet Inspection traffic shaping services, at all.
SSH tunneling services will do a few things:
1) raise the priority of traffic
2) encapsulate packets
3) encrypt packets (SSH)
4) force a port (SSH uses port 22)
5) change the route (you're going to the tunneling service instead of straight to Blizzard through your normal route)
6) send smaller data packets due to the technology involved
as well as a few small other changes. Why, exactly, it is having this effect is currently unknown. This is something being actively investigated at Blizzard currently.
Brighthouse has been very forthcoming and helpful. According to the information they passed to us, they do not employ any sort of Deep Packet Inspection traffic shaping services, at all.
There's one issue -- if you use a tunneling service, you could potentially trigger a false-positive for hacking on your account, since the tunnel changes the IP address Blizzard is communicating with when dealing with you. Sometimes Blizzard's systems will hit you with an account lock until you unlock through a password change on your Battle.net account, so it's one of those use-at-your-own-risk suggestions.

I won't really know if my problem is solved until Sunday night, when I hit the raid scene again. For the past three weeks, though, things have been pretty bad. This whole ordeal has been trying for a lot of WoW players, who were doing nothing but playing their game normally, hampered by what looks like preferred treatment to certain packets of information. Even as I write this and make edits, new information is coming to light about AT&T's servers also playing a role, but we still don't have all of the information we need to figure out what is really going on. That's so uncomfortable to me.
What can you do to help if the situation isn't cleared up yet? If you use Time Warner, Brighthouse, or another affected ISP, live on the east coast and are experiencing this issue, you can help. Call up your ISP, politely ask to talk to a senior technician, and explain the problem you're having (World of Warcraft is running slow on the network, and it's not your setup), or let the lower-level technician know that you want your information elevated to the senior guys. They know that there is a problem, and all of the hardworking people over at Time Warner and the other affected ISPs are doing their best to get the situation resolved. Be polite, offer information, and we should be on our way to a solution.
Very nice people
I'll end this Lawbringer on a sappy note. The people in Blizzard's customer service department who I talked to about these issues (and the SSH tunneling stuff) were completely knowledgeable about the issue, and the solutions and incredibly helpful. Thumbs up also to the guys over at @TWCableHelp for their quick responses about whether there was a problem or not. As customers, we like to know there is a problem and that it's being worked on, as opposed to being lied to about whether there is an issue at all. Thank you, guys, for taking us east coasters' complaints seriously and working on a quick fix.
Hopefully, this saga is at an end. I shouldn't have to pay for a tunneling service to raid effectively on the internet I pay a pretty penny for.
Filed under: Analysis / Opinion, The Lawbringer






Reader Comments (Page 1 of 4)
solidus.obscura Jan 7th 2011 5:12PM
Wow, you too? I was having the exact same issue - east coast customer on the Chicago datacenter. Glad to see its a real issue and not an excuse to shoot my router.
Aanye Jan 7th 2011 5:23PM
I hope the ISPs realize, as issues like this pop up, that they're never going to be able to shape traffic and lock people out of certain activities (and I"m not talking BitTorrent... I'm talking Netflix, Hulu, etc.) because the geeks will always find a way around. And I can only imagine that SSH tunnels, with the extra overhead of encryption and what not, are far more of a drain on the network than the raw packets--though I'm not a networking expert, so I could be wrong. Just like DRM on downloaded content, traffic shaping is only going to inconvenience those who are trying to "do it right".
Tim Jan 7th 2011 5:30PM
I hit lag spikes and have updated every component on my cpu and still get shitty lag. I read your article and it makes sense. East Coast. Time Warner. Shitty internet. Gonna try the smoothlag thing. Thanks Mr. McCurley.
VioletArrows Jan 7th 2011 5:31PM
"Even as I write this and make edits, new information is coming to light about AT&T's servers also playing a role, but we still don't have all of the information we need to figure out what is really going on."
.... I've been spending the past several weeks tearing every cable and box open on my home network (and there's a lot of boxes here) trying to find what the hell's been going on with my connections. I went from 150ms to varying between 600 and *4000* over the past 3 weeks. I thought my connection had gone to crap from inevitable burden of 20 devices on a 5Mbps connection and 8 year old hardware.
I CHANGED IT ALL AND DRILLED A BUNCH OF HOLES IN MY HOUSE AND PROPERTY FOR NOTHING. DAMN IT AT&T!
(lol read the forums next time, noob)
Orkchop Jan 7th 2011 6:07PM
Same here. Although I'm on the west coast. I had just bought a new router, too, and was about to get really mad. I'm going to call up AT&T tonight.
vinniedcleaner Jan 7th 2011 7:05PM
I'm on the west coast also using AT&T and my latency has gone throught the roof to the point that WoW is almost unplayable.
Thomolithic Jan 7th 2011 9:13PM
On talktalk in the UK here and WoW IS unplayable. It says 56ms but the game is totally static but for my character. Complete and total packet failure....
Deborah Jan 8th 2011 5:16AM
Just wondering do you guys on the west coast have dsl or the uverse? Me and my husband both play and we have not been able to play together since 4.0 came out. I sign on and it's fine, he then signs on and it boots me off, and then his is fine, then I sign on and it boots him off. So frustrating and I'm about ready to switch ISP's.
vinniedcleaner Jan 9th 2011 5:20PM
@ Deborah
I have just DSL
PacGamer Jan 7th 2011 5:34PM
Don't mean to be Debbie Downer here but in regards to that Blue Post about things being better, that was posted on Wednesday where it was a lag-free all around. Unfortunately, the Lag Storm came back after that.
corpsehand Jan 7th 2011 5:35PM
So can this article be taken to mean that SmoothPing is a trustworthy service? I've always been a little nervous about rerouting my internet traffic like this...
Dezaris Jan 7th 2011 5:45PM
I'm using Time Warner, and my latency hasn't changed at all recently. Why is this? 0.o
Joe Perez Jan 7th 2011 6:02PM
What server are you connecting to? If you're not connecting to one of the servers located in the Chicago data center you may be just fine.
Dezaris Jan 7th 2011 6:16PM
Vek'nilash is in the LA datacenter, but I love on the East coast. Kind of a tangent, but how much latency could I expect to lose by transferring to a datacenter closer to where I live?
VioletArrows Jan 7th 2011 7:30PM
@Dezaris: My main server's on the LA datacenter, but my rp server *WAS* on the Dallas cluster (it's closer to FL than Boston) and there was a definite difference. It sat just below 100ms even though it's about 1000 miles away.
http://www.wowpedia.org/US_realm_list_by_datacenter Just roll an alt and pick a server from each cluster (don't mind the particular time zones, that just changes the in game clock) and play through a starting zone, keeping an eye on the built in meters. I took a year long break and didn't even realize that Dallas moved to Chicago and Seattle moved to Phoenix, so I might even have a chance at rolling on Wyrmrest Accord like I'd been wanting.
Except now I don't know where I'm being sent that instance servers push my lag into the thousands. :(
zarmy Jan 7th 2011 5:49PM
Interesting, i sent a tip regarding this issue back in December to wowinsider. I've had fluctuating latency (and much higher average latency when it is stable) since 4.0.3. The old forums had 2 ongoing stickied threads about it where customers were upset because Blizzard was saying (in that those threads) that the problem was not on their end, it was on the side of AT&T and their routing paths, or it was our ISP's. These sort of replies really irked a lot of folks in those threads.
One of the blue post actually blamed our ISP's for not purchasing enough bandwidth from AT&T to use their line and thats what was causing a lot of the lag, which made no sense because the lag issued affected multiple people in several countries with different ISP's, and for all of them to have not purchased enough bandwidth from AT&T at the same time is just unfathomably impossible to comprehend.
Blizzard says they can't go back to the old way that their packets were handled before, so it's just something I have to live with now, and I doubt my ISP will change it without a lot of complaining from a lot of customers.
Aruhgulah Jan 7th 2011 5:51PM
THIS is very enlightening; I've been getting massive lag for no apparent reason for the past month, and after doing all the troubleshooting steps usually done (recycle modem, DNS cache flush etc etc). Is there a way to get the list of affected ISPs?
JungleEmber Jan 7th 2011 5:53PM
I'm really glad you guys shined some light on this issue, I though it was so little people affected that Blizzard decided to sweep it under the rug and people unaffected couldn't care less. I've been having this issue since patch 4.0.1 and I tried everything. For a while I played with the create a char, delete the char fix but you can only avoid instances for so long. Then I also joined a tunneling service called reducethelag and I have less lag than I've had in my 3 years of WoW. Their EU option is still in beta so it has failed me for a few days at a time but no problems since.
Wraithanne Jan 7th 2011 5:58PM
Good to know that it may not just be me, bad to know that ISPs may still not be playing nice with our traffic. The intarwebs are meant to be free, maaaaaan.
Julien Jan 8th 2011 7:05PM
Yeah, I thought it was just me! Weird...I thought my home connection has been stupid but I've been just dealing...good thing I'm switching providers tomorrow!
Damn "Streaming Initialization Error" happens every so often too...