The best way to improve your DPS is to attack more. I'm not talking about stacking haste so that your attacks are swift as lightning, but rather about not missing any opportunities. In order for a rogue to deal maximum damage, he needs to be in melee range of something that needs to die at all times.
Staying in melee range can be difficult, as bosses are always tossing out pools of fire and knocking us back into the walls. Rogues need to be tenacious; we need to be vicious dogs that won't let go of our prey for any reason. We need to hustle between targets, minimizing the amount of time we see "Out of Range" at the top of our screen. In order to ensure that we're on our targets at all times, we need to learn about how melee mechanics work.
Thinking outside of the hitbox
Every in enemy in WoW is surrounded by an imaginary circle known as the hitbox. Every mob has its own hitbox, and its diameter varies based on the particular mob. For example, bosses tend to have much larger hitboxes than regular mobs. A rogue's melee range is 5 yards, which means that you need to be within 5 yards of your opponent's hitbox in order to damage it. In order to attack a mob continuously throughout an encounter, you need to always be within striking range of the mob. Learning the true size and features of your opponent's hitbox will greatly improve your effectiveness.
Due to the varying size of enemy hitboxes, there's no way to tell exactly how big a boss' hitbox is without getting in there and trying to stab it. The red targeting circle that is displayed on the ground is not representative of your target's hitbox, even though it is often misrepresented as such. Your target's actual body, its model, also does not reflect the size of its hitbox. It's completely possible to attack a target that appears to be several yards away from you if it has an inflated hitbox size.
Alysrazor, for example, can be attacked from nearly anywhere in the room when she's flying through the center of the room at the beginning of the encounter. You can use this knowledge to start attacking her immediately as the fight begins, knocking a few percentage points off of her health bar. Her Voracious Hatchlings, however, have an incredibly tiny hitbox that requires you to be on top of them in order to land your swings.
In order to play your rogue efficiently, you need to learn the various hitboxes sizes for the enemies you're attacking to ensure you're always able to melee them. Every second that you're out of range is damage lost forever.
Every hitbox has two sides, the front and the back. Most hitboxes are split right down the middle, with 50% of the box dedicated to each side. Certain bosses like Magmaw and Ragnaros have their hitbox split skewed to enlarge their back sides, which makes me feel like they belong in a Sir Mix-a-Lot video. In general, you can assume that the hitbox is split evenly and position yourself accordingly.
As a rogue, you always want to attack your enemies from the rear. Mobs are capable of parrying your blows when you attack from the front, greatly lowering your DPS. If you attack from the rear, you won't see any of these parries, and so rogues are always standing behind the boss. Obviously while you're leveling up, you don't have the luxury of a tank to hold threat for you, so you have to live with the loss of damage while you're solo.
Subtlety and assassination rogues also use Garrote, Ambush, and Backstab as part of their rotations, and all three of those abilities require you to be behind your target. While rogues do have an ability that can only be used when your target is facing you, Gouge is most often used to incapacitate an enemy so that you can subsequently get behind it for a powerful blow. If there is any way for you to get behind a boss, you should.
Cones and cleaves
Many boss abilities are targeted to hit targets in a particular area of their hitbox. Most dragons have a tail swipe ability that will knock down anyone inside of a 90-degree cone behind them, while cleaving anyone in front of them with a claw swipe, while also breathing fire on anyone in a similar 90-degree cone in front of them. You want to stay behind the dragon anyway, but you have to be careful to avoid the tail swipe as well. You typically want to position yourself right behind the line between the front and the back split. You'll be safe from the tail swipe while also avoiding any parries. Some mobs, like Majordomo Staghelm, have cleave-like abilities that break from this pattern, but it's fairly rare.
Expertise's role in positioning
An enemy can parry and dodge your attacks if you engage him from the front but can't parry you if you're behind him. He can still dodge your attacks from behind, though. Rogues can stack up expertise rating to reduce or eliminate the boss' ability to dodge us from behind, which can be useful for specs like combat and subtlety that rely heavily on physical damage.
While you could, in theory, get enough expertise to remove the boss' ability to parry, bosses have such a high parry rate that it's really not feasible to achieve. You would be sacrificing so many other stats in favor of expertise, when you could simply switch our position to negate the parries instead. There have been encounters where it can be favorable to stack expertise due to restrictions in boss positioning and specific mechanics. Stacking expertise for a particular fight might sound clever, but usually the cost of reforging or gemming your gear for one fight isn't worth the small gain you would see from the expertise itself.
Sneak in every Wednesday for our Molten Front ganking guide, a deep-dive into the world of playing a subtlety rogue -- and of course, all the basics in our guide to the latest rogue gear.