Before you can do anything with pet battles, you have to be at least level 5 and have up to 10g, depending on whatever discount you may have.
Now travel to your capital city and chat with the Battle Pet Trainer there. Varzok is located near the flightmaster in Orgrimmar, and Audrey Burnhep hangs out near the Cataclysm portals in Stormwind City.
You only need to train on one of your characters, as all of the training, quests, and battle pets are account-wide. This means that if you train with your main character in Stormwind, your level 1 blood elf can start collecting pets for you in Silvermoon.
But wait! There's more!
Get a pet
You probably already have a pet or 20 if you've been playing for a while, but new players often don't have any. If your Pet Journal is empty, Varzok and Audrey will sell you one of your faction pets. You need a pet that can battle in order to collect pets. Put this pet in your first battle slot in your Pet Journal.
Your first quest
Now you can pick up Learning the Ropes from your trainer, which will ask you to win a pet battle. You have to go out into the wild, so just exit Orgrimmar and look for the blue area on your minimap. There should be plenty of pets there, unless they are camped. Go closer to the troll starting area if pets are sparse, as players don't normally go that far. All pet battle quests are account-wide, which means that the quest can be completed by a different character than the one who picked the quest up.
Choosing a wild pet
You will now be able to see green paw prints on your minimap and on your main screen on top of capturable pets. Move around a bit until they pop up. Approach one of them and mouseover. Your pets all begin at level 1, so you want to fight the same level to have your best chance of success. There are other factors, such as whether your attacks are strong against the target and its attacks are strong against yours, but we'll leave that for a later tutorial.
Right click on the pet to attack.
It's battlin' time!
The pet battle UI will appear, show you some information about the pet you are attacking and also give you access to your one attack for your one pet. (You'll get more attacks per pet and up to three pets per battle as you increase your pet battle accomplishments.) The battle is turn-based, so press 1 to attack and see the results. Repeat until the battle is over. If you lose, go find another level 1 wild pet and try, try again. Regardless of whether your win or lose, any living pet will gain experience. Unfortunately, you only have one pet when you first start out, so defeat means death which also means no XP.
Your trainer will continue to give you more quests until you are familiar with the most important aspects of pet battles. Completing each quest completes it on all of your characters. You have to complete a series of quests across many zones in order to unlock the dailies. As with the other quests, dailies are account-wide and can only be completed once a day by any character.
Battling other players
Once you feel ready to challenge other players, go to your Pet Journal and choose Find Battle on the bottom right. You will be put in a queue, waiting for a somewhat evenly matched challenger. Once the battle is over, your pets will be healed to full. Or you can right-click another player's portrait and choose Pet-Battle Duel.
Wild pets can be found almost anywhere and raise in level along with the zone. As you continue to battle, your pets will gain experience and level up. You'll be caging wild pets as your trainer taught you and they will vary in quality from poor to rare. (There are no epic wild pets at this time.) Try to keep caging pets until you get rares of each kind, unless you are just collecting.
Achievements abound. Your cages will improve. Your Pet Journal will fill. And your addiction will worsen. At least mine did, and most of the players I know. Enjoy and catch 'em all.