This is probably a good question to revisit since we're heading into a patch where lots and lots of us will be running pickup raids and groups. Souldreamer on WoW Ladies LJ asks: "just when do you give up on a raid?" It's a good question, and unfortunately, the answer probably depends on the raid itself. If, going in, you're not sure just how much DPS the raid can do, and your healer says he's actually specced prot, and you were planning on going to bed anyway, one wipe is probably enough to call it. On the other hand, if you've dropped a few bosses and have an issue with the tank losing aggro for a second on a boss, you'll probably go back for another few wipes just to see.
Do any of you have an actual policy? I tend to not get involved in PuGs at all if I think there's a chance they won't make it -- there are too many fish in the sea, and too many other things for me to work on rather than beating my head up against a boss. But maybe it would be good to set up a rule that most of us can agree on, something like, "three wipes and you're out." That might save a lot of time and frustration in the new Dungeon system.
Of course, the good news is that, on the opposite side of things, Blizzard has made it pretty easy to fill in spots for people who leave -- you just queue up again and get another player (though we'll have to see how people feel about signing up for an instance and appearing there only to find that two bosses are already down). But I'm sure quite a few groups will end up having problems, and when they do, and that stranger who's supposed to be healing decides to call it quits rather than continue on, just remember: they've had one too many wipes.