1) As I understand it, its RNG for the most part. Having a higher skill should help your chances of getting a rare artifact, but I've seen people get rares at 175, while I didn't a single one until 240. There are also a handful of rare items you can't get until 450. Ideally though, you should be repeating common items until you have all of them, though I've had a couple times where I've repeated common items without having the rest of them. Essentially, its entirely random.

2) Your fragments never go anywhere, so feel free to go over that 32 limit, the extras will start counting for your next project. I know a lot of people who save up troll fragments, as the only rare troll piece is a raiding-quality 2h sword that you can only see when you hit 450 skill.

3) The only thing exchangeable in archaeology currently are keystones. Highborne scrolls, troll tablets, dwarf rune stones, those kinds of things. Prices on these vary as (in my experience) some keystones are rarer than others. I've got a whole stack of troll keystones, but very rarely do a find the ones in Outland or Northrend, so prices tend to vary wildly from 40g to 500g.
The artifacts you get are stored permanently in the archaeology interface, so you might as well just vendor your finds (sadly they don't sell for much. at all.) unless you have bag/bank space to spare.