Remember, too, that the night elves weren't playing with your run-of-the-mill sorcery; they were frequently playing with magics supercharged (although at that point, probably unintentionally, which certainly isn't any kind of reassuring) by the Well of Eternity. That could easily allow for the kinds excesses that would drive off the Pandarens, without a general antipathy to the arcane.

And then there's this whole issue that there's some evidence that night elves don't really make good neighbors; not just their xenophobic response to Grom's timber cutting parties, but as I understand it, their wars with the Amani and Gurubashi pre-Sundering were more conquest than defense, and those wars took place in the same general time frame the Pandarens left.

Say... isn't there a troll city south of the night elf lands? I suspect Zul'Farrak wasn't a great neighbor, either. Nor would I want to be even vaguely between Zul'Farrak and Eldre'Thalas. While it's certainly possible that Old Pandaria is part of the lands destroyed in the Sundering, it was still almost certainly caught between the night elves and Amani when they went to war.

Keep in mind that the Pandaren had to take Pandaria from the Mogu; and that they apparently regarded that conquest as a better/safer/easier path than sticking around to get crushed between the night elves and the Amani.