It's funny how misconceptions get started.
Are you ready to spill some blood in the name of the Light? For Velen? For Argus?
-- Vindicator Boros, What Argus Means To Me
We know that the orcish Horde led by Blackhand, corrupted by the blood of Mannoroth and manipulated by Gul'dan for his master Kil'Jaeden was a force of unrelenting bloodlust and cruelty, and that they waged a genocidal war on their draenei neighbors that came close to wiping them out. Not a very numerous race in the first place, the draenei were ultimately overwhelmed by the orcs.
Many seem to combine this with the fact that Velen led the first draenei from their homeworld of Argus to prevent them from being corrupted by the offer Sargeras made to Archimonde, Kil'Jaeden and Velen to accept his gifts and become his servants to assume that the draenei are pacifists. That they're a culture that eschews war and flees from conflict. They note that the draenei have numerous times fled before the advance of the Burning Legion, moving from world to world before the dimension ship they were aboard crashed on what would become known as Draenor.
When combined with their mastery of and reverence for The Holy Light and their association with the naaru, who themselves are beings that exude said Holy Light, it's easy to see why people make this mistake. They look at Velen, who prophecizes that one day the mortal races will unite against the Legion, and see this as a desire for peace. But it isn't, exactly. Velen isn't preaching peace - he's arguing that we're fighting the wrong war.