Here's some details on exactly how the multi-target diminishing returns are set to work in 5.4. Warning: here there be maths
The basic, concise explanation is as follows: The Nth strongest (based on pre-mitigation average auto attack DPS) mob that has hit you in the last 5 seconds grants 1/Nth of full vengeance with their attacks. N is recalculated on every hit taken.
So here's an example. Say you're tanking 3 mobs – we'll say it's a boss and two adds. The boss has pre-mitigation average auto attack DPS of 1500k, one add does 400k, and the other add does 300k. The game will form a list of those mobs from 1-3, like so:
1. Boss: 1500k DPS, 1/1 (100%) of normal vengeance is granted
2. First add: 400k DPS, 1/2 (50%) of normal vengeance is granted
3. Second add: 300k DPS, 1/3 (33.333%, repeating of course) of normal vengeance is granted
This would of course continue as more mobs are being tanked (granting 1/4, 1/5, 1/6, and so on). Also, if the second add were to land a hit at any point when the first add hasn't attacked you in the last 5sec (such as if it swings first), that attack would grant 1/2 Vengeance instead of 1/3.
Hope that clears things up for the theorycrafters.