This actually sounds like a really good idea. The only problem is that if lets jut say you dont like me for xyz reason.. you pm your little buddies to gang up on me and neg rep me.. then im toast, no matter what reason. there needs to be other checks and balances.
Neg rep should definitely be a factor.
Maybe you should think about this a bit more and get a more bullet proof plan. But I think our on to something
The only way they could do that is if THEY got a few + reps for themselves. Your rep doesn't count (Neutral) if you don't have any rep yourself ...