Probably the best method would be too use a flat fee, or a percentage of list, and then add bonus commissions for performance in negotiations, using the average 'over/under asking price' in the local market as an index. But this is probably too complicated for most people, and at the end of the day, the Realtor is more concerned about the volume of deals they can close rather than the individual deals, so performance based commissions will still probably have minimal impact on behavior.
That's not too bad an incentive, because all other incentives lead to them closing more often than you'd like.