Decideware

All animals are equal ...

Steve McKee of Business Week makes an interesting point in his article about agency selection (How to Hire an Ad Agency - see below for excerpt) on the point of using a spreadsheet with unweighted criteria to make a decision - in this case selecting a new agency. His point "Not every element on your list will be of equal value and a scoresheet can easily introduce an impressive-looking, but false, equation into the decision" certainly rings true to us.

If importance weighting is not used, or even if it is used but is not done accurately it is very easy to let numbers rule over common sense.

... but some animals are more equal than others 

So it is essential that if you do want to use a rating system to supplement common sense, you must accurately prioritize and weight the criteria you are using. There are quite a few well regarded ways of doing this, such as pair-wise comparison or ranked lists. We use the Analytical Hierarchy Process in our software.

Garbage In, Garbage Out

Whenever we facilitate a decision model we have to be very careful to ensure that our client understands the process and has the correct 'brains trust' at hand to weight the selection criteria. It is always fascinating watching the interactions that occur and the discussion generated during the prioritization process.

Sometimes very significant differences in perspective can surface, often to the surprise of the client group.

We caution reliance on the assessment scores that are generated. Indeed if clients do not feel that the numbers are right, there is almost always a reason. As an example there may be a critical risk that has not been factored into the equation, or the accuracy of the evaluation needs review.

As Steve correctly pointed out in his article, we should use the decision model to help frame and focus our thinking, not simply use the numeric outputs as a silver bullet.

The wonder of gut feel

One point that I would like to make is that in our experience there is an amazingly close correlation between the decision model scoring results and those made by 'gut feel'. Our software is used in the Ernst & Young Entrepreneur of The Year program to help with the judging process and one of the key comments we get from the judges each year is how much alignment there is between the scores produced from the decision models and their own their 'gut feel'.

The "Why?" is fairly obvious. If you have spent a good deal of time looking at what is important, and accurately rated each agency on these criteria, does it not stand to reason that the outcomes should match very closely with your natural judgement.

Author: Richard Benyon, Decideware

("5. Don't let a spreadsheet make your decision. It may help if you develop some sort of checklist to track and evaluate an agency's capabilities, but don't go so far as to develop a scoring system and award your account to the agency with the highest average. Not every element on your list will be of equal value and a scoresheet can easily introduce an impressive-looking, but false, equation into the decision. If it helps you bring some element of discipline to the process, fine, but in the end you have to go with your gut.")

("All animals are equal but some animals are more equal than others" with thanks to George Orwell