Key criteria to consider when developing agency evaluation question sets

To make better decisions, it is vital to have data you can rely on. An evaluation is nothing without a strong, in-depth question set. Designing questionnaires or scorecards can be one of the more complicated portions of the process, as advertisers need to include input from the various departments interacting with the agency, and in many cases the agencies themselves.

Question sets are an integral part to the success of the agency evaluation

The goal of the question set is to balance the right numbers of questions to get actionable information. Those questions should be succinct, to-the-point, and relevant to the specific duties of the agency. Additionally, greater consideration is being given to phrasing questions to ensure they are short, clear and easily understood. For effective assessments, advertisers should consider the following principles to develop a meaningful and comprehensive question set:

Lean – This is the most visible change over the last two years, and one that most advertisers are trending towards. In the past, it was common for evaluations to contain too many questions. These questions were often verbose and overly “bloated” due to multiple stakeholders adding layer upon layer of content. The concern is that too many questions can lead to the same criteria being asked in different ways, causing confusion. Furthermore, too many questions can lead to evaluation “burnout”, particularly for those people (often senior marketing leadership) who are filling out multiple surveys on different agency relationships.  To address this, many advertisers are now dramatically reducing the total number of questions in the survey, perhaps down to around10-15 questions, to make them more impactful. Fewer questions and tight phrasing contribute to lean survey design.

Impactful – It is crucial that every question is linked to an actionable business result, and can drive positive change in the relationship. With the drive towards a leaner question set, “nice to have” questions should to be challenged, and if not absolutely needed, removed from the survey.

Relevant – When developing question sets, consider the types of agencies that are being evaluated. Some advertisers look to create a dedicated question set for each major discipline or channel (e.g. digital, creative, media, PR, etc.)  Keeping the overall theme of the questions similar, yet adjusting them for the particular nuances, allows consistency, yet targeted information gathering, without losing sight of the overall program goals. 

Consistent –Keeping the structure of the question set consistent, specifically using the same sections and keeping as many of the questions the same, has several advantages.  First, it brings added clarity and understanding to evaluators, particularly if they are filling out multiple surveys. This will also be less of a burden as all the surveys will have the same theme so less time is needed to decipher and analyze the questions.  Advertisers will also find that they are better able to analyze the results and agency benchmark performance against their peers.

Reflective – When developing a question set, it is important to understand whether the client is enabling the agency to do its best work. Therefore, each question in the “client-on- agency” evaluation should have a mirrored a question in the "agency-on-client" feedback evaluation. For example – if asking the client about how well the agency understood and implemented the brief, consider a similar question for the agency to complete regarding how well they feel they were briefed for the project. This will be very beneficial to uncover process issues that may be impacting both the efficiency and effectiveness of the relationship. 

For additional information regarding Agency Evaluation, please visit www.decideware.com.