In the last post we focused on the problem definition and in this post we will discuss underlying evidence. Here are links to the articles of the different parts of the book:

Introduction

Step I: Define the problem

Step Two: Assemble some evidence

Bardach boils policy analysis down to two major activities: thinking and data gathering. Whereas the thinking clearly is the more important data gathering, the latter will consume most of the time. To make matters worse time pressure can be as bad or worse than political bias. Therefore data collection should be goal-oriented from the onset and focused on turning information into evidence (i.e. important information that affects existing believes).

Evidence is used for three main purposes:

• Assess the nature and extend of the problem
• Assess particular features of a concrete policy situation
• Assess policies that have been thought to have been worked effectively in apparently similar situations

The range of assessments requires the evidence assembly to be performed several times indifferent stages.

Bardach recommends to actually focus on thinking through what kind of data to collect and how to use it rather than just starting to collect and look where you end up. This is due to the value of evidence. The cost of acquiring data must be compared to the strength of evidence it can achieve:

• likelihood of the new evidence to cause a shift from a original decision to a substitute decision
• likelihood of the substitute decision producing a better policy outcome than the original
• magnitude of the difference between the original and substitute

To reduce costs “educated guesses” may be employed to rule out or require closer investigation. Guiding questions for this guesstimate should be:

• If the data looks different than expected, what will be the implications?
• Compared to the best expectation, how bad could the data be if it was really gathered?
• How much does the evidence improve if I switch from guesstimate to real analysis?

Available Literature can be useful and is often readily available on the internet. However, it but must be used with care as it is often produced by biased advocacy groups that may favour publishing research favouring their cause.

Data can also be gathered by analogy where a problem (and its possible solution) distinct on the surface is queried to gain insides (how to disbar incompetent lawyers by analysing how incompetent physicians loose their license).

Many activities need long preparation times and/or need to be squeezed into busy schedules, so administrative steps should be taken as early as possible (e.g. request access to archives, schedule interviews, etc.).

Policy analysis are not performed in a vacuum and therefore the work should be exposed to critics in the process to mitigate complaints about the resulting policy. Biases can be countered by explicitly contacting critics of sources, sponsors or other directly interested parties.

A more potent feedback cycle can be used when a policy is implemented iteratively and takes in feedback from the participants. In that case the policy analyst becomes a partner/broker in the process.