Developing Indicators and Measuring Achievements AdvancedTraining on Prevention of Corruption Techniques &Methodologies 10...
Applying the tools to the RBM life-cycle approach? IMPACT OUTPUT Planning Evaluation Monitoring Capacity Capacity Capaci...
Overview of presentation <ul><li>Designing indicators </li></ul><ul><li>Measuring/Monitoring your indicators </li></ul><ul...
Overview of presentation <ul><li>Designing indicators </li></ul><ul><ul><li>Basic steps </li></ul></ul><ul><ul><li>Actiona...
Key steps in conducting an effective assessment Identify national institution or civil society organisation for hosting ...
10 features of an effective assessment development <ul><li>Strategy is aligned to national policy priorities and processe...
Why participatory adaptation? <ul><li>Technical benefits </li></ul><ul><li>Customized </li></ul><ul><li>Actionable </li></...
Participation at 2 levels <ul><li>Through process : </li></ul><ul><ul><li>Participation in design </li></ul></ul><ul><ul>...
Stakeholder Analysis <ul><li>Identifying the key stakeholders and their interests in reform (positive or negative) </li></...
Significant influence Some influence Little influence No influence Significantly interested in reform Some interest Little...
Significant influence Some influence Little influence No influence Significantly interested in reform Ministry of Local Go...
THE RBM LIFE-CYCLE APPROACH
Applying the tools to the RBM life-cycle approach? IMPACT OUTPUT Planning Evaluation Monitoring Capacity Capacity Capaci...
Overview of presentation <ul><li>Designing indicators </li></ul><ul><ul><li>Basic steps </li></ul></ul><ul><ul><li>Actiona...
Which, if either, of these indicators are actionable? <ul><li>1. How often do firms make extra payments to influence the...
Activity: two examples <ul><li>Non-actionable </li></ul><ul><li>1. How often do firms make extra payments to influence t...
Actionable or ‘action-worthy’? <ul><li>Actionable does not mean action-worthy </li></ul><ul><ul><li>e.g. Measuring whet...
Determining what is an ‘action-worthy indicator’ <ul><li>Is it measuring interventions that are truly ‘beneficial’? </li>...
Overview of presentation <ul><li>Designing indicators </li></ul><ul><ul><li>Basic steps </li></ul></ul><ul><ul><li>Actiona...
What indicators measure Corruption Integrity Perception Public opinion Experts Public sector Experience / victimisation...
Examples of corruption indicators <ul><li>Number of reported cases of bribery within e.g. the police </li></ul><ul><li>Ran...
Corruption indicators can measure… <ul><li>The incidence of corrupt transactions </li></ul><ul><ul><li>experiencial and p...
Integrity indicators <ul><li>Also called ‘transparency’ or ‘accountability indicators’ </li></ul><ul><li>Measure the rever...
What indicators measure Corruption Transparency/Accountability/Integrity Perception Public opinion Experts Public sector...
Example of a fact-based / experience-based question <ul><li>In some areas there is a problem of corruption among publi...
Example of a perception-based question <ul><li>It is known that in some countries the problem of corruption among public...
Raise your hand every time you agree with a statement: <ul><li>Perceptions are better indicators of progress than facts <...
Drawbacks of perception data <ul><li>Seen as biased, leading some to minimize its use, or rule it out entirely </li></ul><...
But perception data matters! <ul><li>High perceptions of corruption are correlated with low perceptions of state legitimac...
What indicators measure Corruption Transparency/Accountability/Integrity Perception Public opinion Experts Public sector...
Inputs and outcomes <ul><li>Inputs (de jure, laws) </li></ul><ul><li>vs. </li></ul><ul><li>Outcomes (de facto, practice) <...
de jure indicators vs. de facto indicators <ul><li>De jure: </li></ul><ul><li>In law, is there an agency with a legal m...
Using de jure and de facto indicators together <ul><li>Combine de jure and de facto indicators to show discrepancies b...
Discrepancies in law and practice are revealing <ul><li>Are laws and organisational reforms translating into impact on th...
Why generate poverty and gender sensitive indicators <ul><li>Corruption exacts a higher price on women and the poor. </li...
4 ways to make indicators sensitive to vulnerable groups <ul><li>Disaggregating by poverty/gender </li></ul><ul><li>Speci...
Methods of measurement <ul><li>Objective analysis </li></ul><ul><li>Statistics </li></ul><ul><li>Surveys </li></ul><ul><li...
Using benchmarks for Action Plans (vs. baselines) <ul><li>It’s not just what you measure, it’s what you measure it agains...
Developing a scale to quantify qualitative indicators <ul><li>Example of qualitative indicator: In practice, are major...
How are the tools above applicable to the RBM life-cycle approach?
Applying the tools to the RBM life-cycle approach? Planning Evaluation Monitoring Actionable Actionable Corruption Corrupt...
Exercise: Selecting Strategy and Action Plan indicators <ul><li>Goal: To learn how to identify impact and output indicator...
Strategy Goal 1: A decrease in corruption in the judiciary <ul><li>Step 1: Who are the stakeholders who could provide info...
Strategy Goal 2: A decrease in corruption in the judiciary ( perceptions) <ul><li>What questions could you ask of citizens...
Monitoring the Action Plan Objective: Increase in transparency of the legislature <ul><li>What indicators would you propos...
Benchmarking the Action Plan Indicator for Monitoring <ul><li>No compliance: 0%-30% of legislative proceedings are publi...
Evaluating impact (longer term) <ul><li>Corruption data and Integrity data </li></ul><ul><li>Experience- based and Percept...
Don’t reinvent the wheel! <ul><li>Many useful tools are already out there, they just require some adaptation. </li></ul>
<ul><li>Thank you! </li></ul><ul><li>Asmara Achcar </li></ul><ul><li>Regional Specialist, Governance Assessments </li></ul...
of 49

Prevention of corruption, developing indicators and measuring achievements

Advanced Training in prevention of Corruption Systems and Methodologies,10-11 November, Vienna Asmara Achcar
Published on: Mar 4, 2016
Published in: Self Improvement      Technology      
Source: www.slideshare.net


Transcripts - Prevention of corruption, developing indicators and measuring achievements

  • 1. Developing Indicators and Measuring Achievements AdvancedTraining on Prevention of Corruption Techniques &Methodologies 10-11 November 2011 Asmara Achcar
  • 2. Applying the tools to the RBM life-cycle approach? IMPACT OUTPUT Planning Evaluation Monitoring Capacity Capacity Capacity Capacity Capacity Participatory process
  • 3. Overview of presentation <ul><li>Designing indicators </li></ul><ul><li>Measuring/Monitoring your indicators </li></ul><ul><li>Potential models/examples </li></ul>
  • 4. Overview of presentation <ul><li>Designing indicators </li></ul><ul><ul><li>Basic steps </li></ul></ul><ul><ul><li>Actionable indicators </li></ul></ul><ul><ul><li>Different indicator approaches </li></ul></ul><ul><li>Measuring/Monitoring your indicators </li></ul><ul><li>Potential models/examples </li></ul>
  • 5. Key steps in conducting an effective assessment Identify national institution or civil society organisation for hosting initiative Conduct multi-stakeholder dialogue on governance priorities Decide on assessment framework Decide on indicators Decide on sampling Analyse results Disseminate results Conduct multi-stakeholder consultation Develop policy recommendations Conduct policy reform Institutionalize and repeat at regular intervals Identify key stakeholders Decide how to collect data Establish a steering committee Raise funds Decide who will do the research Select type of assessment
  • 6. 10 features of an effective assessment development <ul><li>Strategy is aligned to national policy priorities and processes </li></ul><ul><li>Strategy is country contextualized </li></ul><ul><li>Methodology is rigorous </li></ul><ul><li>Selection of indicators is transparent and participatory </li></ul><ul><li>Results are stored in a public national database </li></ul><ul><li>Indicators are pro-poor and gender-sensitive </li></ul><ul><li>Capacity of national stakeholders is developed </li></ul><ul><li>Assessment is cost-effective and timely </li></ul><ul><li>Results are widely disseminated </li></ul><ul><li>Assessment is repeated </li></ul>
  • 7. Why participatory adaptation? <ul><li>Technical benefits </li></ul><ul><li>Customized </li></ul><ul><li>Actionable </li></ul><ul><li>Political benefits </li></ul><ul><li>Buy-in </li></ul><ul><li>Credibility </li></ul><ul><li>Legitimacy and public trust </li></ul><ul><li>Consensus-building and political will </li></ul><ul><li>Efficiency benefits (usage) </li></ul><ul><li>Adapted to needs </li></ul><ul><li>Integrated in planning </li></ul><ul><li>Sustainable </li></ul>Increase impact of assessment
  • 8. Participation at 2 levels <ul><li>Through process : </li></ul><ul><ul><li>Participation in design </li></ul></ul><ul><ul><li>Participatory monitoring </li></ul></ul><ul><li>Through methodology : </li></ul><ul><ul><li>Inclusive data sources </li></ul></ul><ul><ul><li>Gender-sensitive and pro-poor governance indicators </li></ul></ul>
  • 9. Stakeholder Analysis <ul><li>Identifying the key stakeholders and their interests in reform (positive or negative) </li></ul><ul><li>Assessing the influence and importance of each stakeholder </li></ul><ul><li>Identifying measuring needs of stakeholders (basis for engagement) </li></ul>
  • 10. Significant influence Some influence Little influence No influence Significantly interested in reform Some interest Little interest No interest
  • 11. Significant influence Some influence Little influence No influence Significantly interested in reform Ministry of Local Government state commission for the prevention of corruption citizens NGO 2 NGO 3 The poor Women Some interest President of municipal councils ministry of transport NGO 1 state audit office bureau of public procurement Media 2 NGO 4 Little interest Mayors Business community Media 1 ministry of environmental protection and urban planning No interest Chief of administration Local public servants
  • 12. THE RBM LIFE-CYCLE APPROACH
  • 13. Applying the tools to the RBM life-cycle approach? IMPACT OUTPUT Planning Evaluation Monitoring Capacity Capacity Capacity Capacity Capacity Participatory process
  • 14. Overview of presentation <ul><li>Designing indicators </li></ul><ul><ul><li>Basic steps </li></ul></ul><ul><ul><li>Actionable indicators </li></ul></ul><ul><ul><li>Different indicator approaches </li></ul></ul><ul><li>Measuring/Monitoring your indicators </li></ul><ul><li>Potential models/examples </li></ul>
  • 15. Which, if either, of these indicators are actionable? <ul><li>1. How often do firms make extra payments to influence the content of new legislation? </li></ul><ul><li>2. How many judges and magistrates do you think are involved in corruption? </li></ul><ul><li>How would you make them more ‘actionable’? </li></ul>
  • 16. Activity: two examples <ul><li>Non-actionable </li></ul><ul><li>1. How often do firms make extra payments to influence the content of new legislation? </li></ul><ul><li>2. How many judges and magistrates do you think are involved in corruption? </li></ul><ul><li>Actionable </li></ul><ul><li>In practice, are the regulations restricting post-government private sector employment for national legislators effective? </li></ul><ul><li>In practice, are the requirements for the independent auditing of the asset disclosure forms of members of the national legislature fulfilled? </li></ul><ul><li>In practice, when necessary, does the judicial disciplinary agency initiates investigations? </li></ul><ul><li>In practice, when necessary, does the judicial disciplinary agency impose penalties on offenders? </li></ul>
  • 17. Actionable or ‘action-worthy’? <ul><li>Actionable does not mean action-worthy </li></ul><ul><ul><li>e.g. Measuring whether a country has an independent anti-corruption commission when there is no guarantee it will reduce corruption </li></ul></ul><ul><ul><li>e.g. Measuring the speed of judicial proceedings when not clear that increasing the speed will ensure that justice is done </li></ul></ul><ul><li>There is a risk of measuring things because they are easily measurable </li></ul>
  • 18. Determining what is an ‘action-worthy indicator’ <ul><li>Is it measuring interventions that are truly ‘beneficial’? </li></ul><ul><ul><li>Must be appropriate for country context </li></ul></ul><ul><ul><li>Should be in line with reform priorities </li></ul></ul><ul><ul><li>Must not further entrench corrupt systems </li></ul></ul><ul><li>Can the performance measured be attributed to specific policy inputs? (‘actionable’) </li></ul>
  • 19. Overview of presentation <ul><li>Designing indicators </li></ul><ul><ul><li>Basic steps </li></ul></ul><ul><ul><li>Actionable indicators </li></ul></ul><ul><ul><li>Different indicator approaches </li></ul></ul><ul><li>Measuring/Monitoring your indicators </li></ul><ul><li>Potential models/examples </li></ul>
  • 20. What indicators measure Corruption Integrity Perception Public opinion Experts Public sector Experience / victimisation General population / vulnerable groups Public sector Private sector Diagnostic Assessments Compliance monitoring Institutions Processes Sectors Local level
  • 21. Examples of corruption indicators <ul><li>Number of reported cases of bribery within e.g. the police </li></ul><ul><li>Ranking of most corrupt institutions in public national perception surveys </li></ul><ul><li>Bribe payments as % of income among individuals in low-income neighborhoods </li></ul><ul><li>Proportion of sanctions to registered complaints </li></ul>
  • 22. Corruption indicators can measure… <ul><li>The incidence of corrupt transactions </li></ul><ul><ul><li>experiencial and perception-based data </li></ul></ul><ul><li>The incidence of sanctions for corruption </li></ul><ul><ul><li>statistical comparison </li></ul></ul><ul><li>The impact of corruption </li></ul><ul><ul><li>data sensitive to vulnerable groups & gender </li></ul></ul>
  • 23. Integrity indicators <ul><li>Also called ‘transparency’ or ‘accountability indicators’ </li></ul><ul><li>Measure the reverse of corruption: what is in place to prevent it </li></ul><ul><li>Focus on the effectiveness of anti-corruption mechanisms which are within the control of policy makers </li></ul><ul><ul><li>Therefore tend to be ‘actionable’ </li></ul></ul>
  • 24. What indicators measure Corruption Transparency/Accountability/Integrity Perception Public opinion Experts Public sector Experience / victimisation General population / vulnerable groups Public sector Private sector Diagnostic Assessments Compliance monitoring Institutions Processes Sectors Local level
  • 25. Example of a fact-based / experience-based question <ul><li>In some areas there is a problem of corruption among public officials. During 2011, has any government official, for instance a customs officer, police officer or inspector in your own country, asked you or expected you to pay a bribe for his service? </li></ul><ul><ul><li>yes </li></ul></ul><ul><ul><li>no </li></ul></ul><ul><ul><li>don’t know </li></ul></ul>
  • 26. Example of a perception-based question <ul><li>It is known that in some countries the problem of corruption among public officials is perceived to be high by citizens. Imagine a person who needs something that is entitled to him/her by law. Is it likely or not likely that this person would have to offer money, a present or a favour (e.g. more than the official charge) to get help from: </li></ul><ul><ul><li>Member of Parliament? </li></ul></ul><ul><ul><li>Elected municipal councillors? </li></ul></ul><ul><ul><li>Customs officials? </li></ul></ul><ul><ul><li>Tax/revenues officials? </li></ul></ul>
  • 27. Raise your hand every time you agree with a statement: <ul><li>Perceptions are better indicators of progress than facts </li></ul><ul><li>Facts are better indicators of progress than perceptions </li></ul><ul><li>Both perceptions and fact-based indicators are useful </li></ul><ul><li>Neither is helpful, it is not possible to measure corruption </li></ul>
  • 28. Drawbacks of perception data <ul><li>Seen as biased, leading some to minimize its use, or rule it out entirely </li></ul><ul><li>Politically sensitive, especially in polarized political climates, where the media is seen to have undue influence in magnifying public perceptions of corruption </li></ul><ul><li>Lag effect: perceptions often lag behind reforms </li></ul>
  • 29. But perception data matters! <ul><li>High perceptions of corruption are correlated with low perceptions of state legitimacy </li></ul><ul><li>High perceptions can fuel corrupt practices by: </li></ul><ul><ul><li>encouraging people to believe they must pay bribes </li></ul></ul><ul><ul><li>reducing the likelihood of citizens reporting </li></ul></ul><ul><ul><li>leading those with power to believe that there is nothing wrong with accepting bribes. </li></ul></ul><ul><li>Perceptions and opinion reveal important information about how corruption works in a specific context </li></ul>
  • 30. What indicators measure Corruption Transparency/Accountability/Integrity Perception Public opinion Experts Public sector Experience / victimisation General population / vulnerable groups Public sector Private sector Diagnostic Assessments Compliance monitoring Institutions Processes Sectors Local level
  • 31. Inputs and outcomes <ul><li>Inputs (de jure, laws) </li></ul><ul><li>vs. </li></ul><ul><li>Outcomes (de facto, practice) </li></ul>
  • 32. de jure indicators vs. de facto indicators <ul><li>De jure: </li></ul><ul><li>In law, is there an agency with a legal mandate to address corruption? </li></ul><ul><li>De facto: </li></ul><ul><li>In practice, is the anti-corruption agency effective? </li></ul><ul><ul><li>Does the ACA make regular public reports (e.g. to legislature)? </li></ul></ul><ul><ul><li>When necessary, is the ACA able to independently initiate investigations? </li></ul></ul><ul><li>Can citizens access the anti-corruption agency? </li></ul><ul><ul><li>Does the ACA act on complaints within a reasonable time period? </li></ul></ul><ul><ul><li>Can citizens complain to the ACA without fear of recrimination? </li></ul></ul>
  • 33. Using de jure and de facto indicators together <ul><li>Combine de jure and de facto indicators to show discrepancies between: </li></ul><ul><ul><li>change in law & resources ($, infrastructure, procedures, staff) </li></ul></ul><ul><ul><li>and </li></ul></ul><ul><ul><li>change in practice (‘improved governance’ in practice) </li></ul></ul>
  • 34. Discrepancies in law and practice are revealing <ul><li>Are laws and organisational reforms translating into impact on the ground? If not, why not? </li></ul><ul><ul><li>Inappropriate reforms? </li></ul></ul><ul><ul><li>Economically perverse incentives? </li></ul></ul><ul><ul><li>Vested political interests? </li></ul></ul><ul><ul><li>Lack of capacity (e.g. human resources, information retrieval systems)? </li></ul></ul><ul><ul><li>High tolerance of corrupt conduct by the public? </li></ul></ul><ul><ul><li>Different understandings of what is corrupt conduct? </li></ul></ul>
  • 35. Why generate poverty and gender sensitive indicators <ul><li>Corruption exacts a higher price on women and the poor. </li></ul><ul><li>Rich/middle class men tend to be in positions of power, and thus are beneficiaries of corruption. </li></ul><ul><li>Need to be sensitive to effectiveness of anti-corruption mechanisms on these groups </li></ul><ul><li>Indicators too often tend to be gender and poverty blind </li></ul>
  • 36. 4 ways to make indicators sensitive to vulnerable groups <ul><li>Disaggregating by poverty/gender </li></ul><ul><li>Specific to the poor/women </li></ul><ul><li>Implicitly poverty/gender sensitive </li></ul><ul><li>Chosen by the poor/women </li></ul>
  • 37. Methods of measurement <ul><li>Objective analysis </li></ul><ul><li>Statistics </li></ul><ul><li>Surveys </li></ul><ul><li>Quantifying the qualitative </li></ul>
  • 38. Using benchmarks for Action Plans (vs. baselines) <ul><li>It’s not just what you measure, it’s what you measure it against . </li></ul><ul><li>Benchmarks are imperative to judge progress, demonstrate change, measure success. </li></ul><ul><li>Use international treaties; standards from the World Bank, TI, etc.; or country-specific objectives </li></ul>
  • 39. Developing a scale to quantify qualitative indicators <ul><li>Example of qualitative indicator: In practice, are major public procurements effectively advertised? </li></ul><ul><li>Quantification: </li></ul><ul><li>Score 10: There is a formal process of advertising public procurements. This may include a government website, newspaper advertising, or other official announcements. All major procurements are advertised in this way. Sufficient time is allowed for bidders to respond to advertisements. </li></ul><ul><li>Score 5: There is a formal process of advertising but it is flawed. Some major procurements may not be advertised, or the advertising process may not be effective. The time between advertisements and bidding may be too short to allow full participation. </li></ul><ul><li>Score 0: There is no formal process of advertising major public procurements or the process is superficial and ineffective. </li></ul>
  • 40. How are the tools above applicable to the RBM life-cycle approach?
  • 41. Applying the tools to the RBM life-cycle approach? Planning Evaluation Monitoring Actionable Actionable Corruption Corruption Integrity Integrity Experience-based Fact-based Fact-based Experience-based Perceptions Perceptions Impact Impact Output Output Satisfaction Satisfaction Participatory process
  • 42. Exercise: Selecting Strategy and Action Plan indicators <ul><li>Goal: To learn how to identify impact and output indicators for long-term and short-term results </li></ul>
  • 43. Strategy Goal 1: A decrease in corruption in the judiciary <ul><li>Step 1: Who are the stakeholders who could provide information on this goal? </li></ul><ul><ul><li>Citizens with contact with the judiciary </li></ul></ul><ul><ul><li>Lawyers </li></ul></ul><ul><ul><li>Civil society organizations that work on legal issues </li></ul></ul><ul><ul><li>Civil servants with contact with the judiciary </li></ul></ul><ul><li>Step 2: What questions could you ask of citizens to help monitor this goal? </li></ul><ul><ul><li>Have you had contact with the legal system in the past 12 months? If so, were you asked to pay a bribe? </li></ul></ul><ul><ul><li>How much have you paid in bribes to judges, lawyers, or other legal officers in the last 12 months? </li></ul></ul>
  • 44. Strategy Goal 2: A decrease in corruption in the judiciary ( perceptions) <ul><li>What questions could you ask of citizens to help monitor this goal? </li></ul><ul><ul><li>Do you think corruption in the judiciary has decreased over the past x years? </li></ul></ul><ul><ul><li>In your opinion, do you think there is a higher, equal, or lower level of corruption in the judiciary as compared to [insert another key institution]? </li></ul></ul><ul><ul><li>In your opinion, do you think there is a higher, equal, or lower level of corruption in the judiciary as compared to [insert neighboring country]? </li></ul></ul>
  • 45. Monitoring the Action Plan Objective: Increase in transparency of the legislature <ul><li>What indicators would you propose to monitor this objective? </li></ul><ul><ul><li>What percentage of legislative proceedings are published on a publicly available website? </li></ul></ul><ul><ul><li>What percentage of public requests for legislative information under the FOI law are granted? </li></ul></ul><ul><ul><li>What is the average length of time it takes for an FOI request to be granted? </li></ul></ul>
  • 46. Benchmarking the Action Plan Indicator for Monitoring <ul><li>No compliance: 0%-30% of legislative proceedings are published </li></ul><ul><li>Partial compliance: 30%-90% of legislative proceedings are published </li></ul><ul><li>Full compliance: 90%-100% of legislative proceedings are published </li></ul><ul><ul><li>What percentage of legislative proceedings are published on a publicly available website? </li></ul></ul>
  • 47. Evaluating impact (longer term) <ul><li>Corruption data and Integrity data </li></ul><ul><li>Experience- based and Perception based </li></ul><ul><li>Actionable/not </li></ul><ul><li>Baseline and Benchmarks </li></ul><ul><li>>> Goal to assess BUT ALSO to feeds into next planning stage </li></ul>
  • 48. Don’t reinvent the wheel! <ul><li>Many useful tools are already out there, they just require some adaptation. </li></ul>
  • 49. <ul><li>Thank you! </li></ul><ul><li>Asmara Achcar </li></ul><ul><li>Regional Specialist, Governance Assessments </li></ul><ul><li>[email_address] </li></ul>

Related Documents