Other
Views amidst violence: can perception surveys improve aid in conflict affected states?
This event was organised by the Secure Livelihoods Research Consortium (SLRC) in order to explore the challenges associated with delivering perception surveys in conflict-affected situations and to assess the extent to which they have successfully influenced aid policies and programming.
Interesting ODI seminar in London today #FCASperceptionsurveys: can perception surveys improve aid in fragile states? #aideffectiveness
#FCASperceptionsurveys have challenges but useful tool to assess aid impact in fragile&conflict states, shape policy and inform media.- SLRC's Research Director, Rachel Slater, began by setting out the aim of the event: to explore impacts of perception surveys on policies and programming in conflict affected situations. SLRC is currently in the process of designing a perception survey which will be implemented across its focus countries, from DRC to Pakistan, later this year.
Robert Muggah - Research Director and Program Coordinator for Violence Reduction at the Igarapé Institute and Fellow at the Instituto de Relações Internacionais, Pontifícia Universidade Católica do Rio de Janeiro
Robert Muggah - Fellow at the Instituto deRelações Internacionais, Pontifícia Universidade Católica do Rio de Janeiro - ODI Events on BlipRobert Muggah - Fellow at the Instituto deRelações Internacionais, Pontifícia Universidade Católica do Rio de Janeiro - explores the role...
#FCASperceptionsurveys Muggah has just started by talking about types of images that 'represent' Haiti- The first speaker, Robert Muggah, began by talking about his experiences in Haiti. Haiti has been dominated by prevailing media narratives and images of extreme violence, insecurity, and uncontrolled police, which have been used to justify spending on Security Sector Reform (SSR). It has also been reported that vigilante justice is a major issue for the Haitian people, for NGOs and for investors, but Robert questioned whether Haiti was any worse off in reality than its neighbours.
Robert went on to state that there have been several successive UN missions to Haiti which emphasised stabilisation and peace support, as well as two failed attempts at Disarmament, Demobilisation and Reintegration (DDR). Robert's organisation, The Igarapé Institute, were interested in finding out whether the above measures were legitimate and whether they actually led to improvements in safety and security. Since 2004, 20 surveys have been conducted - most semi-randomised - by highly trained enumerators. Multiple approaches were used, including focus group discussions and discourse analysis tools, in addition to quantitative surveys, although Robert emphasised that these were by no means the only tools to use in such contexts. The research found that, despite the media speculation, there was surprisingly limited evidence to justify the measures that were being implemented.
Haiti survey: key findings- Haiti has one of the lowest homicide rates in the Caribbean
- There has been a downward trend in physical assaults over recent years
- Property crime has generally been low and of small scale
- Most people would turn to the police if they experienced a crime, running somewhat contrary to claims of widespread vigilante justice
- NUDIST tool was used to highlight significance of risk factors
Muggah: Homicide on the (considerable) decline in Haiti 2004-2009. #FCASperceptionsurveys. Point: longitudinal & disaggregated data v.impt.- Muggah: can't use only surveys; multi-method approach more robust. #FCASperceptionsurveys
- Haiti survey: conclusionsRobert concluded that we need to be modest about the extent to which we can influence policy makers. We need to ensure we have credibility and that our work has legitimacy, which can be helped through the use of academic publications. Robert also felt that the media could prove to be an incredibly useful tool in getting survey findings into the public domain. There needs to be an investment in robust empirical methods and there should be transparency in the methods that are used. Surveys can be implemented rapidly and cost effectively, but they need to be supplemented through a multi-pronged approach. It is important to have a well trained local team who have a deep contextual understanding. The main value of longitudinal and geo-tagged datasets is that they allow for measurement over time.
George Varughese - previously The Asia Foundation’s Country Representative in Afghanistan (currently Country Representative in Nepal)
- RT @SLRCtweet: #FCASperceptionsurveys Time, resource, training important when conducting surveys to avoid "putting rubbish in and getting rubbish out"
- The second speaker, George Varughese of The Asia Foundation, started by stating that fragile environments are, by their very nature, highly dynamic and uncertain. Generally speaking, we have insufficient information on them, something which is made worse by local capacities to collect data.
George felt that one of the most important issues to bear in mind when doing perception surveys was the 'survey decision environment', be it pre, during or post-survey implementation: it is important to pay attention to all stages of the process. It is also advisable for survey objectives to be shared by other key actors, particularly those in-country, in order to both gain acceptance and secure buy-in to avoid negative reception of findings. Because of the nature of conflict affected states, researchers will often find themselves working on common problems with multiple actors. George also highlighted the work The Asia Foundation does to help build the capacity of local researchers, especially when conducting longitudinal surveys.
Afghanistan survey: conclusions
The Asia Foundation has been conducting perception surveys in Afghanistan since 2004. Reflecting on his experiences of overseeing many of these surveys, George concluded that it was important to distinguish between 'curiosity questions' and questions that generate 'actionable findings' (i.e. questions that lead to policy-relevant findings). There is also a danger of overstating the findings, meaning that it is important to look at the direction the findings are pointing in rather than focusing solely on their magnitude. George recognised the temptation or pressure researchers often feel to say something the research donor or government likes, but emphasised that this must be resisted. It was also noted that in conflict affected states it is often the case that researchers do not share their findings, which can result in research being duplicated. Finally, cultural sensitivity is a key factor which needs to be taken into consideration when conducting surveys in sensitive contexts. - #FCASperceptionsurveys Varughese: important to brief audiences prior to release of survey data to prevent misinterpretation
Tom Parks - The Asia Foundation's Regional Director for Governance and Conflict
- Parks: different burden of proof in FCAS. Need to manage expectations re:rigour. Still need to improve evidence base. #FCASperceptionsurveys
- The final speaker, Tom Parks, took a wider view on perception surveys, looking at general trends and reflecting the nature of The Asia Foundation's current work. The Asia Foundation have been systematically using surveys to explore a range of governance issues and monitor programmes. In most cases, they are collecting perceptions in order to improve something tangible; hence, the vast majority have a programme and policy implication. More recently, they are being used to monitor programmes to estimate impact. Tom felt that a strength of perception surveys is that they make it possible to identify the micro-level impacts of macro-level policies.
Perception survey challenges
Tom highlighted the tension between the current 'fragile states agenda' and empirically driven official development assistance (ODA), which he felt raises a dilemma: if we really want to understand the dynamics of fragility, we have to ask a lot of questions that are hard to measure. There is also a worrying gap between information regarding project-level monitoring and the real changes taking place on the ground. Ultimately, in an era where we have to justify aid programmes it is becoming very hard to produce the level of evidence that’s expected.
How can we shape the current 'fragile states agenda' so it allows for more rigour and better evidence?
Tom felt that perception surveys could be a large part of the answer. State capacity and service provision are not the only factors that determine violence levels; political factors and perceptions of legitimacy are often the more important drivers of fragility. Tom further explained that surveys are a particularly powerful tool in fragile states in terms of influencing policy debates and informing programme monitoring. In the fields of peacebuilding and state-building, many of us are guilty of making sweeping assumptions (e.g. 'service provision will result in state legitimacy and thereby result in violence reduction'). The lack of data thus far has been a major stumbling block to raising the policy profile of these issues, but, perhaps more troublingly, the weak evidence base leads us blind into a lot of things, such as:1) Local level conflict dynamics: from the outside, practitioners presume some degree of homogeneity in conflict-affected areas. However, perception surveys in Mindanao in the Philippines show there is extreme variation even within districts in terms of the effiects of Community Driven Development (CDD) projects on violence levels.2) Troubled state-society relations: political alliances and identity variables are big determinants of perceptions of governments, who can sometimes alienate certain parts of the population through their policies.
Final reflections
In fragile states, Tom felt an alternative burden of proof is necessary. Perception surveys cannot be held to the same standards as those conducted in non-conflict affected situations, because the kinds of issues looked at were not as straightforward as they might be in stable states e.g. state-society relations. - Parks: question not whether we should do surveys in FCAS, but how to do them better. #FCASperceptionsurveys
