An Introduction to Triangulation
UNAIDS Monitoring and Evaluation Fundamentals
UNAIDS Monitoring and Evaluation Fundamentals
An Introduction to Triangulation
I would like to welcome you to the UNAIDS Monitoring and Evaluation Fundamentals series. As the response to the global HIV epidemic continues to evolve, monitoring and evaluation (M&E) has become more important than ever. Determining what programs do or do not work; implementing programs with proven cost-effectiveness; monitoring progress towards achieving targets; and ensuring accountability are objectives which are especially important now in the HIV response, as well as in other health and development areas. Thus, it is increasingly important that M&E is better understood, communicated in simplified language, and conducted in a coordinated and sustainable manner that generates information that can be easily used. Further, it is essential that M&E addresses the needs of and involves all key stakeholders right from the start and that results are made publicly available and utilized strategically in policy-making, planning, and program improvement. This series provides a common sense introduction to a range of M&E issues. It covers the fundamentals and their practical applications and includes techniques and tools for managing M&E of the HIV epidemic and response. Although the series uses HIV as its focus, the M&E fundamentals are also relevant to other areas of public health and development. As such, these books may also be useful in strengthening national M&E systems designed to track progress in other health and development goals, such as those outlined in the United Nations Millennium Development Goals (MDGs). I hope you find this series useful and welcome your feedback and suggestions on this and future topics for the series. With my best regards, Deborah Rugg, PhD Chief, UNAIDS Monitoring and Evaluation Division
UNAIDS | 05
ACKNOWLEDGEMENTS David Hales was the primary author of this book. He received invaluable input from Greet Peersman (Tulane University, New Orleans), Deborah Rugg (UNAIDS, Geneva) and Eva Kiwango (UNAIDS, Geneva). The following reviewers provided excellent comments during the drafting of this book: David Chitate (UNAIDS, Malawi), Boga Fidzani (National AIDS Coordinating Agency, Botswana), Wayne Gill (UNAIDS, Botswana), Salil Panakadan (UNAIDS, China) and Matthew Warner-Smith (UNAIDS, Geneva).
06 | UNAIDS
CONTENTS Getting Started
The Fundamentals What is triangulation? What are the different types of triangulation? What are the implications for monitoring and evaluation? What are the strengths and weaknesses of the four types of triangulation? Why do triangulation? Who should be involved in triangulation?
11 12 14 18
Tools and Techniques A basic approach to data triangulation Identifying the goals of data triangulation Quantitative versus qualitative data Data sources Determining the usefulness of data Time and resource considerations
31 32 36 38 40 43 46
Questions to consider
Learning more about triangulation
20 25 28
UNAIDS | 07
Abbreviations AIDS HIV IDU M&E MDG MERG NGO TWG UNAIDS WHO
08 | UNAIDS
acquired immunodeficiency syndrome human immunodeficiency virus injecting drug user monitoring and evaluation Millennium Development Goals Monitoring and Evaluation Reference Group non-governmental organisation technical working group Joint United Nations Programme on HIV/AIDS World Health Organization
Getting Started Why this topic? Triangulation is an effective and increasingly popular approach for analysing the HIV epidemic and response at the national level. However, triangulation is not a single approach. In fact, there are many different approaches to triangulation and there are articulate proponents for each approach. The primary purpose of this book is to provide enough background information for readers to be able to think clearly about triangulation in general and to ask good questions about the different ways to do triangulation. If it is done well, triangulation can provide M&E (monitoring and evaluation) professionals and key decision-makers with new insights on the HIV epidemic and response. The challenge is to bring together a good plan, clear goals, committed stakeholders, usable data and insightful analysts. The key to bringing all these elements together is a clear understanding of what triangulation is and is not. What is in this book? This book is a common-sense introduction to triangulation. It is designed to provide the basic information required to understand the main types of triangulation and to spark further investigation and discussion on the merits of the approach and its applicability in specific situations. The Fundamentals section of the book focuses on essential background information that anyone interested in triangulation should know: What is triangulation? What are the implications for monitoring and evaluation? What are the strengths and weaknesses? When and why should it be done? Who should be involved?
UNAIDS | 09
The Tools and Techniques section includes practical information to bear in mind when considering data triangulation as part of monitoring and evaluation: a basic approach to data triangulation, multiple issues related to data (e.g. types of data, sources of data, usefulness of data) and time/resource issues.
THE FUNDAMENTALS The greater the triangulation, the greater the confidence in the observed findings. Norman Denzin
What is triangulation? Triangulation is a method used to determine the location of a fixed point based on the laws of trigonometry. These laws state that if one side and two angles of a triangle are known, the other two sides and angle of that triangle can be calculated.
The exact origins of triangulation are not known, but it was widely used by civilizations in ancient Egypt and Greece. Over the centuries, triangulation was commonly associated with maritime navigation, where sailors used it to track their position and course. Historically, it has also played an essential role in surveying and civil engineering. In addition, triangulation is the principle behind the GPS or Global Positioning System technology. A GPS receiver processes radio signals sent from four different satellites to determine longitude, latitude and altitude. (In theory, the signals from three satellites could be used to fix the location; however, four are used in order to improve the precision of the measurement.)
12 | UNAIDS
Triangulation extended beyond its mathematical roots in the 1970s when it began to be used as a sociological method. In this new sector, triangulation was defined as a process of combining data from different sources to study a particular social phenomenon. In 1978, Norman Denzin identified four basic types of triangulation: (1) data triangulation: the use of multiple data sources in a single study; (2) investigator triangulation: the use of multiple investigators/researchers to study a particular phenomenon; (3) theory triangulation: the use of multiple perspectives to interpret the results of a study; and (4) methodological triangulation: the use of multiple methods to conduct a study. Since the 1970s, triangulation has become widely accepted as a way to improve the analysis and interpretation of findings from various types of studies. More specifically, triangulation has proved to be an effective tool for reviewing and corroborating findings in the surveys, assessments, appraisals, etc., that are an essential part of effective monitoring and evaluation.
UNAIDS | 13
What are the different types of triangulation? Among experts in triangulation in the social sciences, there continues to be a general consensus on the usefulness of the four types of triangulation originally identified by Denzin in the 1970s: (1) data triangulation; (2) investigator triangulation; (3) theory triangulation; and (4) methodological or method triangulation. Data triangulation is the use of a variety of data sources, including time, space and persons, in a study. Findings can be corroborated and any weaknesses in the data can be compensated for by the strengths of other data, thereby increasing the validity and reliability of the results. The approach has been used in many sectors to strengthen conclusions about findings and to reduce the risk of false interpretations. • In a large capital city, three different nongovernmental organizations (NGOs) run comprehensive prevention projects for hotel-based sex workers. Each programme works in a different part of the city where there are multiple hotels that rent space to sex workers and their clients. The projects are generally similar, but the dynamics in each community are slightly different. Triangulating performance data from across these three projects (e.g. frequency of contact with sex workers, percentage of sex acts including proper use of a condom, reduction in prevalence of sexually transmitted infections (STIs) among sex workers) will provide a much clearer picture of the overall situation than simply reviewing the data from one programme and attempting to extrapolate broader lessons from those data. Methods triangulation is the use of multiple methods to study a situation or phenomenon. The intention is to decrease the deficiencies and biases that come from any single method. In other words,
14 | UNAIDS
the strengths of one method may compensate for the weaknesses of another. This type of triangulation is very similar to the mixed method approaches used in social science research, where the results from one method are used to enhance, augment and clarify the results of another. It is also a variation on data triangulation, with an emphasis on using data collected by different methods as opposed to data collected for different programmes, locations, populations, etc. • A country has excellent quantitative data on the availability of ageappropriate HIV and AIDS curricula in its primary and secondary schools. It has equally good quantitative data on the number of teachers trained to deliver these curricula. It also has three separate qualitative studies involving extensive focus group discussions with young people reached by the curricula. Triangulation of the findings from the various data collection methods will highlight the strengths and weaknesses of the different methods and will give the triangulation team more insight than any one method is likely to provide. Investigator triangulation is the use of more than one investigator, interviewer, observer, researcher or data analyst in a study. The ability to confirm findings across investigators — without prior discussion or collaboration between them — can significantly enhance the credibility of the findings. Investigator triangulation is particularly important for decreasing bias in gathering, reporting and/or analysing study data. • Investigators from a condom social marketing project, a local university and a family planning NGO looked at the issue of condom access and condom use among rural populations in a mid-sized country. Each investigator reached different conclusions on the issues. Triangulating the findings from the three investigators allows their approaches, biases and findings to be directly compared and
UNAIDS | 15
contrasted and to identify opportunities to launch and/or improve interventions. (Investigator triangulation can also be used very specifically in the analysis of data. In this case, several different investigators would be asked to interpret the same data sets and to provide their independent analysis for further comparison.) Theory triangulation is the use of multiple theories or hypotheses when examining a situation or phenomenon. The idea is to look at a situation/phenomenon from different perspectives, through different lenses, with different questions in mind. The different theories or hypotheses do not have to be similar or compatible; in fact, the more divergent they are, the more likely they are to identify different issues and/or concerns. • Needle sharing continues to be very high in a border town with a large number of injecting drug users (IDUs). One study of the situation is based on the premise that social norms among these local IDUs encourages sharing as a way to show trust. Another study presumes that existing needle exchange projects are simply not providing enough clean needles to meet a high demand. A third study is looking at the correlation between intense police activity, the corresponding need for IDUs to stay on the move to avoid arrest and the efficiency of sharing drugs and needles. Each theory/hypothesis may be correct; triangulation is an opportunity to compare and contrast the findings from each of them and identify relevant lessons for improving interventions with IDUs. There are other types of triangulation used in the social sciences — for example, data analysis triangulation (i.e. the combination of two or more methods of analysing data) — but the four types mentioned above are the most common and the most widely used.
16 | UNAIDS
It is useful to distinguish triangulation from meta-analysis. Metaanalysis combines the original data from several rigorous scientific studies of similar quality and design for sophisticated statistical analysis. In contrast, triangulation uses findings from diverse sources, bearing in mind the strengths and weaknesses of those findings, and it looks for a convergence of the evidence in order to draw overall conclusions. IMPORTANT NOTE: Although data triangulation is only one type of triangulation, it is the most commonly used type. In this book, the term ‘triangulation’ will generally be used as the common descriptor; ‘data triangulation’ will be used in reference to that specific type of triangulation.
UNAIDS | 17
What are the implications for monitoring and evaluation? Triangulation can and should play a major role in monitoring and evaluation. It is an invaluable way to confirm findings in one study with findings from other sources, methods, investigators and theories. In fact, the ability to compare and contrast different findings and perspectives on the same situation and/or phenomenon is a very effective way to find inconsistencies in data and opportunities for further investigation. In addition, triangulation — particularly data and methods triangulation — can reinforce the validity and credibility of a finding, which makes it much easier to explain and justify. It can also provide a more complete and comprehensive perspective on a given situation and generate new insights into that situation. To gain the maximum benefit from triangulation, it is important to factor its various types into M&E planning from the outset. For example, the core activities involved in monitoring and evaluating the epidemic and response — DHS (Demographic and Health Surveys), behavioural surveillance surveys, disease case reporting, operational research, etc. — should all be part of ongoing triangulation efforts at the country level. In general, triangulation should be a routine activity, not an ad hoc one. However, triangulation can be used very effectively in special situations to provide invaluable information on the response for decision-makers. Triangulation is also an invaluable way to bridge the gap between the need for useful data for decision-making and the vast quantities of data that have been collected in recent years. As funding for HIV increased, so did the quantity of data collected about the epidemic
18 | UNAIDS
and response. Unfortunately, much of the data are not being used because of disparities in the data sets.
“Triangulation is above all a state of mind, which requires creativity from the researcher. The search for convergence is the motto, in order to make propositions more sound and valid.” Alain Decrop
For example, national M&E systems put subnational programmatic and surveillance data in separate databases that are housed in different locations from other relevant information such as research data, national census data and other special studies. Similarly, national surveys generally result in data sets that are analysed in isolation from other information. Integration of different data sets is difficult. In most instances, it is difficult to do a direct comparison or combining of data, which reduces the power of subsequent analyses. At the other end of the spectrum, scientific research is often focused on very specific and/or highly technical questions, with a slow turnaround time for the release of the results and with little bearing on the response. Triangulation can help M&E experts and programme managers find meaningful information in seemingly unrelated data sets, and can help them make timely recommendations for policy development and implementation as well as programme planning and improvement. In other words, triangulation is an effective way for information from across the spectrum to ‘meet in the middle’ and provide useful insights for strategic decision-making.
UNAIDS | 19
What are the strengths and weakness of the four types of triangulation? In general, triangulation can enhance the validity and reliability of existing observations about a given situation. If findings converge, it can also generate new, credible findings about a situation or phenomenon and can create new ways of looking at a situation or phenomenon. Most importantly, it can provide a better understanding of a situation or phenomenon. However, the many strengths of triangulation are counterbalanced by a number of major challenges, including: the amount of additional time required to conduct triangulation activities; the complexities of dealing with large quantities of data; the potential conflicts between different investigators, theories/hypotheses and/ or methods; the difficulties of interpretation when data do not converge into a clean, clear picture; and limited understanding among policy/decision-makers about how triangulation works and why it was used in a given situation. Ironically, it is as possible to do too much triangulation as it is to do not enough. It is also difficult to balance the routine use of triangulation with its use on an ad hoc basis; ideally, triangulation should be institutionalized within the national M&E system, whether it is used on a routine or ad hoc basis. Overall, triangulation is only effective if it is approached as a rigorous and demanding activity.
20 | UNAIDS
Institutionalizing triangulation. One key to institutionalizing triangulation is identifying capable and credible organizations that will effectively ‘own’ the process. In other words, these organizations would have primary responsibility for ensuring the consistent and high-quality use of triangulation to provide useful data on the country’s HIV response. For example, a partnership between the M&E unit in the National AIDS Commission and an umbrella NGO would provide the structure and support needed to institutionalize triangulation in a country. Another key to institutionalizing triangulation is to include it in existing M&E systems. In Malawi, triangulation is included in the annual M&E workplan. Equally important, following a successful triangulation initiative, there is now an annual budget dedicated to triangulation activities.
Data triangulation The core strength of data triangulation is the use of existing data for review and analysis. Rather than drawing conclusions from a single study, data triangulation — by definition — uses multiple data sources to examine a situation. A larger pool of relevant data practically guarantees that areas of convergence and divergence will be discovered; areas of convergence and divergence that may not have been identified or noticed in the data from a single study. A parallel strength is the nature of the data when they are drawn from multiple data sources and data sets. With triangulation, it is likely that the data will be drawn from a much more diverse set of sources and this diversity ensures a more expansive look at the situation. The weaknesses of data triangulation tend to be related to the quantity and quality of the data. For example, having too few data
UNAIDS | 21
means that triangulation is unlikely to provide any meaningful insights. In addition, poor quality data can completely undermine the usefulness of triangulation. There can also be serious problems with data triangulation if qualitative data are analysed from a quantitative perspective. Qualitative data are best analysed using the more flexible qualitative methodologies that look for deeper meaning in individual responses and/or data sets. However, it is not uncommon to code qualitative data in ways that make them more compatible with quantitative findings. This too can lead to false or misleading analysis. Methods triangulation The core strength of methods triangulation is its potential to expose unique differences or meaningful information that may have remained undiscovered with the use of only one approach or data collection technique in the study. Combining quantitative and qualitative methods enhances the ability of analysts to rule out rival explanations of change and improves the validity and reliability of change-related findings. For example, qualitative findings may help explain the success of an intervention when the quantitative data — the numbers — do not provide any corollary information. Many experts believe that across-method and within-method triangulation provide far richer findings than reliance on a single method. Methods triangulation has several weaknesses, including the expense of deploying multiple/mixed methods, the challenges of meshing quantitative and qualitative findings and the varying quality of different studies using different methods. With methods triangulation, it is critical to remember during analysis that inaccuracies of data from one method do not necessarily lessen or offset the inaccuracies of data from a different method. This is precisely why it is so important to use proven methods for both quantitative and qualitative work.
22 | UNAIDS
Investigator triangulation A key strength of investigator triangulation is the reduction of bias in gathering, reporting and analysing data. There is a general sense that having multiple investigators not only reduces bias but can also have a positive impact on both validity and reliability. Also, most investigators are skilled at one type of research and/or data collection methodology; for example, an investigator is most adept at either quantitative or qualitative research. Having multiple investigators using different methodologies, which would actually include investigator triangulation and methods triangulation, would also ensure a broader and potentially more balanced perspective on the situation being examined. In addition, corroborating data and verifying their interpretation across multiple investigators can increase the value of the findings. While a reduction in bias is a strength of investigator triangulation, it can be a weakness as well. For example, different investigators may be resistant or overreact to the known perspective of other investigators, undermining the objectivity of their own findings. Also, if different investigators are working with the same subpopulation, it is possible that conflicts could arise, which would be disruptive to the studies. Theoretical triangulation The primary strength of theoretical triangulation is its ability to look deeper and more broadly at findings. Specifically, using only one theory, perspective or hypothesis can decrease the number of alternative explanations for a situation or phenomenon. In fact, using multiple — even rival — perspectives or hypotheses can challenge analysts to look beyond obvious explanations and identify sharper ways of examining and explaining findings.
UNAIDS | 23
If the theories and/or hypotheses being used in theoretical triangulation are not well defined, this type of triangulation can be confusing and unproductive. It is also possible that the use of opposing theories/hypotheses in triangulation could be equally confusing and unproductive. Conversely, analysts must remember that findings are not automatically more credible because they have been supported by similar theories/hypotheses. While all types of triangulation must be very carefully managed in order to ensure that the process has integrity and the results are credible, there is a heightened need for vigilance with theoretical triangulation.
24 | UNAIDS
Why do triangulation?
“The use of triangulation strategies does not strengthen a flawed study. Researchers should use triangulation if it can contribute to understanding the phenomenon; however, they must be able to articulate why the strategy is being used and how it might enhance the study.” Veronica Thurmond
Triangulation can only be done when data are available, whether they are data from different sources, different investigators, different theories or different methods. However, when data are available, there are a number of different reasons why triangulation can and should be used. Complex questions. When seeking to answer complex questions concerning the quality, implementation, outcome and impact of a programme, the ability to draw from multiple inputs can provide a wider range of information and a significantly broader insight into the issues underlying the complex questions. Dissimilar data. When there are sufficient data but they are dissimilar, triangulation can balance the different perspectives and lead to a valid conclusion or a new hypothesis that can be tested. In fact, triangulation can create opportunities to compare a wide range of data on a particular situation or phenomenon side by side, providing new insights and generating new hypotheses. Poor-quality data. When relevant data from different sources, investigators and methods are available, triangulation can compensate
UNAIDS | 25
for the poor quality of some of the data, assuming that the validity and reliability of the other data can be confirmed. Insufficient data. When directly applicable data are not available, triangulation may be able to use indirectly applicable data to draw a valid conclusion. However, in these cases it is important to consider additional ways to confirm the accuracy of the conclusion. Trend data. When examining trend data on the epidemic and response, triangulating from a range of data types and sources can provide a more precise picture of the overall trend. Rapid response. When there is a need to rapidly respond to a situation, triangulation — using readily available data — can provide a valid perspective far more quickly than collecting and analysing new data. Alternative to research. When the findings from a rigorous, specifically designed research study are not available and when such a study is not feasible, triangulation – again, using available data – can be a viable option, depending on the depth and breadth of the available data. Estimates of population-level outcomes. When no data on population-level outcomes are available, triangulation can be used to piece together population-level conclusions using the available data on subsets of the population. It is important to stress that triangulation can and should be an institutionalized component of monitoring and evaluation. It is possible and desirable to plan to use multiple inputs for triangulation on a regular/recurring basis. However, when specific needs or opportunities arise – e.g., an urgent request to understand the effectiveness of an intervention in order to include those findings in a funding
26 | UNAIDS
proposal – it is also possible and desirable to conduct a focused triangulation exercise.
Triangulation in Botswana. In recent years, triangulation has been used very effectively in Botswana. The benefits have been twofold. First, using pre-existing data enables the work to be done relatively rapidly, which means the findings are more timely and, consequently, more useful. Second, using data from multiple sources has identified new issues to be studied and has provided a valuable check on the quality of data, including a reduced likelihood of data and researcher bias.
UNAIDS | 27
Who should be involved in triangulation? Effective triangulation depends on coordination and collaboration; consequently, a high level of buy-in and cooperation is required from key stakeholders, particularly those who are actively involved in collecting data about the epidemic and response. Regardless of the type of triangulation, it is most successful when stakeholders are involved in all the phases, including deciding the goals for triangulation, collecting and/or aggregating data, analysing the data and drawing conclusions from the analysis. Key stakeholders • National, provincial and district-level policy/decision-makers (e.g. national AIDS commission, ministry of health); • Programme and project managers and staff; • Civil society representatives (e.g. NGOs, community leaders); • Researchers (e.g. universities, research consortia); • Clients and/or beneficiaries of services; • Bilateral and/or multilateral partners (e.g. bilateral donors, Global Fund to Fight AIDS, Tuberculosis and Malaria, UNAIDS, WHO); • Representatives from other constituencies with similar expertise and experience (e.g. experts from neighbouring countries). It is important for stakeholders to play an active role in identifying the goal or goals of the triangulation process. However, it is equally important to focus on the key stakeholders to ensure that the quality of the input is high and the quantity is manageable. It can be more practical to use an existing body – e.g., an M&E technical working group – or to establish a small task force of technically proficient stakeholders who provide regular and active support for triangulation efforts. The working group or task force can serve as a conduit for communication to the larger group of interested parties.
28 | UNAIDS
Ideally, the working group/task force members should represent a range of expertise and have a recognized degree of involvement in the community. The group should have a chairperson whose main responsibility is to facilitate communication between all members of the task force and to establish the necessary political support for triangulation. The chairperson should also be responsible for ensuring that the triangulation goals are met. The group should also include quantitative and qualitative data experts as well as researchers, monitoring and evaluation specialists and others who are familiar with the specific technical areas being assessed by triangulation.
Stakeholders in Botswana. Experience with triangulation in Botswana demonstrated that one of the essential prerequisites is the engagement of high-level policy-makers and administrators. The involvement of these stakeholders during the early stages of the triangulation process was particularly important. However, they proved to be useful partners throughout the entire process.
UNAIDS | 29
Tools and techniques Data triangulation is the most widely used type of triangulation in the HIV response. Consequently, the Tools and Techniques section focuses on data triangulation.
A basic approach to data triangulation The basic approach to data triangulation is the same if the triangulation is a routine activity or an ad hoc one. First, the goal of triangulation must be agreed. Second, the data must be collected and/or aggregated before they can be reviewed. Finally, the data are analysed and conclusions are drawn. 1. Agree on the goal(s) of data triangulation. Since triangulation is a collaborative process, it is essential that stakeholders agree on an achievable goal or goals. For example, a goal of data triangulation could be to understand if risk behaviours are changing among prison populations and whether any changes can be linked directly to interventions. It is important not to overreach with data triangulation. It can be a challenging and time-consuming activity. Consequently, every goal should be: appropriate (i.e. triangulation is the right approach); relevant (i.e. the findings could have a significant impact on the epidemic and/or response); actionable (i.e. the findings could be used to make specific improvements in the response); and feasible (i.e. the data are available or can be collected, the human and financial resources are available and the triangulation can be completed in a reasonable amount of time). 2. Request, collate and aggregate the necessary data. If the data required for triangulation have not been requested from the individuals and/or organizations that collected them and/or control them, the necessary steps should be taken to ask for them. Depending on who collected/controls the data, the steps may be formal (e.g. an official letter from the national AIDS commission), informal (e.g. a telephone call between colleagues) or both.
32 | UNAIDS
Any new data collection exercise is separate and distinct from the triangulation process, since triangulation is not a data collection methodology. However, data triangulation can spur the collection of new data, from other sources or using different investigators, theories/hypotheses or methods. Assuming the data already exist, the individuals and/or organizations that collected them should make the data available. However, for various reasons, they can be reluctant to share their data for use in triangulation. For example, in some cases there are critical issues of confidentiality and consent. Consequently, the ability to successfully deal with these issues can be an essential part of triangulation and it should be addressed from the outset of any triangulation exercise. When all the data required for triangulation are available, they must be collated, aggregated and then presented in a graphical way, which makes it easy to compare the similarities and differences as well as the strengths and weakness of the different inputs. For example, it is important to identify the type and format of the data as well as the data collection methods. This is also an opportunity to document when data were collected. Knowing precisely when data were collected is essential if triangulation is going to compare trends over time. It is also useful for determining if data were too old and no longer relevant. As data are being collated, aggregated and graphed, a preliminary review of their usefulness should also be prepared. This review should consider questions such as: How representative is the data? What are the biases in the data? Were the data collection methods consistent? Was the data collected in accordance with ethical standards? (For more information on determining the usefulness of data, see page ??.)
UNAIDS | 33
During this process, the quality of the data should also be assessed. Is it reliable? Valid? Credible? At the same time, a determination of the level of confidence in the data should be made. • Analyse the data and draw conclusions. While there are no fixed rules for the process of analysing data for triangulation, there are several activities at the heart of the process. First, analysts should make critical observations about the data: for example, prioritizing the findings most relevant to the goal(s) of triangulation, identifying ways that findings from different studies relate to one another and highlighting gaps in the data. Second, analysts should identify any trends in the data and whether they are drawn from a single data source or from multiple sources. (Not all triangulation exercises will provide trend data.) Third, analysts should develop working hypotheses related to the goal(s) of data triangulation. For example, if the goal is to understand if risk behaviours are changing among key populations and whether any changes can be linked directly to interventions, analysts should be developing hypotheses from the available data that are linked to this goal. Hypotheses can be in support of the goal; for example, a supportive hypothesis could be ‘injecting drug use among prisoners has been reduced because of the availability of substitution therapy in prisons’. Fourth, analysts should work to confirm or refute the hypotheses. During this process, analysts should think as openly and creatively as possible. This is a critical point in triangulation when new ideas, perspectives and explanations are likely to emerge. It is also a point when gaps in data will be identified, which could lead to a
34 | UNAIDS
search for additional data. If no additional data are available, a hypothesis may need to be modified or dropped. Any modified hypotheses should then be reconfirmed. If a hypothesis is dropped, analysts should consider whether to add another one, based on their experience during this process and the available data. Fifth, analysts should use the convergence of data supporting or not supporting the hypothesis to draw reasoned conclusions from the triangulation exercise. The conclusions should be linked as closely as possible to the stated goal(s) of triangulation. The key to this process is to make the strongest case for a hypothesis/ goal given the evidence. Questions that may be helpful to consider during the process include: • Which hypotheses are supported by the most rigorous data? • Which hypotheses are supported by independent sources? • Which hypotheses are supported by both quantitative and qualitative data? • Are there important biases or limitations in the available data? • Are there any other possible explanations not covered by the hypotheses? • How confident are the analysts in the conclusion? • Is the conclusion actionable (i.e. does it lead to a specific improvement in the response)? Analysts should carefully and thoroughly document their conclusions before disseminating them. Because data triangulation is a secondary process (i.e. it uses data collected by others to make determinations), it may have few natural supporters, particularly if the conclusions are not in line with findings from other activities; consequently, there is a need for careful and thorough documentation of the conclusions and buy-in from all stakeholders from the beginning of the triangulation exercise.
UNAIDS | 35
Identifying the goals of data triangulation Identifying the goal(s) of triangulation is the all-important first step in making this approach a useful part of an integrated approach to monitoring and evaluation. Given the many issues that are important to tracking the epidemic and response, the process of identifying the goals should be open and inclusive, in order to ensure that stakeholders have an opportunity to share their ideas and participate in the selection of a prioritized set of goals. An effective way to engage stakeholders in the process is to bring them together to brainstorm goals. At the outset of the meeting, participants should be briefed on the overall triangulation process and the importance of clearly defined goals to that process. It is possible that some questions of interest and/or importance will have been identified before the meeting. For example, technical experts within the national AIDS commission may have already developed a list of relevant issues as part of their day-to-day responsibilities. However, it is important that the final list of triangulation goals has had input from and has been agreed by a consensus of key stakeholders. Their input helps to ensure that the goals are broadly inclusive, while their agreement helps to ensure that they will support and participate in the triangulation process. It is critical to allow enough time for the brainstorming. In some cases, one meeting may be sufficient. In other cases, it may be useful to hold multiple meetings in order to give participants more time to consider the options. However, under no circumstances should the brainstorming be open-ended. In general, two or three meetings should be enough to get the necessary inputs and reach a consensus.
36 | UNAIDS
One final and very important point about brainstorming: Do not pass judgement on the feasibility or importance of any suggested goals as they come up. Simply make the list as agreed by the key stakeholders. Remember: This is only the first step in the larger data triangulation process. It is possible that a goal will need to be modified or replaced as the triangulation process moves forward. For example, it may turn out that relevant data are not available or that the cost of collecting the data is too high or that the goal has a lower priority given a change in the epidemic and/or response. If a goal needs to be modified or replaced, every effort should be made to solicit input from the key stakeholders who participated in the initial goal-setting process.
Botswana experience. During a triangulation exercise in 2005, the stakeholder group drafted a list of critical themes/issues that would benefit from the exercise. The stakeholders then prioritized this list based on the availability of data and the importance of developing new policies and programmes or revising existing ones. Ultimately, there was a consensus among the stakeholders about the top item on the list – the effectiveness of antiretroviral therapy and the prevention of mother-to-child transmission – and this item became the goal of the triangulation exercise.
UNAIDS | 37
Quantitative versus qualitative data There are two basic types of data: quantitative and qualitative. Both types of data are equally useful in any triangulation activity. In fact, the use of both types of data is an underlying premise of triangulation. Quantitative data can be measured on a numerical scale and can be analysed using statistical methods and displayed using tables, charts, histograms and graphs. The essential steps in a quantitative exercise are to classify features, count them and construct statistical models in an attempt to explain what is observed. Quantitative data are collected using quantitative methods such as systematic surveys, tests and analysis of records. Qualitative data are not intended or designed to be measured, counted or expressed in numbers. As the term implies, they relate to the quality or character of something and provide an understanding of social situations and interactions, as well as people’s values, beliefs, opinions, perceptions, motivations, behaviours and reactions. Qualitative data are generally expressed in words, pictures or objects and are collected using qualitative methods such as interviews and observation. There are limitations to quantitative and qualitative data. For example, inappropriate sampling can compromise the validity and reliability of both types of data. In addition, quantitative data will never accurately reflect certain aspects of situations (e.g. the motivations behind risk-taking behaviours). Qualitative data can similarly misrepresent the magnitude of the situation (e.g. the percentage of a population at higher risk of engaging in risky behaviour).
38 | UNAIDS
The differences between quantitative and qualitative data Quantitative data
• Numbers as data • Ask: How many? How much? • Data collected through surveys • Generalizability is a goal • Use probability sampling • Use a large sample size • Goal: prove/verify
• Words as data • Ask: How? Why? • Data collected through interviews and observation • Generalizability is not a goal • Use purposive, convenience, snowball, or quota sampling • Use a small sample size • Goal: discover/explore
UNAIDS | 39
Data sources There are many sources for the quantitative and qualitative data used in triangulation. The categories include – but are not limited to – research initiatives, surveillance, surveys, programme records, focus groups, ethnographies, census reports and grey literature. The following table gives examples of specific data sources from different categories. It is not an exhaustive list; there are many different data sources.
40 | UNAIDS
Examples of specific data sources from different categories Data source category
Specific data source examples
• National census • Demographic and Health Surveys (DHS) • AIDS Information Surveys (AIS)
• Seroprevalence surveys (e.g. sentinel surveillance, population-based surveys) • Case reporting • Demographic and Health Surveys (DHS) • AIDS Information Surveys (AIS) • Behavioural surveillance surveys (BSS) • Integrated bio-behavioural surveillance surveys (IBBS)
• Focus groups • Ethnographies • Rapid assessments
• Annual reports, quarterly reports, patient/client records, reviews, interviews, surveys, mapping, etc.
UNAIDS | 41
Evaluation and research
• Published papers and/or reports
• Abstracts from conferences • Programme/project evaluations • Technical reports • Unpublished research papers • Websites • White papers • Working papers • Preprints or drafts of scientific papers
It is important to assess the validity and reliability of the data available from different sources. For example, review the sampling method and sample size, determine if there is any bias and evaluate the data quality. If there is a high level of confidence in the data, regardless of the source, the data’s usefulness in triangulation increases. Conversely, if the level of confidence in the data is low – even if the source is well known – then the data are likely to be less useful for triangulation. However, it is important to remember that a lower confidence level does not necessarily invalidate data. In triangulation, the objective is to combine the full range of reliable data to identify patterns and convergence, and then reach the most accurate conclusion.
42 | UNAIDS
Determining the usefulness of data The usefulness of data to answer a specific triangulation question is dependent on the quality and integrity of the data. When evaluating the usefulness of a data set, it is worth considering the following questions: How representative are the data? For data to be useful in triangulation, they must be representative of the target population (e.g. general population, sex workers, injecting drug users) and their location (e.g. urban, rural). Determining how representative data are may require a review of the sampling methods (e.g. probability and non-probability) and sampling approaches (e.g. cluster, random or convenience) that were used to obtain the data. What are the biases in the data? Bias is common in data sets of all types. However, it is important to understand the bias before using data in triangulation. There are several different types of bias, but observer, confounding and selection bias are three of the most common. • Observer bias is when the investigator’s subjective perspective may influence the objectivity of the data. In ethnographic research, the people being studied are described by the researcher through his or her own cultural thought system, using the researcher’s own terminology. • Confounding bias occurs when the dependent variable of interest is associated with two or more independent variables that are associated with each other. For example, people associating the transmission of malaria (the dependent variable of interest) with eating mangoes (independent variable 1), as mangoes are often available during the rainy season (independent variable 2), when
UNAIDS | 43
malaria is more prevalent. (Although mangoes do not transmit malaria, they are available in the rainy season, when malaria is more prevalent. Assuming mangoes and malaria are associated would be a confounding bias error.) • Selection bias occurs when people selected for a study do not reflect the population of interest. For example, HIV prevalence among blood donors cannot be used as a measure of overall prevalence in a country because blood donors are not necessarily reflective of the overall population. Were the data collection methods consistent? Unfortunately, methods for data collection can change over time; for example, if a new programme manager or new organization assumes responsibility for data collection, they may change the method used by their predecessor. However, the consistency of data collection is crucial, particularly for an accurate modelling of trend data. Consequently, data collection methods should be reviewed to determine if they have been consistent and, if not, how significant the changes are. Were the data collected according to ethical standards? The data used in triangulation should be collected ethically; this will include, when necessary, approval by an institutional review board (IRB). The following issues should be addressed when evaluating the ethics of data collection: • Study participants provided informed consent. • Confidentiality was guaranteed for participants. • The risk of stigma and discrimination for participants was not increased. • There was no increased risk of harm for populations at higher risk, especially if their behaviour is illegal (e.g. sex work or injecting drug use).
44 | UNAIDS
• Participants had access to prevention, treatment and care services. • Standards of professional conduct, practice and the manner in which the studies were consistent with the relevant international, national, and local laws and guidelines. The ethical principles used in public health settings are described in the Belmont Report,1 the Helsinki Agreement,2 the Council for International Organizations of Medical Sciences guidelines3 and UNAIDS guidelines.4
1 Belmont Report. Available at: www.hhs.gov/ohrp/humansubjects/guidance/belmont.htm. 2 Helsinki Agreement. Available at: www.wma.net/e/policy/b3.htm. 3 Council for International Organizations of Medical Sciences. International Ethical Guidelines for Biomedical Research involving Human Subjects. Geneva, Council for International Organizations of Medical Sciences, 2002. Available at: www.cioms.ch/frame_guidelines_nov_2002.htm. 4 UNAIDS. Guidelines on Protecting the Confidentiality and Security of HIV Information. Geneva, UNAIDS, 2007. Available at: http://data.unaids.org/pub/manual/2007/confidentiality_security_interim_guidelines_15may2007_en.pdf
UNAIDS | 45
Time and resource considerations Whether triangulation is being done on a routine or ad hoc basis, it is vital that sufficient time and resources are set aside to ensure the quality and thoroughness of the exercise. Factors to consider when estimating time and resource needs include: • Complexity of the issues being examined; • Availability and quality of the data; • Number of stakeholders involved in the process; • Ability and availability of stakeholders to participate in the process; • Ability and availability of analysts to participate in the process; • Availability and use of consultants to support the process (if necessary). Triangulation is a very efficient and effective way to gain new insights into specific aspects of the epidemic and response. However, it is a mistake to think it can be done quickly, particularly if it is done properly. Consequently, it is extremely important to develop a realistic timeline for the overall process at the outset. In terms of the basic approach to triangulation, the following estimates can serve as a rough guide: • Agree on the goal(s) of triangulation: one to two months. • Collect, catalogue and aggregate the necessary data: two to three months. • Analyse the data and draw conclusions: two to three months. The scope of the triangulation exercise will have a major impact on the timeline, as will the extent of available human and financial resources. In addition, if studies must be undertaken to collect new data, the timeline is likely to be extended. Conversely, if all the data used in an exercise are readily available, the timeline is likely to be
46 | UNAIDS
condensed. (If studies must be undertaken to collect new data, it is reasonable to include the expense of the data collection in the budget for triangulation.) In terms of human resources, it is important to identify a person who can dedicate sufficient time to support the triangulation effort. This person would essentially be the project manager, whose tasks would range from organizing stakeholder inputs, to assisting with the cataloguing and aggregation of the data, to disseminating the conclusions from the exercise. It is equally important to identify a team of analysts, whose ability to develop hypotheses and review findings are at the very core of the triangulation process. The number of analysts will vary based on the nature and size of the triangulation exercise. For example, if triangulation is done on a routine basis, it may be valuable to have a larger pool of analysts who can be brought in as needed in order to avoid overusing any one analyst.
UNAIDS | 47
Notes 50 | UNAIDS
Do you have anything to add?
There are real costs associated with triangulation, including dedicated human resources, workshops and technical assistance, which must be factored into any costed M&E workplan. However, when compared with other approaches for collecting and analysing data, triangulation can be very cost-effective and extremely useful for decision-making.
UNAIDS | 51
52 | UNAIDS
Questions to consider
The following questions may be useful for anyone planning to conduct a triangulation exercise. • Is triangulation a cost-effective exercise?
• Does the political will exist to do effective triangulation?
• Are the financial and human resources available to do triangulation?
• Will triangulation require the collection of new data?
• Are quantitative or qualitative data more valuable? More accessible?
• Will key stakeholders provide relevant data for triangulation?
• How will triangulation add value to existing monitoring and evaluation activities?
• What are the most important or most useful data sources?
• Can more than one plausible hypothesis be developed to support a single triangulation goal?
UNAIDS | 55
1 What are the main types of triangulation used in the social sciences? Indicate all that apply: Data. Data analysis. Investigator. Meta-analysis. Methods. Theory.
58 | UNAIDS
2 Indicate whether the following items are strengths or weaknesses of triangulation: Use of existing data for review and analysis. otential to expose unique differences or meaningP ful information. Meshing quantitative and qualitative findings. orroborating data and verifying their interpretation C across multiple investigators. Ability to look deeper and more broadly at findings. Reduction in bias. orroborating data and verifying their interpretation C across multiple investigators. Varying quality of different studies using different methods.
3 True or false: Triangulation is not useful as a routine activity. riangulation should not be used to analyse indirectly apT plicable data to draw a valid conclusion. Triangulation can provide a valid perspective far more quickly than collecting and analysing new data. ffective triangulation depends on coordination and colE laboration.
60 | UNAIDS
Behavioural surveillance surveys. A method for tracking trends in HIV knowledge, attitudes and risk behaviour in the general population or targeted subpopulations. Bias. Any effect during the collection or interpretation of information that leads to a systematic error in one direction; for example, observer bias in the interpretation of replies to survey questions. Case reporting. Detailed report of the symptoms, signs, diagnosis, treatment and follow-up of an individual patient. Cluster sampling. When the population is divided into groups (clusters), with a subset of the groups chosen as a sample. After groups are chosen, all or a sample of individuals in each group are chosen for inclusion in the study. Confounding bias. Occurs when two factors are closely associated and the effects of one confuses or distorts the effects of the other factor on the outcome. The distorting factor is a confounding variable. Data. Specific quantitative and qualitative information or facts that are collected and analysed. Data set. A collection of data elements. Data triangulation. The analysis and use of data from three or more sources obtained by different methods. Findings can be corroborated and the weakness or bias of any of the methods or data sources can be compensated for by the strengths of another, thereby increasing the validity and reliability of the result. Demographic and Health Surveys (DHS). Population-based survey designed to collect and provide data for a wide range of monitoring
62 | UNAIDS
and impact indicators related to a range of different issues, including child health, family planning, gender/domestic violence, maternal health, nutrition and women’s empowerment. For additional information see: www.measuredhs.com/aboutsurveys/ dhs/start.cfm Disease case reporting. See Case reporting. Epidemiology. The study of how often diseases occur in different groups of people and why. Epidemiological information is used to plan and evaluate strategies to prevent illness. Generalizability. The degree to which results of a specific study or review can be applied to a larger population or in other circumstances. Grey literature. Working documents, preprints, research papers, statistical documents and other difficult-to-access materials that are not controlled by commercial publishers. Hypotheses. The plural of hypothesis. Hypothesis. A tentative explanation for a phenomenon, used as the basis for further investigation. Intervention. A specific activity (or set of activities) intended to bring about change in some aspect of the status of the target population (e.g. HIV risk reduction, improving the quality of services) using a common strategy. An intervention has distinct process and outcome objectives and a protocol outlining the steps of the intervention.
UNAIDS | 63
Mixed method evaluation. The use of different methods (e.g. quantitative and qualitative methods) in the evaluation of findings related to the same or similar situations or phenomena. Non-probability sampling. When the units of a sample are chosen so that each unit in the population does not have a calculable nonzero probability of being selected in the sample. Observer bias. A bias caused by the investigator’s subjective perspective, which may influence the objectivity of the data. Preprint. A draft of a scientific paper that has not been published in a peer-reviewed scientific journal. Probability sampling. The selection of units from a population based on the principle of randomization. Every unit of the population has a calculable (non-zero) probability of being selected. Purposive sampling. When investigators purposely seek out respondents they believe will fulfil the needs of the study. Qualitative data. Data collected using qualitative methods, such as interviews, focus groups, observation and key informant interviews. Qualitative data can provide an understanding of social situations and interactions, as well as people’s values, perceptions, motivations and reactions. Qualitative data are generally expressed in narrative form, pictures or objects (i.e. not numerically). Quantitative data. Data collected using quantitative methods, such as surveys. Quantitative data are measured on a numerical scale, can be analysed using statistical methods and can be displayed using tables, charts, histograms and graphs. The aim of a quantitative study
64 | UNAIDS
is to classify features, count them and construct statistical models in an attempt to explain what is observed. Quota sampling. A sample created by gathering a predefined number of participants from each of several predetermined categories. The selection process within each category may be random; for example, dividing a class into groups of males and females and randomly selecting 25 participants from each category. Random sampling. A subset of the population in which every member of the population has an equal likelihood of being selected. Reliability. Consistency or dependability of data collected through the repeated use of a scientific instrument or a data collection procedure used under the same conditions. Representativeness. The ability of a sample (i.e. a selected subset of a population) to accurately represent or typify a larger population. Research. A study which intends to generate or contribute to generalizable knowledge to improve public health practice; i.e. the study intends to generate new information that has relevance beyond the population or program from which data are collected. Research typically attempts to make statements about how the different variables under study, in controlled circumstances, affect one another at a given point in time. Sample. A selected subset of a population. A sample may be random or non-random and may be representative or non-representative. Typically, a sample is selected as a proxy for the target population for a given experiment/intervention.
UNAIDS | 65
Sample size. The number of people in a sample. Sampling. A technique used to obtain information about a large group by examining a smaller, randomly chosen selection of group members. If the sampling is conducted correctly, the results will be representative of the group as a whole. Sampling method. An approach used to select people from a population for a survey; approaches include random, stratified, systematic, convenience and self-selecting. Selection bias. The sample population chosen is not representative of the population at risk. Sentinel surveillance. Systematic, ongoing collection and analysis of data from certain sites (e.g. hospitals, health centres, antenatal clinics) selected for their geographic location, medical specialty and populations served and considered to have the potential to provide an early indication of changes in the level of a disease. Seroprevalence. The rate of HIV-infected individuals in a given population as measured by blood antibody tests. The number of infected individuals divided by the total number in the population. Stakeholder. A person, group or entity that has a role and interest in the goals or objectives and implementation of a programme. Surveillance. The ongoing, systematic collection, analysis, interpretation and dissemination of data regarding a health-related event for use in public health action in order to reduce morbidity and mortality and to improve health. Surveillance data can help predict future trends and target needed prevention and treatment programmes.
66 | UNAIDS
Snowball sampling. A non-probability sampling technique where existing study subjects recruit future subjects from among their acquaintances (i.e. the first respondent refers a friend, the friend refers a friend, etc.). Trend. The general direction in which tracking data tend to move, either upwards or downwards. Surveillance, for example, involves observing the trend of infection rates to help identify any increases or declines. Validity. The extent to which a measurement or test accurately measures what is intended to be measured. White paper. An authoritative report or guide that argues a specific position or solution to a problem. Although white papers have their roots in governmental policy, they have become a common tool used to educate people and help them make decisions.
UNAIDS | 67
Learning more about triangulation
Notes 70 | UNAIDS
• The Research Act: A Theoretical Introduction to Sociological Methods; Norman K. Denzin (1970) • Triangulation in qualitative research: issues of conceptual clarity and purpose; Kathleen A. Knafl and Bonnie J. Breitmayer (1989) • A critique of the use of triangulation in social research; Norman W.H. Blaikie (1991) • Convergent and discriminant validation by the multitrait-multimethod matrix; Psychological Bulletin, 56, 81-104; Donald T. Campbell and Donald Fiske (1959) • Triangulation in nursing research: issues of conceptual clarity and purpose; Fu-Jin Shih (2001) • Handbook of Data Analysis; Melissa A. Hardy and Alan Bryman (2004)
UNAIDS | 71
• Evidence Based Medicine [online database]. Charleston, SC. Medical University of South Carolina (http://www.library.musc.edu/ page.php?id=726, accessed 13 September 2009) • Monitoring and Evaluation Reference Group, Glossary of M&E Terms, 2009. • University of California, San Francisco. HIV Data Triangulation Resource Guide: Synthesis of Results from Multiple Data Sources for Evaluation and Decision-making. WHO, UNAIDS, Global Fund, 2008. • Instructional Assessment Resources: Glossary [online glossary]. Austin, TX. University of Texas (http://www.utexas.edu/academic/ diia/assessment/iar/glossary.php, accessed 13 September 2009) • Centers for Disease Control and Prevention. Introduction to Program Evaluation for Public Health Programs: Evaluating Appropriate Antibiotic Use Programs: Glossary and Evaluation Resources. 2006. (http://www.cdc.gov/drugresistance/community/programplanner.htm) • Tashakkori, A, Teddlie, C. Mixed Methodology: Combining Qualitative and Quantitative Approaches. Thousand Oaks, Sage Publications, 1998. • Cook, TD Postpositivist critical multiplism. Social science and social policy, 1985, pp. 21-62. • M&E Sourcebook [online publication]. Geneva. Provention Consortium. (http://www.proventionconsortium.org, accessed 13 September 2009)
74 | UNAIDS
• Denzin, N. Sociological Methods: A Sourcebook. New Brunswick, Transaction Publishers, 1970. • Thurmond, VA. The Point of Triangulation. Journal of Nursing Scholarship, 2001, 33:3, 253-258. • Decrop, A. Triangulation in qualitative tourism research. Amsterdam, Elsevier, 1999.
UNAIDS | 75
Notes 76 | UNAIDS
Notes UNAIDS | 77
Notes 78 | UNAIDS
Publications in this series Basic Terminology and Frameworks for Monitoring and Evaluation (2010) An Introduction to Indicators (2010) A National Evaluation Agenda for HIV (2010) An Introduction to Triangulation (2010)
UNAIDS | 79
About the Monitoring and Evaluation Fundamentals Series With the advent of the global financial crisis affecting most countries around the world, monitoring and evaluation (M&E) has become more important than ever before. Determining what programs do or do not work; implementing programs with proven cost-effectiveness; monitoring progress towards achieving targets; and ensuring accountability are objectives which are especially important now in the HIV response, as well as in other health and development areas. Thus, it is increasingly important that M&E is better understood, communicated in simplified language, and conducted in a coordinated and sustainable manner that generates information that can be easily used. Further, it is essential that M&E addresses the needs of and involves all key stakeholders right from the start and that results are made publicly available and utilized strategically in policy-making, planning, and program improvement. This series provides a common sense introduction to a range of M&E issues. It covers the fundamentals and their practical applications and includes techniques and tools for managing M&E of the HIV epidemic and response. Although the series uses HIV as its focus, the M&E fundamentals are also relevant to other areas of public health and development. As such, these books may also be useful in strengthening national M&E systems designed to track progress in other health and development goals, such as those outlined in the United Nations Millennium Development Goals (MDG).
UNAIDS 20, AVENUE APPIA, CH-1211 GENEVA 27, SWITZERLAND Tel.: (+41) 22 791 36 66, Fax: (+41) 22 791 48 35 e-mail: [email protected]