University of South Florida
Scholar Commons Graduate Theses and Dissertations
Evidence-Based Practice Attitudes, Knowledge and Perceptions of Barriers Among Juvenile Justice Professionals Esther Chao Mckee University of South Florida, [email protected]
Follow this and additional works at: http://scholarcommons.usf.edu/etd Part of the Criminology and Criminal Justice Commons, and the Social Work Commons Scholar Commons Citation Mckee, Esther Chao, "Evidence-Based Practice Attitudes, Knowledge and Perceptions of Barriers Among Juvenile Justice Professionals" (2014). Graduate Theses and Dissertations. http://scholarcommons.usf.edu/etd/5271
This Dissertation is brought to you for free and open access by the Graduate School at Scholar Commons. It has been accepted for inclusion in Graduate Theses and Dissertations by an authorized administrator of Scholar Commons. For more information, please contact [email protected]
Evidence-Based Practice Attitudes, Knowledge and Perceptions of Barriers among Juvenile Justice Service Professionals
Esther Chao McKee
A dissertation submitted in partial fulfillment of the requirement for the degree of Doctor of Philosophy School of Social Work College of Behavioral & Community Sciences University of South Florida
Major Professor: David Kondrat, Ph.D. Mario Hernandez, Ph.D. Lisa Rapp McCall, Ph.D. Anne Strozier, Ph.D. Date of Approval: June 26, 2014
Keywords: Practitioner knowledge, implementation science, recommendations for practice, organizational change Copyright © 2014, Esther Chao McKee
Dedication This dissertation is dedicated to my husband Seth C. McKee and our son Jesse Chao McKee.
Acknowledgments I owe a great thanks to my social work professors at the University of South Florida for your investment in me. It has been an honor to learn from you in class and I am grateful for the preparation I received to successfully complete a dissertation. I am particularly grateful to Dr. Allison Salloum and Dr. Roger Boothroyd for their ability to teach research methods and statistics in a way that was relatable and applicable to our own projects. Many thanks to Ms. Teri Simpson and Ms. Ruth Tilden for giving me the opportunity to round out my doctoral training with classroom experience by teaching field seminars. Plus, your kindness and encouragement helped me realize why I was working toward a Ph.D. in the first place! I would especially like to thank those who have been part of my committee, particularly to mention the immense appreciation and gratefulness I have for Dr. Lisa Rapp-McCall, without whom I would not be at this point now. Your leadership and expertise in the juvenile justice field gave me opportunities that have yet to even be realized. Your patient, thoughtful critique and feedback of my work helped me regain my confidence and to keep pressing forward. I really appreciate your commitment to teaching and research and I hope to keep learning from you. Thank you so much! Dr. Anne Strozier, I will always remember and be forever appreciative to you for the opportunity as a new social work researcher to see how “real” research is conducted. You have also been enormously patient as I moved through this long process, but throughout it all, your unflagging encouragement and support has meant so much. Thank you for believing in me from the beginning! Dr. Mario Hernandez, I am so thankful to you for agreeing to be part of my committee! You have many, many other responsibilities and obligations, but yet you took the time to talk to me, offer guidance and provide resources. I really appreciate your kindness and generosity. Lastly, Dr. David Kondrat… words can’t adequately express my gratitude for you! You guided me through the most nerve-wracking part of this process in the midst of major changes within your own life with extreme calm, patience and encouragement. Your statistical prowess helped win the day and you
showed me an exciting future of research and teaching that I could not imagine before. Thank you for everything! Along the way, I have also benefitted from the mentorship and friendship of Dr. Pam Alvarez. You already know how much you mean to me and my personal life and professional career would not be where it is today without you. Thank you for giving me opportunities to grow and tackle challenges that allow me to be a credible social work leader and scholar. But more importantly, thank you for the chance to work at Bay Area Youth Services, Inc. This gave me insight to an interesting research topic and access to a group of dynamic Juvenile Justice Service Professionals whose willingness to participate made this study possible. Of course without Dr. Erica Sirrine, Dr. Patty Sharrock, Dr. Kim Gryglewicz, and Dr. Alicia Mendoza, this journey would not have been nearly as fun or rewarding. Thank you ladies for being the ultimate support system and cheering section as I finally make my way to the finish line to join you in life after Ph.D.! I treasure our friendship and I look forward to seeing all the amazing things you will do. Thank you so much! Last but not least, I am eternally grateful for my beautiful family. My parents, Paul and Ruth Chao who have given me so much and showed me what is important in life. My in-laws, Jim and Marilyn McKee and Daniel and Sue Martin, who loved me unconditionally since the day Seth took me home to meet you nearly two decades ago. My siblings, Moses Chao, Lydia Chao Mann, Matt Mann, John and Sarah Mosqueda, and Brett and Alina McKee, many thanks to each of you for offering support, but more importantly offering free babysitting when necessary and excusing me from many of my family obligations during the last several years. Finally, a huge thanks to my husband Seth C. McKee, you cheerfully bore the brunt of my frustrations and fears for the last 20 years and bolstered me when needed. I love you dearly. I am so grateful for the life we built together and I am looking forward to enjoying the future with you and Jesse without ever being a student again!
TABLE OF CONTENTS LIST OF TABLES......................................................................................................................................... iii ABSTRACT .................................................................................................................................................. iv CHAPTER 1: INTRODUCTION ....................................................................................................................1 Background of Problem ...................................................................................................................1 Statement of Problem ......................................................................................................................3 Purpose of Study .............................................................................................................................5 Significance of Study .......................................................................................................................5 Relevance to Social Work ................................................................................................................7 Research Design and Primary Research Questions .......................................................................8 Scope of Study and Delimitations ....................................................................................................8 Summary of Chapter ........................................................................................................................9 CHAPTER 2: REVIEW OF LITERATURE ..................................................................................................10 Theoretical Framework ..................................................................................................................10 Conceptual Framework ..................................................................................................................13 Evolution and Definition of EBP .....................................................................................................15 Development of EBP in Social Work..............................................................................................17 Development of EBP in Juvenile Justice .......................................................................................21 Characteristics of EBP Interventions Used Within Juvenile Justice...............................................22 Selecting and Assessing EBP........................................................................................................23 Political and Environmental Context of this Study .........................................................................25 Practitioner Attitudes Toward EBP.................................................................................................26 Practitioner Perceptions of Barriers Toward EBP ………………………..........................................34 Practitioner Knowledge of EBP ................................................................. ………………………... 35 Summary of Chapter ......................................................................................................................38 CHAPTER 3: METHODOLOGY .................................................................................................................39 Overview of Study ..........................................................................................................................39 Ethical Considerations ...................................................................................................................39 Research Design ...........................................................................................................................39 Population and Sample ..................................................................................................................40 Description of Participants ................................................................................................40 Sample ..........................................................................................................................................41 Rationale for Selected Sample .........................................................................................41 Criteria For Inclusion .........................................................................................................41 Delimitations .....................................................................................................................42 Measures .......................................................................................................................................42 Attitude Toward EBP .........................................................................................................42 Perceptions of Barriers to Using EBP ...............................................................................45 Level of EBP Knowledge ..................................................................................................47 Pilot Study .........................................................................................................................48 Instrument Reliability Summary ........................................................................................49 Demographic Questions ...................................................................................................49 Data Collection Procedures ..............................................................................................50 Data Analysis and Hypotheses .....................................................................................................54 Analysis 1: Measuring Attitude ..........................................................................................55 Analysis 2: Measuring Knowledge ................................................................................... 57 i
Analysis 3: Measuring Barriers .........................................................................................59 Methodological Limitations.............................................................................................................59 Summary of Chapter .....................................................................................................................62 CHAPTER 4: RESULTS .............................................................................................................................64 Missing Data/Don’t Know Answers ................................................................................................64 Descriptive Analysis .......................................................................................................................64 Demographics ...................................................................................................................64 Quantitative Results .......................................................................................................................67 Research Question 1: Do JJSPs attitudes toward EBP differ by individual factors? ........67 Research Question 2: How knowledgeable are JJSPs regarding the concepts of EBP? .......................................................................................69 Research Question 3: Do JJSPs knowledge of EBP concepts differ by individual factors? ...............................................................................................70 Research Question 4: Do JJSPs perceptions of barriers toward EBP differ by individual factors? ...............................................................................................71 Research Question 5: What barrier is perceived by JJSP as the most significant/severe barrier toward using EBP..................................................73 Qualitative Analysis........................................................................................................................74 Lack of Time .....................................................................................................................75 Lack of Training ................................................................................................................75 Lack of Applicability to Clients ..........................................................................................75 Lack of Interest/Negative JJSP Attitude............................................................................76 Lack of JJSP Knowledge/Research Information ...............................................................76 Lack of Resources ............................................................................................................76 Lack of Model Fidelity .......................................................................................................77 Client Resistance ..............................................................................................................77 Lack of JJSP Understanding/Experience using EBP ........................................................77 Lack of Funding ................................................................................................................78 Lack of Support from Administrators/Colleagues..............................................................78 Supervision Issues ............................................................................................................78 CHAPTER 5: CONCLUSIONS ...................................................................................................................91 Conclusions: Research Question 1- Attitudes ...............................................................................92 Conclusions: Research Question 2- Knowledge............................................................................95 Conclusions: Research Question 3- Barriers .................................................................................97 Study Limitations..........................................................................................................................100 Implications for Social Work Research and Practice ...................................................................102 REFERENCES ........................................................................................................................................105 APPENDICES ........................................................................................................................................120 Appendix A: Study Recruitment Letter .........................................................................................120 Appendix B: Study Survey Instrument .........................................................................................121 Appendix C: University of South Florida Institutional Review Board Approval Letter ..................127
List of Tables Table 3.1: Instrument Statistics and Internal Consistency Reliability .......................................................49 Table 2.1: Characteristics of Sample Population .....................................................................................80 Table 2.2: Attitudes toward Evidence Based Practice/Interventions ..........................................................81 Table 2.3: Perceptions of Barriers toward Evidence Based Practice/Interventions ....................................82 Table 2.4: Requirement to using Evidence Based Practice ........................................................................83 Table 2.5: Knowledge of Evidence Based Practice Concepts ....................................................................84 Table 2.6: Attitude Score by Individual Characteristics ..............................................................................86 Table 2.7: Level of Evidence Based Practice Knowledge ..........................................................................87 Table 2.8: Knowledge Score by Individual Characteristics .........................................................................88 Table 2.9: Perception of Barrier Score by Individual Characteristics ..........................................................89 Table 2.10: Code Categories for Qualitative Question ...............................................................................90
Abstract This mixed methods study examined the attitudes, knowledge and perceptions of barriers toward Evidence-Based Practice (EBP) among Florida Juvenile Justice Service Professionals (JJSP). Previous research established individual factors such as age, gender, years of professional experience and educational attainment are related to attitudes and perceptions of barriers among social service and mental health professionals, but scant research has been conducted among juvenile justice providers (Aarons 2004, 2010; Rubin & Parrish, 2007, 2012; Jette et al., 2003). Most individual factors were found to have no significant effect on attitude and knowledge scores within this population with exception of gender and major of study as predictors to barrier scores. Qualitative analysis to a question asking JJSPs to list their top three perceived barriers confirmed quantitative results and revealed Lack of Time to be the most frequently endorsed barrier among JJSPs. By adapting existing instruments to measure primary research variables with a new population, this study advances knowledge in both social work and criminal justice fields. The study’s results also support the use of Rogers’s Theory of Innovation Diffusion and Ajzen’s Theory of Planned Behavior
CHAPTER 1: INTRODUCTION Background of Problem During the last decade, the proverbial pendulum has swung toward a philosophy of addressing juvenile crime with a restorative and rehabilitative approach and away from one that promoted punitive measures for criminal youth behavior. Consequently, the need to determine successful intervention strategies for youth in the juvenile justice system has led to an explosive growth in the literature examining efficacious programs. However, this accumulated knowledge about ‘what works’ does not immediately translate into practice, as professionals may be poorly informed by the research or inadequately trained and/or evaluated in terms of successful implementation of evidence-based interventions (Jones & Wyant, 2007). While practitioners in juvenile justice, child welfare or mental health systems understand the value of evidence-based treatments, it is another matter to render these in daily practice, as this gap between scholarship and its application in the real world has been well documented. These implementation issues are often attributed to factors such as practitioner attitude regarding evidence-based practice, lack of time and resources for practitioners, insufficient training, lack of access to research journals, lack of feedback and incentives for using evidence-based practices, inaccurate assumptions about the efficacy and effectiveness of programs, and inadequate agency infrastructure to support successful translation of evidence-base practices in real-world settings (Glasgow, Lichtenstein, & Marcus, 2003; National Institute of Mental Health, 1999; Schoenwald & Hoagwood, 2001). Community-level providers of juvenile justice services can be thought of as the end-users of research findings, so further understanding of their role in the implementation process of evidence-based interventions is necessary (Walrath, Sheehan, Holden, Hernandez, & Blau, 2006). Despite the acknowledged desirability of using evidence-based interventions to deliver services to youth in the juvenile justice system, there is also a lack of awareness and consensus among juvenile justice program administrators, clinicians and other front line staff of what evidence-based ‘programs’, ‘practices’ or 1
‘interventions’ are. The differences in how these terms are defined are another factor that contributes to the gap between research and practice, since there is not a universal definition for what each of these terms mean and different organizations define them differently. This simply adds to the difficulty of the adoption and implementation of evidence-based programs among community agencies, since the numerous definitions create confusion for agency leadership trying to establish a clear direction. Agency decision-making hinges on administrators understanding what an evidence-based program is and how to implement one correctly. Successful program implementation depends on strong agency leadership that values an evidence-based practice philosophy, an organization’s readiness for change, practitioners’ experience and attitudes, and funding sources. Further complicating this process are the unique characteristics of client populations, characteristics of usual practice, and organizational and resource availability (Mitchell, 2010). The numerous facets related to the implementation of evidence-based practice (EBP) imply a variety of potential research topics. This study narrows the focus by surveying Juvenile Justice Service Professionals (JJSP) in Florida to learn more about their attitudes toward EBP, knowledge about textbook EBP concepts, and perceptions of barriers to using EBP in their daily work. According to the website of the Florida Department of Juvenile Justice (2013), Florida has managed juvenile delinquents under a rehabilitative model of justice, as there was a time when “proceedings related to children” were under the auspices of the Department of Health and Rehabilitative Services (formerly known as HRS). At that time, the agency’s approaches to dependency and delinquency cases were the same: to provide services to the child and family. In 1994, Florida began its gradual shift from a social services model toward a juvenile justice model and the state legislature created the Department of Juvenile Justice (DJJ). The newly formed DJJ retained HRS employees, so the philosophy of recognizing juvenile delinquents as children in need of treatment and reform rather than as criminals deserving punishment continued. However, by 2000 Florida passed comprehensive legislative known as the “Tough Love” plan and provided statutory authority for DJJ to overhaul its organizational structure. Prevention and intervention programs were maintained, but control and public safety services grew into Offices of Detention Services, Probation and Community Intervention and Residential Services within the DJJ; indicating that the 2
prevailing approach had swung away from rehabilitation and toward punishment, likely in response to the political and economic environment of that time. However, by July of 2007, Governor Charlie Crist authorized the creation of the Blueprint Commission, which was charged with making recommendations to improve Florida’s juvenile justice system. The findings and recommendations were developed in conjunction with the nation’s top researchers at the Center for the Study and Prevention of Violence at the University of Colorado to create more EBPs that emphasized prevention and rehabilitative intervention programs but could also hold the state accountable and ensure that money was allocated to the most efficacious programs. To this end, DJJ subscribed to guiding principles that deemed prevention and education to be paramount and emphasized that it was necessary to promote public safety through effective intervention. The emphasis on “effective” prevention and intervention led to the popularity of using name brand evidence-based programs/interventions that could measure outcomes as long as fidelity to the program model was maintained. Therefore, community agencies bidding for contracts from the DJJ needed to also adopt such programs/interventions to remain competitive for funding from the DJJ since the DJJ awarded grants and contracts based on the extent to which community agencies’ services and interventions could be shown to be “evidenced-based.” Maintaining agency funding is a concrete reality that drives agency decision-making. Thus the rationale for using EBPs is sound, but it takes a great deal more than model fidelity for EBPs to actually succeed and allow an agency to deliver the quality services it promises. Statement of Problem As explained above, this research study was born from the DJJ recognizing that public safety is promoted through effective intervention and education and the need to learn more about how JJSPs perceive EBP in order to help agency leadership tailor training and implementation planning. One of the most basic issues stems from how JJSPs define EBP. Depending on one’s education and background, the definition falls into one of two broad categories. Simply stated, within the context of juvenile justice research, EBP is defined as “using a body of knowledge obtained through scientific method to determine the impact of specific practices on targeted outcomes for youth and their families” (Hoagwood et al., 2001, p. 287). When JJSPs “utilize certain interventions that have already been designated as empirically 3
supported or evidence-based,” this has been described as a top-down approach. A bottom-up approach is when JJSPs “seek and appraise evidence themselves in connection to idiosyncratic practice decisions” (Rubin & Parrish, 2007, p. 112). EBP as proposed by Sackett, Rosenberg, Muir, Gray, Haynes, and Richardson (1996) in Evidence-based medicine was more bottom up in that it was broadly defined as integrating the best research evidence with clinical expertise and patient values. Leaders in the field of social work noted the congruence between Sackett’s description of evidence-based medicine with similar clinical social work values; values such as taking into account the best available research evidence along with the practitioner’s expertise and meshing it with client needs, values and preferences, all within the client’s environmental and organizational context (Gambrill, 2006; Straus et al., 2010; Thyer & Pignotti, 2011). Again, however, within juvenile justice research EBP has tended to refer to an individual program or intervention in a top-down fashion and implementation has entailed practitioners adhering to specific treatment guidelines (Gellis & Reid, 2004). This usage is different from the ‘process’ or ‘philosophy’ of EBP that the social work profession has begun to embrace with efforts to integrate research evidence into practice at the individual level (Stanhope, Tuchman, & Sinclair, 2011). Intervention research focused on structure rather than process generates studies that measure the effects of the structural manualized components of an intervention on outcomes without taking into account variable human processes that are an inevitable part of service delivery (Stanhope et al., 2011). This type of research emphasizes the ‘gold standard’ of findings generated from randomized controlled experiments or ‘bronze standards’ of experimental and quasi-experimental research with comparison groups and meta-analyses to designate which interventions truly work (Byrne & Lurigio, 2009). These data are then published by well-regarded organizations such as the Campbell Collaboration or Cochrane Collaboration that summarize outcome studies and identify which evidence-based programs/interventions have the most solid empirical support and produce systematic reviews to help agencies and practitioners to select ‘the best’ evidence-based programs/ interventions, hence the topdown dissemination of information (Rubin & Parrish, 2007). Further along on this juvenile justice research continuum are studies of implementation problems that occur when agencies adopt the interventions.
Unlike what is the case in social work literature, very little is published in juvenile justice research about the thought process related to, understanding of or attitudes to using EBP among professionals working in this field. While research concerning EBP among social workers in mental health or community settings has increased, social workers employed in juvenile justice settings have not received the same attention. In general, little research has been conducted among all types of juvenile justice professionals regarding their attitudes, knowledge and skills related to EBP. Therefore, this study addressed the knowledge base gap by exploring how the original philosophy of EBP aligns with the attitudes of professionals currently working within juvenile justice provider agencies in the state of Florida. Purpose of Study Thyer and Pagnotti (2011) referencing Sackett’s (1997) definition of evidence-based medicine, asked healthcare practitioners to locate the best available evidence, evaluate its potential applicability to clients and to use this to inform their practice decisions. One of the purposes of this study is to examine how this philosophy of EBP translates into the juvenile justice setting among professionals from various educational and training backgrounds with a quantitative survey design. Because there is a heavy emphasis within juvenile justice literature on the barriers of implementing evidence-based programs or interventions, there appears to be a disconnect between research published within the implementation science field of juvenile justice emphasizing model fidelity and research that examines practitioner values, attitudes or perceptions regarding evidence-based practice and in integrating EBP concepts within the realities of the practice world. This calls for further investigation, as these realities are not mutually exclusive, but rather model fidelity can be affected by practitioner values, attitudes and perceptions of barriers. Significance of Study According to Proctor (2004), The gap between the availability and actual use of evidence-based treatments remains wide and persistent…. This gap compromises the quality of care and threatens professionals’ abilities to achieve their goals of reducing disparities in health, family well-being, and individual functioning in society. Failure to use research-based knowledge may prove costly and harmful, leading to overuse of unhelpful care, underuse of effective care, and errors in execution. (pp. 227-228) Segregation between the research and practice worlds has contributed to the lack of growth of published practice-based evidence. Simply transferring evidence-based “name-brand” interventions into the clinical 5
arena requires major shifts at the policy, agency and clinician level, and real world settings cannot simply absorb new practices because the organizational structure itself and its people are comprised of a multitude of factors that must all accept and adapt to the new practices (Stanhope et al., 2011). Thus, implementation depends on research knowledge that both translates well into and has utility in the real world. Additionally, the dissemination of this knowledge, the receptiveness and skills of practitioners, the leadership and infrastructures of agencies as well as the conditions of local, state and national policies all need to be in alignment to create an environment for successful implementation (Mancini et al., 2009; McHugo et al., 2007). Implementation is defined as “the intentional use of strategies to introduce or adapt evidencebased interventions within real-world settings” (Mitchell, 2010, p. 208). While this particular definition can encompass the social work value of infusing EBPs into client services, this is distinguished from ‘adoption,’ which is merely a formal decision to use an evidence-based intervention. The goal of successful implementation is to achieve regular use of evidence-based interventions (Mitchell, 2010). One would be remiss if one were to discount the importance of practice wisdom, the knowledge emerging and evolving from primarily practical experience rather than from empirical research and gained personally and/or disseminated among practitioners that seems to encompass many of the clinical practice issues related to implementation. Agency decisions and actions often stem from practice wisdom, but because much practice-based knowledge remains tacit and undocumented, the thinking behind such decisions is never formally articulated (Mitchell, 2010). It could be explained that the opposing worldviews of practice wisdom and EBP is the difference between ‘how’ and ‘what,’ or the difference between infrastructure and content. Essentially, core concepts of practice wisdom, such as commitment to client-centered, developmentally appropriate, or relationshipbased care, refer to how services should be delivered, which shapes how assessment, problem formulation and treatment planning at the practitioner-client level are determined as well as how resources are allocated and the workforce is developed at an organizational and systemic level (Mitchell, 2010). On the other hand, the core concepts in the EBP literature focus on ‘what’ is delivered and can include specific treatment modalities with evidenced-based therapeutic content that is behavioral or 6
cognitive and measurable via assessments or instruments. The key is for evidence-based interventions to be incorporated but still to fit under the ‘how’ of working with clients (i.e. client-centered or developmentally appropriate) that an agency subscribes to. Some form of resolution or synthesis between empirical research and practice wisdom that highlights the best knowledge from both would increase the quality and effectiveness of client services and offer some solutions for the gap in knowledge identified within this study (Aarons, 2004; Aarons, Sommerfeld, & Walrath-Greene, 2009; Mitchell, 2010). Relevance to Social Work Recent studies of mental health clinicians and child welfare providers reveal that practitioner attitudes to EBP have a significant effect on the success of program implementation, and these attitudes may stem from practice wisdom. Commonly reported themes of practitioner attitudes include practitioners’ concern for the lack of applicability of research with their client population, a desire for greater emphasis on the therapeutic relationship and the need for flexibility within treatment protocols. Specifically, most evidence-based treatments are perceived to be too long to be effectively implemented in community practice because client dropout rates are so high. Research has also indicated that practitioners perceive EPBs that are found to be efficacious in controlled settings to be impractical or difficult to apply with actual clients (Aarons, Lawrence, & Palinkas, 2007; Gioia & Dziadosz, 2008; Hurlburt & Knapp, 2003; Nelson & Steele, 2007; Nelson, Steele, & Mize, 2006; Pope, Rollins, Chaumba, & Risler, 2011). Although additional studies show that mental health practitioners generally have positive views of EBP, these positive attitudes do not necessarily result in actual use or implementation of an evidencebased process in practice due to system-level issues such as insufficient time, resource constraints and inadequate access to guideline materials (Crisp, 2004; O’Neill, 2003; Plante, Anderson, & Boccaccini, 1999; Shlonsky & Gibbs, 2006). Thus, clinicians’ receptivity to EBPs appears to depend on their perceptions of the fit between EBPs, their agency goals and personal practice values (McColl et al., 1998; Rosenheck, 2001; Sanderson, 2002). One of the primary focuses of this study is whether JJSPs and mental health practitioners have similar attitudes toward EBP. This study adds to current social work research because the level of EBP knowledge among JJSPs was discovered by using surveys to rate their knowledge level as high, medium or low. 7
A wide range of studies point to the need for continued research of implementation barriers as they relate to practitioner attitudes, knowledge and skills. While most of the literature cited thus far has included sample populations of licensed mental health counselors, licensed clinical social workers and other clinical mental health specialists throughout the nation, this study included a group that has not been surveyed on this topic before: social workers and other professionals with different backgrounds working within Florida juvenile justice provider agencies. Research Design and Primary Research Questions Utilizing a cross-sectional design and administering a survey, this study sought to obtain information from community providers of juvenile justice programs to determine what professionals in this field know about EBP as demonstrated by their understanding of evidence-based concepts and definitions; what their attitudes to EBP are and what their perceptions regarding barriers to implementing EBP are. Participants in this study included agency administrators, master’s level clinicians/practitioners, bachelor’s level case managers and other front line staff working in juvenile justice provider agencies across Florida who subscribe to an EBP philosophy or offer an evidence-based intervention as part of their programs for youth and families. Scope of Study and Delimitations This study took a closer look at the differences in attitudes, knowledge and perceptions of barriers among different professionals who provide leadership within agencies and interact with youth in addition to the social workers, counselors, and other mental health practitioners providing therapeutic services. The results of this study are limited to the state of Florida, but by including English-speaking women and men employed in community agencies providing prevention, diversion, and residential services to juvenile justice youth across the state, this study gleaned exploratory data about attitudes, knowledge, and perceptions of barriers regarding EBP among JJSPs for the entire state. Any person currently employed in an agency and occupying a role providing direct services to youth was included in this study, as length of employment and job title were key variables tested. Examples of these job titles are: social worker (licensed or non-licensed), counselor (intake, community, transition services, etc.), therapist (licensed or non-licensed), case manager, shift supervisor, family 8
service worker, mental health worker (licensed or non-licensed), and youth specialist. Additionally, those in leadership positions such as clinical director, program coordinator and program director were also invited to participate in this study. By more clearly articulating the relationship between practitioner attitude, knowledge, and perceptions of barriers to EBP, this study adds to the literature base on implementation issues with what was found among professionals providing services for the Florida Juvenile Justice System. Summary of Chapter While the fields of mental health and social work have made strides toward adopting an evidenceinformed philosophy that includes practitioner clinical judgment and accounts for real-world implementation barriers, much of the research in the field of juvenile justice continues to focus on efficacy studies of evidence-based interventions. The specific objectives of this study were to: •
Identify the attitudes to EBP among JJSPs and compare the differences between them according to years of professional experience, agency type, major studied, and degree level.
Identify the level of knowledge of concepts and process of EBP among JJSPs and compare differences between them according to years of professional experience, agency type, major studied, and degree level.
Identify primary barriers to JJSPs implementing EBP.
Determine whether there is a relationship between JJSPs’ attitudes toward EBP and the extent to which situations are perceived as barriers to their use of EBP in their daily work with clients.
Determine whether there is a relationship between JJSPs’ level of knowledge and attitude to EBP.
Determine whether there is a relationship between JJSPs’ level of knowledge and the extent to which situations are perceived as barriers with regard to their use of EBP in their daily work with clients. The research questions that guided the study were: 1) What are JJSPs’ attitudes toward using EBP? 2) What are the perceptions of barriers to using EBP among JJSPs? 3) What is their actual knowledge of EBP concepts? 9
CHAPTER 2: REVIEW OF LITERATURE This chapter provides a theoretical framework and a brief historical perspective of EBP. It also explores the differences in its current application within the fields of social work and criminal justice on the basis of research published in approximately the last 20 years regarding practitioner knowledge and application of EBP concepts, attitudes of professionals to EBP, and professionals’ perceptions of barriers to EBP. Theoretical Framework This study examined individual level attitudes toward EBP, knowledge of EBP and perceptions of barriers to implementing EBP in the juvenile justice provider setting. However, this research cannot be taken out of the context of how organizational factors contribute to the adoption and diffusion of EBP. Thus, Rogers’s Diffusion of Innovation Theory (2003) undergirds this study, as it illustrates a process in which new ideas, practices or innovations are integrated by an individual and then spread into a social system. The theory holds that innovation diffusion is a “general process, not bound by the type of innovation studied, but by who the adopters are or by place or culture” (Rogers, 2004, p. 16). Because this process of innovation diffusion explains how ideas or behaviors are taken up in a population rather than being focused on the actual idea or behaviors themselves, it has a general application that has allowed many different fields and academic disciplines to find uses for this theory. In this case, Rogers’s theory was a good fit for explaining how an innovation (EBP) has been absorbed (diffused) within a system (JJSP in Florida). Rogers’s insight into the process of social change answered three questions: 1) What qualities allow an innovation to spread successfully? 2) What is the importance of peer-to-peer conversations and/or peer networks in this spread? 3) How does one best understand the needs of different users of the innovation? (Robinson, 2009). To start, Diffusion is defined as “the process in which an innovation is communicated through certain channels over time among the members of a social system” (Rogers, 2004, p. 5) and Innovation is 10
defined as “an idea, practice, or object that is perceived as new by an individual or other unit of adoption” (Rogers, 2004, p. 12). The end result of diffusion and innovation is Adoption, defined as the point at which “an individual acquires new knowledge, determines whether to accept or reject the new information and then accommodates and uses the new information” (Rogers, 2004, p. 12); Implementation is defined as putting the innovation into practice and testing it, and Institutionalization is when an innovation is fully supported and incorporated into typical practice routines. When enough individuals have adopted, implemented and institutionalized a new practice, then diffusion occurs because innovation ideas have become widespread and acceptance has been created, resulting in integration of a practice into an organization. To answer Rogers’s first question of what qualities make innovations spread, innovations must be perceived as better than the idea that they supersede by a particular group of users. In the context of this study, juvenile justice providers from the front line worker to the agency administrator must perceive that EBPs have value and that the process itself allows for better service delivery than by not using EBPs or interventions. Secondly, innovations must be perceived as being consistent with the values, past experiences, and needs of potential adopters. This was a key element of this study as attitudes to EBP hinged on the personal values, needs and past experiences of juvenile justice providers, as well as the values and practices of an agency. Three additional qualities of innovations must be present in order for adoption to occur. Understandably, the degree to which an innovation is perceived as simple and easy to use is important, since innovations that display these qualities are adopted more quickly than innovations that require the adopter to develop additional comprehension and new skills. Within the realm of juvenile justice, this was likely to be a central factor, as this study intended to use “EBP test questions” on a survey to explore whether knowledge of EBP was demonstrated among agency leadership and upper ranks as well as among front line employees. This component intended to answer the question of how simple and easy is it to understand the concepts of EBP by assessing what the sample’s baseline understanding was, and was also used to determine if EBP was considered a more complicated innovation and was thus slower to be adopted among JJSPs.
Rogers (2004) also suggested that the degree to which an innovation can be experimented with on a limited basis, or the “trialability” of an innovation, affects the rate of adoption. When an innovation can be tested and used on a smaller scale, potential adopters become more receptive as the innovation is perceived as less uncertain to the individual who is considering it. To this end, trial efforts to implement EBPs within a system as large as the Florida DJJ may be reasonable, as there are many different discrete programs, offices or contracted agencies within the DJJ that could be early adopters of EBPs. This could then lead to evaluation and replication among other programs, offices or agencies within the DJJ. However, at the individual level, employees who work in any one of these particular programs, offices or contracted agencies who have been tasked with implementing the new EBPs may not necessarily feel that this is a trial effort, but rather a huge change that affects their entire agency and the way they complete their daily work. The concept of “trialability” was considered important as it related to JJSPs’ attitudes and perceptions of barriers to using EBP. Rogers’s (2004) fifth and final quality explaining innovation adoption is the ability to produce observable results. This straightforward explanation implied that the more visible the results of an innovation are, the more likely it is for individuals to adopt it because visible results produce lower uncertainty and stimulate peer engagement and discussion of the innovation (Robinson, 2009). Given that an observable result of innovation is critical, an individual’s attitude to this visible result can be a precursor to the decision of whether or not to try a new practice. According to Aarons (2007), developer of the Evidence-Based Practice Attitude Scale (EBPAS), if service providers do decide to try a new practice, then the affective or emotional component of attitudes can affect decision processes regarding the actual implementation and use of the innovation. The EBPAS measures four dimensions: 1) the intuitive appeal of the EBP, 2) the likelihood of adopting an EBP given the requirements to do so, 3) the individual’s openness to new practices, and 4) the perceived divergence between research-based interventions and current practice. These dimensions paralleled Rogers’s qualities of innovation, as Aarons (2007) reported that individual factors such as professional experience and training are as important as organizational factors such as program type and presence of written policies in their effect on both individual professionals’ attitudes and the rate of EBP
adoption and diffusion within an organization. More information related to the EBPAS is provided in subsequent sections describing the instruments used in this study. Likewise, Funk, Champagne, and Tornquist (1995) studied barriers to research utilization among nurses. They also used Rogers’s innovation diffusion theory to explain different types of barriers and reported analogous themes among clusters of barriers. These themes included: a) issues related to the adopter, b) the innovation itself, c) how the innovation is communicated, d) the system or organization in which an individual works, and 5) time, which is the dimension that cuts across all four previous areas. Funk et al. (1995), Hefferin et al. (1982), and Miller and Messenger (1978) each noted similar barriers to research utilization, such as organizational and workplace characteristics, time to read and implement research, research values and skills of clinicians. The process of communicating research findings, the rigor of research and research accessibility were also potential barriers. Funk et al. (1995), in their study of nurses in various subfields and roles, concluded that there was a stable picture across time that predominant barriers to the use of research findings in nursing practice were due to both limitations in their organizational setting as well as within individual nurses’ capacity for understanding research. This finding corroborated Rogers’s theory that innovation diffusion occurs more rapidly when these barriers are reduced. In this study, Rogers’s (2003) Diffusion of Innovation Theory was used to organize theoretical elements regarding system change. However, in the following section, theories of individual change are included to provide a conceptual framework for the actual variables under study. Conceptual Framework Ajzen’s (1991) Theory of Planned Behavior and Bandura’s (1977) Social Learning Theory were used to further guide this study at the individual level. Ajzen argued that the best predictors of an individual engaging in a specific behavior are dependent on his/her attitudes, subjective norms and perceived behavioral control. Individual behavior stems from intentions, which are a function of an individual’s attitude about the behavior. A person may have positive or negative feelings about performing a particular behavior and this is reflected in his/her attitude to it. A person’s attitude is developed via an internal self-assessment of one’s beliefs regarding the consequences of performing the behavior and the
desirability or undesirability of these consequences (Eagly & Chaiken, 1993; Wade & Schneberger, 2005). Subjective norms are the perceptions of how others value the behavior, and they are specifically defined by an individual’s perception of what people important to him or her would think were the behavior to be performed or not. The weight of the opinion of others is supplemented by the individual’s desire to comply with what he or she believes the other person wants them to do. Perceived behavior control is the individual’s perception of his/her ability to overcome potential obstacles or how difficult one believes it will be to perform the behavior. This perception of difficulty lies on a continuum of behaviors that range from easy to perform to those requiring considerable effort, resources, time etc. (Eagly & Chaiken, 1993; Wade & Schneberger, 2005). With respect to practitioner attitudes toward using EBP in their daily work, the Theory of Planned Behavior explains how an individual’s attitude about using EBP is mediated by her perception of how her colleagues feel about using EBP. If a practitioner believes most of her co-workers have positive attitudes about using EBP, then she is more likely to engage in EBP herself, as defined by the concept of subjective norms. Furthermore, if a practitioner perceives there to be few barriers to integrating EBP into her own practice and does not expect it to require a significant effort or considerable time and resources, then perceived behavior control has not negatively affected her subjective norm or attitude to EBP. Ajzen’s (2011) work is frequently cited as a model for predicting human behavior and has become more widely known in recent years. Yet in spite of this theory’s popularity, it has detractors as some clinicians think it offers an inadequate explanation of human behavior and the theory has received criticism for reducing human behavior to simple terms. However, the concept of planned behavior does not stray far from Bandura’s Social Learning Theory (1977). According to Bandura (1977), expectations such as motivation and feelings of frustration result from repeated failures or successes, which then determine effect and behavioral reactions. Expectations are separated into two types: self-efficacy and outcome expectancy. Self-efficacy is the belief that one can successfully perform the behavior required to produce the desired outcomes. Outcome expectancy pertains to how a person estimates that a given behavior will lead to successful outcomes. Bandura (1977) theorized that self-efficacy was the most important aspect of behavioral change because without it, 14
a person cannot develop coping skills. Ajzen’s (2011) perceived behavior control originated from the concept of self-efficacy, because if one’s sense of self-efficacy led to the belief that one can successfully perform a behavior with desired outcomes, then one is inclined to perceive the difficulty of performing that same behavior as being lower on the continuum. At the heart of Social Learning Theory are three concepts: people can learn through observation; people’s internal mental states are an essential part of the learning process; and the fact that a concept has been learned does not mean that it will result in a change in behavior (Bandura, 1977). Bandura (1977) summarized: Learning would be exceedingly laborious, not to mention hazardous, if people had to rely solely on the effects of their own actions to inform them what to do. Fortunately, most human behavior is learned observationally through modeling: from observing others as one forms an idea of how new behaviors are performed and on later occasions this coded information serves as a guide for action (p.123). In integrating and using Social Learning Theory to explain individual behavior toward using EBP, the basic tenets of this theory described the process of how an individual learns new knowledge that determined their behavior, which in turn determined how adoption and integration of new knowledge could occur. Both the Theory of Planned Behavior and Social Learning Theory explained how humans respond to new knowledge and information, which was what this study hoped to capture in regard to the use of EBP among JJSPs. Evolution and Definition of EBP For this study, EBP is defined as a “way of assessing, intervening, and evaluating based on a set of assumptions and values” (Mullen, Bledsoe, & Bellamy, 2008, p. 326). EBP was originally developed in medicine as a way to train residents in a new form of practice that emphasized critical assessment skills to strengthen the scientific base used by physicians in decision-making (Evidence-Based Medicine Working Group, 1992; Sackett, Rosenberg, Gray, Haynes, & Richardson, 1996; Straus, Richardson, Glasziou, & Haynes, 2005). Sackett et al. (1997) extended this definition to “the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients” (p. 71). This definition was further conceptualized by Sackett et al. (1997) as not only using the available best evidence for decision making but also allowing a physician to evaluate both their own personal experiences and 15
external evidence in a systematic and objective manner when identifying effective treatment approaches for their patients. As for the integration of best evidence, this referred to clinically relevant research pertaining to diagnostic tests, prognostic markers, efficacy and safety of therapeutic, rehabilitative, and preventive interventions. Clinical expertise included using physicians’ skills and experiences to quickly identify each patient’s unique state of physical health and the individual risks and benefits of possible interventions while accounting for patients’ personal values and expectations (Institute of Medicine, 2000). In the medical field, practitioner accountability is vital, and is established through a standardization process requiring efforts to follow specific protocols when administrating a treatment. Medical trials administering the same medication or intervention at the same dose to treat the same symptoms time after time ensured the efficacy of that particular medicine or intervention. This model of evidence-based medicine was eventually integrated into psychotherapeutic care and evolved into the empirically supported treatment (EST) movement (Reynolds, 2000). EST refers to specific interventions that have demonstrated efficacy for treating specific problems as a result of many randomized trials (Wachler, Kalodner, Wampold, & Lichtenberg, 2000). This led to the creation of treatment manuals as the best way to ensure standardization in the mental health arena (Patel, 2010). In 1993, the APA created the Task Force on Promotion and Dissemination of Psychological Procedures in response to the justification required for therapeutic interventions (Madson, 2005). The task force intended to identify efficacious interventions for the purposes of training graduate students (Chambless et al., 1998). However, a number of clinicians strongly opposed ESTs, claiming that the identified interventions did not include aspects of psychotherapy that were relevant to successful client outcomes such as clinician flexibility and the therapeutic relationship (Garfield, 1996). The long-standing debate was mirrored among prominent social work authors (Briggs & Rzepnicki, 2004; Gibbs & Gambrill, 2002; Howard, McMillen, & Pollio, 2003; Kirk & Reid, 2002; Mullen & Bacon, 2006; Mullen & Streiner, 2004; Proctor, 2006; Roberts & Yeager, 2006; Rosen & Proctor, 2004; Thyer, 2004). Because of this controversy, the APA introduced the concept of EBP and defined it as the integration of best available research with clinical expertise and client values. Current definitions of EBP range from a narrow conceptualization of EBP as a practice model to a broader process-oriented 16
definition that encompasses more than just interventions (Woody et al., 2006), but the idea that EBP is a process rather than a model is more commonly accepted within social work, mental health and other practice-oriented fields. This process has now become part of the philosophy of practice fields that includes practitioner experience and client values and balances them with the importance of scientific evidence. Essentially, clinicians who use EBPs take into account the dynamics of the therapeutic relationship and client variables before determining what specific interventions or approach to use (Patel, 2010). Development of EBP in Social Work The divide between social work practitioners and researchers was evident as early as the late 1800s when charitable organizations attempted to use a more “scientific” approach not only to give financial assistance to families but also to approach charity as an effort to advocate and organize various resources. Documentation methods were developed to prevent duplicity and duplication of services for the client and the service provider. Also at this time, journals were published to disseminate successes, concerns and future goals (Boyer, 1978; Katz, 1996; Popple, Phillip, Leighninger, 2002; Richmond, 1969). By 1955, the National Association of Social Workers (NASW) was formed and its bylaws stated that social workers were to “further the broad objective of improving conditions of life in our democratic society through utilization of professional knowledge and skills of social work, and to expand through research the knowledge necessary to define and attain these goals” (NASW, 1955, p. 3). However, from the beginning these efforts to create structure around practice continued to be resisted by those who did not want to approach social problems with a scientific method. During the 1950s and 1960s, few studies examined the effectiveness of social work practice, with exception of the landmark study by Myer, Borgatta, and Jones (1965), Girls at vocational high-An experiment in social work intervention; that examined the effectiveness of casework. But these types of studies grew in number as time passed. Interest in effectiveness studies increased and thus the rigor of research methods evolved from pre-experimental designs lacking internal validity to single case designs that became the start of empirically based practice, which charted the progress of individual clients toward goal attainment (Garvin & Tropman, 1992).
In the 1970s, researchers began encouraging eclectic practitioners to use empirically based interventions that would lead to measurable outcomes. Unfortunately, although the trend of offering empirical training using research to guide practice and evaluations continued in schools of social work, a similar trend was not seen in practice settings (Reid, 2004). Opponents of EBP argued that “it denigrates the clinical experience, ignores patients’ values and preferences, promotes a cookbook approach to practice, is merely a cost cutting tool and it leads to therapeutic nihilism” (Mullen & Streiner, 2004, p. 117118). The social work profession has tried to define itself ever since Flexner’s (1915) challenge in his momentous speech that “social work was not a profession but an intellectual activity with a mediating function that linked individuals with social functioning problems to helpful resources” (cited in Holosko, 2003, p. 265). Although what identifies social work as a profession includes having a professional organization, using a code of ethics and employing standards of practice that are governed by an accrediting body that monitors training, credentialing and licensing, the debate on the professional status of social work continues to evolve (Hackney & Cormier, 2005). One key component determining the status of the field as a profession is the degree to which empirically derived knowledge guides practitioners (Levy, 2010). Social work leaders such as Eileen Gambrill (2006a, 2006b) and Bruce Thyer (2004) have written extensively about the importance of EBP in both practice and research, and it is difficult to deny that using EBP will increase effectiveness and credibility in a social worker’s practice. It is also hard to argue against the importance of integrating evidence into interventions as a way to identify the best available practices (Zayas, Drake, & JonsonReid, 2010). Hence, within the last 20 years emphasis on the use of EBP in the social work literature has partly been a response to the fact that many national organizations will not fund agencies unless they are using programs that are evidence-based or supported by scientific data (Mullen, 2004). Funding bodies are also demanding more accountability, efficiency and effectiveness in the provision of social services (Pope, Rollins, Chaumba, & Risler, 2011). Yet the crux of this issue within social work remains the relationship between the producers of research (researchers) and the users of research (practitioners).
The gap between research and its use by practitioners has been well documented over the past 50 years (Kirk & Reid, 2002) and despite efforts to encourage practitioners to use research to guide their practice, studies have continued to show that practitioners do not like to read research, do not consistently read it or use it to evaluate their practice, and they rarely use empirically supported interventions (Kirk & Reid, 2002; Mullen & Bacon, 2006; Sanderson, 2002). Other studies have noted that this gap is not just found in social work, but also throughout other health and human services, in that there is a discrepancy between what research has demonstrated to be effective and what actually occurs in practice (Fixen, Naoom, Blasé, Friedman, & Wallace, 2005; Panzano & Herman, 2005; Torrey & Gorman, 2005). Shon (1983) reported that social workers think of their practice as more of an art than a science. Zayas, Drake, and Jonson-Reid (2010) stated that, among clinical social workers, EBP is believed to be too narrow in focus and possibly irrelevant to their client population, and that they held that their clinical judgment “supersedes the use of scientifically tested techniques” (p. 400). Rosenblatt (1968) wrote that social worker practitioners have a low level of research skill and considered it as one of the last sources of information to turn to when looking for guidance to practice. More recent studies have shown that clinicians have still not changed the way they make clinical decisions based on best available external clinical evidence from systematic research. Davis, Gervin, White, Williams, Taylor, and McGriff (2013) found that graduating MSW students from a large southeastern university are producing less evaluation-oriented scholarship and field instructors possess insufficient knowledge and skills of EBP to teach students in the field setting. Their findings are similar to those of McNeill’s (2006 study), which reported that social workers are uncertain about how to implement EBP in practice. Davis et al. (2013), as one explanation of their findings, raised the possibility that the average work experience of their sample was 14 years, with a maximum of 35 years, which meant many of their participants began their careers well before policy and practice trends embraced outcomes and EBP, and their social work education hence probably did not include training and skills commensurate with EBP. Mathiesen and Hohman (2013) adapted and revalidated an instrument that assessed knowledge of, attitudes to and behaviors toward EBP that was originally created for medical students in Hong Kong. 19
They surveyed BSW and MSW students as well as field instructors to learn more about their knowledge of and attitude to and behavior toward EBP from social workers who were beginning their careers and those who had many years of experience in the field. The most noteworthy finding they reported was the significantly higher self-rating of use and knowledge about EBP by MSW students as compared to BSW students and field instructors. Mathiesen and Hohman also stated that the recent emphasis on EBP in the field and in schools of social work could explain the lack of EBP training field instructors have received since beginning their career many years ago. Conversely, undergraduate social work students may also lack EBP training given the shorter length of their education. Mathiesen and Hohman recommend additional training and access to EBP resources for field instructors and graduating students when a university library is not readily accessible . Ultimately, the translation of research findings into the practice setting often fell short of the ideals of EBP (Kam & Midgley, 2006; Morrow-Bradley & Elliott, 1986). A recently published study conducted among 364 Australian social workers found that a third of the sample self-reported their literature searching skills as adequate and that participants had conducted a literature search within the past month. However, mean scores for skills in critically appraising the literature were below adequate, supporting the contention that social workers have an unsophisticated grasp of research evidence and the authors in turn recommended additional skill building or professional development in assessing the strength and application of research findings (Gray, Joy, Plath, & Webb, 2014). On the one hand, researchers may be asking the wrong questions, being overly technical when describing their findings, or be unable to make findings known to the clinical community. On the other hand, clinicians do not value research because of not having the skills or time to access available evidence, or they may not have the agency support or infrastructure to prioritize research utilization (Midgley, 2009). Kirk and Reid (2004) concurred, and suggested five reasons explaining the divide between researchers and practitioners: •
There was a sudden influx of practice research literature, but lack of knowledge on how to disseminate this information to practitioners.
The vast differences in worldview and orientation among practitioners are not sufficiently captured within research literature. 20
The perception among practitioners that research lacks real world applicability.
Practitioners are generally disinterested in research and research utilization.
Research methodology and designs proved to be too complex to be useful in practice settings. This study investigated Kirk and Reid’s (2004) reasons for the divide between research and
practice within social work by examining the relationships between JJSPs level of knowledge of EBP and their perceptions of implementation barriers. Development of EBP in Juvenile Justice The social work field has recognized EBP as both a problem-solving process and the use of an empirically supported treatment. However, studies published within criminal justice or criminology related to EBP tends to focus on efficacy research (when a particular treatment or program is tested under controlled conditions) or on effectiveness research (when an efficacious intervention is delivered in community settings without a controlled setting) to gauge how well a treatment works in a real-world setting (Chambless et al., 1998). Much has been written on implementation of specific evidence-based interventions rather than on practitioners’ thoughts about the EBP philosophy. While there are plenty of efficacy and implementation studies concerning interventions published within social work, it is simply more common in the criminal justice literature for EBP-related studies to review which practice models or interventions have passed rigorous standards to be deemed “evidence-based” or how an “evidenced-based program” was implemented rather than discussion of the EBP process itself. Again, there are a variety of definitions of EBP in criminology and juvenile justice. Similarly complicating matters is the interchangeable use of terms such as “research-based”, “empirically supported”, “best practice”, “promising practice” or “innovative practice”. Fortunately, there is a thread of consistency among the different definitions within juvenile justice, and the term ‘evidence-based’ is appropriately defined as “conscientious, explicit and careful utilization of the best available evidence in professional decision making. More specifically, it is the use of scientifically validated assessment, intervention and evaluation procedures with specific client groups” (Juvenile Justice Sourcebook, 2004, p. vii). The emphasis is on selecting an evidence-based assessment, intervention and evaluation process rather than using EBP as a problem-solving approach. Therefore, the characteristics of interventions used 21
within juvenile justice systems are described in the next section, along with a description of current EBP initiatives within the Florida DJJ, as this provides a better understanding of the context and types of interventions used by JJSPs sampled in this study. Characteristics of EBP Interventions used in Juvenile Justice Guerra, Kim, and Boxer (2008) indicated that multiple influences on juvenile delinquency varied over time and across contexts. Consequently, it was found to be important to tailor interventions to dynamic or changeable risk factors that had the greatest influence on a youth. This involved focusing on family dynamics, individual skills and beliefs that impacted behavior in a wide range of situations and environments. There are multiple pathways to delinquency, and variations in “readiness to change” must be considered when selecting interventions since a particular program may not be effective alone but be more effective in conjunction with other interventions. The three major principles of effective intervention – risk, need and responsiveness – should undergird most effective interventions. These principles translated into practice specified the need to use behaviorally-based interventions with higher risk offenders; targeting interventions to their criminogenic needs while being responsive to their temperament, learning style and level of motivation to determine the most appropriate interventions (Andrews et al., 1990; Andrews, Bonta, & Hoge, 1990; Andrews, Bonta, & Wormith, 2006; Lowenkamp & Latessa, 2005; Lowenkamp, Latessa, & Smith, 2006; Lowenkamp, Pealer, Latessa, & Smith, 2006; Lowenkamp, Hubbard, Makarios, & Latessa, 2009). Successful interventions are typically highly structured and employ a cognitive-behavioral approach (Cullen, 2005; Losel, 1995; McGuire & Priestly, 1995). They are based on a rehabilitative or therapeutic philosophy (Lipsey et al., 2010); are frequently evaluated by well-trained staff who adhere to a program model (Dowden & Andrews, 2004; Gendreau et al., 2006; Lipsey 2009); are delivered in the community rather than in an institutional setting; and are of sufficient duration and intensity (Izzo & Ross, 1990; Lipsey 2009). Furthermore, Julian (2011) reported that programs that accurately assessed risk and needs in order to match juvenile offenders with appropriate interventions, used skill-building approaches, and provided feedback to program participants were more successful (Andrews et al., 2006; Bonta & Andrew, 2007; Cullen & Gendreau, 2000; Howell, 2009).
Selecting and Assessing Evidence-Based Programs Regardless of whether interventions are focused on the individual or the family, juvenile justice systems and community agencies have the difficult task of determining what interventions and strategies to implement in the first place. It is critical that agency leaders or decision-makers recognize the importance of incorporating research into practice. In some cases, specific programs have been evaluated, replicated and “proven” to be effective. In other cases, programs were deemed “promising” because they were consistent with the principles of effectiveness but did not meet the necessary standards to become an evidence-based program. The ultimate questions administrators and funders asked were “what works,” “how well does it work,” “do we have qualified personnel who can administer the program” and “how much does it cost”? Greenwood (2008) placed answers to these questions into two categories. One way to determine program effectiveness used a “generic” approach that included reviewing a number of generalized strategies and methods that had been tested by various investigators in different settings. Meta-analyses produced findings based on the results of hundreds of other research studies of individual homegrown programs that pointed consistently in the same direction, which allowed one to summarize “what works” and “how well” it did so from the efforts of many independent researchers. The work of Mark Lipsey (2009, 2010) and his collaborators is well known and often cited, as juvenile justice systems take this approach to evaluating and selecting interventions. Whether developing prosocial skills is accomplished by strictly using and maintaining fidelity to an evidence-based program or by simply applying elements of evidence-based programs in the work with clients, Lipsey et al. (2010) indicated that as long as intervention activities were informed by evidence-based principles and applied effectively and consistently, it could lead to successful client outcomes, even if a name-brand intervention is not exclusively used. This claim rested on Lipsey’s meta-analysis research that captured the evidence from many other research studies and offered elements of what worked (Lipsey et al., 2010). This appears to be a flexible alternative for community juvenile justice systems needing to integrate research into programs and services currently utilized. Conversely, it was argued that this method of synthesizing research into practice was insufficient, and the only way to gauge true change or successful outcomes
was to rely on specific adherence to a model program evaluated by a randomized or quasi-experimental design. As a result, Greenwood’s (2008) second category to answer the question of “what works” consisted of implementing “brand name” programs developed by a single investigator or team, rigorously tested over the years through replications, and typically funded by large grants to meet criteria established by various groups that identified evidence-based or proven programs. Essentially, a program is determined to be evidence-based if (a) evaluation research shows that the program produced the expected positive results; (b) the results can be attributed to the program itself, rather than to other extraneous factors or events; (c) the evaluation is peer-reviewed by experts in the field; and (d) the program is “endorsed” by a federal agency or respected research organization and included in their list of effective programs (Cooney, Huser, Small, & O’Conner, 2007). The Center for the Study and Prevention of Violence (CSPV) at the University of Colorado, Boulder with support from the Office of Juvenile Justice Delinquency Prevention (OJJDP) created the Blueprints for Violence Prevention initiative in 1996 to identify effective youth violence prevention programs across the United States. CSVP and other similar entities advocated using only rigorously evaluated and endorsed evidence-based programs to maintain measurable outcomes. However, many agencies simply could not afford program licenses, materials and manuals or sustain agency infrastructure with high staff turnover or improper training to maintain program fidelity. Practitioners were known to use homegrown programs and/or deconstruct model programs by picking or choosing program elements to suit client needs. Over time a shift occurred in how to use research to inform practice in juvenile justice. Although the gold standard remained implementing name-brand model programs that had the backing of successful outcome research, many agencies continued to rely on homegrown programs supported by practice wisdom and adapted to client needs, so some researchers looked for ways to integrate this type of information into an evidence-based framework. Ideal characteristics of evidence-based interventions have been described along with a framework for how juvenile justice systems can approach program selection. The next section describes the political and environmental context surrounding the DJJ and reviews initiatives Florida currently has in place to integrate EBP within its juvenile justice system. 24
Political and Environmental Context of This Study Florida was one of four states selected in 2011 to participate in a national initiative to reform the juvenile justice system by translating "what works" into everyday practice and policy. The Florida DJJ was chosen for the Juvenile Justice System Improvement Project (JJSIP) administered by Georgetown University's Center for Juvenile Justice Reform. The JJSIP provided a framework for implementing best practices throughout the entire juvenile justice system and intended to: • Produce an evaluation tool to identify any shortcomings in juvenile programs or services. •
Evaluate how closely those programs or services aligned with the most prominent research in the field.
Make concrete recommendations for improvement.
The DJJ invested significant resources, time and personnel to ensure the success of this partnership. Individual provider agencies along with services and programs operated by the DJJ were eventually all under review using the evaluation tool developed within the course of the project. The evaluation process began in Pinellas County, as this was the pilot setting where programs and services used were compared with findings from meta-analyses conducted with research published from the last few decades. Currently used programs were scored according to how closely they matched with qualities or components deemed effective by the meta-analyses. The goal of participation in the project was to generate concrete recommendations for how the DJJ could improve the current system. Because this partnership was an opportunity for Florida to emerge as a national leader in juvenile justice reform, there was an expectation by the DJJ that community agency providers would support this partnership by participating fully in the evaluation process. This information was relevant to this study because there is a natural tension between the DJJ, as the funder, and the community agencies contracted by the DJJ as providers of services to youth involved in the juvenile justice system. Politics and alliances within the DJJ and between community providers were part of the context of this study. To some extent, the DJJ has the power to dictate the types of evidence-based interventions used by agencies through the way it chooses to award contracts that fund programs. Those agencies using EBPs that are familiar to the DJJ and that also conduct program evaluations that determine successful outcomes typically rank higher during the contract bidding process. 25
Federal and state policies also factor in the adoption of evidence-based programs. Policies can arise from political values and be aligned with political agendas, which change frequently but have a significant impact on agency funding and service delivery. Political leaders who make funding decisions often lack understanding of the needs of specific populations and they have been found to be unaware of the research that supports evidence-based programs (Henggeler & Sohoenwald, 2011). Participants in this study were likely to be aware of many of these political issues stemming from the different reasons why the DJJ may support EBP and why agencies implement them. Given that this study explored practitioner attitudes to EBP and their perceptions of barriers to implementing EBP, these issues are highlighted in this section to provide contextual information. Previous sections of this study summarized widely accepted standards regarding evidence-based interventions used in the juvenile justice setting. Background information about the Florida DJJ’s partnership with Georgetown University to evaluate the effectiveness of its current programs and interventions was also discussed. Finally, environmental and political forces that affect the population under study were described. The next sections of this literature review focus specifically on research related to the variables considered in this study. Practitioner Attitudes toward EBP Understanding effective dissemination and implementation of EBP in general has been studied in a variety of ways, but if the most efficacious and effective interventions are to be disseminated and implemented in community-based settings, a better understanding of the attitudes of individual providers is needed (Aarons, 2004). In his seminal study, Aarons (2004) identified four dimensions of attitudes to the adoption of EBP: •
Intuitive appeal of EBP
Likelihood of adopting EBP given requirements to do so
Openness to new practices
Perceived divergence of usual practice with research- based/academically developed interventions.
Aarons (2004) developed the Evidence-Based Practice Attitude Scale (EBPAS) for use with providers in community mental health settings because, while much research has examined service 26
provider attitudes regarding organizational change, little research has been conducted on individual attitudes. This instrument has been used in other studies and has remained valid and reliable over time (see Aarons, 2006a, 2006b; Aarons, Cafri, Lugo, & Swatzky, 2010; Aarons, McDonald, Sheehan, & Walrath-Greene, 2007; Aarons & Palinkas, 2007; Stahmer & Aarons, 2009). Aarons (2004) surveyed 322 clinicians from 51 programs in the public sector. He conducted an exploratory factor analyses (EFA) with 50% of the sample and a confirmatory factor analysis with the other 50%. He found that a four-factor solution supported the four hypothesized dimensions of attitudes toward adoption of EBPs. Fifteen out of 18 items on the instrument were retained and the EFA model accounted for 63% of the variance in the data. Chronbach’s alpha measuring internal consistency was .77 overall, but ranged from .50 to .90 among individual items. Only the divergent scale had low internal consistency reliability (< .60), but this subscale was retained given the importance of the construct shown in other studies. Participants in this study were direct service providers and managers (one-third of the sample was comprised of social workers) and approximately 90% responded to the survey. With the factor analysis completed to find the four dimensions, regression analysis was then conducted to examine different independent variables with each of the dimensions. Aarons (2004) discovered that several variables influenced the attitudes of clinicians toward EBP, among which are clinicians’ level of educational attainment, years of experience, and the organizational context in which the clinician is employed. Aarons (2004) found that clinicians with a higher educational status were positively associated with having a favorable attitude toward adopting EBP (β= .06, SE β= .042, p< .05). Interestingly, intern status also indicated more positive attitudes to adopting EBP (β= .169, SE β= .098, p< .05) and more extensive clinical experience was associated with less favorable attitudes. Additionally, organizational factors such as higher levels of perceived agency bureaucracy and policies increase negative attitudes toward adopting EBP. A hallmark of Aarons’s (2004) study is its highly successful response rate (96%). Research funding made it possible to employ a research assistant who remained on-site while participants completed surveys to answer questions and ensure that correctly completed surveys were submitted. While this may be a common practice with projects of this size, this level of research support is not always 27
available to non-academic researchers or program administrators who do not have access to university research grants or the capability to write grants for federal/state funding. This is not necessarily a critique of the study; it is simply important to note that one must bear in mind that similar response rates may not occur if replicating this study requires the survey to be administered via mail or website without the personalized attention of an on-site research assistant. This exact phenomenon occurred in 2007 when Aarons attempted to replicate his original work with a more geographically diverse sample, since his primary sample comprised of mental health workers in only one California County. With the second study, Aarons sampled service providers from agencies affiliated with communities funded by the federal Comprehensive Community Mental Health Services for Children and Their Families Program in 17 states. However, rather than obtaining results that contradicted the original results, Aarons discovered that the new factor structure supported the original hypothesized dimensions and internal consistency of the EBPAS increased, thus increasing reliability of this scale. Predictably, the main critique of this study is the much lower response rate (40%) The data collection for the study was web-based, and used Dillman’s (2000) multi-stage emailing process, consisting of a pre-survey email, followed by a survey invitation the next week that included the web link, a username and password. A reminder email was sent the following week to the full sample and finally another reminder email was sent the week after to the non-respondents. The last step of the process was making phone calls to the final non-responders a month after the previous reminder email had been sent. Given that Aarons obtained a 41% response rate for his 2007 study, he states that this rate is higher than other published response rates for web-based surveys, which he claimed was the only feasible method to reach such a geographically diverse sample. Prior to beginning his 2004 survey, Aarons took time to conduct a brief pre-survey with clinicians in supervisory roles to gauge their level of familiarity with the term “EBP.” This pre-survey revealed a lack of familiarity with the term among the supervisory clinicians, and even after being provided with a brief description of EBP, these supervisory clinicians continued to show limited understanding of the concept, showing a mean familiarity rating of 1.4 (SD = 1.39).
Aarons then assumed that other clinicians under the supervision of the supervisory clinicians would also have limited familiarity with the term. JJSPs who would be potential respondents for this study were assumed to also have limited familiarity with the tern EBP. Therefore, steps to define the term and examples were provided within questions on the survey instrument and in the cover letter to address this issue. Based on findings from Aarons’s 2004 study, Aarons and Swatzky (2006) investigated organizational culture as it related to attitudes to EBP. This study further reiterated that individual level variables such as educational attainment and level of professional development are important and need to be controlled for in order to understand organizational culture. For example, the difference in attitude toward EBP between interns and experienced clinicians among the 301 mental health providers in 49 programs sampled in this study again found that interns had more positive attitudes to adopting EBP than experienced clinicians. In this same study, a negative correlation was also discovered between job tenure and willingness to adopt EBPs. This suggested that people who were newer to their profession are more open to adopting EBPs. Aarons and Sawitzky (2006) cite Rogers’s (1995) fourth edition of Diffusion of Innovations, in which he asserts that more formal education is associated with favorable attitudes to change and increased adoption of innovation. While this may initially seem to contradict Aarons’s findings indicating that interns are more receptive to EBP; it must be clarified that training for most clinical professionals involves an internship at the master’s level. In addition, many mental health professions classify ‘registered interns’ as master’s prepared professionals in the midst of earning requisite clinical supervision hours before sitting for a licensure exam. While these studies included do not specify the exact qualification status of interns, it is common practice in many states, including Florida, to use the designation ‘registered intern,’ and it was defined as such for this study. In another study, Stahmer and Aarons (2009) used the EBPAS among a sample of Autism Early Intervention (EI) providers and Children’s Mental Health providers and hypothesized that clinicians who had positive attitudes to innovations would promote effective dissemination and implementation of the most efficacious and effective interventions. They investigated the effect of individual factors such as educational attainment and clinical experience on the adoption of EBPs among 309 mental health 29
professionals working with children. One relevant result of their study showed that years of experience were negatively associated with willingness to adopt EBPs, indicating that younger clinicians were more open to adopting EBPs. As noted in the studies described above, practitioner attitude toward EBP has been shown to vary according to educational level and experience. Aarons’s (2004) work demonstrates that those who have more education but less experience tend to have the most favorable attitudes to using EBPs. These outcomes are also seen in literature published across other disciplines such as psychiatry, psychology and other behavioral health care subfields. Thus, level of education attainment and years of professional experience were key variables in this study. Each of the studies reviewed thus far uses Aarons’s (2004) Evidence-Based Practice Attitude Scale (EBPAS) with samples of clinical professionals in different mental health or educational settings. Aarons’s work has critically informed this study and in this research I intend to further his work by surveying JJSPs with questions adapted from the EBPAS. Results of similar surveys of this population have not been published, and this is a primary distinction between this study and others that have been completed in the past few years. Outside of mental health and education fields, practitioner attitudes toward EBP and implementation barriers are widely studied. Jette, Bacon, Batty, Carlson, Ferland, and Hemingway (2003) examined attitudes among physical therapists by surveying 488 members of the American Physical Therapist Association to determine their beliefs, attitudes, knowledge and behaviors regarding EBP. The authors developed a self-report questionnaire patterned after one created by McColl, Smith, White, and Field (1998), who surveyed British general practitioners on their perceptions of evidence-based medicine. Jette et al. (2003) conducted logistic regression analyses to examine the following univariate associations: responses to items measuring attitudes and beliefs; interest and motivation; education, knowledge, and skills; and access and availability of evidence with items measuring age, years since licensure, education level, and whether a respondent was a clinical instructor. One level of the independent variable served as a reference against which the odds of the other levels occurring were determined with the confidence interval set at 95%.
Based on self-report, 90% of respondents were in agreement that it is necessary to use evidence in practice and that the quality of patient care is better when grounded in research evidence. Most of the demographic factors were not associated with attitudes and beliefs, except amount of formal training. Familiarity with and confidence in search strategies, use of databases, and critical appraisal of literature tended to be associated with younger physical therapists with fewer years since licensure. Those respondents with fewer than 5 years since licensure tended to have more knowledge of EBP terminology than those with more than 15 years since licensure. Those with fewer than 5 years of experience were also 2.1 times more likely to understand the term “meta-analysis” and 4.2 times more likely to understand the term “confidence interval” than those with more years of experience (Jette et al., 2003). However, physical therapists with a baccalaureate degree or certificate as their first professional or highest degree were less likely to understand the terms than those with a post-baccalaureate professional degree or an advanced master’s or doctorate degree as their highest degree. This is also true in that bachelor-level physical therapists were less likely to have received training on EBPs and they were reportedly less confident in their EBP skills than those with a post-baccalaureate professional degree or advanced master’s or doctorate degree as their highest degree. Thus, Jette et al. (2003) also concluded that education and skills related to EBP were mutually exclusively associated with age, years since licensure, and advanced academic degrees. Similarly, Iles & Davison (2006) surveyed 124 Australian physiotherapists working in private and public hospital settings to investigate self-reported practice, skills and knowledge of EBP. They found that physiotherapists licensed 5 years or fewer were more supportive of adopting EBP and were more confident in their EBP skills. Recent graduates (post-1998) rated their EBP skills more highly than experienced graduates (pre-1998), but did not perform EBP tasks more often. Although 69.4% of respondents claimed to read research literature at least once a month, only 10.6%-26.6% used a database search, and of those who used a database search, only 35.8% reported critically appraising the research reports. Iles and Davidson (2006) concluded that respondents had a positive attitude toward EBP, but the main barriers to using EBP skills regularly were due to the time required to keep up to date with literature,
access to easily understandable summaries of evidence, journal access and lack of personal skills in searching and evaluating research evidence. Both Jette et al. (2003) and Iles & Davidson (2006) described the evidence-based process with an assumption that American and British physical therapists should abide by this process or philosophy in their practice. Both studies outlined issues related to teaching and training the EBP process in undergraduate and graduate programs and the barriers to understanding and using this process in their daily work. However, these studies did not consider a simple alternative in which the training and education of current graduate physical therapists may be focused on interventions deemed to be evidence-based rather than on the process itself. Therefore, the individual physical therapist does not necessarily learn to search for and evaluate evidence in response to a question about a single patient. Rather, the physical therapist understands how to use an evidence-based intervention(s) and/or technique(s) to treat their patient based on a certain symptom or cluster of symptoms that he or she has learned in school or while on the job rather than to approach their patient assessment with an evidencebased process in mind (Hoge et al., 2003). Attitudes toward the use of EBP among physicians and nurses have been well studied. Given the consistency in the number of years it typically takes to complete school, residency and additional training to become a physician, education level or years of education are not typically selected as variables related to attitudes. The nursing field follows a similar pattern. Nevertheless, general attitude has been correlated with the type of practice setting, type of methods physicians and nurses use to learn, seek, and apply skills of evidence-based medicine in their practice along with the frequency with which these skills are applied (Hutchinson & Johnson, 2006; McColl et al., 1998; Retsas, 2000). Knowledge of EBP and barriers to using EBP, as well as perceptions of barriers to implementing EBP in practice, are more commonly evaluated and will be discussed in the following sections. Beasley & Woolley (2002) assessed the attitudes of faculty members in medical schools. A more extensive research background was positively correlated with a positive attitude toward EBP, whereas number of years since residency correlated negatively with EBP attitudes. Additionally, those further removed from their residency were less likely to incorporate EBP in their teachings. Weissman and Sanderson (2002) examined the amount of inclusion of EBPs in graduate training programs in helping 32
professions and found that clinicians who had been formally trained 10 years or more before the study was conducted were unlikely to be familiar with EBPs. This indicates that more experienced faculty would be less likely to disseminate EBPs in their teachings. Interestingly, a study by Nelson and Steele (2007) examining the use of EBP among mental health practitioners found that significant predictors of self-reported use of EBP included whether or not the practitioner took an EBP class while in school, the culture of the practitioner’s clinical setting, and the practitioner’s identified theoretical orientation. Practitioners’ academic degrees had no influence and years of experience was not significantly correlated with use of EBP. However, this study did not intend to measure attitude toward EBP use, but was rather a measure of self-reported use of EBP. Hoge, Tondora, and Stuart (2003) reported that students who received training in EBP during their formal education establish and maintain fidelity to EBPs more consistently than students who either did not receive training or received post-graduate training in EBPs. However, based on reviews of literature from the fields of mental health, social work, physical therapy and nursing, the EBP process and philosophy remains at the core of research and these studies emphasized what is supposed to happen and efforts to discover why it does not happen. Conversely, in criminology and criminal justice literature, the focus is on model fidelity and barriers to implementing an EB intervention correctly with little to no emphasis of the EBP process at all. Nevertheless, while studies on practitioner attitude generally report positive attitudes toward EBP, this does not necessarily result in actual use of EBP in practice. This suggested that favorable views of EBP do not equal actual implementation of EBP (McColl et al., 1998; Sanderson, 2002). As a movement toward even more integration of EBP into different fields occurs, the attitudes of practitioners in applied settings will have an even larger impact on the success of implementation efforts. Understanding practitioner attitudes toward EBP and their perceived challenges to implementing EBPs will be crucial to the successful integration of EBPs into clinical settings (Nelson, Steele & Mize, 2006). This study explored attitudes among practitioners in the juvenile justice setting as this is another field where there is a strong emphasis on and motivation for the use of evidence-based practices and interventions, but this particular population has not been surveyed before nor has there been an exploration of their attitudes to EBP. 33
As the Florida DJJ faces tough decisions about how to allocate funds with budgets reduced by state legislators, community juvenile justice provider agencies must continue to provide services with less funding from the DJJ. Using EBPs to track program success and efficacy holds the DJJ and community agencies accountable to the legislature and taxpayers. With the increased interest in using EBP and requirements by the government or agency policy to do so, the attitudes of juvenile justice providers toward EBP are an important factor to take into account as agencies look for ways to successfully implement EBP. Practitioner Perceptions of Barriers toward EBP Literature on research utilization tends to focus on three areas: 1) institutional support (organizational dynamics), 2) availability of information (technology), and 3) attitude to research (Champion & Leach, 1989). These three variables have been identified as having an impact on the ability of practitioners to adopt EBP (Fishbein & Ajzen, 1975; Horsley et al., 1978; Miller & Messenger, 1978). While practitioner attitude certainly is a barrier to adoption of EBP, this section will focus on barriers to research utilization in terms of institutional support and availability. NASW (2003) identified lack of time and access to research as barriers to using research in daily practice among social workers after conducting a study of 22 oncology social workers employed in several different clinical settings. Only three out of the 22 social workers indicated that they were currently involved with oncology research. Participants stated that the major barriers were lack of time and difficulty of access to sources of research materials. Participants also perceived their institutions and organizations not to support research because they did not see it as relevant to their clinical social work roles. Social workers in this study also reported a lack of practice guidelines and a lack of knowledge regarding research methods as additional barriers (Holosko & Leslie, 1998; Levy, 2010; Schoenwald & Hoagwood, 2001; Weissman & Sanderson, 2002). McGuire (2006) attempted to identify barriers and attitudes about EBP among masters level social workers working in clinical settings in Texas. He surveyed 1,728 licensed clinical social workers and found that, while most of the participants (78%) had heard of EBP and the majority had access to the internet at work to review professional literature, lack of time was still the predominant barrier to adopting EBP. 34
Bennett et al. (2003) also identified lack of time and lack of knowledge of EBP as barriers to implementation among occupational therapists. Retsas (2000) surveyed 400 nurses in Australia and reported that the primary barriers to using research evidence in nursing practice were lack of accessibility of research findings in the workplace, lack of anticipated outcomes for using research, lack of support from colleagues to use research, and, most importantly, lack of organizational support to use research. Upton and Upton (2005) created a widely adapted instrument for nurses based on their knowledge that clinical effectiveness is best achieved through EBP. With the understanding that barriers to EBP included workforce attitudes, lack of time, and appropriate skills, the instrument intended to be a valid and reliable self-report measure that considered three aspects of EBP: day-to-day application, individual attitudes, and relevant skills. These studies represented a largely consistent theme across different professions regarding barriers to using EBP. Lack of time, lack of knowledge regarding concepts of EBP, lack of accessibility of research findings, lack of individual motivation, and lack of institutional support are repeatedly reported. Given the acknowledged importance of evidence-based practices within juvenile justice settings, further studies to increase understanding of JJSPs’ perceptions of barriers to their use of EBP are warranted. Practitioner Knowledge of EBP The rationale for conducting a study on JJSPs’ attitudes toward EBP and perceptions of barriers has been discussed in previous sections. Lack of knowledge has often been indicated to be a perceived barrier to using EBP, but even less is known about JJSPs’ actual level of knowledge regarding EBP. Aarons (2004) identified mental health professionals’ lack of familiarity with the term EBP; does the same hold true among Florida JJSPs? Two forms of knowledge were of interest in this study: first, the knowledge and understanding of EBP concepts, and second, knowledge and understanding of how to apply EBP when working with clients. Studies in mental health contexts have included one of doctoral level licensed psychologists’ attitudes regarding use of treatment manuals in traditional clinical settings (Addis & Krasnow, 2000; Addis, Wade, & Hatgis, 1999; Morrow-Bradley & Elliott, 1986; Prochaska & Norcross, 1983), but within community mental health agencies, most providers do not have doctoral level training (Aarons,
Woodbridge, & Carmazzi, 2003). This was an important piece of information to consider, as the JJSPs included in this study did not have doctoral level degrees or licensure. Pope, Rollins, Chaumba, and Risler (2011) found that, similarly, few researchers have evaluated the actual knowledge, skills and use of EBP within social work. They reported that literature from social work and several other related fields have mostly evaluated attitudes toward EBP, the ability to access and interpret the evidence, awareness of practice guidelines, and perceived barriers to implementation (Aarons, 2004; McColl et al., 1998; Mullen & Bacon, 2006; Vallino-Napoli & Reilly, 2004). However, Rubin and Parrish (2007) examined views of EBP among social work faculty and found that while 73% supported EBP, their definitions of it varied and a lack of consensus made it impossible to discern what percentage truly knew the correct definition. In spite of many studies indicating that social workers have supportive attitudes toward EBP, when Mullen and Bacon (2006) compared knowledge and use of EBP among social workers, psychiatrists and psychologists, they found that 42% of social workers reported having heard about EBP, compared to 94% of psychiatrists and 81% of psychologists in their sample. When asked about their knowledge of EBP guidelines, only 14 out of 81 social workers in their sample reported knowing a particular practice guideline that was evidence-based. Mullen and Bacon concluded that “social workers were poorly informed; typically not even aware of the meaning of practice guidelines and that social workers were generally not using research findings or research methods in their practice” (2006, p. 90). However, Mullen and Bacon (2006) only surveyed one large agency with 697 employees: 42 psychiatrists, 53 psychologists, 386 social workers and 19 other mental health professionals. Respondents included 17 psychiatrists, 16 psychologists, 81 social workers and 10 other mental health professionals. Given this response rate, Mullen and Bacon also concluded that social workers were somewhat less responsive than the other professions. The overall low response rate could not ensure a representative sample of the entire agency and generalizations from their study can thus only be made with extreme caution. Pope et al. (2011) conducted a study that further explored social work practitioners’ knowledge of EBP and use of EBP as represented by a mean score on the Social Work Evidence-Based Practice Scale
(SWEBPS) and compared this score with demographic and professional characteristics. The SWEBPS consists of questions such as: •
I go to the literature to answer questions regarding my clients.
I incorporate clinical judgment with social work research.
I use relevant research to answer my clinical questions.
I access EBP guidelines online.
I incorporate patient preferences with research findings.
An electronic questionnaire was emailed to over 1500 social workers using a convenience sample within the US and the United Kingdom, securing 201 respondents. One respondent was a student intern, so that questionnaire was not included, and a total of 200 questionnaires were completed. Pope and her team (2011) concluded that there was a weak positive relationship between social workers’ years of professional experience, on the one hand, and knowledge and use of EBP on the other. The correlation between years of practice and score on the SWEBPS was r (198) = .071, p = .323. For age and score, r (198) = .049, p = .493, showing that the relationships were not statistically significant. A scatterplot confirmed that there was not a linear relationship between either age or years of practice and score on the SWEBPS. Pope et al. (2011) used one-way Analysis of Variance (ANOVA) with the highest level of social work education obtained and divided the respondents into three groups (BSW, MSW and PhD). Based on statistical results, social workers’ knowledge and use of EBP did not differ according to their area of practice or social work education. Contrary to Mullen and Bacon’s (2006) study, the majority of social workers in this sample had mean scores on the SWEBPS that indicated moderate knowledge and use of EBP. Unfortunately, there are no comparison scores, so it is impossible to determine whether this sample exhibits a high or low average SWEBPS score. The studies mentioned above included a variety of questions and analysis procedures that informed the creation of the instrument for this study. In summary, the literature suggests that there are obstacles to the adaptation and implementation of EBP within social work. The primary issue of resistance among social workers is attributed to practitioners’ lack of time, lack of access to practice guidelines, and lack of knowledge of research methods (Holosko & Leslie, 1998; Schoenwald & Hoagwood, 2001; Weissman & Sanderson, 2001). 37
Additionally, a few researchers have evaluated actual knowledge and skills of EBP among social workers, and this corresponds to what is reported regarding EBP knowledge within the criminology and criminal justice literature. This study tested JJSPs on their understanding of EBP definitions, conceptions and principles. As these providers may have many different fields, including both criminology and social work, the study attempted to add to the existing literature by exploring how much actual knowledge juvenile justice providers in Florida have of EBP. Summary of Chapter This chapter presented Rogers’s (2003) Diffusion of Innovation Theory, Ajzen’s (1991) Theory of Planned Behavior, and Bandura’s (1977) Social Learning Theory to provide a theoretical structure to the study. This was followed by a brief historical account of the evolution of evidence-based medicine and how EBP came to be integrated into the field of social work. Differences between the social work philosophy of viewing EBP as a process and guiding principle rather than as a set of research-tested and -proven interventions, as is the case in the field of juvenile justice, were also described. Contextual information regarding current efforts by the Florida DJJ to establish an evidence-informed structure was included to highlight the utility of this research. Finally, relevant studies that have shed light on practitioner attitudes, perceptions of barriers to using EBP, and practitioner level of knowledge were reviewed. However, there is a dearth of research regarding providers within a juvenile justice setting and consequently little is known about the attitudes, perceptions of barriers and knowledge of EBP among JJSPs. This study contributed to filling this gap by surveying employees working in juvenile justice residential facilities, prevention programs and other community agencies that receive funding from the Florida DJJ to provide services for youth who have become involved in the juvenile justice system.
CHAPTER 3: METHODOLOGY Overview of the Study The overall purpose of this study was to evaluate Florida JJSPs attitudes to, and knowledge and perceptions of barriers toward using EBP in their daily work with clients. This study also described the differences between attitudes, knowledge and perceived barriers according to individual factors such as age, professional status, years of professional experience, degree type and degree level. Ethical Considerations I applied for review and approval of the study with the University of South Florida (USF) Institutional Review Board (IRB). USF IRB educational requirements were completed to ensure appropriate procedures were followed for obtaining consent, instrument administration and data collection. Approval from the DJJ’s IRB was not sought as no data were collected from the DJJ or any other state offices. Confidentiality was maintained by storing all research study documents, including completed surveys and additional demographic information, in a locked file in my office and any computer files with survey data were password protected. Research Design This study was a non-experimental research design, as control groups or comparison groups were not used and a cross-sectional design using a survey instrument was administered at one point in time. Data was obtained on the association of JJSPs’ attitudes to the adoption of EBPs with their level of educational attainment, years of professional experience, age, licensure type, primary discipline and type of agency setting. The sample was intended to adequately represent the population of JJSPs in Florida, but findings are not generalizable to practitioners outside of the state. Little research has been conducted regarding attitudes and perceptions of EBP among this population, and this study added to the social work knowledge base given that Florida is considered a leader in juvenile justice reform and there is interest in this state’s efforts to implement EBPs at both state and federal levels. 39
Population and Sample Description of participants. The target population sampled for this study consisted of community stakeholders in partnership with the DJJ. Many agencies across the state contract with the DJJ to provide a range of services for delinquent youth and these agencies pay dues to become members of the Florida Juvenile Justice Association (FJJA). The FJJA brings together juvenile justice system professionals and agencies to promote public awareness and education of juvenile justice issues. The FJJA also contributes to the development of public policy regarding juvenile justice issues and it supports evaluation and research. With 24 member agencies and 10 associate member agencies, the FJJA has become an informal lobbying entity to represent the interests of these agencies during the state’s legislative budgeting sessions. These member agencies provide prevention, civil citation, diversion, and residential services for youth throughout the state, and all receive funding from the Florida DJJ. However, in some cases, juvenile justice services are merely a portion of services and programs some agencies provide to adolescents and adults in general. Nevertheless, participants in this study are service providers working in agencies subscribing to a philosophy of EBP or that use evidence-based intervention(s) as defined in earlier sections of this dissertation. A thorough review of each FJJA member agency’s website and online job postings was conducted along with discussions with key informants familiar with staffing patterns of member agencies to determine that participants qualifying for this study have job titles such as case manager, clinical director, counselor (intake, community, transition services etc., licensed and non-licensed), site manager, mental health worker (licensed or non-licensed), program coordinator, program director, facility administrator, social worker (licensed or non-licensed), therapist (licensed or non-licensed), or wilderness instructor.
Sample Rationale for selected sample. The time and cost needed to generate a probability sample is difficult to overcome and randomization was not indicated given the research questions for this study. As this study has a cross-sectional design and intended to obtain data at one point in time to better understand the attitudes, knowledge and perceptions of barriers of JJSPs in community agencies across Florida, a nonprobability convenience sample was considered more suitable. While a nonprobability sample does not allow for calculations of sampling statistics that gauge the precision of the results, a convenience sample of JJSPs in FJJA member agencies is sufficient to answer the research questions. Based on phone calls and emails to residential facilities operated by FJJA member organizations to determine staffing patterns and job titles, it was estimated that there were approximately 500 JJSPs eligible to complete the survey. A power analysis determined the sample size needed to detect an effect of a given size with a degree of confidence. Cohen (1998) stated that five factors must be determined in order to perform a statistical power analysis: a) significance level, b) effect size, c) desired power, d) estimated variance, e) sample size. The significance level for most studies in the social work field is fixed 2
at .05. Cohen has standardized effect sizes into small, medium and large, and for regression analysis f = .02, .15 and .35 respectively. In this study, the highest level of analysis was multiple regressions to analyze ordinal-level response variables that compared the relationship of the independent and dependent variables. Using the traditional .05 criterion of statistical significance, it was anticipated that the largest number of variables in the regression analysis was thirteen. Using online calculators available through danielsoper.com, and entering the commonly accepted parameters of .15 for a medium effect size with 0.8 for desired statistical power level with thirteen predictors, and a probability level set at .05, this yielded a minimum sample size of 131 required to achieve adequate statistical power. Criteria for inclusion. Any person who provided direct services to youth receiving intervention or prevention services in a program or agency funded by the DJJ and who either personally used an evidence-based intervention or worked within an agency that subscribes to an EBP philosophy was eligible to participate in this study. There were no specifications related to gender, age, years of employment, level of education, educational background/degree type, licensure or agency type, as these 41
were all variables of interest. Moreover, the list of position titles indicated above is not exhaustive as different agencies have different titles for similar positions. Delimitations. Agencies have additional positions that do not qualify as full or part time positions, but are rather contract positions that do not require the same level of training. Therefore, these contract employees were not eligible for this study. Likewise, many agencies have positions allocated to provide other direct care activities such as nutrition services, physical therapy, and medical services. Physicians, nurses, medicine technicians, teachers and transportation providers are examples of such, and were not included. While these professionals may certainly be aware of evidence-based interventions and probably participate in delivering them, they fell outside of the sampling frame for this study. Measures The survey instrument was researcher-constructed and comprised of questions from Aarons’s Evidence Based Practice Attitude Scale (2004), Jette’s Evidence Based Questionnaire (2003) and Parrish’s Test Questionnaire of EBP Knowledge (2008). A single validated and published instrument measuring these components could not be located, and this survey instrument was therefore created. While this compilation had not been tested as a stand-alone instrument, the psychometric properties of each separate instrument are discussed in greater detail below. The adaptation of these scales and questionnaires for this study was based on a comprehensive review of the literature and published studies in the fields of social work, mental health, nursing and physical therapy. This researcher measured the following components: •
Attitudes of JJSP toward EBP
JJSPs’ level of perceived barriers to using EBP
JJSPs’ Knowledge of EBP concepts
Demographic composition of sample
The validity and reliability of the three original instruments is reported in the following sections. Attitude toward EBP. Aarons’s (2004) Evidence-Based Practice Attitude Scale (EBPAS) had 15items designed to measure mental health service providers’ attitudes about adopting new or different therapies or interventions. The EBPAS contained response categories ranging from 0 (not at all) to 4 (to a very great extent) (Aarons, 2004; Rice, Hwang, Abrefa-Gyn, & Powell, 2010) and identified four domains 42
of attitude that Aarons found to be important in understanding the process of adopting EBPs. These domains were intuitive Appeal, attitudes toward organizational Requirements, Openness to innovation, and perceived Divergence of research-based innovation. Aarons (2004) argued that these domains “represent measurably distinct aspects of attitudes toward adoption” (p. 63). For example, general openness to innovation was likely to be due more to one’s attitudinal disposition than being contingent on requirements of the workplace. Put another way, it was expected that an instrument should be sensitive enough to measure a respondent’s attitudinal openness to EBP differently from the respondent’s intuitively positive perception of an EBP (Appeal). Furthermore, it was likely that perceived divergence of current practice with EBPs would be inversely associated with more favorable attitudes, such as Openness and Appeal. Aarons’s instrument was designed to identify not only the domains but the associations between them as well. Initially the EBPAS was tested among 322 public-sector clinical service workers from 51 programs and approximately one-third of the sample comprised of social workers. Exploratory factor analysis and confirmatory factor analysis revealed a four-factor solution determining the four attitude domains with subscale Chronbach’s alphas as follows: intuitive Appeal of EBP (alpha = .80); likelihood of adopting EBP given Requirements to do so (alpha = .90); Openness to new practices (alpha = .78); and perceived Divergence of usual practice with research-based/academically developed intervention (alpha = .59) with an overall scale reliability of .77 (Aarons, 2004). Subsequent studies using the EBPAS among 303 public-sector mental health clinicians and case managers, including 99 social workers from 49 programs, further tested the reliability of this scale (overall alpha of .77 with subscale alphas ranging from .59 to .90) (Aarons, 2004, 2006). Aarons et al. (2007) conducted another study of 207 mental health service providers, including 99 social workers again, and found an overall alpha level of .79 for the scale with subscale alphas ranging from .66 to .93 (2007). Based on findings from the pilot study described below, items 1-8 on this survey instrument also gauged JJSP attitudes toward EBP but the questions were modified from Aarons’s EBPAS (2004) using fewer clinical terms that would not be clearly understood by those without a social work or mental health background, Items 19-23 reflected attitudes toward EBP being required as part of JJSPs’ daily work
responsibilities. The answers to these Likert questions were rated on an ordinal scale so that 1 = strongly disagree; 2 = disagree; 3 = agree and 4 = strongly agree; 0 = don’t know. Given that this ordinal scale ranged from 0-4, the following attitudinal questions assessed whether JJSPs agreed or disagreed that using EBP is necessary in their daily work, whether they believed that using literature and research evidence is useful in their daily work, whether they were willing to use new and different types of EBP or interventions, whether they believed their own professional/clinical experience is more important than using EBP or interventions, and, finally, their willingness to try a new EBP or intervention even if it is very different from their current practice. Items 1-8 were summed to represent a general attitude score by adding the total number of points according to the scale. Two exceptions were questions 3 and 5, which were reverse coded and scored as such. The highest possible number of points for this section was 32, with 1-8 points representing a strongly negative attitude to EBP, 9-16 representing a negative attitude, 17-24 representing a positive attitude and 25-32 representing a strongly positive attitude to EBP. An option to select 0 for “don’t know” as it related to attitude toward EBP was included on the survey to counteract potential skipped or missing answers. Giving respondents an option to answer honestly rather than to skip the question entirely made it possible to still garner useful information that could not otherwise be assumed with a skipped question. By using a mean replacement strategy (Saunders, Morrow-Howell, Spitznagel, Dore, Proctor, & Pescarino, 2006), each respondent’s total score was calculated, and a mean score for each individual was derived and then substituted for his/her “don’t know” responses. Calculating the individual mean and substituting that score rather than using a series or overall mean was a more accurate calculation for each respondent. Out of 7800 data points, only 109 “don’t knows” were substituted and only two respondents out of the total 195 actually skipped questions, which caused their surveys to be excluded from this portion of the analysis as missing data. Given that the midpoint of this scale was 16.5, it was possible to infer that if a respondent had a score of below 16.5, then that respondent had a negative attitude to adopting EBP. If the score was above 16.5, then that respondent had a positive attitude to adopting EBP.
Perceptions of barriers toward using EBP. Jette et al. (2003) adapted their questionnaire from one developed by McColl et al. (1998). The original questionnaire was used to study general practitioners’ perceptions of EBP in the Wessex region of south England. Jette and colleagues further refined the original questionnaire for use in surveying physical therapists in the United Kingdom. Content validity of this instrument was evaluated after a draft of the questionnaire was sent to 10 experienced physical therapists practicing in pediatrics (n = 1), acute care (n = 1), orthopedics (n = 2), and rehabilitation (n = 3). Slight modifications were made on the basis of this feedback, and the final instrument was a self-reported questionnaire designed to explore attitudes and beliefs about EBP; interest and motivation to engage in EBP; educational background, knowledge and skills related to accessing and interpreting information; level of attention to and use of the literature; access and availability of information to promote EBP; and perceived barriers to using evidence in practice. The questionnaire used a five-point Likert scale that ranged from “strongly disagree” to “strongly agree.” Fifty-four survey respondents completed the questionnaire twice, on occasions between 2 weeks and 2 months apart, to assess the reliability of the items. Intraclass correlation coefficients (ICC) were determined for the ordinal items and percentages of agreement were determined for categorical and rank items. The ICCs ranged from .37 to .90, with 50% of the items having ICCs of >70. Dichotomous items had 68%-93% agreement and rank items were also used to inform selection of questions for this study (Jette et al., 2003). Items 9-17 on this survey instrument were adapted from the Evidence-Based Questionnaire (Jette et al., 2003). These questions were modified to capture perceptions of barriers to the application of the EBP concept (i.e. JJSPs using literature and research findings in their daily work, rather than simply adhering to an evidence-based intervention or whether lack of research skills or statistical analyses skills were barriers). Answers to these Likert ranged from 1 = strongly disagree; 2 = disagree; 3 = agree and 4 = strongly agree; 0 = don’t know/not sure. Items 9-17 were summed to represent a general perception of barriers score by adding the total number of points according to the scale. The maximum number of points for this section was 36 with 1-9 points representing a strong disagreement of barriers to using EBP, 10-18 representing disagreement of
barriers to using EBP and 19-27 representing agreement of barriers to using EBP, and 28-37 representing a strong agreement of barriers to using EBP. Again, the option of selecting 0 for “don’t know” as it related to whether one perceived something was a barrier to using EBP was included on the survey to counteract potential skipped or missing answers. The “don’t know” answer was usable information and it was not necessary to consider it as missing data. By using a mean replacement strategy again (Saunders, et al., 2006), each respondent’s total score was calculated and a mean score for each individual was derived and then substituted for his/her “don’t know” responses. Calculating the individual mean and substituting that score rather than using a series or overall mean was a more accurate calculation for each respondent. Out of 8,775 data points, only 67 “don’t knows” were substituted and only two respondents out of the total 195 actually skipped questions in which their surveys were excluded from this portion of the analysis as missing data. Given that the midpoint of this scale was 18, it was possible to infer if a respondent had a score of below 18, that respondent disagreed that there were barriers to adopting EBP. If the score was above 18, then that respondent agreed that there were barriers to adopting EBP. There is one qualitative question in this survey instrument that 128 participants answered. Content analysis was used to verify coherence of answers to this open-ended question within the quantitative survey (Hogan & Desantis, 1992). A coding scheme was developed and tested according to guidelines recommended by Weber (1985). A total of 339 statements were analyzed, with the unit of analysis being each of the responses (reasons) to the question, “Please list the top three (3) reasons you think are barriers to implementing EBP(s) in your current work with clients.” Sample responses included: “Lack of time,” “Lack of training,” “Lack of support from agency,” “Individual differences across client populations do not mesh with some EBP” and “Most don’t apply to the clients we have.” I methodically reviewed all responses, which were entered into a Microsoft Excel spreadsheet. Subsequently, an open coding scheme was developed based on themes in the data and specific code categories were determined. To ensure reliability, I initially coded all data and the same text was recoded by a second person. This technique is referred to as “intercoder reliability” (Krippendorf, 2013, p. 271; Weber, 1985, p. 17). The second analyst was a recent graduate with a master’s in social work who interned in an FJJA affiliated agency two semesters ago and has been familiar with this study from its 46
inception. As recommended by Krippendorf (2013), I trained the second coder on analysis procedures. She was given a written list of codes and their definitions and was instructed to assign one of the 12 codes to all units of analysis. Following the second coding, I compared her coding assignment with that of the second coder and it revealed a 94.6% agreement between coders. This indicated acceptable intercoder reliability. Although there was already high agreement between coders on the 12 categories, another code category was discovered after further discussion and data analysis, and 13 categories are reported in the following chapter. Level of EBP knowledge. The next section of the survey instrument assessed a respondent’s knowledge of EBP with six multiple-choice questions and four true/false questions, each with one correct answer. The aim behind using test questions is to capture the baseline range of JJSP knowledge of EBP concepts and processes without relying on JJSPs’ self-reported knowledge. Using the same test questions Parrish (2008) created for her dissertation, respondents’ scores fell into categories of low, moderate or high, in order to determine their level of knowledge regarding the concepts and processes of EBP. According to Parrish (2008), these questions broadly captured the essential concepts of EBP that were consistently used to define EBP across numerous professional fields. One of the questions focused on the general philosophy of the EBP model, four others were related to appraisal of research evidence (one of which overlapped with the philosophy of EBP), one on the hierarchy of evidence for EBP effectiveness, one on posing an EBP question and one on searching for evidence to answer an EBP question. Parrish wrote these questions to be used in a pretest/posttest design within her dissertation research and her hypothesis indicated that the introduction of a daylong EBP training workshop (independent variable) would improve the posttest scores immediately after the workshop and then again at 3 months after the workshop (dependent variable). Parrish tested these questions among field instructors at the University of Texas at Austin and used their feedback to improve clarity and conciseness of the instrument. She reported internal
consistency reliability coefficient alphas of .92-.94 overall for her instrument and has subsequently published several additional studies testing her scale (Rubin & Parrish, 2007, 2010, 2011, 2012). Because the definition and use of EBP varies between the social work and criminal justice professions, the inclusion of these test questions was intended simply to provide a sense of what JJSPs’ knowledge base of EBP concepts are, as this had not been studied before among this population. The number of correct answers participants identified were summed to calculate scores for this portion of the instrument. The highest potential score for this portion is 10 (one point for each correct answer). Low level of knowledge ranged from 0-3, moderate level of knowledge ranged from 4-6, and high level of knowledge ranged from 7-10. Pilot study. A portion of the survey instrument was pilot tested to determine feasibility of administering the instrument, to learn more about why potential participants may or may not want to take part in the survey, whether the questions on the instrument were clearly worded, formatted and easily understood, and if the instrument was of an acceptable length. The pilot study was conducted with 12 key informants. Seven women and two men employed as JJSPs in a community provider agency funded by DJJ participated as this approximately represented the ratio of women to men among the 68 agency staff members. Two academic professionals also reviewed the survey instrument and provided feedback regarding order of questions, formatting and minor editing issues. These individuals range in age, level of educational attainment, years of professional experience, and degree type. Their feedback indicated that the original questions on the EBPAS were confusing because clinical terms used were unfamiliar to three of the individuals. These individuals had several years of professional experience, but none had a master’s degree in a counseling or social work field and they did not understand what was meant by “manualized therapies.” Therefore, the wording was modified with less clinical terminology but without changing the substantive meaning. As for feedback received for items 9-17 regarding barriers to implementing EBP, five participants expressed negative feelings about using EBP, but these were general statements regarding the overall topic of this study, not specifically critiquing the proposed survey questions. Examples include, “I don’t understand the importance of EBP,” “I do not get to choose what EBP to use, most programmatic decisions are made by agency administrators and I just do what I am told,” and “Real life is not taken into 48
account by administrators when EBP are mandated and does not allow me to help a client in the way that I know works.” These statements indicated that the cover letter and instructions must specify the purpose of this study and clearly define what EBP are within the scope of this study. However, feedback from the pilot study participants revealed that the actual questions were easily understood and reflected current themes participants shared with me regarding their perceptions of barriers in their daily work. Similarly, four participants commented that they did not think they scored well on the “test” questions used to gauge JJSPs’ knowledge of EBP. This feedback was important as it specified that further testing of JJSP knowledge was useful and may provide a baseline of how much knowledge this population has of EBP concepts. Instrumentation Reliability Summary. The scales used in this study demonstrated acceptable reliability coefficients when administered to the sample. Table 3.1 displays the basic statistics and internal consistency reliability coefficients for all instruments. Table 3.1. Instrument Statistics and Internal Consistency Reliability (Chronbach’s Alpha) Scale M SD α Attitudes 26.21 4.69 .67 Barriers 20.47 5.21 .83 Knowledge 25.61 4.61 .65 Demographic questions. Nine demographic questions were included to learn about the characteristics of the sampled population. They were not included in the pilot study as these questions are routinely asked within social work research. Age. Research has shown that younger professionals are more likely to be open to EBP. This variable was defined by asking respondents to select their current age category, defined in 10-year bands. Gender. This variable described the sample and determined selectivity bias. This variable was defined by asking respondents to state whether they are male or female. Years of experience. Research has shown that people who are newer in their professional role may be more adaptable and learn new practice methods more quickly (Aarons, 2004). Respondents were asked to indicate how many years it had been since they had obtained their highest degree.
Level of educational attainment. Aarons (2004) reported that those who have received more education have a more favorable view of the EBP process. Respondents were asked to select their highest completed degree from high school, associate’s degree, bachelor’s degree, master’s degree, doctoral degree (Ph.D., Psy.D., Ed.D. or M.D.). Discipline/major. This variable is also used to describe the sample and determine selectivity bias between groups. Respondents were asked to describe their discipline/major using the following responses: social work, criminology/criminal justice, psychology, sociology, rehab counseling, education or other. Job title. This variable not only described the sample, but helped ensure respondent eligibility to participate. Respondents were asked to select the job title that most closely matched their own. Licensure. This is another variable used to describe the sample and ensure respondent eligibility to participate. Respondents were asked to select which practice license(s) he/she held from a list. Agency type. Aarons (2004) found differences in practitioner attitudes depending on their practice setting or agency type. Thus, respondents were asked to indicate their current employment agency type: residential, detention, day treatment, prevention, juvenile assessment center and other. Data Collection Procedures Given the qualities of the sample population, a mixed mode survey comprised of the same version of a mailed instrument and a web-based instrument was used for this study. There were five contact points to increase response rates: a pre-notice email, the questionnaire mailing with incentive, a thank you postcard, a replacement questionnaire with the web-based survey option, and a final contact with both the paper survey and information for the web-based version included (Dillman et al., 1974; Herberlein & Baumgartner, 1978). The study followed Dillman’s Tailored Design Method (2009), which included three fundamental considerations: a) Reducing four sources of survey error that undermined the quality of information collected-coverage, sampling, non-response, and measurement; b) Developing a set of survey procedures that engaged and encouraged people in the sample to respond to the survey; c) Developing survey procedures that build positive social exchange and encouraged response by taking into
consideration elements such as survey sponsorship, the nature of and variations within the survey population, and the content of survey questions. Dillman’s basic assumption stemmed from a social exchange perspective of human behavior and posited that respondents would be more willing to participate and answer more accurately when they believe their completion of the self-administered questionnaire would accrue rewards that outweigh the anticipated costs of responding (2009). The Tailored Design Method (2007) stressed the importance of creating questions and questionnaires that are well designed and carefully implemented with procedures that result in a successful survey response rate. In 2007, Dillman conducted a study with a random sample of households in a small metropolitan region of Idaho in which participants were asked to complete a questionnaire with 81 items. This questionnaire was sent to half of the sample via regular mail along with the usual reminders. The other half of the sample received a postal letter inviting them to participate in a web-based version of the questionnaire. Dillman stated the postal mail version of the questionnaire yielded a response rate of 71% whereas the web-based questionnaire yielded a response rate of 55%. With each initial version of the questionnaire, respondents were given the other option at the second reminder contact. For example, the postal recipients were given the option to take the web-based version and the web-based participants were told a paper version would be mailed in 2 weeks if they did not have access to the internet. Of those who initially received paper surveys, 41% elected to complete the web version but only 1% of initial web responders elected to use the paper format. Dillman, Smyth, Christian, and O’Neill (2008) also found web-based surveys to have a 55% response rate. Additionally, Dillman et al. (2009) offered several guidelines to strengthen response rates. First, personalize all contact attempts to the greatest extent possible, even when individual names are not available. I accomplished this by using official university letterhead on heavier paper and blue ink signatures for all correspondence. Next, Dillman et al. (2009) recommended sending a token of appreciation with the questionnaire mailing as that brought social exchange into play and encouraged respondents to reciprocate by completing the questionnaire. A novel or unexpected gesture was thought to bring additional attention to the request so respondents may be more inclined to read and consider the questionnaire rather than just 51
throwing it away. “Token financial incentives included with original survey request(s) have been shown to be significantly more effective than much larger payments promised to respondents after they complete their questionnaires “(Dillman et al., 2009, p. 241). This method effectively changed the process from an economic exchange in which people expected to be paid for completing a questionnaire. Unfortunately, if a person believed the price was too low, was not worth their effort or if a person was simply not interested, then they did not feel a social obligation to respond since it was a socially acceptable choice to refrain from responding. However, if $1 lottery tickets were included with the first questionnaire mailing as a goodwill gesture then this was no longer seen as an economic exchange but rather a social exchange. It was then expected that response rates would increase for this study (Bailey et al., 2007; Church, 1993; Gendall & Healey, 2007). In following Dillman’s Tailored Design Method (2007), a pre-notice email was sent on September 16, 2013, a few days prior to the survey period on my behalf by the FJJA executive committee chair to each of the chief executive officers, chief operating officers, and other top administrators or leaders of FJJA member agencies. This served as a letter of introduction and support of this research study. It included an attachment with a letter from me providing a brief description of the study and a request for assistance from the agency administrator to, within 7 days, provide the number of employees in their agency who would qualify for the study according to job title and to identify a contact person within his/her agency if the administrator did not want to remain the contact person for the duration of the study. This contact person maintained contact with me and helped distribute questionnaires among the agency’s eligible staff. He or she also ensured that each member of his/her organization completed the survey only once. During the second contact phase, from September 25 through November 6, 2013, I mailed packages to 17 agencies containing the correct number of questionnaire packets to each contact person with instructions to distribute each packet to each eligible employee. The packets contained copies of a detailed cover letter providing a description of the study explaining its confidential nature, copies of a description of the risks and benefits of participation, instructions for completing the questionnaires, the questionnaires, and contact information for me and the Institutional Review Board. Prepaid postage envelopes and token incentives (lottery tickets) were also part of this package. 52
Within 1 week of the second contact, depending on when the original packet was mailed out, an email with a thank you letter attached was sent to each agency administrator and/or contact person that was forwarded to all participants. This letter expressed appreciation to those who had already responded and was a reminder for those who had not completed the questionnaire to do so and return their completed survey. Two weeks after the second contact, I sent another email to the administrator/contact person asking that he or she to forward an attachment that offered another reminder to complete the questionnaire. In this attachment, a hyperlink to the web-based version of the questionnaire was provided for the first time. However, extra paper questionnaire packets were prepared in case they were needed, but none of the administrators/contact persons requested additional packets. Given that a relationship was established between myself and the contact person within each participating agency, the contact person was asked to assist with distributing the surveys only. There was no need to collect or return the surveys as the respondent placed each paper survey instrument in a stamped envelope with no other identifying information. Survey information remained confidential because respondents returned their completed surveys directly back to me via mail and the agency contact person was not able to access the completed surveys. Finally, the fifth contact occurred 2 weeks after the fourth contact and was the last email I sent to the administrator/contact person to forward to their agency’s participants. This final contact indicated that the study concluded on December 1, 2013, and I thanked each participant again and reiterated the importance of their contribution to the study with the last request for their prompt completion and return of the questionnaire. As this was a self-funded study, I budgeted $500 for token incentives, office supplies, paper, envelopes and copying the survey instrument. Postage costs for mailing 500 surveys with stamped return envelopes was budgeted for another $1,000. Actual expenses were as follows: •
Lottery tickets: $263.00
Postage costs: $76.74
Return envelope stamps: $386.61
Manila Envelopes: $39.28 53
Return Envelopes: $5.84
Office supplies: $39.55
Grand Total: $1,143.52
Data Analysis and Hypotheses The Statistical Package for the Social Sciences (SPSS) was used for data analysis. Initial frequency distributions, measures of central tendency and dispersion of all variables were obtained. A rejection level of p < .05 was set prior to assessment of relationships between variables as this is the conventional level in most social science research (Patel, 2010). Prior to conducting statistical analyses, SPSS data were crosschecked with each hard copy of returned survey instruments to ensure accuracy. Then the data were screened and assumptions of each parametric test were verified (Mertler & Vannatta, 2005). Assumptions for Pearson’s Product-Moment Correlation required all variables be continuous and to have linearity, normality, homoscedasticity, and only minimal outliers. Assumptions for multiple regression included normality, linearity, and homoscedasticity (Mertler & Vannatta, 2005; Tabachnick & Fidell, 2007). Additionally, prior to conducting the multiple regression analysis, a power analysis was obtained to ensure the sample size was adequate. A sample of 195 respondents with thirteen predictor 2
variables would yield a level of power of .73 for a cumulative R = .20 while a sample of 50 participants 2
with three predictor variables would yield a level of power of .83 for a cumulative R = .20. The first step in the data analysis process was to obtain frequency distributions and measures of central tendency and variability of all variables. Box plots and frequency distributions were used to assess for outliers and missing data. The skewness and kurtosis of each variable was analyzed to assess the univariate normality of the distribution. According to Mertler and Vannatta (2005), skewness and kurtosis values closest to 0 are ideal but can be acceptable between -1 and +1. Inspecting the bivariate scatterplots assessed for linearity and homoscedasticity and an oval or elliptical shaped scatterplot indicated that the variables were linearly related (Mertler & Vannatta, 2005;
Tabachnick & Fidell, 2007). A scatterplot in which the variables were close to the same width indicated that the variables were homoscedastic (Tabachnick & Fidell, 2007). In addition, Levene’s Test was used to examine homogeneity of variances (Mertler & Vannatta, 2005). All assumptions for the analyses were met. Tolerance statistics were also obtained prior to the multiple regression analysis to assess for multicollinearity, which would cause statistical problems when correlations between predictor variables are .90 and above (Tabachnick & Fidell, 2007). Bivariate correlations between the dependent variables (attitude score, barrier score and knowledge score) and independent variables (age, gender, license type, educational attainment, years of professional experience, major studied and agency type) were conducted. P-values were obtained to determine whether the relationships between attitudes, knowledge and perceptions of barriers and the covariates were statistically significant (Rubin & Babbie, 2011; Mertler & Vannatta, 2005). In short, the analysis involved assessing numerous descriptive statistics and eventually proceeded to a multivariate analysis in order to move beyond simple bivariate relationships. Analysis 1: Measuring attitude. Research Question 1: Do JJSPs’ attitudes toward EBP differ by individual factors? Hypothesis 1a: As JJSPs’ ages increase, they will score lower on the Attitude Subscale. Independent variable: JJSP age Dependent variable: Attitude score A Pearson’s bivariate correlation with a one-tailed level of significance test (p < .05) was used to determine whether there is a negative association between JJSP age and attitude score. Hypothesis 1b: JJSPs’ attitude score will vary based on level of educational attainment. Independent variable: Degree type Dependent variable: Attitude score A one-way ANOVA was used to assess whether there was a significant difference in attitudes toward EBP and JJSPs’ degree type. Hypothesis 1c: As JJSPs’ years of experience increase, they will score lower on the attitude subscale. Independent variable: Years of professional experience 55
Dependent variable: Attitude score A Pearson’s bivariate correlation with a one-tailed level of significance test (p < .05) was used to determine whether there is a negative association between years of professional experience and attitude score. Research Question 1d: Do males and females have significantly different attitude scores? An independent-sample t-test with a two-tailed level of significance test was used to measure difference in-group means. Research Question 1e: Do JJSPs with licensure and those without licensure have significantly different attitude scores? An independent-sample t-test with a two-tailed level of significance test was used to measure difference in-group means. Hypothesis 1f: There are significant differences in attitude scores among JJSPs that majored in social work, criminology/criminal justice and psychology/mental health counseling fields. Independent variable: Major of study Dependent variable: Attitude score A one-way ANOVA was used to assess whether there are significant differences among majors studied. Hypothesis 1g: There are significant differences in attitude scores among JJSPs between those who work in residential facilities, day treatment programs and prevention programs. Independent Variable: Employment setting Dependent Variable: Attitude score A one-way ANOVA was used to assess whether there are significant differences among employment setting. Beyond descriptive statistics, inferential statistical analysis can be conducted given that Aarons (2004) established that the first eight questions of his scale are closely related and measure attitude. Hence, a simple additive index from the eight questions was created. For instance, Respondent 1 registers a score for each of the eight questions and the total points will be tallied. Therefore, the range of scores on the index was from 1 to 32 for each of the respondents. Any “don’t know” responses were 56
replaced with the individual’s mean score, which was derived from the average of their other item scores. There were only two missing cases, which were omitted from the analysis. Frequencies were run on the additive index. Based on the frequencies, the index was recoded to include the mean replacement and this became the dependent variable. After the dependent variable was recoded, key relationships between variables such as age, educational attainment, degree type, years of practice and license type were assessed with a multivariate analysis. Hypothesis 1h: JJSP attitude score can be correctly predicted from age, degree type, years of professional experience, gender, licensure, major studied and employment setting. Independent variables: Age, degree type, years of professional experience, gender, licensure, major studied and employment setting Dependent variable: Attitude score Because the dependent variable constructed is an attitude index on an interval scale, it was appropriate to use a multiple linear regression model to determine if average item scores for Attitude could be predicted from individual factors. In this model, the attitude index variable was regressed on the following covariates: age, degree type, years of professional experience, gender, licensure, major studied and employment setting. Again, the data were screened for missingness and violations of assumptions prior to analysis and missing data were addressed as described above. The major studied variable was dummy coded and all majors were compared against social work. In addition a set of n-1 employment setting dummies were included in the model. The dummies included were: Residential, Detention, Day Treatment, Prevention and JAC. The omitted category that these variables were compared against was Redirections as this agency setting is known to exclusively use evidence-based interventions for all of their programming and was expected to differ from other agency settings. Analysis 2: Measuring knowledge. Research Question 2: How knowledgeable are JJSPs regarding the concepts and skills of EBP Raw scores were calculated to determine how many out of 10 knowledge questions each respondent answered correctly: 0-3 correct = low level of knowledge; 4-6 = moderate level of knowledge; 7-10 correct = high level of knowledge. 57
Research Question 3: Is there a significant difference between JJSPs’ levels of knowledge regarding EBP based on individual factors? Hypothesis 3a: As JJSPs’ age increase, they will score lower on the knowledge subscale. Independent variable: JJSPs’ age Dependent variable: Knowledge score A Pearson’s bivariate correlation with a one-tailed level of significance test (p < .05) was used. Hypothesis 3b: JJSPs’ knowledge scores will vary based on level of educational attainment. Independent variable: Degree type Dependent variable: Knowledge score A one-way ANOVA was used to assess whether there was a significant difference in level of EBP knowledge and JJSPs’ degree type. Hypothesis 3c: As JJSPs’ years of experience increase, they will score lower on the knowledge subscale. Independent variable: Years of professional experience Dependent variable: Knowledge score A Pearson’s bivariate correlation with a one-tailed level of significance test (p < .05) was be used. Research Question 3d: Do males and females have significantly different knowledge scores? An independent-sample t-test to measure group differences was used to determine if there were significant differences between men and women on knowledge scores. Research Question 3e: Do JJSPs with licensure and those without licensure have significantly different knowledge scores? An independent-sample t-test to measure group differences was used to determine if there were significant differences between those with or without licensure on knowledge scores. Hypothesis 3f: Are there significant differences in knowledge scores among JJSPs that majored in social work, criminology/criminal justice and psychology/mental health counseling fields? Independent variable: Major studied Dependent variable: Knowledge score
An ANOVA was used to assess whether there was a significant difference in knowledge toward EBP among social workers, psychology/mental health counselors, and criminologists. Hypothesis 3g: Are there significant differences in knowledge scores among JJSPs that work in residential facilities, day treatment programs and prevention programs? Independent variable: Employment setting Dependent variable: Knowledge score Another one-way ANOVA was used to assess whether there was a significant difference in knowledge of EBP concepts among JJSPs who work in residential facilities, day treatment programs and community prevention programs. Hypothesis 3h: Can JJSP knowledge score be correctly predicted by age, degree type, years of professional experience, gender, licensure, major studied and employment setting? Independent variables: Age, degree type, years of professional experience, gender, licensure, major studied and employment setting Dependent variable: Knowledge score Since the raw knowledge scores could be added to create an interval scaled knowledge index as the dependent variable, it was appropriate to use a multiple linear regression model again to determine if average item scores for Knowledge could be predicted from individual factors. In this model, the knowledge index variable is regressed on the following covariates: age, degree type, years of professional experience, gender, licensure, major studied and employment setting. Analysis 3: Measuring barriers. Research Question 4: Do JJSPs’ perceptions of barriers toward EBP differ by individual factors? Hypothesis 4a: As JJSPs’ ages increase, they will score lower on the barrier subscale. Independent variable: JJSP’s age Dependent variable: Barrier score A Pearson’s bivariate correlation with a one-tailed level of significance test (p < .05) was used. Hypothesis 4b: JJSPs’ barrier score will vary based on level of educational attainment Independent variable: Degree type Dependent variable: Barrier score 59
A one-way ANOVA was used to assess whether there was a significant difference in barrier score and JJSPs’ degree type. Hypothesis 4c: As JJSPs’ years of experience increase, they will score lower on the barrier subscale. Independent variable: Years of professional experience Dependent variable: Barrier score A Pearson’s bivariate correlation with a one-tailed level of significance test (p < .05) was used. Research Question 4d: Do males and females have significantly different barrier scores? An independent-sample t-test was to measure group differences was used to determine if there were significant differences between men and women on barrier scores. Research Question 4e: Do JJSPs with licensure and those without licensure have significantly different barrier scores? An independent-sample t-test to measure group differences was used to determine if there were significant differences between those with or without licensure on barrier scores. Hypothesis 4f: Are there significant differences in barrier scores among JJSPs that majored in Social Work, Criminology/Criminal Justice and Psychology/Mental Health Counseling fields? Independent variable: Major studied Dependent variable: Barrier score A univariate ANOVA was used to assess whether there was a significant difference in perceptions of barriers toward using Evidence Based Practices among Social Workers, Psychology/Mental Health Counselors, and Criminologists. Hypothesis 4g: Are there significant differences in barrier scores among JJSPs that work in Residential Facilities, Day Treatment Programs and Prevention Programs? Independent variable: Employment setting Dependent variable: Barrier score Another univariate ANOVA was used to assess whether there was a significant difference in perceptions of barriers toward using Evidence Based Practices among JJSPs who work in Residential facilities, Day Treatment programs and community Prevention programs. 60
The Barrier score is calculated in the same way as the Attitude score. Items 9-17 on the survey instrument were closely related and measured perceptions of barriers so another additive index from the 9 questions was created. As each respondent registered a score for each of the 9 questions, the total points were tallied. The range of scores on the index was from 1 to 36 for each of the respondents and “Don’t know” responses were replaced with the individual’s mean barrier score, which was derived from the average of their other barrier item scores. The same two missing cases were omitted from the analysis. Frequencies were run on the additive index again and recoded to include the mean replacement to become the dependent variable. After the dependent variable was recoded, relationships between variables such as age, gender, licensure, advanced degree, major studied and years of professional experience, were assessed with multivariate analyses. Hypothesis 4h: JJSP barrier score can be correctly predicted from age, degree type, years of professional experience, gender, licensure, major studied and employment setting. Independent variables: age, degree type, years of professional experience, gender, licensure, major studied and employment setting Dependent variable: Barrier score Because the barrier index as the dependent variable is on an interval scale, it was again appropriate to use a multiple linear regression model to determine if average item scores for barriers scores could be predicted from individual factors. In this model, the barrier index variable was regressed on covariates: age, degree type, years of professional experience, gender, licensure, major studied and employment setting. Research Question 5: What barrier is perceived by JJSPs as the most significant/severe barrier toward using Evidence-Based Practices? Descriptive analysis of responses of the Barrier Subscale questions was completed to determine which barrier was the modal category. Qualitative analysis of Question 18 on the survey instrument asking respondents to list their top 3 reasons to why barriers exist in implementing EBP in their current work with clients was also completed.
Methodological Limitations Potential limitations of this study included threats to internal validity with a cross-sectional design as the confidence in concluding that a relationship exists between these independent and dependent variables can be challenged by issues such as selection bias and design contamination. For example, participants may be more inclined to participate in a study about EBP based on strong negative or positive opinions regarding EBP. Or, given the 4-week timeframe in which survey data was collected, the chance of one participant indicating to another what is actually contained within the questionnaire could have dissuaded the other from participating because they believed the survey was too difficult, too time consuming or due to some other factor (Rubin & Babbie, 2011). Other possible limitations of the study may be due to using self-reported data from participants. Issues of selective memory, selective attribution of positive and negative events or outcomes, and exaggeration by JJSPs can potentially occur when self-reporting attitudes and perceptions of barriers to using EBP (Rubin & Babbie, 2011). However, one effort to account for this was by using test questions to measure knowledge of EBP rather than relying on a self-report of proficiency of this variable. Additionally, time and resources limit a longitudinal study tracking different effects among participants for a longer period of time or over the course of repeated surveys. Summary of Chapter A quantitative survey research design with an overall sample of 195 participants was obtained. The primary instrument utilized in this study was adapted from a questionnaire developed by Aarons (2004) for use with community mental health providers to measure attitudes toward EBP. An instrument created by Jette et al. (2003) based on the work of McColl et al. (1998) informed and shaped the questions designed for use in this study to capture JJSP perceptions of barriers to EBP and the actual type of evidence-based interventions JJSPs identify as being used in their agency. Other questions used in this study were based on Parrish’s (2008) dissertation work that gauged JJSPs’ knowledge of EBP concepts and skills. With regard to the empirical analysis, the three substantive areas of the survey (attitudes, perceptions of barriers, knowledge) were evaluated using standard descriptive measures. Multivariate analysis was conducted in those instances where the relative impact of certain factors on components 62
such as attitudes and barriers of implementation of EBP is assessed. This quantitative assessment utilizing a survey provided a better understanding of the attitudes, primary barriers to EBP and knowledge of EBP concepts of service professionals working in the Florida Juvenile Justice System.
CHAPTER 4: RESULTS Missing Data/Don’t Know Answers Missing data from skipped questions is a common concern with survey research. To guard against potential skipped or missing answers by participants were offered the option of selecting 0 for “Don’t Know” as a response to survey questions about their attitude toward EBP and perceptions of barriers toward using EBP. In order to capture more specific information since it can be argued that an answer of “Don’t know” represents more information than a skipped answer containing no information at all. To this end, every “Don’t know” option was substituted with a score by using a mean replacement strategy (Saunders, et al., 2006; Mertler & Vannatta, 2005; Tabachnick & Fidell, 2007; Salloum, 2008). Each respondent’s total score was calculated for each scale, and a mean score for each individual was derived and then substituted for his/her “Don’t know” responses. Calculating the individual mean and substituting that score rather than using a series or overall mean was a more accurate calculation for each respondent (Salloum, 2008, Saunders, et al., 2006). Out of a possible 7,800 data points for the Attitude subscale, 109 (.01%) “Don’t knows” were substituted. Out of 8,775 data points for the Barriers subscale, 67 (less than .01%) “Don’t knows” were substituted. Only two respondents out of 195 skipped 12 or more questions; meaning these participants skipped at least 50% of the total number of questions for these two sections on the instrument. This is within an acceptable range for missing data (0-4.5%; Tabachnick & Fidell, 2007) Therefore, their surveys were excluded from the Attitudes and Barriers portions of the analysis and categorized as missing data (.01%). Descriptive Analysis Demographics. People employed by member agencies of the Florida Juvenile Justice Association comprised the convenience sample (n=195). Eight out of 24 member agencies participated, including the two largest residential facilities with the greatest number of employees in the state. A total of 64
263 paper survey instruments were mailed and 170 were returned for a response rate of 65%. An electronic survey link was emailed to all 24 member agencies and yielded an additional 29 completed survey instruments; but, a calculation of the electronic survey response rate is impossible to determine as it is unknown how many total people received the email survey link. However, with a total of 199 returns, adequate statistical power was achieved with this sample size. Table 2.1 shown at the end of this chapter provides demographic information about participants. Of respondents, 132 (67.7%) were female and 56 (28.7%) were male. Seven participants did not answer the gender question. Respondents’ ages ranged from 18 to over 70 years old with ten (3.1% of respondents) between 18-24; thirty-two (5.1% of respondents) between 25-29; Seventy-eight (40% of respondents) were aged 30-39; Forty-one (21% of respondents) were aged 40-49; eighteen (9.2%) were aged 50-59; nine (4.6% of respondents) were aged 60-69 and one (.5% of respondents) was 70 years or older. Six participants did not answer this question. One hundred and twenty four respondents (63.3%) did not hold a practice license. Of the 30.3% that held a professional license one had an LCSW (.5%); 17 had LMHCs (8.7%); two LPCs (1%); one had an LMFT (.5%); four had CAPs (2.1%), 20 were LMHC registered interns (10.3%) and 14 were LCSW registered interns (7.2%). Twelve participants did not answer this question. Years of professional experience/number of years since obtaining highest degree ranged from 1 year to 41 years with 2 years being the modal category representing 9.7% of respondents. Twenty-two participants did not answer this question. Regarding normality of this variable, the standardized skewness coefficient (skewness divided by standard error of skewness) and the standardized kurtosis coefficient (kurtosis divided by standard error of kurtosis) showed this variable was highly kurtotic, positively skewed and not normally distributed. The highest level of education for participants varied. Thirteen (3.6%) participants had a high school diploma; 12 (6.2%) earned an Associates Degree; 72 (36.9%) earned a Bachelors Degree; 89 (45.6%) earned a Masters Degree, and one respondent (.5%) obtained a doctorate degree. Eight participants did not answer this question. The variable looking at different college majors was unique as this researcher was particularly interested in differences, if any, of attitudes, level of knowledge, and perceptions of barriers among 65
participants with different majors. Thirty respondents (15.4%) were social work majors; 33 (16.9%) were criminology/ criminal justice majors; 35 (17.9%) were psychology majors; 2 (1%) were rehab counseling majors; 14 (7.2%) were sociology majors; 6 (3.1%) were education majors; 15 (7.7%) were business/management majors, and 25 (12.8%) respondents had other majors not listed above. Eighteen (9.2%) participants did not respond to this question. Type of employment setting was another variable included to see what relationship and/or differences exist with JJSP attitudes, knowledge and perception of barriers. Ninety-nine (46.2%) participants worked in a residential facility. The high number of respondents employed by just one setting was not surprising given that the two largest agencies in the state included in the sampling frame are residential facilities. Thirteen (7.1%) respondents worked in a detention facility; 15 (8.2%) were employed in Day Treatments; 28 (14.4%) worked in prevention programs; 14 (7.6%) worked in Redirections programs; 10 (5.4%) were employed by Juvenile Assessment Centers; and 5 (2.6%) worked in other settings. Eleven (5.6%) participants did not respond to this question. The majority of respondents, 167 (86.1%), worked during day shift hours between 8am and 5pm but twelve (6.2%) respondents worked the night shift between 3pm and 11pm, and three (1.5%) respondents worked during the graveyard shift hours between 11pm and 8am. Twelve (6.2%) participants did not answer this question. Table 2.2 at the end of the chapter describes the percentage of participants’ answers toward each of the attitude subscale items with the largest percent of strong agreement being with the statement “using EBP is necessary in my daily work” at 47.2%. The strongest disagreement was with the statement “my professional/clinical experience is more important than using EBP/interventions. Two additional attitude items were asked in a different format as Aarons (2004) specified it was important to gauge how easily understood an EBP was as this related to one’s overall attitude toward EBP. In this case, 99.5% of respondents agreed or strongly agreed that any EBP training they received made sense and was intuitively appealing to them. Table 2.3 shows the frequency distribution of the barrier questions with the majority of respondents disagreeing or strongly disagreeing that the list of potential barriers were actually barriers for themselves. The only exceptions were that 63.3% agreed or strongly agreed that insufficient time was a 66
barrier toward using EBP in their daily work with clients and 49.2% agreed or strongly agreed that lack of informational resources was a barrier for them. Participant responses on Aarons (2004) Evidence Based Practice Attitude Scale indicated being required to use EBP is a factor that also affects overall attitude scores. Thus, these questions were included in the adapted survey instrument. It is clear in Table 2.4 that the majority of respondents were required to use EBP among agencies funded by the Florida Department of Juvenile Justice and this may have an influence on attitudes and perceptions of barriers with implications discussed in the following chapter. Table 2.5 shows the frequency distribution of how many respondents answered each knowledge question correctly, with the correct answer to the question in bold. Each of the knowledge questions were broad and expected to cover basic EBP concepts taught across all disciplines or majors. With 82.1% of respondents answering Question 10 correctly, this question was answered correctly most frequently. Only 17.9% answered Question 1 correctly, making this the least frequent correctly answered question. A knowledge index (Table 2.7) created to assess overall knowledge scores revealed 38 (19.5%) respondents answered 1 to 3 questions correctly placing them at a low level of EBP knowledge. One hundred and six (54.4%) respondents answered 4 to 6 questions correctly, placing them at an average level of EBP knowledge and 48 (24.6%) respondents answered 7 to 10 questions correctly, placing them at a high level of EBP knowledge. Quantitative Results Research Question 1: Do JJSPs attitudes toward EBP differ by individual factors? H1a: As JJSPs age increase, they will score lower on the Attitude Subscale It was hypothesized that attitude scores would be negatively associated with age. Specifically, older JJSPs would be associated with lower attitude scores. However, results of the bivariate correlation indicated there was no statistically significant association between JJSP age and attitude score, r (193) = .01, p = .87. H1b: JJSPs attitude score will vary based on level of educational attainment A one-way Analysis of Variance (ANOVA) was used to assess whether there was a significant difference in attitudes toward Evidence Based Practices and degree type of JJSPs. Results indicated 67
there was not a statistically significant difference in attitude scores by degree type, F (4, 181) = .59, p = .67. In other words, attitudes towards Evidence Based Practices do not statistically differ by those with Associates, Bachelors or Masters degrees. H1c: As JJSPs years of experience increase, they will score lower on the Attitude Subscale It was hypothesized that attitude scores would be negatively associated with years of experience. Results of the bivariate correlations showed no statistically significant association between years of experience and attitude score, r (193) = -.02, p = .77. RQ1d: Do males and females have significantly different attitude scores? An independent sample t-test was used to determine if there were significant differences between men and women on attitude toward EBP scores. Results indicate that there was not a statistically significant difference in mean attitude scores according to gender, t (186) = -.24, p = .82). RQ1e: Do JJSPs with licensure and those without licensure have significantly different attitude scores? An independent sample t-test was used to investigate if there were significant differences between those with or without licensure on attitude scores. There was not a statistically significant difference in mean attitude scores by licensure status, t (193) = -1.27, p = .21). H1f: Are there significant differences in attitude scores among JJSPs that majored in Social Work, Criminology/Criminal Justice and Psychology/Mental Health Counseling fields? A one-way Analysis of Variance (ANOVA) was used to assess whether there was a significant difference in attitudes toward Evidence Based Practices by different majors (Social Work, Psychology/Mental Health Counseling, Criminology/Criminal Justice). Results indicated that there was not a statistically significant difference in attitude scores by profession, F (3, 191) = .99, p = .40. In other words, the results indicated that attitudes towards Evidence Based Practices do not statistically differ by these three different professional backgrounds. H1g: Are there significant differences in attitude scores among JJSPs that work in different employment settings (Residential Facilities, Detention, Day Treatment Programs, Prevention Programs and Juvenile Assessment Center)? To assess whether there was a significant difference in attitudes toward Evidence Based Practices among JJSPs who work in Residential facilities, Day Treatment programs and community 68
Prevention programs, the researchers used a one-way ANOVA. Results showed no statistically significant difference between type of facility and attitude toward EBPs, F (5,178) = .33, p = .90. As with the previous analysis, the null finding suggested attitudes among JJSPs employed in different types of agency settings are essentially the same. H1h: Can JJSP attitude score be correctly predicted by individual factors (age, gender, licensure, degree, major) and type of employment setting (Residential Facility, Detention Center, Day Treatment, Prevention program, Redirections program, and Juvenile Assessment Center (JAC))? A hierarchical linear regression was conducted to see if results would differ by demographics and by place of employment (see Table 2.6). The model was not statistically significant F(8, 155) = .592, p < .05 and explained only .03% of variance of attitude scores. In the first step, the Attitude Index score was regressed on age, gender, licensure, degree type and major studied with only gender and type of major (Social Work) significantly related to attitude (β = -.15, SD = .06, p<.01, β =.14, SD =.07, p<.01). In the second step, type of employment setting variables (Residential, Detention, Day Treatment, Prevention, Juvenile Assessment Center) were added to the model. The introduction of employment setting variables explained an additional .03% variance among attitude scores after controlling for individual factors. As shown in Table 2.6, none of the additional covariates (employment settings) in the second model had a statistically significant effect on JJSP Attitude. The total model was not a significant predictor of attitude, F (8, 155) = .592, p>.01) and given it explained so little overall variance, the two significant results could still be due to chance and was regarded cautiously. Research Question 2: How knowledgeable are JJSPs regarding the concepts of EBP? Raw scores were calculated to determine how many out of 10 knowledge questions each respondent answered correctly, and an index was created with 0-3 (0 to 30 percent) correct = Low level of knowledge; 4-6 (40-60 percent) correct = Moderate level of knowledge; 7-10 (70 to 100 percent) correct = High level of knowledge. Table 2.7 presents a distribution of scores by level of knowledge. Based on collected data, JJSPs employed among agencies funded by the Florida department of Juvenile Justice appear to have a moderate to high level of understanding of core concepts of EBP that are universal across different disciplines.
Research Question 3: Do JJSPs knowledge of EBP concepts differ by individual factors? H3a: As JJSPs age increase, they will score lower on the Knowledge Subscale It was hypothesized that knowledge scores, as measured by number of correct answers, would be negatively associated with age. Specifically, older JJSPs would be associated with lower knowledge scores. The results of the bivariate correlation indicated there was not a significant association between JJSP age and knowledge score, r (193) = .10, p = .19. H3b: JJSPs knowledge scores will vary based on level of educational attainment A one-way ANOVA was used and there was not a statistically significant difference between educational attainment/degree type and knowledge score, F (4, 183) = 1.44, p = .22. The null finding suggests knowledge scores among JJSPs with Associates, Bachelors or Masters degrees are essentially the same. H3c: As JJSPs years of experience increase, they will score lower on the Knowledge Subscale It was hypothesized that knowledge scores would be negatively associated with years of experience and again, results of the bivariate correlation showed no significant association between years of experience and knowledge score, r (193) = .01, p = .93. RQ3d: Do males and females have significantly different knowledge scores? An independent sample t-test was also used to determine if there were significant differences between men and women and knowledge scores. Results indicate that there was not a statically significant difference in knowledge scores according to gender, t (186) = -.99, p = .33. RQ3e: Do JJSPs with licensure and those without licensure have significantly different knowledge scores? Another independent sample t-test was used to determine if there were significant differences between those with licensure or not and knowledge scores. Results indicated there was not a statistically significant difference in knowledge scores according to licensure, t (193) = -1.18, p = .24. H3f: Are there significant differences in knowledge scores among JJSPs that majored in Social Work, Criminology/Criminal Justice and Psychology/Mental Health Counseling fields? A one-way Analysis of Variance (ANOVA) was used to assess whether there was a significant difference in knowledge toward Evidence Based Practices in the case of Social Workers, Psychology/Mental Health Counselors, and Criminologists. This model indicated there was not a 70
statistically significant difference, F (3, 191) = .72, p = .54. In other words, the three different professional backgrounds were essentially the same with respect to knowledge of Evidence Based Practice concepts. H3g: Are there significant differences in knowledge scores among JJSPs that work in different employment settings (Residential Facilities, Detention, Day Treatment Programs, Prevention Programs, Juvenile Assessment Center)? A one-way ANOVA was used to assess whether there was a significant difference in knowledge of Evidence Based Practice concepts among JJSPs who work in different employment settings. Results showed there was not a statistically significant difference between the groups, F (5, 178) = 1.81, p = .11, indicating level of knowledge among JJSPs employed in different types of agency settings are essentially the same. H3h: Can JJSP knowledge score be correctly predicted by individual factors (age, gender, licensure, degree, major) and type of employment setting (Residential Facility, Detention Center, Day Treatment, Prevention program, Redirections program, and Juvenile Assessment Center (JAC))? Another hierarchical linear regression (see Table 2.8) was performed to determine if results would differ by demographics and by place of employment. The model was not statistically significant F(8, 155) = .592, p < .05 and explained only .03% of variance among knowledge scores. In the first step, the Knowledge Index score was regressed on age, gender, licensure, degree type and major studied with no statistically significant predictors found. In the second step, type of employment setting variables (Residential, Detention, Day Treatment, Prevention, Juvenile Assessment Center) were added to the model. The addition of employment setting variables explained an additional .03% variance of knowledge scores after controlling for individual factors. As shown in Table 2.8, none of the covariates (employment settings) in the second model had a statistically significant effect on knowledge scores either. The total model was not a significant predictor of knowledge, F (8, 155) = .592, p>.01) given it explained so little overall variance. Research Question 4: Do JJSPs perceptions of barriers toward EBP differ by individual factors? H4a: As JJSPs age increase, they will score lower on the Barrier Subscale It was hypothesized that barrier scores would be negatively associated with age. Specifically, older JJSPs 71
would be associated with lower barrier scores meaning they would perceive fewer barriers to using EBP. However, results of the bivariate correlation indicated there was no significant association between JJSP age and barrier score, r (193) = -.04, p = .56. H4b: JJSPs barrier score will vary based on level of educational attainment A one-way Analysis of Variance (ANOVA) was used to assess whether there was a significant difference in perceptions of barriers toward using EBP and degree type of JJSPs. Results indicated there was not a statistically significant difference in barrier scores by degree type, F (4, 181) = .59, p = .67. In other words, the results indicated perceptions of barriers towards using EBP do not statistically differ by those with Associates, Bachelors or Masters degrees. H4c: As JJSPs years of experience increase, they will score lower on the Barrier Subscale It was hypothesized that barrier scores would be negatively associated with years of experience. But results of the bivariate correlation showed no significant association between years of experience and barrier score, r (193) = -.04, p = .55. RQ4d: Do males and females have significantly different barrier scores? An independent sample t-test was used to determine if there were significant differences between men and women on barrier toward EBP scores. Results indicate that there was not a statistically significant difference in mean attitude scores according to gender, t (184) = 1.82, p = .07). RQ4e: Do JJSPs with licensure and those without licensure have significantly different barrier scores? An independent sample t-test was used to investigate if there were significant differences between those with or without licensure on attitude scores. There was not a statistically significant difference in mean barrier scores by licensure status, t (191) = .47, p = .64). H4f: Are there significant differences in barrier scores among JJSPs that majored in Social Work, Criminology/Criminal Justice and Psychology/Mental Health Counseling fields? A one-way Analysis of Variance (ANOVA) was used to assess whether there was a significant difference in perception of barriers toward Evidence Based Practices in the case of Social Workers, Psychology/Mental Health Counselors, and Criminologists. Results indicated that there was not a statistically significant difference in barrier scores by major, F (3, 189) =1.63 p = .19). In other words, the results indicated that perceptions of barriers towards Evidence Based Practices do not statistically differ 72
by these three different professional backgrounds. H4g: Are there significant differences in barrier scores among JJSPs that work in Residential Facilities, Day Treatment Programs and Prevention Programs? To assess whether there was a significant difference in perceptions of barriers toward Evidence Based Practices among JJSPs who work in Residential facilities, Day Treatment programs and community Prevention programs, the researcher ran a one-way ANOVA. Results show there was not a statistically significant difference between type of employment setting and perceptions of barriers toward EBPs, (F (5,176) = .13, p. = .99). As with previous analyses, the null finding suggested perceptions of barriers among JJSPs employed in three different types of agency settings are essentially the same. H1h: Can JJSP barrier score be correctly predicted by individual factors (age, gender, licensure, degree, major) and type of employment setting (Residential Facility, Detention Center, Day Treatment, Prevention program, Redirections program, and Juvenile Assessment Center (JAC))? Once again a hierarchical linear regression was completed to find out if results would differ by demographics and by place of employment (see Table 2.9). Again, the model was not statistically significant F(8, 155) = 1.561, p < .05 and explained only 7.5% of variance of barrier scores. In the first step, the Barrier Index score was regressed on age, gender, licensure, degree type and major studied with only gender and type of major (Criminology/Criminal Justice) significantly related to perception of barriers (β = -.146, SD = .06, p<.01, β =.209, SD =.07, p<.01). In the second step, type of employment setting variables (Residential, Detention, Day Treatment, Prevention, Juvenile Assessment Center) were added to the model. The introduction of employment setting variables explained an additional 9.8% variance among barrier scores after controlling for individual factors. As shown in Table 2.9, none of the additional covariates (employment settings) in the second model had a statistically significant effect on JJSP perception of barrier scores. The total model was not a significant predictor of perception of barriers F (8, 155) = 1.561, p<.01) and given it explained so little overall variance, the two significant results could still be due to chance. Research Question 5: What barrier is perceived by JJSPs as the most significant/severe barrier toward using EBP? Descriptive analysis of responses (Table 2.2) to the Barrier Subscale questions found 63.6% of 73
respondents agreed or strongly agreed that lack of time was a barrier to using EBP in their daily work. About 49.2% of respondents agreed or strongly agreed that lack of informational resources was a barrier and 43.6% agreed or strongly agreed that lack of applicability of literature to their client populations was a barrier making these the most significant or severe barriers toward using EBP in daily work with clients. Interestingly, respondents showed the highest level of disagreement or strong disagreement at 71.3% with the statement that lack of research skills was a barrier. Disagreement and strong disagreement with the statement that lack of understanding statistical analysis as a barrier was indicated by 62.1% of respondents and it may be somewhat promising to note that 72.8% disagreed or strongly disagreed that lack of interest was a barrier. Qualitative Analysis One instrument question was designed to collect qualitative data to enrich quantitative findings regarding JJSPs’ perceptions of barriers. Table 2.10 shows results of the qualitative content analysis for the question; “Please list the top 3 reasons you think are barriers to implementing EPB(s) in your current work with clients” was conducted to explore JJSP perceived barriers further. Most participants provided two to three short statements in response to the above question while two participants provided lengthier explanations for their answers. One respondent took time to write additional comments at the end of the survey offering critiques of this survey instrument. Of the 195 participants, 129 answered the question and were included in the qualitative analysis with a total of 339 individual reasons. Fifteen of the responses/reasons were too vague to be coded in the analysis, such as “Professional position does not always apply”, “constraints”, “surveys” and “interviewing family”. Because elaboration and/or clarification was not provided by the respondent and because the interviewer was unable to seek further clarification, these 15 statements were not coded in an effort to avoid making any erroneous assumptions about the true meaning of the data. As mentioned in Chapter 3, an open coding scheme was initially developed based on the prominent themes identified in the data. All data were coded by into twelve defined codes and were coded again by a second coder to ensure reliability. Interrater reliability reached 94.6% after both coders thoroughly discussed the data and confirmed the twelve code categories were sufficient. However, a new category was also added after the second round of coding discovered there were four statements that 74
could be grouped together as one additional barrier even though that comprised only 0.3% of the data. The categories include: Lack of time, lack of training, lack of resources, lack of funding/high cost, lack of interest/negative JJSP attitude, lack of JJSP understanding/experience with EBP process, lack of JJSP knowledge of EBP concepts and research information, lack of applicability to clients, lack of model fidelity/inconsistency of implementation, lack of support from administrators and colleagues, client resistance, lack of supervision/supervision issues and finally, environmental issues. Each code category is listed in descending order of greatest number of responses to smallest number and described with definitions below. Results are summarized above in Table 2.10. Lack of Time. This category was fairly straightforward and included statements such as “lack of time”, “insufficient time” and “time constraints”. With 52 response statements (15.3%), this reason was the most frequently cited barrier to implementing EBP in current work with clients. Not surprisingly, this high occurrence rate corresponded with quantitative findings of 63% of respondents indicating “strongly agree” or “agree” on questionnaire item number 9: “Insufficient time is a barrier for me”. Lack of Training. Nineteen out of 48 statements (14.2%) in this category simply stated “lack of training”, “limited training”, “lack of continued training”, “lack of training provided”, “lack of available training”, “not enough training on EBP”, “lack of refresher courses given by company” and “no training”. These statements were easy to categorize although it was not always possible to discern what type of training or training topic the respondent had in mind because the respondent did not elaborate. Also included in this category were statements that implied lack of training as a barrier, but included another barrier within the statement such as, “lack of time for trainings”, “not enough time for research and training”, or “lack of funding to attend valuable trainings” and “no funds to train”. These types of statements were categorized as “lack of training”, but were also included in “lack of funding” or “lack of time” categories as appropriate. Lack of Applicability to Clients. Forty-four (13%) statements specified lack of applicability to clients as a top barrier to using EBP with clients. Besides that exact statement, other phrases included in this category are: “evidence based research does not always apply to minorities”, “individual differences across client populations do not mesh with some EBP”, “does not always apply to every client”, “not always applicable to clients’ insight/level of understanding”, “clients do not fit neatly into one diagnostic 75
category”, “research may not be relevant to clients’ needs”, “clients may not understand curriculum, have to work on their level”, “most is difficult to understand for kids, over their heads”, and “may not be appropriate for those with developmental problems”. While exact meaning may be slightly different, this category is clear-cut and the statements fit easily without much overlap with other categories. Lack of Interest/Negative JJSP Attitude. Lack of interest /negative JJSP attitude began as two categories, but upon further review and discussion between coders, the two were collapsed into one since many of the statements specified “attitudes of clinicians”, “some feel it doesn’t work”, “the procedures get a little stale”, “lack of enthusiasm by the facilitator”, “can be boring-too informational and not experiential enough/lacks relevance”, “unable to think outside of the box, “not willing to change”, “feeling that the system in place is adequate”, “doesn’t matter”, “waster of time”, “better methods exist”, “not enough staff believe in the practices” and simply “lack of interest” imply similar meaning and none of the 36 (10.6%) statements could be viewed as anything other than negative attitudinal remarks toward EBP as a barrier to implementation. Lack of JJSP Knowledge/Research Information. During initial coding it seemed logical to collapse “JJSP knowledge” and “JJSP experience” into one category but upon further analysis and discussion between coders, it became evident the data was split between statements pertaining to clinical experience knowledge and knowledge related to EBP concepts, understanding statistics/research as well as a lack of research information about specific populations or lack of access to research literature. Thirty-five (10.32%) statements belonged in this category and examples include: “NOT described in simple terms-too much stat stuff”, “lack of historical data”, “lack of education of EBP”, “lack of research”, “my co-workers/supervisors do not have much knowledge and cannot instruct me”, “lack of knowledge”, “understanding research analysis”, “inexperience in critically appraising literature” and “inexperience in critically appraising literature, more specifically, scientific research in the way it is often published”. Lack of Resources. Lack of resources is another category with multiple interpretations for what “resources” could mean. Thirty-two statements (9.44%) fell in this category because 14 out of the 32 statements simply stated “lack of resources”, “lack of informational resources” or “limited resources” so the category was developed with an open definition for what “resources” could include and other statements such as “lack of awareness of resources”, “current EBP not available”, “need more EBPs to 76
implement”, “lack of availability of EBP”, “lack of materials” and “the resources to effectively train our team in delivering the information/curriculum” were also added to this category. Lack of Model Fidelity/Inconsistent Implementation. Lack of model fidelity/inconsistent implementation is a unique category because it is evident from the statements that only respondents who have direct experience with using evidence-based curriculum would provide this type of response as a top barrier. With 30 statements (8.85%) such as “ training and fidelity of models in EBP sometimes value procedure over therapeutic relationship”, “my co-workers/supervisors do not all seem to agree”, “lack of consistency within the group”, “lack of consistency with instructions” “closed groups”, “group size”, “meeting frequency” and “lack of flexibility in implementation as EBP requires you to maintain 100% fidelity to the curriculum” cited as top barriers, this category implies the respondent currently uses an EBP program/curriculum or has used one in the past and it sheds light on what JJSPs think about evidenced based programs and curricula used in Florida to date. Client Resistance. Client resistance was another straightforward category because emerging statements were quite specific with 21 statements accounting for 6.19% of all qualitative responses. Statements such as “clients may not be open to intervention” or “lack of participation from clients” may not mean specifically to EBP, but since the question posed asked participants to list their top three barriers to implementing EBP, it is assumed the resistance pertains to using evidence based interventions. Other examples include: “uncooperation by participants”, “lack of buy-in from client”, “client is not receptive”, “client resistance”, “lack of rapport building”, “lack of client interest”, “resistance of clientele”, “EBP does not account for resistance of client” and “family resistance to session involvement”. Lack of JJSP Understanding/Experience Using EBP. Lack of clinical understanding/experience using EBP is the corresponding category to previous “lack of clinician knowledge/research information” category. Rather than include statements related to conceptual or technical knowledge, this category included statements pertaining to lack of clinical experience with using EBP and/or applying it to client base. Examples include: “lack of understanding of how to implement”, “I don’t have clinical experience”, “lack of experience”, “lack of understanding of direct care”, “staff operating outside of their skill set”, “because of small caseload-difficult to practice some of the requirements/skills as often, may affect skill level”, “lack of full understanding of EBP by counselor” and “lack of experience 77
with EBP itself”. With 19 statements (5.60%) in this category, future analysis of degree type and years of experience could be conducted to determine if these types of statements correspond to participants with less education and years of experience in the field. Lack of Funding. There was some overlap between lack of funding and others such as “lack of resources” and “lack of training”, but 18 statements (5.31%) included specific phrases pertaining to funding so they were coded in all appropriate overlapping categories. Samples include: “lack of funding”, “lack of funding to attend valuable trainings”, “lack of funding for implementation”, “no funds to train”, “lack of money”, “materials and literature that should be available are not due to funding”, “cost of training”, “cost”, “cost in obtaining appropriate EBP materials”, “too expensive”, “too costly to buy rights” and “the financial expense of training and implementation”. Lack of Support from Administrators/Colleagues. Fifteen statements (4.42%) indicated a lack of support from an organizational standpoint or from colleagues and administrators. While there were only 15 statements, the researcher and second coder agreed, there were enough data and enough consistency among the statements to create this category. Samples include: “lack of agency support”, “lack of support from non-clinical staff on interventions used”, “lack of interagency support”, “not supported”, “lack of support from third parties”, “lack of collective support among colleagues in my agency/facility”, “the buy-in from everyone so that we can introduce new curriculum” and “there is lack of support from management on understanding how one takes a curriculum/EBP and make it more individualized”. Supervision Issues. There were 23 statements (6.78%) that indicated supervision issues as a barrier, therefore, it was not the smallest category, but it was the most loosely defined. While most of the above mentioned categories had distinct definitions, this category included a broader range of issues related to supervision with three sub-categories: 1) lack of supervision, 2) supervisor responsibility and 3) supervisor expectation. The first sub-category included: “lack of supervision”, “not enough qualified supervisors on-site to train”, “transitional- without supervision over and release from program-not being more structured”. The second sub-category included caseload/case assignment issues such as “heavy client load”, “case load too high”, “I don’t have a regular caseload” and the last sub-category is related to performance 78
expectations and pressure from a supervisor. Examples included: “the demand to meet contract goals”, “pressure to be productive”, “contract requirements of DJJ activities”. Four statements were placed into an additional “environmental issues” code category developed after the initial 12 categories because enough consistency was found among the statements: “Settingenvironment not conducive to EBP”, “environment”, “geographic” and “environment” to warrant this category. But since the 4 statements constituted only 0.3% of the data, not much can be inferred from this category. Finally, there were 11 statements that were too vague to be coded into any of the categories without risk of making incorrect assumptions about the meaning. Examples are: “Professional position does not always apply”, “constraints”, “surveys” and “interviewing family”. With only 11 out of 339 statements (3.24%) not coded, the majority of the qualitative data were analyzed and allowed this researcher to tap into the deeper meaning of what the quantitative data represents (Rubin & Babbie, 2011) and will be fully discussed in the final chapter.
Table 2.1 Characteristics of Sample Population Sample Population Characteristics Gender Male Female Age 18-24 25-29 30-39 40-49 50-59 60-69 70+ Practice Licensure LCSW LMHC LPC LMFT CAP LMHC Intern LCSW Intern Do not hold license Degree Type High School Diploma Associates Bachelors Masters Doctorate Majors Studied Social Work Criminology/Criminal Justice Psychology Rehab Counseling Sociology Education Business/Management Other Years of Professional Work Work Setting Residential Detention Day Treatment Prevention Redirections JAC Other Work Shift Day Night Graveyard
Mode or Frequency (%)
10 32 78 41 18 9 1
3.1% 5.1% 40% 21% 9.2% 4.6% 0.5%
1 17 2 1 4 20 14 124
0.5% 8.7% 1% 0.5% 2.1% 10.3% 7.2% 63.3%
13 12 72 89 1
3.6% 6.2% 36.9% 45.6% 0.5%
35 2 14 6 15 25 173
17.9% 1% 7.2% 3.1% 7.7% 12.8 (2 years) 9.7%
99 13 15 28 14 10 5
46.2% 7.1% 8.2% 14.4% 7.6% 5.4% 2.6%
167 12 3
86.1% 6.2% 1.5%
Did not respond 7 (0.4%) 6 (0.3%)
22 (8.9%) 11 (5.6%)
Table 2.2. Attitudes Toward Evidence Based Practice/Interventions How much do you agree Strongly Disagree Agree with the following items? Disagree
Using evidence-based practice is necessary in my daily work.
Literature and research findings are useful in my daily work.
I know better than academic researchers how to care for my clients.
I am willing to use new and different types of EBP/Interventions developed by researchers.
My professional/clinical experience is more important than using EBP/Interventions.
I would try a new EBP even if it were very different from what I am used to doing.
If you received EBP training in an EBP/Intervention, please rate the following:
It was intuitively appealing to you?
It “made sense” to you? 1.0% *Note modal categories are in bold
Table 2.3. Perceptions of Barriers Toward Evidence-Based Practice/Interventions How much do you agree Strongly Disagree Agree Strongly with the following items? Disagree Agree
Insufficient time is a barrier for me.
Lack of informational resources is a barrier for me.
Lack of research skills is a barrier for me.
Inexperience in critically appraising literature is a barrier for me.
Lack of applicability of the literature to my client populations is a barrier for me.
Lack of experience in applying research findings to individual clients with unique characteristics is a barrier for me.
Lack of understanding statistical analysis is a barrier for me.
Lack of collective support from my colleagues in my agency/facility is a barrier for me.
Lack of interest is a barrier 19.5% for me. *Note modal categories are in bold
Table 2.4. Requirement to using Evidence Based Practices If you received training in an Yes EBP/Intervention, Please answer Yes or No to the following questions:
It was required by your supervisor?
It was required by your agency?
It was required by FL DJJ?
Have you personally EVER provided an EBP/Intervention?
Have you personally ever worked in 86.7% a setting where you saw EBP being used? *Note modal categories are in bold
Table 2.5. Knowledge of Evidence-Based Practice Concepts Questions Practitioners who engage in EBP will: 1) Rely most heavily on the practice wisdom of supervisors and experts 2) Scientifically monitor client outcomes 3) Utilize one may theoretical framework and intervention strategy with most clients 4) Only select interventions that are on the list of empirically supported treatments 5) Answers 2 and 3 only
35 out of 195 answered correctly
The best scientific evidence might: 1) Be only good enough to guide practice in a tentative manner 2) Not be relevant to a client 3) Be an indication of what not to do 4) Answers 1 and 2 only 5) All of the above
93 out of 195 answered correctly
Which of the following statements is/are true about applying the intervention supported by the best research evidence? 1) Because it is the most effective, you should implement this treatment the best that you can, even if you cannot get training or supervision 2) Because it is the most effective, you should make the clinical decision to select this intervention for your client 3) If you decide to implement this intervention, you should monitor client progress with it 4) All of the above
107 out of 195 answered correctly
If it is determined there is strong evidence of the effectiveness of an intervention in an outcome evaluation, it is important to consider: 1) How the outcome was measured 2) Whether a control group was used 3) Whether participants across different treatment conditions are really comparable to begin with 4) All of the above
156 out of 195 answered correctly
A good systematic review: 1) Fully discloses its search and inclusion strategy 2) Assess the quality of selected outcome studies 3) Is done by someone within the field who has a vested interest in the findings of the review 4) All of the above 5) Answers 1 and 2 only
94 out of 195 answered correctly
Table 2.5 (Continued) A hierarchy of evidence exists in the scientific world, which is the most accurate order from highest to lowest to determine an effective intervention (assuming all studies are implemented perfectly, with good measurement and are unbiased)? 1) Experimental studies, quasi-experimental studies, meta analyses, qualitative studies 2) Quasi-experimental studies, meta-analyses, qualitative studies, experimental studies 3) Meta-analyses, quasi-experimental studies, experimental studies, qualitative studies 4) Meta-analysis, experimental studies, quasiexperimental studies, qualitative studies
61 out of 195 answered correctly
If a study has been published and it appears in a reputable literature base, you can be sure it has been conducted objectively and rigorously. 1) True 2) False
117 out of 195 answered correctly
When measuring client outcomes, it might be okay to develop your own individualized rating scale instead of using an instrument that researcher have validated. 1) True 2) False
50 out of 195 answered correctly
111 out of 195 answered correctly
162 out of 195 answered correctly
EBP questions are restricted to asking about the effectiveness of interventions, programs or policies. 1) True 2) False EBE includes professional judgment and behavior that is guided by the client system over time. It also includes evaluating the practice to determine if the predicted results have been attained as a direct consequence of the practitioner’s actions. 1) True 2) False
Table 2.6. Attitude Score by Individual Characteristics Model 1 Variable
Model 2 B
Age .018 .042 .047 .017 Masters Degree -.015 .051 -.028 -.028 Years of Experience -.008 .006 -.154 -.166 Gender -.146 .076 -.154 -.166 Licensure (Yes) .031 .095 .033 .055 Social Work .073 .119 .061 .069 Criminology/Crim .213 .098 .191 .236 Justice Psychology/Mental -.032 .101 -.032 -.017 Health Counseling Residential -.099 Detention -.300 Day Treatment -.096 Prevention .013 JAC -.127 2 R .030 .061 2 F for change in R .030 .031 Note: Dependent variable as follows: Attitude Index Score
.043 .053 .078 .078 .095 .120 .100
.044 -.051 -.175 .044* .057 .058* .212
.137 .197 .175 .153 .195
-.111 -.153 -.062 .011 -.068
Table 2.7. Level of Evidence Based Practice Knowledge by Percentage Level of Knowledge Low
Percent % 19.5%
Table 2.8. Knowledge Score by Individual Characteristics Model 1 Variable
Model 2 B
Age .194 .168 .131 .127 .172 Masters Degree .126 .361 .036 .257 .369 Years of Experience -.012 .023 -.156 -.008 .024 Gender .357 .309 .095 .320 .316 Licensure (Yes) -.102 .416 -.027 -.050 .423 Social Work .185 .477 .039 .140 .428 Criminology/Crim -.078 .395 -.018 -.177 .402 Justice Psychology/Mental -.103 .404 -.026 -.083 .410 Health Counseling Residential .609 .555 Detention .836 .796 Day Treatment 1.062 .108 Prevention 1.003 .624 JAC 1.450 .797 2 R .023 .053 2 F for change in R .023 .031 Note: Dependent variable as follows: Knowledge Index Score
β .085 .073 -.041 .086 -.013 .030 -.040 -.021 .173 .108 .174 .223 .198
Table 2.9. Perception of Barriers Score by Individual Characteristics Model 1 Model 2 Variable
Age .017 .042 .044 .039 Masters Degree .013 .089 .015 .005 Years of Experience -.008 .006 -.152 .008 Gender -.146 .076 -.154 -.166 Licensure (Yes) .013 .103 .013 .033 Social Work .063 .118 .052 .055 Criminology/Crim .209 .098 .188 .229 Justice Psychology/Mental -.040 .100 -.040 -.029 Health Counseling Residential -.092 Detention -.285 Day Treatment -.093 Prevention .019 JAC -.112 2 R .075 .098 2 F for change in R .075 .023 Note: Dependent variable as follows: Barrier Index Score
.043 .091 .006 .078 .105 .119 .099
.039 .005 -.143 .-175* .034 .045 .206*
.137 .197 .175 .154 .197
-.103 -.146 -.060 .017 -.060
Table 2.10. Code Categories for Qualitative Question: What are the top 3 reasons you think are barriers to implementing EBP(s) in your current work with clients? Code Category
Lack of time
Lack of training
Lack of applicability to client population
Lack of interest/Negative JJSP attitude toward EBP
Lack of JJSP knowledge of EBP/research information Lack of resources
Lack of model fidelity/Inconsistent implementation Client Resistance to EBP
Lack of JJSP understanding/experience with EBP Lack of funding/High cost
Lack of support from administrators and colleagues Supervision Issues
CHAPTER 5: CONCLUSIONS This chapter offers a discussion of the results presented in Chapter 4. Limitations are also included followed by a discussion of implications for clinical social work practice and research. Rogers’ (1962) Diffusion of Innovation Theory provided a framework to explain how EBP as an innovation is communicated among service professionals in the Florida Juvenile Justice System to create a system change. This dovetailed with Ajzen’s Theory of Planned Behavior explaining how individual behavioral change is necessary for system change to occur. Primary research variables of JJSP Attitude, JJSP Knowledge and JJSP Perceptions of Barriers were selected since they are key elements influencing individual change as well as essential qualities for innovation diffusion within a system. At the core of this study, the attitude variable represented how much JJSPs value EBP and to what extent they perceived the EBP process allowed for better service delivery than not using EBP(s) or interventions. The knowledge variable was selected because the degree to which an innovation is perceived as simple and easy to use was another quality of Rogers’ (1962) theory that explains innovation diffusion. Similarly, the third variable of JJSP perception of barriers was explained by Ajzen’s concept of perceived behavioral control or rather, how easy or difficult JJSPs felt it would be to overcome potential obstacles of integrating EBP into their daily work. Survey instrument selection and design was also informed by both theories as dimensions on Aarons’ Evidence-Based Practice Attitude Scale (2004) mirror Rogers’ qualities of innovations regarding individual and organizational factors. Studies specifically related to perceptions of barriers toward using EBP frequently cite Rogers’ Theory of Innovation Diffusion to explain findings concerning clinician attitudes, how innovations are communicated and organizational system changes. Several key conclusions were drawn from the study data. Keep in mind, approximately 46% of the sample had Masters degrees and 37% reported either having professional licensure or was seeking licensure for LCSW, LMHC, LPC or LMFT status, so this indicated recruitment efforts to target correct participants seemed largely successful. There was also roughly equal representation among Social Work 91
(15%), Criminology/Criminal Justice (17%) and Psychology/Mental Health Counseling (18%) majors who worked in a variety of agency settings from least restrictive (community prevention programs, 14%) to most restrictive (residential facilities, 46%). The large number of participants working in residential facilities was not surprising since out of all the possible agencies that receive funding from the DJJ, these residential facilities employ the greatest number of employees. It was not possible to capture more participants from community prevention programs because the selection parameters of this study only included agencies funded by the DJJ. In many cases, community prevention programs are funded by the city, county and/or receive funding from other public and private entities. These non-DJJ funded agencies work in partnership with DJJ funded agencies to provide a continuum of services to youth and to avoid an overlap in juvenile justice services but were not included as part of this study. Conclusions: Research Question 1 - Attitudes The first research question focused on attitudes of Juvenile Justice Service Professionals toward using Evidence-Based Practices. It was hypothesized individual factors such as age, degree type, years of experience, gender, licensure, major of study and employment setting may be associated with attitude level or would determine differences among the sample but the majority of results showed no significant relationships or differences between individual factors and JJSP attitude. Aarons et al. (2010) reported individual reliability coefficients so that the scale could be used in portions and so that the factor structure is generalizable to a variety of service provider contexts and service settings. However, when the EBPAS was adapted for this study, portions of the scale were omitted because the pilot study found the questions were too specifically related to clinical/therapeutic concepts that would not be applicable to a broader sample of JJSPs who may not have had mental health or social work training. In doing so for this study, this may have reduced its ability to capture data as sensitively. Because attitudes toward EBP is such a complex variable and not specifically measured among JJSPs before, the individual factors Aarons tested were used for this study as well. Although Aarons et al. (2010) found older mental health providers were more open to adopting EBP; this hypothesis did not show a significant correlation between JJSPs age and attitude score. Sixty-one percent of this sample 92
was between the ages of 30-49, compared to Aarons’s mean age of 38.22, but respondents were asked to select their age in ten-year increments for this study so it was not possible to determine the exact mean age. Measuring age in this manner could have been constrained by the variability of responses. Different results may have been found if age was measured in a continuous variable instead. In spite of the literature stating otherwise, it is also plausible that in this JJSP population, age has nothing to with attitude. It was also hypothesized that years of experience would be negatively associated with attitude, but again no relationship existed. Aarons (2004) found the most experienced practitioners possessed the least favorable attitudes toward EBP and his 2010 study reported the same results among practitioners with 10.66 mean years of experience (SD = 8.51; range: 0-50). JJSPs in this study had 10.45 mean years of experience (SD = 16.64; range: 0-41) and while simpler in years of experience, the findings suggest that years of experience did not predict attitude toward EBP. As for the hypothesis that higher education attainment/degree type or licensure would be positively correlated with attitude, this study did not find any significant associations even though 46% of the sample obtained masters degrees and 37% obtained or were seeking licensure. Again, these two variables were significant predictors in in Aarons’s research as he found practitioners with higher educational attainment endorsed more positive attitudes (β = .106, SEβ = .042, p < .05) as well as those with a registered intern status (β = .169, Seβ = .098, p< .05). Similar to the current study, Nelson and Steele (2007) did not find any differences between doctoral and master’s level practitioners on selfreported EBP use. However these authors did not compare between practitioners with bachelor level degrees and advanced degrees so while their findings are related, their results were interpreted with caution in the context of this study. Many other studies reiterate experienced practitioners and supervisors may not support or use evidence-based practices/interventions because they lack training and do not feel EBP approaches align with their accumulated practice expertise (Bilsker & Goldner, 2004, Chwalisz, 2003, Dulcan, 2005, Rubin & Parrish, 2007, Sanderson, 2002). While this study’s quantitative data did not reveal similar results to Aarons el al.’s (2010) findings, the qualitative data as discussed below were much more aligned with literature findings.
Aarons (2010) reported women have more positive attitudes toward adopting EBP but an independent t-test determined men and women do not have significant differences on attitude toward EBP even though approximately 68% of this sample was female, compared to the 76% in Aarons and colleagues’ study. Nevertheless, because it is known that the mental health and social service workforce is predominantly female (Aarons, 2010, Sheehan, Walrath, & Holden, 2006), it is recommended that greater attention to why gender differences occur should be made in effort to better support EBP training and education across genders among JJPS. Because 50% of this sample was comprised of Social Work (15%), Criminology/Criminal Justice (17%) and Psychology/Mental Health Counseling majors (18%), the next research question asked if differences in JJSP attitude scores existed among those who majored in in the three different fields. Similarly, participants employed by detention facilities, community prevention programs and day treatment programs encompassing about 69% of the total sample were also compared to see if JJSPs from each group differed in their attitude scores. While no significant differences could be reported to answer either research question, this may be due to the reality of participants’ job functions. A person who has a masters degree in Social Work could conceivably be teaching classes in a day treatment program, providing therapeutic group services in a residential facility, or be in an agency administrative leadership position. These different roles require different applications of EBP and thus attitude could widely vary. Consequently, more careful consideration of this dynamic is necessary in future research. Lastly, it was hypothesized that JJSP attitude could be correctly predicted by individual factors such as age, gender, licensure, degree type, major and employment setting. Only Social Work majors were significantly more likely to have higher positive attitude scores than any of the other potential predictors. This is encouraging because the finding may indicate social work majors are more willing to adopt EBP given the four elements on Aarons’s Evidence-Based Practice Attitude Scale (2004). In other words, Social Workers’ may perceive the intuitive appeal of an EBP as more positive, they may be more willing to adopt new practices if required by their agency and they may be more open toward new or innovations while perceiving a lesser degree of divergence from usual practice when introduced to empirically developed or research-based practices (Aarons, 2004). 94
Some possible explanations for this were found by examining the data obtained from just the 30 respondents who were social work majors. Sixty percent (18 out of 30 respondents) had 5 years or less of professional experience, 70% (21 respondents) earned masters degrees and while only one respondent already obtained clinical licensure, 47% (14 respondents) were registered interns. Although, this sample size does not allow generalizability or statistical inference, these characteristics were congruent with Aarons’s results suggesting those with longer years of experience endorsed a more negative attitude toward EBP and those with advanced degrees and were licensed or licensure-seeking experienced more positive attitudes. Conclusions: Research Question 2 - Knowledge Because self-efficacy and perceived behavioral control are at the center of Ajzen’s Theory of Planned Behavior, it was important to find a baseline level of EBP knowledge among JJSPs to offer some context to findings related to their attitudes and perceptions of barriers toward EBP. These two variables are theoretically contrived by an individual’s belief of how much he or she knows about EBP and if he or she understands EBPs well enough to integrate these practices into their daily work. Few studies emphasizing practitioner knowledge of EBP concepts were found, but Mullen and Bacon’s (2006) conclusion that “social workers were poorly informed; typically not even aware of the meaning of practice guidelines and that social workers were generally not using research findings or research methods in their practice (p. 90) was compelling enough to see if the same were true among juvenile justice service professionals and social workers of this group in particular. As it turns out, about 25% of the sample (48 respondents) were categorized at a high level of EBP knowledge by answering at least 7 out of 10 survey questions correctly, 54% (105 respondents) were categorized as moderate by answering 4 to 6 questions correctly and about 20% (38 respondents) were categorized as low level of knowledge by answering only 0-3 questions correctly. This means that if this particular group of JJSPs were asked to pass a test on EBP using a typical grading scale, 64% would fail by getting fewer than 6 correct answers out of 10. Testing individual factors with knowledge scores showed no significant relationships between knowledge scores and age or with years of experience. There were no differences among groups within degree types, gender, major studied, licensure, and different employment settings nor did a regression 95
model find any of these coefficients to predict knowledge scores. Moreover, the overall index scores for Attitudes and Barriers were not significantly correlated with the Knowledge index score, but interestingly, among Social Work majors and Psychology/Mental Health Counseling majors, there was a negative relationship between knowledge score and barrier score, meaning as these respondents’ EBP knowledge increased, their perceptions of barriers to using EBP decreased (Social work, r = -.44, p = .02, N = 30 and Psychology/Mental Health Counseling, r = -.31, p = .02, N = 57). However, no significant relationship was found among Knowledge score and Barrier score for Criminology/Criminal Justice majors. Rubin and Parrish’s work (2006, 2010, 2011, 2012) offered some insight when interpreting these results because within their research on teaching EBP in schools of social work, they often outline the history of practitioner opposition toward EBP. Viewing EBP as a process that denigrates clinical expertise, ignores patient values, promotes a standardized approach to practice and used simply as a cost-cutting tool are common arguments reported in the literature. Supporters of EBP counter that these views are misconceptions of how EBP was truly represented in the original definition by Sackett, Rosenburg, Gray, Haynes and Richardson (1996). EBP fundamentally included using practitioner expertise in clinical decision-making to determine which treatments would likely work best while negotiating with the client to incorporate his/her preferences in the name of building a therapeutic relationship. With this original definition, it was even acceptable to use the “best available” evidence if one proceeded cautiously and evaluated outcomes along the way when little evidence existed. With the evolution of this controversy among social work and psychology/mental health professions, more emphasis has been placed on teaching and training BSW and MSW students the EBP philosophy and concepts from a bottom-up approach in which practitioners seek and appraise evidence themselves to inform their practice decisions. This may have some bearing on the results found in this study because a literature review of criminology/criminal justice literature did not reveal a similar discourse, but rather an emphasis on the top-down approach in which practitioners use only certain interventions that have already been designated as empirically supported or evidence-based (Drake et al., 2001; Mullen, 2004; Proctor & Rosen, 2004; Rubin & Parris, 2007).
Out of 973 survey responses, Rubin and Parrish (2007) found social workers almost equally defined EBP as top down (22.5%) or bottom up (24.5%) and 46.2% of social workers said they use both definitions. But this definition discrepancy may explain why participants in this study who majored in Criminology or Criminal Justice did not show a correlation between their knowledge scores and barrier scores. Their background and training may have included some information about the philosophy and evolution of EBP in their field, but if the focus of their education was more likely to be more about which interventions work best with what clients and how to address implementation issues such as model fidelity, an increase in knowledge would not decrease perceptions of barriers if one does not subscribe to a broader, more flexible definition of EBP that allows for a JJSP to modify practices to circumvent barriers. Care was taken to ensure EBP was defined in the cover letter as a concept that included specific evidence-based interventions as well as a broader philosophical process. Additionally, efforts to create a survey instrument with broad knowledge-base questions that were assumed to be familiar to all potential participants regardless of their educational background were used but future research may still be better served by accounting for the definition discrepancy among the sample. Conclusions: Research Question 3 - Barriers In spite of inconsistencies in the literature of how EBP is defined, the concept of barriers to using EBP in daily practice does not need such clear-cut parameters. The literature across social work, nursing, physical therapy and medicine consistently find barriers are related to organizational support, accessibility to resources, applicability of research findings and individual attitude. Nine questions on the survey were adapted from the work of Jette, et al. (2003) and respondents were asked to rate each of the nine common barriers by how much they agreed or disagreed that it was a barrier to using EPB in their daily work with clients. The quantitative analysis was completed in the same manner as with attitude and knowledge scores using exploratory bivariate analysis, but individual factors (age, licensure, degree type, and employment setting) resulted in no difference findings. However, a linear regression model including all the individual factors to predict barrier scores revealed women scored significantly lower on the barrier index. Women perceived fewer barriers to using EBP in their
daily work than men. In addition, criminology/criminal justice majors were significantly more likely to perceive barriers than social work and psychology/mental health counseling majors. Possible explanations for these findings are analogous to what has been discussed regarding results for attitudes and knowledge. It is difficult to extrapolate meaning from a single model showing women in this sample were less likely to perceive barriers to using EBP in their daily work, but if known differences exist across genders in the mental health and social service workforce, it could be beneficial to further compare such differences within the juvenile justice workforce. Although the sample for this study was comparable to what is typically seen in social work and mental health settings, a higher proportion of men could be employed in the juvenile justice arena than in social services. As for criminology/criminal justice majors perceiving more barriers to using EBP in their daily work, this is also likely due to differences in their education and background training. While there was no relationship found between knowledge level and perceptions of barriers for criminologists, it is interesting this group was still predicted to score significantly lower on the barrier scale than other majors. These findings suggest further on the job training could be helpful to improve attitude and decrease perception of barriers. The qualitative findings offered rich support to the quantitative results and were corroborated in the literature. A single question on the survey instrument asked participants to “list the top 3 reasons you think are barriers to implementing EBP(s) in your current work with clients”. One hundred twenty-nine respondents offered a total of 339 individual reasons with the top 5 reasons being a lack of time (52 statements, 15.3% of total), a lack of training (48 statements, 14.2%), a lack of applicability to client population (44 statements, 13%), a lack of interest/negative JJSP attitude (36 statements, 10.6%) and a lack of JJSP knowledge of EBP/research information (35 statements, 10.3%). Nearly 64% of all respondents agreed or strongly agreed that lack of time was a barrier on the quantitative portion of survey so it is no surprise it was also the top barrier among qualitative findings. Studies by Funk et al., (1995); Jette et al., (2003); Retsas, (1999); Upton & Upton, (2005) all report lack of time as a barrier to implementation of EBP but it is difficult to find any resolution to this barrier other than the suggestion that further EBP education and training may promote more awareness of its value and so practitioners may give it a higher priority during their busy workday. 98
Almost 49% of all quantitative responses indicated agreement or strong agreement that lack of applicability of literature to their client populations was a barrier and this corresponded with 13% of the qualitative statements. This is also a common barrier across disciplines, but it speaks to the disparate definitions of EBP among social work and other allied professions as well as the history of skepticism towards the feasibility of EBP in daily practice (Rubin & Parrish, 2007). As EBP is more than just finding a specific piece of evidence to support a particular practice or intervention, the better JJSPs can understand this, the greater the likelihood of appraising research correctly. It is interesting to note that over 71% of all respondents indicated disagreement or strong disagreement that lack of research skills was a barrier and yet, of those who chose to respond to the qualitative question, over 10% stated it as a top barrier. Furthermore, one of the multiple choice knowledge questions asked respondents to select the correct order of research hierarchy and this question was only answered correctly by 31% of the sample. So if JJSPs have an inconsistent view of what constitutes EBP and they do not understand how to determine if research is empirically supported, one runs the risk of assuming anything published is evidence based. Essentially, barriers such as lack of training, lack of knowledge and lack of research skills all signify organizational efforts are needed to provide more EBP training opportunities for JJSPs so they are able to discern which practices/interventions are truly evidence-based and make practice decisions based on correct information. Almost 73% of all respondents indicated they disagreed or strongly disagreed that lack of interest was a barrier, but again, it was yet another top qualitative barrier. Perhaps social desirability explains this finding, as participants do not believe their research skills are lacking but even if they were lacking, they did not want to admit that it was due to a lack of interest. Nevertheless, if JJSPs state lack of time was a primary barrier and they understood that “applying rigorous evidentiary standards when selecting interventions” was necessary but more time-consuming, it would not be surprising if there was a strong lack of interest to change their current practices to keep up with new evidence-based interventions as they emerge (Rubin & Parrish, 2007, p. 118). With a high number of statements pertaining to lack of training (48 statements (14.2%), plus 35 statements (10.3%) for lack of JJSP knowledge of EBP and 19 statements (5.6%) for lack of JJSP 99
experience with EBP, it is not difficult to see that there is a certain amount of discomfort with EBP knowledge and experience but JJSPs are eager to seek out training opportunities. Additionally, for those who see lack of resources (32 statements, 9.4%) and lack of funding (18 statements, 5.3%) as top barriers, a lack of desire to learn more about EBP is not the problem. Fortunately, it is easier for agency leadership to resolve these kinds of issues because a positive attitude is already in place among these JJSPs. According to Rogers’s Theory of Innovation Diffusion and Ajzen’s Theory of Planned Behavior, attitude was the best predictor of individuals engaging in a specific behavior that would allow for an innovation to become adopted. As for the JJSPs who stated lack of agency support (15 statements, 4.4%) was a top barrier, this also implies the individual is willing to put forth effort toward using EBPs, but could not successfully do so because of organizational issues. These findings suggest agency leadership could benefit from a closer assessment of their agency’s resources to see if more could be allocated toward EBP education and training given the positive attitude of many JJSPs eager to learn more. As for the statements related to a lack of model fidelity (30 statements, 9%) and client resistance (21 statements, 6.2%), these appear to have been shared by clinicians or therapists who have enough clinical experience and knowledge of using some type of EBP process or intervention to list these as top barriers. With 15% of these statements representing those familiar with current EBP practices, agency administrators are encouraged to consider offering resources and support to those with this type of academic training or background to cultivate their interest and develop them into early adopters/leaders for future innovations. Although most of this study’s hypotheses resulted in no relationship findings, the significant results showing social workers as more likely to have positive attitude scores as well as their knowledge scores negatively correlated with barrier scores seems promising. The qualitative data support these quantitative results, so this study has fulfilled its objectives and taken a tentative step toward closing the literature gap. Study Limitations Nevertheless, there were several limitations to this study. First, the availability sampling technique limits generalizability of findings and affects the external validity of this study (Rubin & Babbie, 2011). 100
Although the entire population of JJSPs in Florida was theoretically eligible to participate, the reality of reaching everyone was not feasible so the sampling frame included only JJSPs working in agencies funded by the DJJ and affiliated with the FJJA. This eliminated potential participants working in agencies funded by other entities. Threats to internal validity are also possible given these convenience sampling methods simply due to selection bias. Those who were willing to participate may truly have differences in attitudes, knowledge and perceptions of barriers toward EBP than those who were not interested enough to participate (Rubin & Babbie, 2011). The cross sectional design is a limitation as it does not allow causal inferences to be made about the relationships between variables because time ordering of the variables is not possible. This means while it was hypothesized that level of attitudes, knowledge and perception of barriers would be affected by individual factors, it is impossible to know if the opposite may also be possible. A cross sectional design also allows more risk for design contamination as the survey data was collected over a four-week period so the chance of one participant influencing another potential participant in a negative way, dissuading the second person from participating or answering questions differently is unfortunately plausible (Rubin & Babbie, 2011). Relying on self-report from JJSPs increased the risk of social desirability and may limit interpretation of results since JJSPs could have answered survey questions differently. The popularity of using EBP remains high and the pressure to produce measurable outcomes continues, therefore participants may have been fearful to appear unsupportive of their organization and answered survey questions accordingly. Because access to participants was critical, agency leaders were asked to help distribute the survey, but this would be a limitation as well if participants felt obligated to participate due to their boss asking them to do so. Although Aarons’s EBPAS has been established as a measure with strong reliability and validity as a whole and among the individual subscales; the EBPAS was adapted for use with JJSPs. This resulted in an instrument with an internal consistency reliability coefficient (α = .67) that is on the lower end of acceptable (Rubin & Babbie, 2011). This lower alpha score could limit the validity and generalizability of findings pertaining to this instrument as Aarons originally intended for groups of questions to measure particular constructs such as Openness to EBP, Appeal of EBP and Requirements 101
to use EBP while also generating an overall score. But if not all the subscale items were included, this could result in a different overall score that does not accurately capture JJSP attitude. Similarly, in the time since Parrish wrote her dissertation using the knowledge questions, she and Allen Rubin have created the Evidence-Based Practice Process Assessment Scale based on her dissertation findings but the new scale questions have changed from the original knowledge questions. While the Barriers subscale had a higher alpha coefficient at .83, the predominantly null results suggest other unmeasured variables may account for the findings in all three subscales. Furthermore, while the sample size allowed adequate power for basic statistical tests, it may not have been large enough to complete the regression analysis at a .05 probability level with 13 predictors in the model for a medium effect size at the desired statistical power level (http://www.danielsoper.com/statcalc, 2014). Post hoc analysis suggested that the sample size resulted in a statistical power of .099 with the addition of the second set independent variables in the hierarchal regression model, meaning this value is the observed power for the additional set of independent variables to the overall model. The lack of statistical power to detect true relationships between the independent and dependent variables increased the probability of a type II error, that is, the probability of not finding a statistical relationship between variables when a true relationship exists (Kondrat, 2008). Implications for Social Work research and practice As evidenced by the low pass rate on the knowledge test, JJSPs are not aware of the basic tenets of EBP and there is likely a discrepancy among their definitions of EBP. Efforts to bridge the gap between research and practice should include helping agency leaders understand and believe in the value of EBP in order to integrate it into their agency’s scope of services. But to do that, the inconsistency or discrepancy of definitions for EBP at the academic level must somehow first be resolved. The on-going divide between a top down and bottom up definition of EBP requires a synthesis of the two points of views among academicians since they teach future practitioners. If students enter into the workforce with an ill-defined concept of EBP, they will be less inclined to overcome expected barriers to using EBP and worse, make incorrect assumptions about the effectiveness of potential interventions when using them with clients (Rubin & Parrish, 2007). More research is needed to determine those factors that explain knowledge because lack of understanding and knowledge perpetuates negative attitudes toward EBP. 102
Findings in this study suggest the demographic and degree variables were mostly unrelated to attitude and knowledge. However, some potential explanatory variables worthy of further study should examine the degree to which attitudes toward adopting EBP predict JJSP behaviors such as how often JJSPs take initiative to seek out EBP training and education opportunities, how frequently they use EBPs and how sustainable EBPs are in their practice (Aarons, 2010). From a theoretical standpoint, it is important to understand what factors might moderate the relationship between attitude and behavior (Aarons, 2005, Ajzen & Cote, 2008, Ajzen & Fishbein, 2005) because when JJSPs seek additional education and training in EBP and become more confident along with becoming more knowledgeable, their sense of self-efficacy may actually influence their behaviors, that is, dictate the frequency of their EBP use with clients. Perhaps an assessment of JJSP attitude prior to any new EBP implementation efforts would identity potential organizational weaknesses and help agency leadership develop activities that foster buy-in and generate JJSP support during implementation phases. Implications for further research suggest the necessity of studies that address not only education and training on EBP, but to investigate organizational challenges to inform front-line service delivery in the juvenile justice setting. With appropriate interagency support, replicating this study with a larger sample to include more JJSPs working in community agencies providing prevention programming is recommended. Also, if more CEOs, Executive Directors, and other high-ranking agency leaders among JJSPs could participate, additional analysis of their attitudes, knowledge and perceptions of barriers could add insight into current findings. Results and recommendations from these lines of inquiry are useful as agency leaders who have the ability to make funding and resource allocation decisions would benefit from this type of information. From a practice standpoint, agencies can make efforts to change their organizational culture by offering educational opportunities in the form of workshops, brown bag lunch series, CEUs or include this discussion as part of clinical supervision for licensure. To increase relevancy of EBP and reduce resistance to change, agency outcomes and current services offered should be used to set goals on how to apply evidence-based interventions to specific client situations (Stanhope, Tuchman & Sinclair, 2010). Perhaps as evolution of EBP continues, agencies will encourage staff to participate in research 103
opportunities and research entities would consider more “civilian” participation. Researchers should explore the best ways to involve frontline workers in the research processes because if more JJSPs and social workers in particular were to understand they are more than “passive recipients” of research, this would increase the relevance of research, reduce resistance and allow JJSPs to become both the originators and recipients of research (Marsh & Fisher, 2008; Ragins, 2005).
References Aarons, G.A. (2004). Mental health provider attitudes toward adoption of evidence-based practice: The evidence-based practice attitude scale (EBPAS). Mental Health Services Research, 6(2), 61-74. Aarons, G.A., & Palinkas, L.A. (2007). Implementation of evidence-based practice in child welfare: Social service perspectives. Administration in Policy in Mental Health and Mental Health Services Research, 34, 411-419 Aarons, G.A, McDonald, E.J., Sheehan, A.K., & Walrath-Greene, C.M. (2007). Confirmatory factor analysis of the evidence-based practice attitude scale (EBPAS) in a geographically diverse sample of community mental health providers. Administration and Policy in Mental Health and Mental Health Services Research, 34, 465-469 Aarons, G.A., Sommerfeld, D.H., & Walrath-Greene, C.M. (2009). Evidence-based practice implementation: The impact of public vs. private sector organizational support, provider attitudes, and adoption of evidence-based practice. Implementation Science, 4(1), article no. 83. Aarons, G.A. & Swatzky, A.C. (2006). Organizational culture and climate and mental health provider attitudes toward evidence-based practice. Psychological Services, 3(1), 61-72. Aarons, G.A., Woodbridge, M., & Carmazzi, A. (2002). Examining leadership, organizational climate, and th
service quality in a children’s system of care (pp. 15-18). Paper presented at the 15 Annual Research Conference. A System of Care for Children’s Mental Health: Examining the Research Base, Tampa, FL. Addis, M.E., & Kransnow, A.D. (2000). A national survey of practicing psychologists’ attitudes towards psychotherapy treatment manuals. Journal of Consulting and Clinical Psychology, 68, 331-339. Addis, M.E., Wade, W.A., & Hatgis, C. (1999). Barriers to dissemination of evidence-based practices: Addressing practitioners’ concerns about manual-based psychotherapies. Clinical Psychology: Science and Practice, 6(4), 430-441.
Ajzen, I. (1991). The theory of planned behavior. Organizational Behavior and Human Decision Processes, 50, 179-211. Ajzen, I. (2002). Perceived behavioral control, self-efficacy, locus of control and the theory of planned behavior. Journal of Applied Social Psychology, 32(4), 665-683. Ajzen, I. (2011). The theory of planned behavior: Reactions and reflections. Psychology and Health, 26(9), 1113-1127. Andrews, D.A., Bonta, J., & Wormith, J. (2006). The recent past and new future risk and/or need assessment. Crime and Delinquency, 52(1), 7-27. Andrews, D.A., Zinger, I., Hoge, R.D., Bonta, J., Gendreau, P., & Cullen, F.T. (1990). Does correctional treatment work? A clinically relevant and psychologically informed meta-analysis. Criminology, 28, 369-404. Bandura, A. (1986). Social foundations of thought and action: A social cognitive theory. Englewood Cliffs, NJ: Prentice-Hall. Beasley, B.W., & Woolley, D.C. (2002). Evidence-based medicine knowledge, attitudes and skills of community faculty. Journal of Drug Education, 17, 632-640. Bennett, S., Tooth, L., McKenna, K., Rodger, S., Strong, J., Ziviana, J., Mickan, S., & Gibson, L. (2003). Perceptions of evidence based practice: A survey of Australian occupational therapists. Australian Occupational Therapy Journal, 50, 13-22. Bilsker, D., & Goldner, E.M., (2004). Teaching evidence-based practice: Overcoming barriers. Brief Treatment and Crisis Intervention, 4, 271-275. Bonta, J., & Andrews, D.A. (2007). Risk-need-responsivity model for offender assessment and rehabilitation. Ottawa, Ontario: Public Safety Canada. Boyer, P.S. (1978). Urban Masses and Moral Order in America, 1820-1920. Cambridge, Mass.: Harvard University Press. Byrne, J.M., & Lurigio, A.J. (2009). Separating science from nonsense: Evidence-based research, policy and practice in criminal and juvenile justice settings. Victims and Offenders: An international journal of Evidence-Based Research, Policy and Practice, 4(4), 303-310.
Chambless, D.L., Babich, K., Crits-Christoph, P., Frank, E., Gilson, M., Montgomery, R. & Weinberger, J. (1993). Task force on promotion and dissemination of psychological procedures. A report adopted by the Division, 12. Chambless, D.L., & Crits-Christoph, P. (2005, March). What should be validated? In J. C. Norcross, L.E. Beutler, & R.F. Levant (Eds.), Evidence-based practices in mental health: Debate and dialogue on the fundamental questions (pp. 191-256). Washington, DC: American Psychological Association. Champion, V. L., & Leach, A. (1989). Variables related to research utilization in nursing: An empirical investigation. Journal of Advanced Nursing, 14, 705-710. Chwalisz, K. (2003). Evidence-based practice: A framework for twenty-first-century scientific-practitioner training. The Counseling Psychologist, 31, 497-528. Cooney, S.M., Huser, M., Small, S.A., & O’Conner, C. (2007). Evidence-based programs: An overview. What works, Wisconsin research to practice series, 6. Madison: University of Wisconsin-Extension. Retrieved October 12, 2012 from http://www.uwex.edu/ces/flp/families/whatworks_02.pdf. Corrigan, P.W., Steiner, L, McCracken, S.G., Blaser, B., & Barr, M. (2001). Strategies for disseminating evidence-based practices to staff who treat people with serious mental illness. Psychiatric Services, 52, 1598-1606. Crisp, B.R. (2004). Evidence-based practice and the borders of data in the global information era. Journal of Social Work Education, 40, 73-85. Cullen, F.T. (2005). The twelve people who saved rehabilitation: How the science of criminology made a difference. Criminology, 43, 1-42. Cullen, F.T., & Gendreau, P. (2000). Assessing correctional rehabilitation: Policy, practice, and prospects. In J. Horney (Ed.), NIJ criminal justice 2000: Changes in decision-making and discretion in the criminal justice system (pp. 109-175). Washington, DC: US Department of Justice, National Institute of Justice. Davis, S., Gervin, D., White, G., Williams, A., Taylor, A., McGriff, E. (2013) Bridging the gap between research, evaluation, and evidence-based practice, Journal of Social Work Education, 49(1), 1629.
Dillman, D.A., Smyth, J.D., Christian, L.M. (2009). Internet, mail and mixed mode surveys: The tailored rd
design method (3 ed.). Hoboken, NJ: John Wiley & Sons, Inc. Dillman, D.A. (2007). Mail and Internet surveys: The tailored design method (2
ed.). Hoboken, NJ: John
Wiley & Sons, Inc. Dillman, D.A., Christenson, J.A., Carpenter, E.H., & Brooks, R. (1974). Increasing mail questionnaire response: A four state comparison. American Sociological Review, 39, 744-756. Dowden, C., & Andrews, D. (2004). The importance of staff practice in delivering effective correctional treatment: A meta-analytic review of core correctional practice. International Journal of Offender Therapy and Comparative Criminology, 48(2), 203-214. Dulcan, M.K. (2005). Practitioner perspectives on evidence-based practice. Child Adolescent Psychiatric Clinics of North America, 14, 225-240. Eagly, A.H., & Chaiken, S. (1993). The psychology of attitudes. Orlando, FL: Harcourt Brace Jovanovich College Publishers. Evidence-Based Medicine Working Group. (1992). Evidence-based medicine. A new approach to teaching in the practice of medicine. Jama,268(17), 2420-2425. Fishbein, M., & Ajzen, I. (1975). Belief, attitude, intention, and behavior. An introduction to theory and research. Reading, MA: Addison-Wesley. Fixen, D.L., Naoom, S.F., Blasé, K.A., Friedman, R.M., & Wallace, F. (2005). Implementation Research: A Synthesis of the Literature. (Financed by the William T. Grant Foundation No. FMHI Publication #231). Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network. Flexner, A. (1915). Is social work a profession? Research on Social Work Practice, 11(2), 152-165. Funk, S.G, Champagne, M.T., Wiese, R.A., & Tornquist, E.M. (1991). Barriers to using research finding in practice: The clinician’s perspective. Applied Nursing Research, 4(2), 90-95. Gambrill, E. (2006a). Evidence-based practice and policy: Choices ahead. Research on Social Work Practice, 16(3), 338-357. Gambrill, E. (2006b). Transparency as the route to evidence-informed professional education. Paper presented at the meeting of the Improving the Teaching of Evidence-Based Practice. Austin, TX. 108
Garfield, S.L. (1996). Some problems associated with “validated” forms of psychotherapy. Clinical Psychology: Science and Practice, 3, 218-229. Garvin, C., & Tropman, J. (1992). Social work in contemporary society. Englewood Cliffs: Princeton Hall, Inc. Gellis, Z., & Reid, W.J. (2004). Strengthening evidence-based practice. Brief Treatment and Crisis Intervention, 4(2), 155-165. Gelso, C.J., & Hayes, J.A. (1998). Components of psychotherapy relationship: Their interaction and unfolding during treatment. The Journal of Counseling Psychology, 41, 239-306. Gendall, P., & Healey, B., (2007, May). Alternatives to prepaid monetary incentives in mail surveys. Paper presented at the annual conference of the American Association of Public Opinion Research, Anaheim, CA. Gendreau, P., Smith, P., & French, S.A. (2006). The theory of effective correctional intervention: Empirical status and future directions. In F. T. Cullen, J. P. Wright & K. R. Blevins (Eds.), Taking stock: The status of criminological theory, vol 15: Advances in criminological theory (pp. 419446). New Brunswick, NJ: Transaction. Gibbs, L. (2003). Evidence-based practice for the helping professions: A practical guide with integrated multimedia. Pacific Grove, CA: Brooks/Cole-Thompson Learning. Gibbs, L., & Gambrill, E. (2002). Evidence-Based Practice: Lessons from medicine and the Allied Health Professions. Research on Social Work Practice, 12(3), 452-476. Gioia, D., & Dziadosz, G. (2008). Adoption of evidence-based practices in community mental health: A mixed-method study of practitioner experience. Journal of Community Mental Health, 44, 347357. Glasgow, R.E., Lichtenstein, E., & Marcus, A.C. (2003). Why don’t we see more translation of health promotion research to practice? Rethinking the efficacy-to-effectiveness transition. American Journal of Public Health, 93(8), 1261-1267. Greenwood, P. (2008). Prevention and intervention programs for juvenile offenders. Thefutureofchildren.org, 18, 185-210.
Guerra, N.G., Kim, T. & Boxer, P. (2008). What works: Best practices with juvenile offenders. In R. Hoge, N.G. Guerra, and P. Boxer (Eds.). The juvenile offender: Treatment approaches. New York: Guilford Press. th
Hackney, H.L., & Cormier, L.S., (2005). The Professional Counselor (5 ed.). Toronto: Allyn and Bacon. Hayward, R.S., Wilson, M.C., Tunis, S.R., Bass, E.B., Guyatt, G., & Evidence-Based Medicine Working Group. (2000). Users’ guides to the medical literature: VII. How to use clinical practice guidelines: A. Are the recommendations valid? JAMA, 274(7), 570-574. Henggeler, S.W., & Schoenwald, S.K., (2011). Evidence-based interventions for juvenile justice offender and juvenile justice policies that support them. Social Policy Report, 25(1), downloaded from www.mtfc.com/2011_EB-Interventions_for_Juv_offenders.pdf. Herberlein, T.A., & Baumgartner, R. (1978). Factors affecting response rates to mailed questionnaires: A quantitative analysis of the published literature. American Sociological Review, 43, 447-462. Hoge, M.A., Tondora, J., & Stuart, G.W. (2003). Training in evidence-based practice. Psychiatric Clinics of North America, 26(4), 851-865. Hoagwood, K., Burns, B.J., Kiser, L., Ringeisen, H., Schoenwald, S.K. (2001). Evidence-based practice in child and adolescent mental health services, Psychiatric Services, 52(9), 1179-1189. Holosko, M.J. (2003). Guest editor’s forward. Research on Social Work Practice, 13(3), 265-266. Holosko, M.J., & Leslie, D. (1998). Obstacles to conducting empirically based practice. In J.S. Wodarski & B.A. Thyer (Eds.), Handbook of empirical social work practice: Social problems and practice issues, 2, 433-451. New York: John Wiley & Sons. Horsley, J., Crane, J., & Bingle, J. (1978). Research utilization as an organizational process. Journal of Nursing Administration, 8, 4-6. Howard, M. O., McMillen, C. J., & Pollio, D. E. (2003). Teaching evidence-based practice: Toward a new paradigm for social work education. Research on Social Work Practice, 13(2), 234-259 Howell, J.C. (2009). Preventing and reducing juvenile delinquency: A comprehensive framework (2nd ed.). Thousand Oaks, CA: Sage Publication. Hurlburt, M., & Knapp, P. (2003). The new consumers of evidence-based practice: Reflections of providers and families. Data Matters, Special Issue #6, 21-33. 110
Hutchinson, A.M., & Johnson, L. (2006). Beyond the BARRIERS scale: Commonly reported barriers to research use. Journal of Nursing Administration, 36(4), 189-199. Iles, R., & Davidson, M. (2006). Evidence based practice: A survey of physiotherapists’ current practice. Physiotherapy Research International, 11(2), 93-103. Izzo, R., & Ross, R. (1990). A meta-analysis of rehabilitation programs for juvenile delinquents: A brief report. Criminal Justice and Behavior, 17, 134-142. Jette, D.U., Bacon, K., Batty, C., Carlson, M., Ferland, A., Hemingway, R.D., Hill, J., C., Ogilvie, L., & Volk, D. (2003). Evidence-based practice: Beliefs, attitudes, knowledge, and behaviors of physical therapists. Physical Therapy, 83(9), 786-805. Jones, P.R. & Wyant, B.R. (2007). Target juvenile needs to reduce delinquency. Criminology and Public Policy, 6(4), 763-771. Julian, S. (2011). Juvenile diversion programs in North Carolina: A mixed methods exploratory study. Unpublished Masters of Public Administration, University of North Carolina at Chapel Hill, University of North Carolina at Chapel Hill. Kam, S.E. & Midgley, N. (2006). Exploring ‘clinical judgment’: How do child and adolescent mental health professionals decide whether a young person needs individual therapy? Clinical Child Psychology and Psychiatry, 11(1), 27-44. Katz, M. (1996). In the Shadow of the Poorhouse: A Social History of Welfare in America (2
New York: Basic Books. Kirk, S. A., & Reid, W. J. (2002). Science and social work: A critical appraisal. New York: Columbia University Press. Kondrat, D. (2008). Expanding understanding of mental health recovery: Effects of stigma and working alliance on the quality of life of persons with severe mental disabilities receiving community-based case management services. (Order No. 3313045, The Ohio State University). ProQuest Dissertations and Theses, , 214. Retrieved from http://search.proquest.com/docview/230674495?accountid=14745. (230674495). rd
Krippendorf, K (2013). Content analysis: An introduction to its methodology (3 ed.). Thousand Oaks, CA: Sage Publications, Inc. 111
Levey, S.R. (2010). The use of research by social workers in agency based settings. Yeshiva University. ProQuest Dissertations and Theses, 142. Retrieved from http://search.proquest.com/docview/839309584?accountid=14745. (839309584). Lipsey, M.W. (2009). The primary factors that characterize effective interventions with juvenile offenders: A meta-analytic overview. Victims and Offenders, 4, 124-147. Lipsey, M.W., Howell, J.C., Kelly, M.R., Chapman, G., & Carver, D. (2010). Improving the effectiveness of juvenile justice programs: A new perspective on evidence-based practice. Center for Juvenile Justice Reform, Georgetown University, Washington, DC: Georgetown University Press. Long, K.J., Homesley, L., & Wodarski, J.S. (2007). The role of social workers in the managed health care system: A model of evidence-based practice. In B.A. Thyer & J.S. Wodarski (Eds.), Social work in mental health: A evidence-based approach (pp. 1-28). Hoboken, NJ: John Wiley & Sons. Losel, F., (Ed.). (1995). The efficacy of correctional treatment: A review and synthesis of metaevaluations (What works: Reducing and reoffending-Guidelines from research and practice ed.). New York: John Wiley. Lowenkamp, C.T., & Latessa, E.J. (2005). Evaluation of Ohio's CCA funded programs. Unpublished manuscript. Lowenkamp, C.T., Latessa, E.J., & Smith, P. (2006). Does correctional program quality really matter? The impact of adhering to the principles of effective intervention. Criminology and Public Policy, 5, 575-594. Lowenkamp, C.T., Hubbard, D., Makarios, M.D., & Latessa, E. J. (2009). A quasi-experimental evaluation of thinking for a change: A real world application. Criminal Justice and Behavior, 36(136-146) Lowenkamp, C.T., Pealer, J., Latessa, E.J., & Smith, P. (2006). Adhering to the risk principle: Does it matter for supervision-based programs? Federal Probation, 4, 3-8. Madson, M.B. (2005). Psychometric evaluation of supervisor feedback measure with motivational interviewing: Reliability and validity of the Motivational Interviewing Supervision and Training Scale (MISTS). Unpublished doctoral dissertation, Marquette University.
Mancini, A.D., Moser, L.L., Whitley, R., McHugo, G.J., Bond, G.R., Finnerty, M.T. & Burns, B.J., (2009). Assertive community treatment: Facilitators and barriers to implementation in routine mental health settings. Psychiatric Services, 60(2), 189-195. Marsh, P., & Fisher, M. (2008). The development of problem-solving knowledge for social care practice. British Journal of Social Work, 38(5), 971-987. Mathiesen, S. G., & Hohman, M. (2013). Revalidation of an Evidence-Based Practice Scale for Social Work. Journal Of Social Work Education, 49(3), 451-460. doi:10.1080/10437797.2013.796793 McColl, A., Smith, H., White, P., & Field, J. (1998). General practitioners’ perceptions of the route to evidence based medicine: A questionnaire survey. British Medical Journal, 316, 361-365. Retrieved from: http://bmj.com/cgi/content/full/316/7128/361 McGuire, D. (2006). Attitude and Barriers to evidence-based social work. Unpublished doctoral dissertation, University of Houston. (Publication No. AAT 3207044). McGuire, J., & Priestly, P. (Ed.). (1995). Reviewing 'what works past, present and future' (What Works: Reducing reoffending-Guidelines for research and practice ed.). Chichester, Sussex, England: Wiley. McHugo, G.J., Drake, R.E., Whiteley, R., Bond, G.R., Campbell, K., Rapp, C.A., Goldman, H.H., Lutz, W.J., Finnerty, M.T. (2007). Fidelity outcomes in the national implementing evidence-based practices project. Psychiatric Services, 58(10), 12791284. Mertler, C.A. & Vannatta, R.A. (2005). Advanced and multivariate statistical methods. Glendale, CA: Pyrczak Publishing. Midgley, N. (2009). Editorial: Improvers, adapters and rejecters-the link between ‘evidence-based practice’ and ‘evidence-based practitioners’. Prevention Action, 14(3), 323-327. Miller, J. R., & Messenger, S. R. (1978). Obstacles to applying nursing research findings. The American Journal of Nursing, 78(4), 632-634. Mitchell, P.F. (2010). Evidence-based practice in real-world services for young people with complex needs: New opportunities suggested by recent implementation science. Children and Youth Services Review, 33(2), 207-216.
Morrow-Bradley, C. & Elliot, R. (1986). Utilization of psychotherapy research by practicing psychotherapists. American Psychologist, 41(2), 188-197. Mullen, E. J., & Bacon, W. (2006). Implementation of practice guidelines and evidence-based treatment. In A. R. Roberts & K. R. Yeager (Eds.), Evidence-Based Practice Manual: Research and Outcome Measures in Health and Human Services. (pp. 210-218). New York: Oxford. Mullen, E. J., Bledsoe, S. E., & Bellamy, J. L. (2008). Implementing Evidence-Based Social Work Practice. Research on Social Work Practice, 18(4), 325-338. Mullen, E. J., & Streiner, D. L. (2004). The evidence for and against evidence-based practice. Brief Treatment and Crisis Intervention, 4(2), 111-121. Murray, C. E. (2009). Diffusion of innovation theory: A bridge for the research-practice gap in counseling. Journal of Counseling & Development, 87, 108-116. Myer, H.J., Borgatta, E.F., & Jones, W.C. (1965). Girls at Vocational High-An Experiment in Social Work Intervention. New York, NY: Russell Sage Foundation. National Association of Social Work (1955) National Association of Social Work (2003, April). New advocacy resources: Child welfare workforce; human services workforce; data on children & families. Government Relations Update. Retrieved Feb. 15, 2013 from http://socialworkers.org/advocacy/updates/041803.asp National Institute of Mental Health. (1999). www.nimh.gov. Nelson, T.D. & Steele, R.G. (2007). Predictors of practitioner self-reported use of evidence-based practices: Practitioner training, clinical setting and attitudes toward research. Administration and Policy in Mental Health Services Research. 34, 319-330. Nelson, T.D., Steele, R.G., & Mize, J.A. (2006). Practitioner attitudes toward evidence-based practice: Themes and challenges, Administration and Policy in Mental Health Services Research, 33, 398409. Norcross, J.C. (2002). Empirically supported therapy relationships. In J.C. Norcross (Ed.), Psychotherapy relationships that work: Therapist contribution and responsiveness to patients (pp. 3-16). New York: Oxford University Press, Inc.
Panzano, P., & Herman, L. (2005). Developing and sustaining evidence-based systems of mental health services. Evidence-Based Mental Health Practice, 167-187. Parrish, D. E. (2008). Evaluation of the impact of a full-day continuing education training on how practitioners learn about, view, and engage in evidence-based practice. The University of Texas at Austin. ProQuest Dissertations and Theses, 180. Retrieved from http://search.proquest.com/docview/276160212?accoountid=14745. (276160212). Patel, S.H. (2010). An investigation of counselor educators’ attitudes towards evidence-based practices and perceived barriers to the incorporation of evidence-based practices in counselor education curricula. University of Central Florida. ProQuest Dissertations and Theses, 185. Retrieved from http://search.proquest.com/docview/733885236?accoundid=14745. (733885236). Patterson, C.H. (1984). Empathy, warmth, and genuineness in psychotherapy: A review of reviews. Psychotherapy, 21, 431-438. Pope, N.D., Rollins, L., Chaumba, J., & Risler, E. (2011). Evidence-based practice knowledge and utilization among social workers. Journal of Evidence-Based Social Work, 8(4), 349-368. th
Popple, P., & Leighninger, L. (2002). Social Work, Social Welfare, and American Society (5 ed.). Boston: Allyn and Bacon. Prochaska, J.O., & Norcross, J.C. (1983). Contemporary psychotherapists: A national survey of characteristics, practices, orientations, and attitudes. Psychotherapy: Theory, Research and Practice, 20(2), 161-173. Proctor, E. K. (2006). Implementing evidence-based practice in social work education: Principles, strategies, and partnerships. Paper presented at the Symposium on Improving the Teaching of Evidence-Based Practice: University of Texas at Austin School of Social Work. Reid, W.J. (2002). Knowledge for direct social work practice: An analysis of trends. Social Service Review, 76(1), 6-33. Retsas, A. (2000). Barriers to using research evidence in nursing practice. Journal of Advanced Nursing, 31(3), 599-606. Reynolds, S. (2000). Evidence based practice and psychotherapy research. Journal of Mental Health, 9(3), 257-266. 115
Rice, K., Hwang, J., Abrefa-Gyan, T., & Powell, K. (2010). Evidence-based practice questionnaire: A confirmatory factor analysis in a social work sample. Advances in Social Work, 11(2), 158-173. Richmond, M. (1969). Friendly Visiting among the Poor: A Handbook for Charity Workers. New York: Macmillian, 1899. Reprint, Montclair, NJ: Patterson Smith. Roberts, A.R. (Ed.). (2004). Juvenile justice sourcebook: Past, present and future. Oxford University Press, USA. th
Rogers, E. M. (2003). Diffusion of Innovations (5 ed.). New York: Free Press. Rosen, A., & Proctor, E. K. (Eds.). (2004). Developing practice guidelines for social work intervention. New York, NY: Columbia University Press. Rosenblatt, A. (1968). The practitioner’s use and evaluation of research. Social Work, 13, 53-59. Rosenheck, R.A. (2001). Organizational process: A missing link between research and practice. Psychiatric Services, 52(12), 1607-1612. Rubin, A. & Babbie, E. (2009). Essential research methods for social work (2
ed.). Pacific Grove, CA:
Brooks/Cole. Rubin, A. & Babbie, E.R. (2012). Brooks/Cole Empowerment Series: Essential Research Methods for Social Work. Brooks/Cole Publishing Company. Rubin, A., & Parrish, D. (2007a). Challenges to the future of evidence-based practice in social work. Journal of Social Work Education, 43, 405-428. Rubin, A., & Parrish, D. (2007b). Problematic phrases in the conclusion of published outcome studies: Implications for evidence-based practice. Research on Social Work Practice, 17(3), 334-347 Rubin, A., & Parrish, D. (2007c). Views of Evidence-Based Practice Among Faculty in Master of Social Work Programs: A National Survey. Research on Social Work Practice, 17(3), 110-122. Sabus, C.H. (2007). The impact of modeling evidence-based practice on physical therapist and student knowledge and clinical practice. Unpublished doctoral dissertation, The University of Kansas. Sackett, D.L. (1996). Evidence-based Medicine – What it is and what it isn’t. Retrieved from: http://www.cebm.net/ebm_is_isnt.asp. Sackett, D.L., Straus, S.E., Richardson, W.S., Rosenberg, W., & Haynes, R.B. (2000). Evidence-based medicine: How to practice and teach EBP (2
ed.). Edinburgh: Churchill Livingstone.
Sackett, D.L., Rosenberg, W.M.C., Gray, J.A.M., Haynes, R.B., & Richardson W.S. (1996). Evidencebased medicine: What it is and what it isn’t, BMJ, 312(7023), 71-72. Salloum, A. (2008). Group therapy for children after homicide and violence. Research on Social Work Practice, 18(3), 198-211. doi: 10.1177/1049731507307808 Saunders, J.A., Morrow-Howell, N., Spitznagal, E., Dore, P. Proctor, E.K. Pescarino, R. (2006). Imputing missing data: A comparison of methods for social work researchers. Social Work Research, 30 (1), 19-31. Sanderson, W.C. (2002). Why we need evidence-based psychotherapy practice guidelines. Medscape General Medicine, 4(4). Schoenwald, S.K. & Hoagwood, K. (2001). Effectiveness, transportability, and dissemination of interventions: What matters when? Psychiatric Services, 52(9), 1190-1197. Schon, D. A. (1983). The reflective practitioner. How professionals think in action. New York: Basic Books. Shlonsky, A., & Gibbs, L. (2006). Will the real evidence-based practice please stand up? Teaching the process of evidence-based practice to the helping professions. In A.R. Roberts & K.R. Yeager (Eds.) Foundations of evidence-based social work practice (pp. 103-121). New York: Oxford University Press. Soper, D.S. (2014). A-priori Sample Size Calculator for Multiple Regression [Software]. Available from http://www.danielsoper.com/statcalc Stahmer, A.C., & Aarons, G.A. (2009). Attitudes toward adoption of evidence-based practices: A comparison of autism early intervention providers and children’s mental health. Psychological Services, 6(3), 223-234. Stanhope, V., Tuchman, E., & Sinclair, W. (2011). The implementation of mental health evidence based practices from the educator, clinician and researcher perspective. Journal of Clinical Social Work, 39(4), 369-378. Staudt, M. (2005). Harold E. Briggs and Tina L. Rzepnicki, eds., Using Evidence in Social Work Practice: Behavioral Perspectives Using Evidence in Social Work Practice: Behavioral Perspectives. Edited
by Harold E. Briggs and Tina L. Rzepnicki. Chicago: Lyceum Books, 2004. Pp. 400. The Social Service Review, (4), 729. doi:10.1086/498921 Straus, S.E., Richardson, W.S., Glasziou, P., & Haynes, R.B. (2010). Evidence-based medicine: How to th
practice and team EMB? (4 ed.). NY: Churchill-Livingstone Straus, S. E., Richardson, W. S., Glasziou, P., & Haynes, R. B. (2005). Evidence-based medicine: How to rd
practice and teach EBM (3 ed.). Edinburgh: Elsevier Churchill-Livingstone. th
Tabachnick, B.G., & Fidell, L.S. (2007). Using Multivariate Statistics (5 ed.). Boston: Pearson Education. Thyer, B.A. (2004). What is evidence-based practice? Brief Treatment and Crisis Intervention, 4(2), 167176. Thyer, B.A., & Pignotti, M. (2011). Evidence-based practices do not exist. Journal of Clinical Social Work, 39, 328-333. Torrey, W.C., & Gorman, D.W. (2005). Promoting the implementation of practices that are supported by research: The national implementing evidence-based practice project. Child and Adolescent Psychiatric Clinics of North America. Upton, D.R., & Upton P.M. (2005a). Development of an evidence-based practice questionnaire for nurses. Journal of Advanced Nursing, 54(4), 454-458. Upton, D.R., & Upton, P.M. (2005b). Knowledge and use of evidence-based practice of GPs and hospital doctors. Journal of Evaluation in Clinical Practice, 12(3), 376-384. Vallino-Napoli, L.D. & Reilly, S. (2004). Evidence-based health care: A survey of speech pathology practice. International Journal of Speech-Language Pathology, 6(2), 107-112. Wade, M., & Schneberger, S. (2005). Social Network Theory. Retrieved from http//www.istheory.yorku.ca/socialnetworktheory.htm Waehler, C.A., Kalodner, C.R., Wampold, B.E., & Lichtenberg, J.W. (2000). Empirically supported treatments (ESTs) in perspective: Implications for counseling psychology training. The Counseling Psychologist, 28, 657-671. Walrath, C.M., Sheehan, A.K., Holden, E.W., Hernandez, M., & Blau, G. (2006). Evidence-based treatments in the field: A brief report on provider knowledge, implementation, and practice. The Journal of Behavioral Health Services and Research, 33(2), 244-253. 118
Weissman, M.M., & Sanderson, W.C. (2002). Promises and problems in modern psychotherapy: The need for increased training in evidence-based treatments. In M. Hager (Ed.), Modern psychiatry: Challenges in educating health professionals to meet new needs (pp. 132-165). New York: Josiah Macy, Jr., Foundation. Whiston, S.C., & Coker, J.K. (2000). Reconstructing clinical training: Implications from research. Counselor Education & Supervision, 39(4), 228-254. Woody, J.D., D’Souza, H.J., & Dartman, R. (2006). Evidence-based practices in health care: Social work possibilities. Health & Social Work, 29(4), 259-261. Yeager, K., & Roberts, A. R. (2006). Foundations of Evidence-based Social Work Practice. Oxford: Oxford University Press.
Dear Juvenile Justice Service Professional: I am a doctoral student conducting a research study to learn more about your attitudes, your level of knowledge regarding Evidence-Based Practice (EBP) and your perceptions of barriers to using EBP in your daily work. This survey is called: Evidence-Based Practice Attitudes, Knowledge and Perceptions of Barriers Among Juvenile Justice Professionals (IRB Study #: Pro00013478). As a juvenile justice service professional in Florida, you have valuable information concerning this topic and have been selected to participate in this survey because you are an employee of an agency contracted with the Florida Department of Juvenile Justice (DJJ) to provide services to youth. Given contractual obligations to DJJ, your agency may use different types of evidence-based practices and/or interventions such as Motivational Interviewing (MI), the Positive Action Change Tool (PACT), or any of the many interventions listed in the Florida Sourcebook for Juvenile Justice Interventions; such as, MultiSystemic Therapy (MST), Brief Strategic Family Therapy (BSFT), Thinking For A Change (T4C), Aggression Replacement Therapy (ART), TruThought, and Girls Circle. I am asking your help to complete a questionnaire even if you do not use these types of programs on a daily basis as I am interested in your attitudes and perceptions toward using them, not whether you actually currently use them. The questionnaire should take no more than 30 minutes to complete. Your participation in this project is completely voluntary. If you participate, you may choose not to answer any given item, and discontinue your participation at any time. Your informed consent is implied upon completion and return of the questionnaire. Since all responses are anonymous, please do not write your name or address anywhere on the questionnaire or return envelope. In approximately one week, you will receive a postcard reminder to return the questionnaire. Items on the questionnaire are designed to capture your opinion toward EBP and your perception of barriers to using EBP in your daily work. There are also 10 “knowledge” questions that gauge your level of understanding of EBP concepts and 9 additional background questions. Please remember there is no consequence to answering any question incorrectly and your responses will not be used to make comparisons between individuals or agencies. Data will not be collected on which agency you are affiliated with; therefore, individual or agency level data and/or results will not be reported. Your individual responses to the survey are confidential. Again, only aggregated results will be reported. The more that is learned about the different attitudes and perceptions of barriers among Florida juvenile justice service providers regarding Evidence-Based Practices, the more opportunity for this state to develop policies and initiatives to support future use of evidence-based practices in the juvenile justice field. Your participation in this survey may benefit other people now or in the future as I hope to offer 120
recommendations to the Florida Juvenile Justice Association based on what is learned from this research study. The results of this survey may also be presented at national and state conferences and I may publish what I learn from this survey. However, if I do this I will not include your name and I will not publish anything that would let people know who you are. This research is considered to be minimal risk. That means the risks associated with this research study are the same as what you may face doing your daily activities. A breach of confidentiality is the only possible risk. But as you will not include any identifying information on your returned survey, a breach is unlikely to happen. I hope that you will choose to participate in this survey. A high return rate is needed to assure that my data are representative. You will receive no payment or other compensation for taking part in this survey. However, a $1.00 Florida lottery ticket will be included with each questionnaire that is yours to keep whether you choose to complete the questionnaire or not. There will be no financial cost to you as a result of being in this survey. Your survey records will be kept private and confidential. But certain people may need to see these records. By law, anyone who looks at your records must keep them completely confidential. The only people allowed to see these records are: •
The research team consisting of: myself as the Principal Investigator and my faculty advisor.
Certain government and university people who need to know more about my research project. For example, individuals who provide oversight on this survey may need to look at your records. This is done to make sure I am doing the research in the right way. They also need to make sure I am protecting your rights and your safety.
Any agency of the federal, state, or local government that regulates this research. This includes the Food and Drug Administration (FDA), Florida Department of Health, and the Department of Health and Human Services (DHHS) and the Office for Human Research Protection (OHRP).
The USF Institutional Review Board (IRB) and its related staff, who have oversight responsibilities for this study, staff in the USF Office of Research and Innovation, USF Division of Research Integrity and Compliance, and other USF offices that oversee this research.
Again, you should only take part in this study if you want to volunteer. You should not feel that there is any pressure to take part in the study. You are free to participate in this survey or discontinue the survey at any time and choose not submit it. There is no penalty or loss of benefits you are entitled to receive if you stop taking part in this research study. Participating in this survey will not in any way endanger your employment. That is, your job(s) will not be in jeopardy if you choose to participate or not participate. If you have questions, concerns or complaints about this study, or experience an unanticipated problem, please call me: Esther Chao McKee, LCSW at (512) 619-9526. The distribution of the questionnaire has been approved by the University of South Florida Institutional Review Board for the Protection of Human Subjects. If you have questions about your rights as a participant in this study, general questions, or have complaints, concerns or issues you want to discuss with someone outside the research team, please call the USF IRB at (813) 974-5638. Thank you for your time and cooperation. Sincerely, Esther Chao McKee, LCSW Student Researcher David Kondrat, Ph.D. Faculty Advisor
EVIDENCE-BASED PRACTICE ATTITUDE SCALE (AARONS, 2004; JETTE ET AL., 2003) The following items ask about your feelings toward using evidence-based practices/interventions. Please read each section carefully and follow the instructions as indicated. Please remember there is no right or wrong answer. Scale of Agreement For items 1-6: Circle the number to the right that best represents how much you agree or disagree with each item.
Don’t Strongly Disagree Neither Agree Strongly Know/ Disagree Agree nor Agree Not Disagree Sure
Using evidence-based practices is necessary in my daily work
Literature and research findings are useful in my daily work
I know better than academic researchers how to care for my clients.
I am willing to use new and different types of evidence-based practices/interventions developed by researchers.
My professional/clinical experience is more important than using evidencebased practices/interventions.
I would try a new evidence-based practice/intervention even it were very different from what I am used to doing.
For items 7-8: If you received training in an evidence-based practice or intervention, please circle the number to the right that best represents how much you agree or disagree with each item.
Don’t Neither Strongly Strongly Know/ Disagree Agree nor Agree Disagree Agree Not Disagree Sure
It was intuitively appealing to you?
It “made sense” to you?
Research reveals common barriers to implementing evidence-based practices exist across different professions. For items 9-17, circle the number to the right that best represents how much you agree or disagree with each item.
Don’t Know/ Not Sure
12. Inexperience in critically appraising literature is a barrier for me
13. Lack of applicability of the literature to my client population is a barrier for me
14. Lack of experience in applying research findings to individual clients with unique characteristics is a barrier for me
15. Lack of understanding of statistical analysis is a barrier for me
16. Lack of collective support among my colleagues in my agency/facility is a barrier for me
17. Lack of interest is a barrier for me
Neither Agree nor Disagree
10. Lack of informational resources is a barrier for me
11. Lack of research skills is a barrier for me
Insufficient time is a barrier for me
18. Please list the top 3 items you think are barriers to implementing evidence-based practice(s) in your current work with clients: 1. 2. 3.
For items 19-21, If you received training in an evidence-based practice or intervention, please circle the number that corresponds to Yes, No, or Don’t know/Not sure.
Don’t know/ Not sure
19. It was required by your supervisor?
20. It was required by your agency?
21. It was required by the Florida Department of Juvenile Justice?
EVIDENCE-BASED PRACTICE KNOWLEDGE QUESTIONS (PARRISH, 2008)
For the next section, please read each statement and circle the response that best represents your answer. 1. Practitioners who engage in evidence-based practice will: 1. 2. 3. 4. 5.
Rely most heavily on the practice wisdom of supervisors and experts. Scientifically monitor client outcomes. Utilize one major theoretical framework and intervention strategy with most clients. Only select interventions that are on the list of empirically supported treatments. Answers B and D only
2. The best scientific evidence might: 1. 2. 3. 4. 5.
Be only good enough to guide practice in a tentative manner. Not be relevant to a client. Be an indication of what not to do. Answers A and B only All of the above
3. Which of the following statements is/are true about applying the intervention supported by the best research evidence? 1. Because it is the most effective, you should implement this treatment the best that you can, even if you cannot get training or supervision. 2. Because it is the most effective, you should make the clinical decision to select this intervention for your client. 3. If you decide to implement this intervention, you should monitor client progress with it. 4. All of the above 4. If it is determined there is strong evidence of the effectiveness of an intervention in an outcome evaluation, it is important to consider: 1. How the outcome was measured. 2. Whether a control group was used. 3. Whether participants across different treatment conditions are really comparable to begin with. 4. All of the above 5. A good systematic review: 1. Fully discloses its search and inclusion strategy. 2. Assesses the quality of selected outcome studies. 3. Is done by someone within the field who has a vested interest in the findings of the review. 4. All of the above 5. Answers A and B only
6. A hierarchy of evidence exists in the scientific world, which is in the most accurate order from highest to lowest to determine an effective intervention (assuming that all studies are implemented perfectly, with good measurement and are unbiased)? 1. 2. 3. 4.
Experimental studies, quasi-experimental studies, meta-analyses, qualitative studies Quasi-experimental studies, meta-analyses, qualitative studies, experimental studies Meta-analyses, quasi-experimental studies, experimental studies, qualitative studies Meta-analyses, experimental studies, quasi-experimental studies, qualitative studies
7. If a study has been published and it appears in a reputable literature base, you can be sure it has been conducted objectively and rigorously. 1. True 2. False 8. When measuring client outcomes, it might be okay to develop your own individualized rating scale instead of using an instrument that researchers have validated. 1. True 2. False 9. Evidence-based practice questions are restricted to asking about the effectiveness of interventions, programs or policies. 1. True 2. False 10. Evidence-based practice includes professional judgment and behavior that is guided by the client system over time. It also includes evaluating the practice to determine if the predicted results have been attained as a direct consequence of the practitioner’s actions. 1. True 2. False
In this final section, please read each question and circle the response that best represents your answer or fill in the blank with your answer. 1. What is your age? __________ 2. What is your gender? 1. Male 2. Female 3. Do you hold a practice license? 1. Yes 2. No 4. How many years have you been working in the juvenile justice field? (Please list number of years since obtaining your highest educational degree)___________ 5. What is your highest completed educational degree? 1. 2. 3. 4. 5. 6.
High School Associate’s Degree Bachelor’s Degree Master’s Degree Ph.D., Psy.D., Ed.D. or MD Other: ________________________
6. What discipline did you major in? 1. 2. 3. 4. 5. 6. 7.
Social Work Criminology/Criminal Justice Psychology Sociology Rehab Counseling Education Other: ________________________
7. What is your job title? 1. Case Manager 2. Clinical Director 3. Counselor (Intake, Community, Transition Services etc.) 4. Family Service Worker 5. Mental Health worker (licensed or non-licensed) 6. Program Coordinator 7. Program Director 8. Shift Supervisor 9. Social worker (licensed or non-licensed) 10. Therapist (licensed or non-licensed) 11. Youth Specialist
8. What type of agency do you work in? 1. 2. 3. 4. 5. 6. 7.
Residential Detention Day Treatment Probation Diversion Prevention Other: ___________________________________________________________
Thank you for completing this survey. Your participation and honesty are greatly appreciated! If you have any questions or feedback for me that you would like to submit confidentially, please provide it below. If you need additional information or would like to contact me about this survey, please email me at [email protected]
____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________
9/9/2013 Esther Chao McKee, LCSW University of South Florida School of Social Work 4202 E. Fowler Avenue Tampa, Florida 33620 RE: Expedited Approval for Initial Review IRB#: Pro00013478 Title: Evidence-Based Practice Attitudes, Knowledge and Perceptions of Barriers Among Juvenile Justice Service Professionals Study Approval Period: 9/9/2013 to 9/9/2014 Dear Ms. McKee: On 9/9/2013, the Institutional Review Board (IRB) reviewed and APPROVED the above application and all documents outlined below. Approved Item(s): Protocol Document(s): EBP Among Juvenile Justice Service Professionals Accepted Item(s) Consent/Assent Document(s): Cover Letter Pre-notice Email It was the determination of the IRB that your study qualified for expedited review which includes activities that (1) present no more than minimal risk to human subjects, and (2) involve only procedures listed in one or more of the categories outlined below. The IRB may review research through the expedited review procedure authorized by 45CFR46.110 and 21 CFR 56.110. The research proposed in this study is categorized under the following expedited review category: (7) Research on individual or group characteristics or behavior (including, but not limited to, research on perception, cognition, motivation, identity, language, communication, cultural beliefs or practices, and social behavior) or research employing survey, interview, oral history, focus group, program evaluation, human factors evaluation, or quality assurance methodologies. Your study qualifies for a waiver of the requirements for the documentation of informed consent as 128
outlined in the federal regulations at 45CFR46.117(c) which states that an IRB may waive the requirement for the investigator to obtain a signed consent form for some or all subjects. As the principal investigator of this study, it is your responsibility to conduct this study in accordance with IRB policies and procedures and as approved by the IRB. Any changes to the approved research must be submitted to the IRB for review and approval by an amendment. We appreciate your dedication to the ethical conduct of human subject research at the University of South Florida and your continued commitment to human research protections. If you have any questions regarding this matter, please call 813-974-5638. Sincerely, Kristen Salomon, Ph.D., Vice Chairperson USF Institutional Review Board