Evidence-based practice questionnaire for physical therapists : Portuguese translation , adaptation , validity , and reliability

| INTRODUCTION: Evidence-Based Practice (EBP) is becoming increasingly important in Physical Therapy (PT). For proper designing, implementing, disseminating and evaluating EBP in PT, a valid and reliable questionnaire measuring attitudes, knowledge, behavior, prerequisites, and barriers related to EBP and guidelines is needed. One questionnaire that could be used to collect this information is the “EBP Questionnaire”, developed by Jette et al. 2003. However, to our knowledge, no Portuguese version or published study with the Portuguese PT population was performed using this questionnaire. OBJECTIVE: The aim of this study was to translate, cross-culturally adapt and validate the “EBP Questionnaire” to European Portuguese and for the PT population. MATERIAL AND METHODS: A draft version was pilot tested for content validity (n=17), and a revised version was tested for test-retest reliability (n=72). The percentage of agreement and the Kappa coefficients between the 2 tests were analyzed. Additionally, the internal consistency was calculated. RESULTS: The preliminary final version of the European Portuguese EBP questionnaire was well accepted (only the items 22, 23, 45-51 needed to be reviewed). The mean average percentage of agreement was 82% (ranged 58–97%), and the Kappa coefficients were 0.658 (ranged 0.336–0.844). In the internal consistency, the mean average Cronbach’s α coefficients were 0.665 (ranged 0.365–0.879). CONCLUSION: The results suggested that this questionnaire can be a useful instrument for measuring self-reported beliefs, attitudes, knowledge, and behaviors related to EBP in the Portuguese PT's context.


Introduction
Access to proper health-care is a fundamental human right 1 . Globally, billions of dollars are spent each year in both public and private sectors on biomedical, clinical, health services research, undergraduate healthcare professional training, continuing professional development, quality improvement, patient safety, and risk management 2 . So, it is expected that healthcare systems, organizations, and professionals use the best practice available to treat patients. To achieve the best practice possible, it is essential to include 3 : firstly, the patient's clinical, emotional, psychological, social and physical circumstances in order to establish what is wrong and what treatment options are available; secondly, this factors to be tempered by research evidence concerning the efficacy, effectiveness, and efficiency of the interventions; thirdly, given the likely consequences associated with each option, the clinician must consider the patient's preferences and likely actions (in terms of what interventions she or he is ready and able to accept); fourthly, what are the available resources for accommodating patient's needs; and finally, clinical expertise is needed to bring these considerations together and recommend the treatment that the patient is able to accept. When all of those considerations are successfully balanced a paradigm called evidence-based practice (EBP) could be achieved.
Originally developed in the 80's, EBP can be defined as the conscientious, explicit and judicious use of current relevant available evidence combined with the healthcare providers' clinical expertise and the patients' preferences, guiding clinical decisions about patients' care 4,7 . The concept of EBP marks a shift among health-care professionals from a traditional emphasis on ''eminence-based'' practice (actions based on the opinions to guide clinical practice) to a more evidence practice (emphasis on data-based, systematic reviewed, clinically relevant studies and research) 8,9 . The rise of EBP was prompted in part by the existence of unexplained wide variations in clinical practice patterns, the poor uptake of therapies of known effectiveness, and the use of technologies/interventions that were known to be ineffective, by several health-care professionals, such as Physical Therapists (PT) 10,11 .
To obtain the main variables and relationships associated with EBP, for designing interventions to support the implementation of EBP, to evaluate if EBP changes have been successfully implemented, to measure and compare relevant EBP outcomes, and constantly increase the quality-of-care, PT's beliefs, attitudes, knowledge and behaviors about EBP have to be studied. One instrument that can be used to gather this information is the "Evidence-Based Practice Questionnaire", developed originally in 2003 by Jette et al. 8 . This questionnaire has already been used in several studies 8,12,19 and has been successfully translated to Spanish 20 and Swedish 21 . However, to our knowledge, no Portuguese version or published study with the Portuguese PT population were performed using this questionnaire. So, the aim of this study is to translate, cross-culturally adapt and validate Jette et al. 8

questionnaire to European
Portuguese and for the Portuguese PT population.

Methods
The questionnaire development process was conducted by a 4 member expert panel. Formal permission to use and adapt the questionnaire to survey Portuguese PTs was granted from its developers. Translation, cross-cultural adaptation and validation were performed in several stages according to established guidelines 22,24 .

Description of the Evidence-Based Practice Questionnaire
The questionnaire is self-administered and have a total of 51 items, divided into 4 main sections 8 : personal attitudes toward, use of, and perceived benefits and limitations of EBP (14 items); personal use and understanding of clinical practice guidelines (6 items); availability of resources to access information and personal skills in using those resources (12 items); and demographic information (19 items). The questionnaire explores the respondents' attitudes and beliefs about EBP (survey items 1, 2, 4, and 6-11); interest in and motivation to engage in EBP (survey items 3 and 5); educational background, knowledge and skills related to accessing and interpreting information (survey items [25][26][27][28][29][30][31]; level of attention to and use of the literature (survey items [12][13][14]; access to and availability of information to promote EBP (survey items 18, 19, and 21-24); access to practice guidelines (survey items [15][16][17][18][19][20]; their perceived barriers to using evidence in practice (survey item 32); and demographic information (survey items 33-51) 8 . Responses to most items were addressed using a 5-point Likert scale with "Strongly Disagree", "Disagree", "Neutral", "Agree" and "Strongly Agree" 8 . Other items that collect data on access to information use a dichotomous "Yes" or "No" answer 8 . The questions about the understanding of scientific terms related to EBP require multiple choice answers with 3 possible choices 8 : "Understand Completely", "Understand Somewhat" or "Do Not Understand". Also, the questions linked to the barriers related to the use of EBP, require multiple choice answers with a 1 to 3 ranking scheme (1 as the most important barrier) 8 .

Questionnaire Development and Cross-Cultural Adaptation
The original questionnaire version was first translated and cross-culturally adapted from English to Portuguese independently by 2 translators who were native Portuguese speakers and had English as their second language. The two obtained translations were discussed in a first consensus panel and synthesized into one Portuguese preliminary version. Also, in this first consensus, the first preliminary version of the questionnaire was analyzed to determine whether it covered all aspects of the content or domains intended to be measured and contained no irrelevant items. Then, this consensus version was translated back to English by another translator (one English native professional translator) without prior knowledge of the original version. The Portuguese preliminary version and the back translation were compared with the questionnaire's original version and discussed in a second consensus panel to achieve a second Portuguese preliminary version. Next, this second Portuguese preliminary version was evaluated by 2 health care experts to comment on the quality of the translation and suggest improvements to the current version. The suggestions gathered were discussed in a third consensus panel and synthesized into a preliminary Portuguese final version.

Validity
Content's validity was established in consensus discussions by the expert panel members. The preliminary Portuguese final version was pilot tested by 17 PTs who completed the questionnaire and were asked to, through individual interviews, comment on the items, instructions and response options in: clarity; length; readability; comprehensiveness; understandability; redundancy; relevance; included all the expected concepts; completion time; and whether they were reluctant to answer any of the questions. The PTs' comments were reviewed and discussed in a fourth consensus panel, which analyzed the possibility of rephrasing, changing of the items' order and/or adding clarifications. Also, responses from the pilot testing were analyzed with regard to missing responses and distribution patterns. After, the information was synthesized into the Portuguese final version of the questionnaire.

Reliability
For assessment of test-retest reliability, the Portuguese final version of the questionnaire was distributed to PTs working in health care units. A sample of 122 PTs comprising a strategic mix of sex, age, and other demographic, working and educational factors was used. The questionnaire was administered on 2 occasions, separated by 2-weeks interval. The time interval was chosen to minimize the probability of occurrence of relevant changes in the PT work context. To participate in the questionnaire, the PTs had to be able to read and write Portuguese; need to have a valid PT working license; and work or have worked (in the last 6 months) as a PT in Portugal. The PTs were excluded of participation if: have another profession than PT; do not work as a PT in Portugal; be a PT bachelor student. To decrease the risk of bias, the participating PTs had no personal relationship with the expert panel members. Before filling the questionnaire, in the first page, information of the questionnaire purpose and context; the data protection rights and how the results will be used; the criteria for selecting the participants and the reasons for non-participation; and the possibility to terminate the questionnaire at any time was given. Additionally, this study followed the basic ethical principles set by the Declaration of Helsinki and received prior approval from the Ethic Commission (CEFADE24-19). All PTs signed a written informed consent, and received no compensation. Data collected was analyzed anonymously and confidentially, used only for statistical information in an academic environment. As the questionnaire is anonymous, in the first page a serial number was created to recognize the PT for the assessment of the test-retest.
The test-retest reliability was analyzed for the individual items in the questionnaire. Also, the sub-group items were evaluated in their internal consistency. Demographic and workplace data were not included in the analysis. After the data analysis, all items with poor reliability were revised in further consensus discussions by the expert panel members, for additional clarity and specificity.

Data Analysis
Data from a psychometric test was analyzed with respect to response distribution. Data quality was assessed with an analysis for missing data. Test-retest reliability for ordinal data was analyzed in terms of the percentage of agreement (PA) between the 2 test occasions with the Svensson method 25 . Items with poor agreement, defined as a PA of less than 60%, were revised. Also, systematic and individual disagreements were calculated. These calculations included systematic relative disagreement in position (RP) and in concentration (RC). Values for RP and RC can range from -1 to 1, with a value closed to 0 representing a small systematic disagreement (i.e., a high level of reliability) of the analyzed item. These were considered statistical significant when they did not include the value 0. Additionally, the so-called relative rank variance (RV), was calculated. Values for RV can range from 0 to 1, with a low RV value representing a small individual disagreement (i.e., a high level of reliability), attributing values < 0.1 as negligible. The strength of agreement, of the ordinal and nominal items was calculated by the linear weighted and unweighted Kappa coefficients, respectively. The coefficients were interpreted following the Ladis et al. 26 table range values (< 0.00= Poor; 0.00-0.20 = Slight; 0.21-0.40 = Fair; 0.41-0.60 = Moderate; 0.61-0.80 = Substantial; 0.81-1.00 = Almost Perfect). For Svensson's method and for the Kappa coefficients, 95% confidence intervals (CI) were used. The internal consistency of the questionnaire was calculated through the Cronbach's α coefficient 27 .
The calculations for the Svensson

Cross-Cultural Adaptation and Validation
The preliminary final version of the European Portuguese EBP questionnaire was well accepted in the pre-test. The questionnaire mean average conclusion time, was 10:38 minutes and the interviews duration was 14:24 minutes. Overall, the majority of the PTs that participate in the preliminary version were females, in the 20-29 years age group, with less than 5 years of PT working license and a bacalaureano degree. Furthermore, most of the PT worked in private outpatient units, in an urban context. All the questions and response options were considered satisfactorily understandable by the PTs, where only small changes in semantics and syntax of the written Portuguese were made. Even so, some items and response options were removed or adapted to better suit the Portuguese context. The questions removed were the original items 37 and 40, as they do not suit the PT working context in Portugal. Regarding the adaptations, there were several items that had some minor revisions and adaptations, yet it is necessary to highlight the items 22 and 23 where the most significant modifications occurred. Specifically, the original "… databases and the Internet…" to electronic databases (in Portuguese -bases de dados eletrónicas) were changed since, as it was originally written, it was a "double-barreled question" and it should be avoided 29 . Also, some adaptations in response options in the original items 45 to 51 occurred. From those, the most significant adaptations occurred in the items 45 and 46, where the option 0 was added and the options "<5" and "5-10" were changed to 1-5 and 6-10, respectively. It was opted to add the option 0 and adapt the others since, as the response options were originally presented, they were not a sensitive enough scale to the question 29 . It is expected that a PT that not treat patients has different behaviors and attitudes regarding EBP, comparing to others that treat at least one (treating the patient may lead to a greater search for evidence for the specific case). In the same way, it is expected that a PT that works alone has different behaviors and attitudes regarding EBP, comparing to others that work with at least one co-worker (one of the most common methods of evidence divulgation is through professional social diffusion 30 ). In the other referenced items, the response options were changed to better suit the Portuguese sociodemographic and working class context (for example, in the original item 49, it was chosen to change the open-ended response option to a close-ended response option -as open-ended questions should be avoid 31 -providing the list of Portuguese districts). Furthermore, as already occurred in the Spanish translation 20 and indicated as a possible limitation of the original questionnaire 8 , it was decided to reduce the original 5-point Likert scale to a 4-point Likert scale, extinguishing the "Neutral" option, in an attempt to avoid the central tendency 31 .

Reliability
Of the original 122 PTs only 72 (59%) were included in the reliability test ( Figure 1).

Figure 1. Flow Diagram of the PTs' Recruitment and Retention
The majority of the PTs were young females (59.7% in the 20-29 years age group), with a valid PT working license obtained less than 5 years ago (45.8%). Furthermore, as expected, most of them only had the baccalaureate degree (77.8%) and were not clinical instructors for PT students (62.5%). Regarding their practice, most of the PTs worked more than 40 hours per week (50.0%), focusing the time taking care of the patients (90.3%), leaving research and teaching as a low priority (97.1% and 98.6% within the 0-20% range, respectively). The majority worked in physical medicine and rehabilitation centers (34.4%), in an urban environment (86.1%), treating more than 15 patients per day (43.1%), most of them adults with orthopedic conditions (75.9% and 76.1%, respectively). For a more detailed data, the descriptive statistics of the PTs' personal and practice characteristics are presented in Table 1. Additionally, the respondents had generally positive attitudes and beliefs regarding EBP. For a more detailed information and further analysis the items sub-groups responses were gathered in Table 2. For the reproducibility tests, 41 items were analyzed. Only one item (32 3º ) had a PA less than 60%. The other items had a moderate to high PA, namely tree items (2, 6 and 322º) were between 61 to 70%, twelve (3-5, 7-11, 13, 16, 38 and 31d) between 71 to 80%, twenty one (1, 12, 14, 15, 17, 20-22, 24-27, 29-  Although the scores in this questionnaire are intended to be interpreted item by item, as the scores of each item do not contribute to produce full scale or subscale scores, the internal consistency was evaluated with the Cronbach's α coefficients according to the original questionnaire sub-grouping strategy 8 . The Access to and availability of information to promote EBP sub-group reach a Cronbach's a of 0.365, considered as unacceptable. Even if the item 23 was deleted, with an α increase to 0.367, it will be still considered as unacceptable. Also, four items (18, 19, 21, 23 and 24) did not reach the conventionally acceptable values in the correct item-total coefficients. Regarding the Interest in and motivation to engage in EBP and the Access to practice guidelines sub-groups they were considered as poor, as they reach an α of 0.577 and 0.564, respectively. None of the items if deleted, could increase their values. Furthermore, the items 15, 18, 19 and 20 did not reach the conventionally acceptable values in the correct itemtotal coefficients. Concerning the Attitudes and beliefs about EBP sub-group the α was 0.767, therefore it could be considered as acceptable. Once again, none of the items if deleted, could increase its values. Finally, the Level of attention to and use of the literature and the Educational background, knowledge and skills related to accessing and interpreting information sub-groups were considered as good, as reach an α of 0.837 and 0.879, respectively. In the Educational background, knowledge and skills related to accessing and interpreting information sub-group, if the item 31b was deleted it could raise the α to 0.882, however it will still be considered as good. None of the evaluated sub-group reached an α greater or equal to 0.95, showing no redundancy in the items 32 . For a more in-depth analysis, the internal consistency data was gathered in Table 5.

Discussion
The results of the present study indicate that, in general, the translation, cross-cultural adaptation, validity and reliability of the questionnaire to European Portuguese and for the Portuguese PTs were successful.
The questionnaire was found to be valid in terms of face and content validity for the measurement of EBP aspects. Several instruments for measuring EBP have been developed 33 , however few of them targeted toward beliefs, attitudes and knowledge, or focused specifically on PT working class. We have found no other questionnaire measuring EBP in PT practice against which we can compare our findings. Therefore, for the content validation, we used a sample of 17 PTs, 2 health care experts and the expert panel (also PTs and psychometricians), to assess the comprehensiveness of the questionnaire and the relevance of the items. So, it should be possible to draw inferences with regard to the target population and context. The interactive process in which the questionnaire was reviewed and consensus was reached by the expert panel on several occasions strengthens the validity conclusions.
In the reliability test, the results obtained in the final questionnaire showed that the PA values for dichotomous items were higher in this study (minimum 88%; maximum 94%) compared to the original questionnaire 8 (minimum 68%; maximum 93%) . The same was also true regarding ranked items, where the minimum in this study was 65% and the maximum 94%, contrasting with the original questionnaire 8 59% to 80% range values. Also, the PA values for the final questionnaire are in line with those found in the Bernhardsson et al. 21 study -ranged from 60% to 81%. It should also be highlighted that, even with the reduction of the Likert scale from 5 to 4, the PA values from the items that could be compared where generally better relatively to the Bernhardsson et al. 21 study (the only exception was our item 10 compared with the Bernhardsson et al. 21 item 13 -75% compared to 80%, respectively). The good achieved PA values were also confirmed by the RV results (as they were always inferior to 0.1, showing no major occasional disagreements) and were in line with the Kappa coefficients (only the 2 and 6 items had a fair Kappa).
In the internal consistency analysis the sub-group Access to and availability of information to promote EBP was the one that achieved unacceptable values. This may be explained by the differences in the access to EBP. For instance, 86.1% did not have access to current research through professional journals in their paper form and, although the majority access EBP by the use of electronic databases, this access was very different between work (65.3%) and home (94.4%). Despite the unacceptable results, we decided not to remove sub-group items since: (1) although the items are part of a sub-group (according to Jette et al. 8 ) responses in some items do not necessarily have to be coherent with responses to other items (for example, accessing evidence by paper form does not necessarily needs to be coherent with accessing it online and vice-versa); (2) and the information gathered in these particular sub-group items are essential to understand how evidence is assembled by the PTs and it should not be discarded.
Analyzing item by item, it was found that item 32 in our questionnaire needs to be reviewed. The item 32 PA values went down as the respondent chose the least important barrier (88% for 32 1º , 68% for 32 2º and 58% for 32 3º ). This may be explained by PT poor ability to respond correctly to the item. Most of responders marked the three most important barriers with an "X" in the boxes, not indicating their order. Furthermore, in other cases, only one barrier was marked, still existing those that marked more than three boxes or even decided not to answer the item in the second occasion. Therefore, in an attempt to increase the number of valid responses (the item T0-T1 mean percentage of missing values was 26.9%), it was decided to change the boxes to -(marked blank spaces). With this new form of response space, it is expected that the responders leave the "X" type responses and chose to fill the space with the appropriated barrier number. We are aware that this form of response space will not solve all missing values, however as it will cover the most common response mistake, it is expected that it will dramatically decrease the number of missing responses and, consequently, increase the PA.
Still in the items agreement proprieties, the items 6, 8 and 31f were the only evaluated items that show a statistical significant disagreement (in the other items the 95% CI were rather narrow, indicating that the disagreements were not substantial). In both 6 and 31f items, the responders chose a change in attitudes toward higher categories on the second questionnaire administration. These differences could be explained by the sample mean age (20-29 years) and academic degree (baccalaureate) since, after the first questionnaire filling, younger PTs with a lower academic degree could had a greater EBP attitudes awareness, as well as they have a greater desire for knowledge, incite searching some keywords that were less understood in T0 (better EBP knowledge are generally achieved later in life with a higher academic degree 8 ).
Regarding the PT personal demographic information items, some important missing responses in the type of facility, majority of patient condition and majority of patient age group were identified (the items T0-T1 mean percentage of missing values were 13.2%, 36.1% and 25%, respectively). The reason for those high missing values relied in the responders' misinterpretation of the items. Most of the PTs misread the item question and instructions, and selected more than one box in each item. So, in order to overcome this issue, it was decided to highlight the importance of selecting only one response, adding the instruction select only one facility (in Portuguese -selecione apenas um local de trabalho) in the type of facility item and the instruction select only one intervention area and one age group (in Portuguese -selecione apenas uma área de intervenção e um grupo etário) in the majority of patient's condition and patient's age group item. After adding and highlighting these instructions, it is expected that the number of missing responses considerably drop, hoping to reach irrelevant missing values in a next questionnaire filling.
Another item issue identified and further discussed after the questionnaire administration was our item 45. Our item 45 corresponds to the original questionnaire 8 item 47 and it is an open-ended question, where the responder is asked to fill the percentage of time dedicated to patient's care, research and teaching. As already discussed, openended questions should be avoided 31 . A major reason is variation in willingness and ability to respond in writing 34 . Unless the sample is very homogeneous with respect to these two characteristics, response bias is likely 34 . As this questionnaire is designed to all national working PTs, a heterogeneous sample with very different perspectives related to EBP is expected. Furthermore, the main reason for using open-ended questions is to capture unsuspected information 34 . However, in this item, only answers in each type of activity between 0 and 100% are considered as valid. Therefore, no unsuspected information will be gathered. Moreover, in questions where responders are likely to be estimating their answers, such as item 47, it is usually better to offer response categories 34 . So, we suggest that the item includes, in each type of activity, five boxes with the options 0%, 5-25%, 30-50%, 55-75% and 80-100%. The percentage range options were similar to those used in the Jette et al. 8 study results presentation and in the Bernhardsson et al. 21 corresponding item response options (item 8).
In conclusion, the final translated and adapted European Portuguese EBP questionnaire has a good face validity, content validity and reliability. The results suggested that this questionnaire can be a useful instrument for measuring self-reported beliefs, attitudes, knowledge, and behaviors related to EBP in working Portuguese PTs.

Limitations
Among the limitations, this study had a relatively low response rate (59%) in the reliability test, especially in the early phase, where many PTs were not interested in participating in the study. Also, the number of missing responses in some items was high (for example, 36.1%) influencing the correct item appreciation. The questionnaire return rate and missing items responses may be explained by its large number of items (49 items) and completion time (approximately, 14 minutes 29 ). Regarding the questionnaire as a whole, as already mentioned in the Bernhardsson et al. 21 study, we also think that the methods used to evaluate the validity of our questionnaire were limited to content and face validity evaluations. For example, construct validity was not assessed with factor analysis because the instrument was intended to measure item by item and not in factors. Also, testing of criterion validity was considered to be difficult because no other instrument measuring selfreported attitudes and behavior related to PT EBP could be found, and measuring self-reported behavior against actual or observed behavior was not deemed possible in the present study. Furthermore, as this is a self-report questionnaire it is difficult to know whether respondents are expressing what they really believe and do, or what they think they are expected to believe and do. This social desirability bias is difficult to control in this type of questionnaire, even more when professional attitudes and behavior are being measured. A further reliability issue concerns the implicit assumption that professional attitudes, knowledge, and behavior are relatively stable. There is a possibility that some of the disagreement between the 2 test occasions (even in such a short time interval as 2 weeks), can be explained by social and professional networks, such as a discussion of the questionnaire between colleagues during a coffee break. Even the act of completing the questionnaire for the first time may cause a change in attitudes. Reviewing, at work or at home, some of the questionnaire keywords that raised more personal/practice questions or, for some reason drew a more in-depth attention, may also explain some of the disagreements between the two testing times. Another limitation of the present study is that the reliability of the revised version of the questionnaire was not tested again.