Perceived impact of a one-week journalology training course on scientific reporting competencies: prospective survey

| INTRODUCTION: The debate on scientific research and reporting integrity issues in Brazil is incipient. Literature suggests that a journalology training course could help to improve the competencies of the participants. OBJECTIVE: To evaluate the immediate impact of a journalology training course on perceived academic competencies, comprised of knowledge, attitudes, and skills. METHODS: The course was taught in 5 consecutive days to an online audience of individuals from the health sciences. A self-applied questionnaire was employed before and immediately after the course, which included initial and acquired perceived knowledge, attitudes, skills. The Wilcoxon non-parametric test for paired samples was used for analysis. RESULTS: A total of 45 individuals participated in the course, with a 53% response rate before and after. The number of participants in each course session ranged between 32 and 45. There was an improvement in perceived knowledge of: (1) writing review articles; (2) ethical aspects of research; (3) scientific authorship; (4) predatory practices; (5) publication bias and spin, and (6) researcher evaluation. There was no improvement in self-reported attitudes towards any item. There was an improvement in the perception of skills relating to: (1) writing a response letter and (2) writing an opinion as a reviewer. CONCLUSIONS: Overall, attendees who participated in the survey reported perceived improved knowledge and skills in some items but not in their attitudes. Therefore, the course appears to have been unable to modify perceived scientific reporting competencies.


Introduction
Published papers and scientific journals are centuriesold sociocultural artifacts 1 that remain the primary tool of disseminating scientific knowledge. 2Journalology, also known as publication science, studies all aspects of scholarly publishing. 3Journalology can be placed within an expanding field of meta-research, aimed to evaluate, and improve the efficiency of research and reporting practices 4 to avoid unnecessary wasteful research. 5For efficiency gains in science, especially regarding transparency and reproducibility, it is essential to ensure that: (a) authors have the necessary competencies to write transparent reports; (b) reviewers have the competencies required to assess scientific quality; and (c) editors know and aim for the best, more ethical editorial practices. 6Editors, however, do not always agree on what constitutes questionable research reporting. 7en it comes to authors, it has been reported that students who underwent training in scientific reporting may report improvement in their knowledge about it, and their skills and attitudes towards it. 8hallenges abound, however, as questionable practices in research and in reporting are valued differently by authors from different cultures.That is, some behaviors are considered graver than others depending on in which culture they occur. 9,10 systematic review has recommended that future research in journalology should focus on developing more rigorous studies on the effects of training professionals in all aspects of publication science. 6e debate of scientific research and reporting integrity issues in Brazil is incipient.Plagiarism and conflicts of interest are prominently discussed 11 , while methodological 12 , managerial issues 13 and incompleteness of reporting 14 , known culprits behind avoidable research waste 5 , have yet to be more thoroughly addressed.National educational guidelines for undergraduate courses within the health sciences were also found to be lacking. 15Much like in the English-speaking world 16 , the national discussion on questionable practices in research and publication usually highlights moralizing and punitive solutions 17 , meanwhile effects of formative and educational efforts, which could forge more effective paths forward 18 , remain understudied. 19Thence, little is known about effectively training authors on scientific reporting competencies.Knowledge, skills, and attitudes are considered the three main elements that constitute competencies 20 , being manifested in the individual's way of thinking, feeling, and acting when faced with any challenging situation.Those three aspects of competencies are directly linked and constantly interacting with one another.Based on this rationale, it is believed that a journalology course could be pivotal in helping to increase transparency and integrity in the scientific record.Therefore, considering the scientific community's demand for efficiency, the present study evaluated the immediate perceived impact of a fiveday journalology course targeting academic reporting competencies (i.e., knowledge, attitudes, and skills) of the participants.Participants' satisfaction with the course was also assessed.

Ethical Aspects
The study was approved by the Faculdade de Medicina/Universidade Federal de Pelotas (CAAE approval number 57884822.0.0000.5318).All participants signed written informed consent terms before participating in the study.The full research protocol and dataset are available at the Open Science Framework (https://osf.io/6qrg5/).

Study design
A pre/post-course survey with a convenience sample of participants in a journalology course was designed.The study aimed to answer the following research question: "What is the perceived impact of a short synchronous online course on academic competencies (knowledge, attitudes, and skills) of participants on aspects of journalology?".
The questionnaires (forty close-ended questions, with a 15-minute response time) were given to participants of a journalology course at two different timepoints: before and immediately after the course.The questionnaires were structured as follows: initial (pre-course) and perceived acquired (post-course) knowledge, initial and perceived acquired attitudes, initial and perceived acquired skills, and finally, satisfaction with the course (avaiable at Open Science Framework).To the best of our knowledge, there are no validated questionnaires available to address those issues, hence we have developed our own for the sake of an initial exploration of the matter in the Brazilian higher education context, as questionable research practices appear to be context/culturally dependent. 9,10e present study has been reported following the SURGE (Survey Reporting Guideline) and CROSS (Checklist for Reporting of Survey Studies) reporting guidelines (avaiable at Open Science Framework).

Participants
The study employed a convenience sample, using a single stage sampling, and it was carried out in two steps.All regularly matriculated Ph.D. or Master Students of the Graduate Program in Dentistry of Federal University of Pelotas were invited.
Advertisements on e-flyers were also placed on the respective graduate program website and social media channels.All participants enrolled in the journalology course were eligible and invited to participate in the surveys.Participants with incomplete responses to the questionnaire (either before or after) were excluded from the final analysis.

Course description
An online synchronous course on journalology applied to the health sciences was conducted for five consecutive days through an online platform (Zoom Video Communications, San Jose, California, USA).The course covered a daily 3.2 hours of group discussions and four hours of individual or group exercises and self-study, totaling thirty-six hours.The exercises were made available only for the students enrolled in the Graduate Program in Dentistry of the Federal University of Pelotas for scoring reasons.The course was taught in English by most non-native speakers (Brazilian with a C1 English level and experience in teaching in English) and one native English speaker.The course was offered by the Graduate Program in Dentistry of the Federal University of Pelotas, Brazil.
Five lecturers were professors and researchers in the health sciences, one lecturer was a professor and researcher in information science.All of them have experience as authors and reviewers, and some as editors in legitimate indexed international peerreviewed journals.During the course, a minimum of three lecturers were always available in the video call room and actively participated in all the sessions and discussions.
The course covered a training program in aspects of journalology related to the following topics: [1] introduction to journalology (addressed by an experienced researcher in the field); [2] ethics in science (current issues in publication ethics); [3] predatory practices (identifying and avoiding predatory journals); [4] how biomedical journals operate, and how to submit; [5] scientometrics (understanding scientific output evaluation metrics); [6] questionable research practices, publication bias and spin; [7] data sharing; and [8] the peer-review process.
The main purpose of the course was to introduce participants to the best practices in publication science.To do so, it aimed to provide guidance on scientific writing and publishing, raising awareness among the attendees on issues of ethics in science (such as behavior deviations and inequities), data sharing, open science and predatory publishing.It also aimed at preparing the attendees to employ reporting guidelines, to write down research protocols, to prepare articles for submission, to write cover and response letters.The editorial and peer review processes in journals in the health sciences and scientific output metrics, such as Impact Factor and H Index, how they are used and misused, were also explained.
The course, as planned and advertised, targeted health science researchers (details in table 1) in training or in their early career stages (e.g., postgraduate students, postdoctoral fellows, medical residents, etc.), since most postgraduate curricula in the health sciences in Brazil do not usually include formal training in scientific writing and publishing. 11,15,17Modules' titles, formats, objectives, and facilitators have been detailed in Box 1.

Questionnaire characteristics
Electronic questionnaires hosted at Google Forms (Google LLC, Mountainview, California, USA) were administered before and after the journalology course to the participants (form available at supplementary file 1).The questionnaire was the assessment form.It was developed based on a previous study 8 that aimed to assess students' satisfaction with a 2-day course on scientific writing in the health sciences and to evaluate their perceptions of knowledge, attitudes, and skills.
The first questionnaire, applied before the course (at baseline), was structured to identify: [1] participant's characteristics (age, gender, professional identification, etc.) and [2] knowledge, attitudes, and skills on the topics addressed in the course.Details have been reported below in the outcomes section.The second questionnaire, applied immediately after the course was completed, contained the following aspects: [1] acquired knowledge, attitudes, and skills and [2] satisfaction with the course.It was kept available online for the subsequent seven days after the ending of the course.Both questionnaires 'links were e-mailed to all participants, and the identity and privacy of all participants were protected.
The questionnaire was comprised of forty questions divided into three sections, which have been described in Box 2. Section 1 of the questionnaire was comprised of 11 questions about demographic characteristics of the participants; section 2 focused on 22 questions about academic competencies; and section 3 covered 7 questions about satisfaction.

Outcomes
Our primary outcome was attendees' perception of their knowledge, attitudes, and skills before (baseline) and immediately after the journalology course employing a Likert scale (1 to 5 points).This outcome was reported as a continuous variable ranging from 1 (totally disagree) to 5 (totally agree).Details on each item have been described in Figure 1.The secondary outcome was student satisfaction measured immediately after the course by the questionnaire employing a Likert scale.

Statistical analysis
Descriptive data were expressed as median and interquartile ranges or absolute and relative frequencies.Data normality was verified with skewness, kurtosis, and Q-Q plot tests.In the next step, to analyze the responses to the items in the questionnaire about attendees' perceptions of their perceived knowledge, attitudes, and skills before and after the course, the Likert scale as a continuous variable (1 to 5 points) was employed.The Wilcoxon non-parametric test for paired samples (after-before comparisons) was used.Finally, data have been expressed as pre-test median, post-test median, and z-test statistics.The significance level was pre-defined at p < 0.05 for all analyses.Participants who did not answer either the pre or post-course questionnaire were excluded from the final analysis, as it could preclude direct comparison (missing data).Statistical analyzes were performed using Stata Software (version 15.1, Stata Corp).

Results
The number of attendees in each course session ranged between 32 to 45.A total of 24 (53%) were included in the final analysis.A total of 45 participants attended the course; a total of 33 (73.3%) participants answered the questionnaire before the course; and a total of 24 (53.3%)participants answered the questionnaire after the course.Additionally, a total of 23 (51.1%) participants answered the satisfaction questionnaire.
Most participants were female (83.3%), graduate students (62.5%), dentists (87.5%), and PhD candidates (54.1%).Approximately 50% of the participants had clinical research as their main domain of research.More than onethird of participants had a basic level of experience authoring manuscripts (58.3%), about 49% never had peerreviewed a scientific manuscript, and 75% had not ever served as journal editors.Finally, half of the participants (50%) reported to have an intermediate level of English (B1-B2).Table 1 detailed their demographic characteristics.Participants' perceptions about knowledge, attitudes, and skills based on a Likert scale have been reported in Table 2. Regarding to each specific domain, among the items about perceived knowledge, we observed improvement in the writing of review articles (1 point; p<0.01); ethical aspects (1 point; p=0.02); scientific authorship (1 point; p=0.03); predatory practices (1 point; p=0.02); publication bias and spin (1 point; p<0.01) and researchers' evaluation (1 point; p=0.05).Regarding attitudes, no change in the Likert scale based on a Wilcoxon signed-rank test was found.Regarding perceived skills, writing a response letter (0.5 points; p=0.02) and writing an opinion as a reviewer (2 points; p=<0.01)displayed improvement respectively.Table 2 shows the results of the knowledge, attitudes, and skills of the included participants.Table II.Pre and post-test median on knowledge, attitudes, and skills of the attendees, based on Likert scale, before and after the course on Journalology (n=24) Overall satisfaction among attendees was high: 65.2% were very satisfied with the course; 75% of the participants were very satisfied with the materials and content of the lectures.About the theoretical classes, about 65% of the participants were very satisfied; and 52% were very satisfied with the practical activities during the course.Regarding course organization, 78% were very satisfied.About the course tutors, 83% of the participants were very satisfied, and about 70% of the participants would recommend the course to a friend.

Discussion
The present study aimed to assess the immediate perceived impact of a five-day training course on journalology targeting specific academic competencies (i.e., knowledge, attitudes, and skills) of the participants.In addition, the participants' perception of the course was also assessed.In the knowledge domain, perceived improvements in writing review articles; ethical aspects; scientific authorship; predatory practices; publication bias and spin; and researchers' evaluation was observed.In the skill domain, writing a response letter and writing peer review also displayed perceived improvement.In both knowledge and skills, an increase of ~1 point was observed.No improvement in perceived attitudes toward any item was found.Overall, attendees were very satisfied with the course for all items in a scale ranging from 52% to 83%.
A systematic review investigating the effectiveness of training programs in journalology found inconclusive results as few studies were available to be included therein. 6Our results have shown that the journalology course had small positive perceived effects on some of the participants' academic knowledge and skills.As attitudes were unchanged it is correct to affirm that the course had no impact on perceived academic scientific reporting competencies because competencies are made simultaneously of knowledge, skills, and attitudes.This outcome falls in line with the puzzling mixed results the meta-science field has been presenting regarding authorship behavior after training on questionable research and reporting practices.
An intervention aimed at promoting greater adherence to completeness of reporting guidelines available from the EQUATOR Network Initiative resulted in improvements in completeness of reporting among authors publishing with the journal BMJ Open. 21On the other hand, initiatives aimed at reducing the incidence of other questionable practices had no effect on authorship behavior. 22,23It has been hypothesized by previous studies that surveying the participants perceptions may have induced among them greater rationalization about the questionable research practices in question, generating a backfire effect in which little to no effect from the training courses could be found in general. 22ditionally, not all questionable research practices manifest equally, and some may be more pervasive and difficult to change than others.It may be easier to concoct a more complete report 22 than to resist the temptation to spin results to overstate the positive all the while disregarding null results. 23Research climate has also been shown to be more associated with questionable research practices than individual factors or publication factors. 24However, training courses usually address individual factors, such as the author level of knowledge, their agency and decision-making.This happens perhaps under the premise that questionable research and publication practices take place due to a lack of formal training on those topics alone.And/or that individual training could override research climate and culture.
Confronting the findings of the present study with the literature points to the possibility that improving academic reporting competencies alone may play a small part in avoiding research waste and cleaning up the literature from unreliable reports, especially if the research environment and culture remain unchanged.Future research on this topic should employ larger more diverse samples, within and across different fields of the health sciences.That could perhaps help to better assess the interplay among discipline of origin, their research culture and environment, author agency, competencies, and decision-making regarding adherence to best research and reporting practices.
The slight effect found in the before-after comparisons in this study could perhaps be explained by the high levels of knowledge on the topics by the attendees at baseline.As a little over a half of the attendees were Ph.D. students (54.1%), it could be hypothesized that they had previously experienced the process of peer review, publication, data sharing, etc., at some moment in their academic careers.However, as attitude was not modified as self-reported by attendees, they either also (a) presented best authorship behavior at baseline, (b) or the course was insufficient to change behavior, (c) and/or their perception of knowledge and skills differed from their actual knowledge and skills either at baseline and/or after the course.
This latter hypothesis should be addressed in future research employing a control group within experimental settings.It is important to notice that, typically, researchers receive little to no formal instruction on the issues covered by our journalology course during their academic training, particularly in Brazil. 11,15,16That alone would make it unlikely that attendees' actual scientific competencies (the triad knowledge, skills, and attitudes) at baseline were optimal.
It could also be that measurement right after the course may not have left room for a proper selfassessment of eventual attitudinal change in attendees.There could had been no opportunity yet for them to self-asses any behavioral change.To had done so, they would have to had been exposed to author/journal interaction after the journalology training course had taken place.Such experience could present attendees with the possibility of confronting perception with actual improvement -or lack thereof -in scientific competencies.
It could also be that the researchers had the knowledge and skills of the best research and reporting practices.However, they could not properly turn that knowledge and skills into attitudes either at baseline or after the course, pointing perhaps to a systemic cultural explanation as culprit of research waste, in alignment with some findings from previous literature that shows research climate factors more associated with questionable research practices than individual or publication factors. 24So, although ours is clearly a study with null results, its exploratory nature leaves room for further investigations aimed at confirmatory evidence under controlled settings.
A follow-up survey of our course's attendees after a lengthy timespan could also prove to be helpful in unscrambling this puzzle.
Other limitations are as follows.Both questionnaires used in this research have not undergone validation.
To overcome that limitation, the present study measured perceived gains before and after the course rather than objectively assessed gains.The number of respondents was small (53.3%; n=24) and not diverse regarding disciplines within the health sciences, possibly because they stemmed from the same graduate program.If research climate forces were at play, a broader audience from different majors and from different graduate programs would be necessary to assess the impact (perceived or actual) of the course among attendees.A lack of a control group is also an issue in our report, whichwhether present -could perhaps decrease the level of uncertainty of the results.
As both questionnaires employed had not previously undergone validation, only perceived self-assessed alterations were measured.Yet, they are prone to many cognitive biases arising from how attendees may expect to be perceived by their own peers. 25t could be that attendees, fully aware that they were being surveyed on scientific integrity issues, unconsciously portrayed themselves as more apt in said issues, placing themselves in a better light in their responses to the survey.The professors of the course are also authors in this research.Evidence that strengthens this argument is the fact that satisfaction rates related to event organization and approval of professors of our journalology course were the only rates around 80% in our post-test responses, while all other approval rates were between 65% and 75%.
The language may have also played a part as a limitation for course enrolment and participation as the course was offered in English to non-native speakers.It is possible that the sample was prone to self-selection bias (only those fluent in English as a second language), which could have resulted in responses prone to overestimation of perceived knowledge, skills and attitudes and a consequent underestimation of results of the intervention.More robust replications or expansions should address those issues.Future studies should also consider other models of training, such as in-person courses and workshops, and employ the same mother tongue as that shared by attendees.

Conclusion
As competencies are comprised of knowledge, skills and attitudes, our findings suggest no impact on perceived academic scientific reporting competencies among attendees when measured before and right after a training course in journalology.Course satisfaction was found to not be associated with improvement in reporting competencies.If that is confirmed by future studies, research waste would have to start being tackled as a systemic problem which should require reform on the research culture and environment overall, and not only at the writing and publishing stages.That would make the problem much more complex and harder to address and solve.
Source: the authors (2023).Box 2. Detail on the three sections of the questionnaire

Table I .
Characteristics of the attendees who answered the baseline questionnaire on scientific writing in Brazil (n=24)