Assessment of authors understanding of the STROBE-nut reporting guidelines

Objectives: To test the experience of authors of applying the STrengthening the Reporting of Observational Studies in Epidemiology-nutritional epidemiology (STROBE-nut) on a recently published paper. Correct understanding of the items on the reporting guidelines could lead to appropriate use. Design: a cross-sectional study, with a convenient sample. Setting: Participants were asked to return the STROBE-nut data collection template filled on recently published papers. Next, an interview was organised to collect feedback on the process of applying the guidelines. Two researchers involved in the development of STROBE-nut completed the template using the published papers by participants. The filled templates were then compared to assess the measure of agreement of the STROBE-nut items as a proxy of understanding. Participants: Authors who recently published papers reporting dietary assessment or food intake data. Results: We recruited 12 participants between May 2018 and June 2019. Five participants never used reporting guidelines before, while ten reported intention of future use. Half of the participants reported that the use of filling STROBE nut was useful, but some modifications are needed. Agreement between participants and experts on items reporting was generally low. Only two items had moderate weighted kappa agreement nut 1 (Kappa= 0.4, P 0.02) and nut 22.1 (Kappa= 0.47, P 0.01). Conclusions: There is a need to ensure the correct understanding of STROBE-nut by authors. Revisions of STROBE-nut that make the items shorter and simpler can increase understanding. Training researchers at an early stage on the aim of reporting guidelines could potentially increase proper use and decrease subjective interpretation.


Introduction
A sizeable body of published papers does not add value to science (1). An essential effort to improve the impact of research is to ensure that all essential information is present in research manuscripts to enable correct interpretation of the study and its findings. Reporting guidelines have been developed to remind researchers to ensure essential information is included in manuscripts. Moreover, they could improve the editorial and peer review processes as they direct readers to key elements of the study. To date, over 400 reporting guidelines are available (2). Despite this however, a large proportion of researchers is still unaware of the existence and utility of reporting guidelines (3).
Efforts are needed to promote correct use of reporting guidelines in the user community i.e. authors, reviewers and editors. A few initiatives were developed to increase the use of reporting guidelines and improve their uptake including endorsement by journals. Among the initiatives are writing aids, a technological extension of a reporting guideline to increase the ease of use (4)(5)(6).
At the same time however, considerations are also needed on the side guideline developers to ensure maximum utility of reporting guidelines (7). There is a need to ensure reporting guidelines are up-to-date and provide timely recommendations (8).
No previous initiative has assessed the understanding of the reporting guidelines by early career researchers. Thus, an assessment of how reporting guideline are used by authors, their perceived added value and burden is a timely effort in this regard.
STROBE-nut was developed in 2016 to improve reporting in nutrition epidemiology and dietary assessment and contains a set of 24 items (9). The guideline endorsed by various journals (10)(11)(12)(13)(14), provides instructions for authors to submit a checklist that indicates the page numbers where the STROBE-nut items are reporting in the manuscript. As part of the team that led the development of STROBE-nut, we monitor its use and understanding, to inform further updates and continuous improvements (15).

Objectives and hypothesis
The objective of this study is to test authors' experience of applying a data collection template based on the STROBE-nut table on a recently published paper and collecting their feedback on the overall experience and understanding of the items included on the checklist. We hypothesis that better understanding of the checklist's items could lead to correct use.

Study design, procedures and participants
We performed a cross sectional study with a convenience sample of researchers.
First, we assessed understanding of the STROBE-nut items using a set of recently published papers. The authors were invited to participate by filling a data collection template based on the STROBE-nut checklist, completing the page number for each item for their paper using the information in their published papers in one of the following categories: 1-Irrelevant/not applicable 2-Fully reported on page/pages 3-Partially reported on page/pages 4-Reasons for exclusion (Annex 1), and sending it to the lead researcher of the present study. A baseline questionnaire was sent to gather participants' characteristics (Annex 2). Second, an interview was scheduled to obtain author's feedback on the application of the STROBE-nut on their paper in phase one (Annex 3). Third, two experts (DH and CL) who took part in the development of STROBE-nut filled the checklist using the information provided in the published paper. The purpose was to assess the agreement between the experts and the authors regarding the reporting of STROBE-nut items in the text. If the authors applied the same category where the items are in paper as the experts, this was considered as an agreement on the reporting of the item.
Participants remained in their research environment. Follow-up was conducted via online communications. The communication between the lead investigator and the participants were through email (sending the baseline questionnaire, the STROBEnut guidelines). The interview was done over MS Skype.
Eligible criteria included authors who published papers reporting dietary assessment or food intake data. Invitations were disseminated to recruit potentially eligible authors. Ghent University academic bibliography https://biblio.ugent.be was searched using the term "nutrition" to find relevant papers. We conducted the search in May 2018. Moreover, we disseminated the invitation within our personal network and conferences including colleagues in different departments at Ghent University. Ghent University is known to have a large community of food and nutrition research, and a high Shanghai ranking in life and agriculture ranking(16).

Pilot testing
The baseline questionnaire, the table and the Skype interview were tested with two PhD students at the UGent department of Food Technology, Safety, and Health.

Study procedures
A baseline questionnaire was used to gather participants' characteristics including their research experience (PhD/Post Doc), previous use of reporting guidelines, frequency of use, and motivation of use of reporting guidelines. We also assessed participant's prior knowledge regarding reporting guidelines using a tool to assess knowledge regarding checklists (17). Subjective knowledge considering the utilization and content of the reporting guidelines was measured with two questions on a 5-point Likert scale ranging from very unknowledgeable (1) to very knowledgeable (5). Objective knowledge was measured using six true or false statements, same questions used in a previous study (6). Informed consent was collected electronically in the baseline questionnaire. Baseline information and informed consent are provided in annex 2. Participants confirmed that they have filled the baseline questionnaire during the Skype call.

Study outcomes and measurements
Authors were asked to fill out a checklist (Annex 1, table1) and complete the page number for each item for their paper: 1-Irrelevant/not applicable 2-Fully reported on page/pages 3-Partially reported on page/pages 4-Reasons for exclusion. Example for illustration is in Annex 1.
Experts (DH and CL) only filled the first three columns: not applicable, fully reported, and partially reported on page/pages. Consensus between the experts (DH and CL) was reached through discussions for each manuscript's items for each manuscript.
The results were compared against the submitted answer for each respondent.
Agreement between the author's and the experts' answer was calculated for every item.
After the return of the filled STROBE-nut checklist, a short semi-structured interview was conducted in English to understand the experience of the users with the reporting guidelines. The questions can be found in annex 3. We were mainly interested to have a better view on the experience of using STROBE nut, barriers, added value of and the intention for use in the next manuscript. As no previous validated questionnaire was available, we developed an interview guide for the purpose of the study. The questionnaire was pretested with two PhD students at Ghent University to assess face validity.
The interview started with a quick reminder of the study objective, intended uses of the interview results, and reassurance that the confidentiality and anonymity were protected. Permission to take notes and record the conversation electronically was requested. The Skype call took between 30 minutes and 1 hour.

Statistical methods
Data cleaning and analysis were conducted using Stata version 14.1 (StataCorp).
Descriptive statistics from the baseline questionnaire were reported using absolute numbers and percentages. For each binary question in the interview, answers were calculated, and results were reported using absolute numbers and percentages.
Agreement between participants and experts was assessed for each paper, and for each item across all papers. Three different level of agreement were calculated.   (18). The results were reported as percent agreement, and Cohen's weighted kappa. To calculate weighted Kappa on the agreement level on the columns filled, the answers of the participants/experts were coded in ordinal order. "Not applicable" and "fully reported" were coded as 2, "partially reported" reported as 1 and "reason for exclusion" as 0. Complete reporting is regarded hence when the item is reported as fully reported or not applicable. The frequency of positive fully reported items/ not applicable were calculated for each item across studies.

Qualitative data analysis
To extract results from interviews, anonymised answers were analysed qualitatively by two researchers (DH, EQ). DH and EQ coded all answers to each question independently. For all questions, the two researchers then discussed differences in coding per question until they reached agreement. The analysis from all questions was combined, similarities where grouped and organized in themes. Striking answers are quoted in the results to illustrate the findings. The data were anonymised after data cleaning and analysis.

Ethics
The study was approved by the Ethics Committee of the Ghent University Hospital number EC /2018/0636. Informed consent was electronically collected and the study protocol was registered prior to the study (2 nd May 2018) on https://biblio.ugent.be/publication/8563182.

Participants
We found 168 published papers after the search of Ghent University academic repository of papers on 4 th May 2018. The majority (n=132) papers did not report food intake data and were excluded. Finally, 3 papers were published by the same first author, 2 authors published 2 papers. Invitation emails were sent to the first authors of the remaining 32 papers to participate. Seven authors responded positively and participated in the study. Five other authors were included in the study through dissemination within personal networks. It was not possible to track response rate, as recruitment methods used personal networks, and invitations were also spread in conferences. We recruited 12 participants between May 2018 and June 2019. Data collection was completed by June 2019.
As shown in Table 1, 75% (n = 9/12) of the sample was PhD students. Almost half (42%) of the sample had rarely or never used any reporting guideline before considered himself or herself un-knowledgeable regarding the guidelines content or its utilization (n = 3). Only two (17%) participants correctly answered that reporting guidelines should not be used as an evaluation tool for the quality of the paper.
Almost all participants correctly answered the three statements regarding the aim of the reporting guidelines, and the way it needs to be filled. One third stated that they sometimes used reporting guidelines. Table 2 is a summary of the participants' answers regarding the ease of use of a data collection template checklist. Overall, there was a positive perception of reporting guidelines by the participants. Participants expressed motivation and interest with using a reporting guidelines and/or STROBE-nut for their next manuscripts, despite certain hesitations and confusion on some sections of the data collection template.

Skype interview results
Regarding the experience of filling STROBE-nut, completing the checklist was perceived fast (n= 4), easy (n=4), helpful (n=3), and comprehensive (n=2) while (n=4) authors stated that the checklist was not the most suitable for their study, as dietary assessment was not the main measurement in the study. The duration of filling the STROBE-nut checklist varied from 10 minutes to 1 hour and 55 minutes.
Applying the data collection template on already published work, made some (n=3) authors realize they missed an opportunity to include certain piece of information that could make their research more comprehensive. A participant stated having to fill the STROBE-nut checklist very quickly upon submission. When filling it again for the study, the participant realized it wasn't filled correctly initially. Participants also expressed further uncertainties on the use of STROBE-nut, i.e.
when in the timeline of writing the manuscripts should STROBE-nut be applied, and reporting on all items with a limited word count. Participants also suggested introducing introduce a more user-friendly interface, such as an online system to fill in the checklist e.g. applying the checklist as an online survey during the submission process.
Generally, the authors experienced difficulties to understand the difference between "fully reported" and "partially reported" for the data collection template describing it with words like "vague, difficult and confusing''.
Participants attributed the difficulties in filling the "fully reported" column on the checklist, to several reasons i) journal policy and word count, ii) person's understanding of the items, the need for high level of understanding the purpose of the checklists and how to fill them iii) certain items like nut 8.1 and nut 5, nut 9 ask to report different elements in one item, however authors tend to sometimes address one element only iv) the reason why the information is excluded or partially excluded is arbitrary, subjective and depends on personal behaviour.
Authors thought that the opportunity to add reason for exclusion was quite helpful, as it helped them realize that they had missed adding available information to their Participants explained that pieces of information were not reported unintentionally.
They simply forgot to include it, because it was deleted by co-authors or they did not perceive it as essential content.
In addition, participants also expressed confusion on certain items, in terms of its phrasing and applicability to their respective study. These items are summarized in annex 4.
Participants were also asked to describe the barriers of using a reporting guideline.  (Table 5).

Discussion
This study assessed user experience with the use of STROBE-nut reporting guidelines in nutrition research and agreement with expert classification. Participants in this study seem to understand the added value of using STROBE-nut when reporting their studies and are willing to use it. Nevertheless, they faced many obstacles applying STROBE-nut on their published papers, including the difficulty to apply STROBE-nut as an extension of STROBE. Updates that make the items shorter and simpler can reduce subjectivity according to authors' feedback, as well as integrating reporting guidelines in the learning process at earlier stage e.g.
Bachelors or Masters degrees could increase the levels of understanding.
To our knowledge, our study was the first to assess the user's understanding from Most of our participants are PhD students, and almost half of the sample rarely or never used reporting guidelines before. The application of a STROBE-nut table by researchers on their published manuscript has potentially increase their intention to use the tool in further writing exercises. The observed high intention of future use is similar to result we have obtained studying the use reporting guidelines as writing aid (6).
As the application and endorsement of STROBE-nut is increasing, there is a need to ensure it is used correctly. Participants suggested developing an education module on STROBE-nut. Thus we have created a module which is shared on STROBE-nut website http://www.strobe-nut.org/content/strobe-nut-lecture. We are currently piloting it and welcome contributions. Also, participants suggested applying the checklist as an online survey within the submission process. This is in line with a few initiatives that are working on improving adherence to reporting guidelines including integrating them in the writing process (6,26).
Our study has several strengths including the application of STROBE-nut on recently published papers and collecting users' feedback that could be integrated in further updates. It provides insights into the behaviour of filling out a checklist. The study also served as a learning experience for the participants. Doing an exercise like this is beneficial and can be seen as an interactive learning experience. On the other hand, the study only involved a small sample size, smaller than what was indicated in the protocol (27). This could be due to limited search using Ghent University academic bibliography and studies reporting food intake data and dietary assessment within the timeframe of the study. Kappa values are affected by the prevalence of the incidence under consideration, thus the small sample size could possible resulted in lower Kappa levels (28).
Finally, this study is an reminder to be vigilant about the added value and potential burden due to the introduction of reporting guidelines on researchers (29). To ensure optimal application and timely revisions of reporting guideline, it is essential to consider the primary target audience: authors of research manuscripts during the development and modifications of reporting guidelines. Research experience: • PhD student 9 (75%) • Post Doc 3 (25%)

Previous reporting guidelines use*
• Yes, to write or co-write a paper, specify which guidelines 9 • Yes, to review a paper, specify which guidelines 3 • More than one answer is possible ^ one missing answer    given the complexity of diet or nutrition as an exposure." Strobe nut-22.1 "Describe the procedure for consent and study approval from ethics committee(s)."

(67%)
Strobe nut-22.2 "Provide data collection tools and data as online material or explain how they can be accessed." 2 (17%)