How to appraise a body of evidence using GRADE system and GRADEPro: A Step by Step tutorial for the perplexed

This is a step by step tutorial on how to use GRADEpro. Step 1: Sign up and sign in Step 2: Name the new project Qeios, CC-BY 4.0 · Article, October 7, 2020 Qeios ID: 2C2WG8 · https://doi.org/10.32388/2C2WG8 1/14 Step 3: Searching and selecting studies Let's say we want to study if mindfulness meditation is beneficial for treating anxiety and depression in people, and we have searched Pubmed/Medline to find out what randomised controlled trials have listed such studies in their database. We conducted a simple search and we found this: Table 1. Search results of Pubmed/Medline database on mindfulness meditation and anxiety and depression Search number Query Sort By Filters Search Details Results 3 #1 AND #2 Free full text, Randomized Controlled Trial, English "mindfulness meditation"[Title/Abstract] AND ("loattrfree full text"[Filter] AND "randomized controlled trial"[Publication Type] AND "english"[Language]) AND (("Anxiety"[Title/Abstract] OR "Depression"[Title/Abstract]) AND ("loattrfree full text"[Filter] AND "randomized controlled trial"[Publication Type] AND "english" [Language])) 26 2 (Anxiety[Title/Abstract]) OR (Depression[Title/Abstract]) Free full text, Randomized Controlled Trial, English "Anxiety"[Title/Abstract] OR "Depression"[Title/Abstract] 9,062 1 Mindfulness meditation[Title/Abstract] Free full text, Randomized Controlled Trial, English "mindfulness meditation"[Title/Abstract] 82 Let's say after some search, we identify the following three studies from which we would like to extract data and put into GRADEpro and fill in the table. The three articles are as follows: Hoge EA, Bui E, Marques L, Metcalf CA, Morris LK, Robinaugh DJ, Worthington JJ, Pollack MH, Simon NM. Randomized controlled trial of mindfulness meditation for generalized anxiety disorder: effects on anxiety and stress reactivity. J Clin Psychiatry. 2013 Aug;74(8):786-92. doi: 10.4088/JCP.12m08083. PMID: 23541163; PMCID: PMC3772979. [1] Hoge EA, Bui E, Palitz SA, Schwarz NR, Owens ME, Johnston JM, Pollack MH, Simon NM. The effect of mindfulness meditation training on biological acute stress responses in generalized anxiety disorder. Psychiatry Res. 2018 Apr;262:328-332. doi: 10.1016/j.psychres.2017.01.006. Epub 2017 Jan 26. PMID: 28131433; PMCID: PMC5526744.[2] van der Zwan JE, de Vente W, Huizink AC, Bögels SM, de Bruin EI. Physical activity, mindfulness meditation, or heart rate variability biofeedback for stress reduction: a randomized controlled trial. Appl Psychophysiol Biofeedback. 2015 Dec;40(4):257-68. doi: 10.1007/s10484-015-9293-x. PMID: 26111942; PMCID: PMC4648965. [3] Step 4. Read and abstract information from the studies and set up an outcomes table Specifically from each study note their participants, the intervention, the comparison they conducted and the outcome Qeios, CC-BY 4.0 · Article, October 7, 2020 Qeios ID: 2C2WG8 · https://doi.org/10.32388/2C2WG8 2/14 measure. In particular, note the outcomes they studies, and how the outcomes were studied. We will input those in the GRADE tables. Hence it is important to put these items right away in a table to make more sense. Use a spreadsheet such as Google Docs to write these. Here is our Google Docs document where we have documented these elements. You have to do this based on the full text of the article and within the full text, read the methods and results sections carefully. All information should be available there. In particular, collect the following information: Author-year Population Intervention (as much details as you can) Comparator (Who was compared) Outcomes In the outcomes, pay attention to how the outcome was measured. Did they measure the outcome as a binary variable, or did they measure the outcome as a continuous variable? These elements are the very basic elements to start with. Based on the above three articles, we have filled in the following table: <!--td {border: 1px solid #ccc;}br {mso-data-placement:same-cell;}--> Table 2. Lists of outcomes studied in the body of literature reviewed Qeios, CC-BY 4.0 · Article, October 7, 2020 Qeios ID: 2C2WG8 · https://doi.org/10.32388/2C2WG8 3/14 Author-year Population Intervention Comparator Outcome Outcomemeasure Type of outcome Hoge-2013 99 people, who were general members of the community and otherwise healthy or on stable medications MBSR SME Anxiety symptoms at baseline and at week 8 using 14 item HAM-A scale, CGI-S and CGI-I scales CGI-S scale Continuous HAM-A scale Continuous Beck Anxiety Inventory Continuous Hoge-2018 70 people, who were general members of the community with no prior symptoms MBSR SME Stress ACTH Continuous Cortisol Continuous TNF-alpha Continuous IL-6 Continuous vanderZwan2015 General members of the public, Netherlands Mindfulness Meditation Pre-intervention state when the person was not practicing mindfulness meditation Stress Depression and Anxiety Stress scale Stress Continuous DASSAnxiety Continuous DASSDepression Continuous This step is essential because in GRADE the focus is on outcomes. If there are multiple studies that study the same outcome measure in the same way, then you should either pool the results of the studies together OR at least you should put them under the same outcome/outcome measurement group. If this is not possible, as we see in this example, then each study will be listed under one specific outcome measure, and we will select individual results or results that are not pooled. In our case, we see that there are 11 outcomes. Step 5: Now set up the study questions, the PICO, and outcomes in GRADE First, click “Add Management Question”, and fill in the boxes as follows: Qeios, CC-BY 4.0 · Article, October 7, 2020 Qeios ID: 2C2WG8 · https://doi.org/10.32388/2C2WG8 4/14 When you save this box, you will see the question and if you click on the question, you will see a box like the one below: Now click “Add outcome”, when you do you will see the following set of eight boxes. Fill or address each one of the eight boxes for EVERY outcome-outcome measurement pair you have created in the spreadsheet (see Table 2). In our case this will mean doing this 11 times over! After filling in each time, press “Add outcome” to add a new outcome. Qeios, CC-BY 4.0 · Article, October 7, 2020 Qeios ID: 2C2WG8 · https://doi.org/10.32388/2C2WG8 5/14 Let's go over each of the numbered boxes one by one: 1. Fill in the outcome. In our case, the first time we do this, we will name our outcome “Anxiety symptoms at baseline and at 8 weeks” 2. Give your outcome a short name, or a nickname. In our case, for the first outcome the name might be “Anxiety Symptoms” 3. In box 3, write how this outcome is measured. In our case, for the first row of outcomes to fill in, we will write “CGI-S scale” 4. Box 4, 5, and 6 correspond to a drop down box. Qeios, CC-BY 4.0 · Article, October 7, 2020 Qeios ID: 2C2WG8 · https://doi.org/10.32388/2C2WG8 6/14 This is the confusing one. If your study does not mention any of the above (i.e., “mean”, “median”, or “range”), click “clear” and then enter the number. This number will correspond to the right hand side drop down box of weeks, days, months, etc. 7. In boxes numbered 7 and 8, you have several choices. If you choose “continuous”, it will ask you to specify what kind of scale. Read the paper(s) carefully to select the specific type of scale reported and fill in the values. Some times, the authors do not mention the lower and upper limit of the values. In that case, do your own research. For example, in their paper, Hoge et.al. (2013) did not describe CGI-S scale. We searched Wikipedia on CGI-S scale to identify the upper and lower values and found that the lowest value is 1 and the highest value is 7. In box 8, you will need to select how you want to present the studies. If you will be working with individual studies, I recommend you use “single studies”. Where you will be working with pooled analyses as in meta-analyses and systematic reviews, I'd suggest you use “pooled estimates”. The rest of the choices or options are more or less self-explanatory. Once you have filled in these boxes, you should save the page by clicking the “floppy disk” icon. This will bring you to the following window where you will be required to fill in about 9 more boxes (see figure): Let's go over these boxes in turn: 1: Click on this box and enter the number of studies. If you selected to use one study or single study, then you will enter 1 in the box. Otherwise, if you choose to use pooled estimates, then click as many studies as are in the pool 2: Identify the study design. There are two choices here: (1) randomized trials or (2) observational study design. Select the appropriate option 3: The third box is about risk of bias. In this box, you have three choices: (1) Not serious, (2) serious, (3) very serious. Let's explain this. Both observational studies and randomised trials are open to several different kinds of biases including selection bias, response bias, and measurement bias. In an RCT, look for if they have reported how they conducted the randomisation procedure. If you cannot find a clear information about their randomisation procedure, then it is a source of bias. Then, look for if this was a single or double blind study. If you cannot find information whether the study has used any form of blinding, then this is another source of bias. Finally, look for if the authors have used intention to treat analysis. If you cannot find any Qeios, CC-BY 4.0 · Article, October 7, 2020 Qeios ID: 2C2WG8 · https://doi.org/10.32388/2C2WG8 7/14 information to that effect, then this is another source of bias. If you find a single bias (for example the authors have used a double blind trial and used intention to treat analysis but did not report how they randomised the participants), then the bias is serious, and if there are two or more such biases, you may signal that the study has very serious risk of bias. In our case, for CGI-S, and for the first study we have investigated, we did not find any information whether the authors used any form of randomisation, nor did we find any information about blinding as mindfulness meditation versus other forms cannot be blinded to anyone. We also did not find any mention whether they used intention to treat analysis. For these reasons, we will rate this study's risk of bias as “very serious”. When you do this, GRADEpro will ask you to write explanations. See this: Write the explanation in the resulting box. In our case it looks as follows: Qeios, CC-BY 4.0 · Article, October 7, 2020 Qeios ID: 2C2WG8 · https://doi.org/10.32388/2C2WG8 8/14 Click “Save” on the blue button to return to the main page and continue with the next item. 4: Inconsistency too has three options: not serious, serious, and very serious. If you are appraising the outcome on the basis of single studies for a single outcome, then write “not serious”. If on the other hand, you find different measures even for a single study have different results, then the entry could be “serious”. For many studies, evaluate the outcomes reported and write whether you find that the direction and magnitude of the results in the individual studies vary a lot and enter “very serious”. As an alternative, for an outcome, you can study the results of all studies and take a call to call it serious or very serious if you find gross inconsistencies in the values of the results and the direction of the results themselves. 5: Indirectness. – Assess how the outcome was measured. Did the authors obtain data directly from the participants in some way or did the authors or investigators depended on third party observations or indirect observations? If the authors obtained data directly from the participants, then the risk is “not serious”. In our case, for example, the authors directly administered the instrument to the participants and obtained data from them, hence “not serious”. 6: Imprecision. – Now read the point estimate. The phrase “point estimate” refers to Odds Ratios or Risk Ratios or any other effect measure. Then notice the 95% confidence interval. Does the 95% confidence interval straddle the null value? Notice the p-value of the effect measure reported by the authors. Does the p-value indicate that the results are statistically not significant? If the 95% confidence interval straddles null estimate or if the p-value is not significant, then enter “serious” or “very serious” (this is subjective indeed). Otherwise, enter “not serious”. In our case, for the outcome measure, we find that the p-values are very low, see Qeios, CC-BY 4.0 · Article, October 7, 2020 Qeios ID: 2C2WG8 · https://doi.org/10.32388/2C2WG8 9/14 7: Other considerations. – Here, they will provide a drop down box with several options. Of this, the publication bias is only important when you are working on a meta-analysis or systematic review. Here, publication bias refers to the fact that generally negative or equivocal studies or small studies are under-represented in meta-analysis and this constitutes publication bias. This is usually detected with Funnel plot or Egger's plot [4] Rest of the options are self-explanatory. In our case, there was no question of detecting publication bias, and everything else was in the negative. When you do this, you will get a certainty estimate of “low”. “Low” certainty estimates tell us according to GRADE principles that more evidence is needed to ascertain whether mindfulness meditation will improve anxiety. After this, we will enter the figures in 8, and 9: 8, 9, and 10: Enter the numbers of people in the intervention and the control arm. You will get the figures from the results and methods tables. Enter the number of people whose data were analysed. Then for continuous outcome, enter the absolute values: Qeios, CC-BY 4.0 · Article, October 7, 2020 Qeios ID: 2C2WG8 · https://doi.org/10.32388/2C2WG8 10/14 Sometimes, as in our case, these are not straightforward. For example, in our case, the authors reported F values, and they did not report 95% confidence intervals, so we cannot enter 95% confidence intervals and we will use “other”. For our case it was entered as follows: Here, Qeios, CC-BY 4.0 · Article, October 7, 2020 Qeios ID: 2C2WG8 · https://doi.org/10.32388/2C2WG8 11/14 So, here, we have shown you how to enter data in GRADEpro to appraise a body of evidence for an outcome. Step 5: Export the table to a word processor At the end of this exercise, you will export the table to a word processor. To do so, click on the arrow on the right top end of the document. You will see the following: Qeios, CC-BY 4.0 · Article, October 7, 2020 Qeios ID: 2C2WG8 · https://doi.org/10.32388/2C2WG8 12/14 Export the table to a word document and open it in word. The final table will be as follows: Qeios, CC-BY 4.0 · Article, October 7, 2020 Qeios ID: 2C2WG8 · https://doi.org/10.32388/2C2WG8 13/14 Conclusion This was a show and tell of how to fill in a GRADEpro table to generate grading of evidence. You can then fill in multiple entries in the table to comment whether we need more evidence (depending on the number of plus signs, the lower the number of plus signs the more likely we will need additional evidence, and if you have four stars, then the current evidence is sufficient of itself) for the effectiveness of the intervention. References 1. ^Elizabeth A. Hoge, Eric Bui, Luana Marques, Christina A. Metcalf, Laura K. Morris, Donald J. Robinaugh. (2013). Randomized Controlled Trial of Mindfulness Meditation for Generalized Anxiety Disorder. J. Clin. Psychiatry, vol. 74 (08), 786-792. doi:10.4088/jcp.12m08083. 2. ^Elizabeth A. Hoge, Eric Bui, Sophie A. Palitz, Noah R. Schwarz, Maryann E. Owens, Jennifer M. Johnston. (2018). The effect of mindfulness meditation training on biological acute stress responses in generalized anxiety disorder. Psychiatry Research, vol. 262 , 328-332. doi:10.1016/j.psychres.2017.01.006. 3. ^Judith Esi van der Zwan, Wieke de Vente, Anja C. Huizink, Susan M. Bögels, Esther I. de Bruin. (2015). Physical Activity, Mindfulness Meditation, or Heart Rate Variability Biofeedback for Stress Reduction: A Randomized Controlled Trial. Appl Psychophysiol Biofeedback, vol. 40 (4), 257-268. doi:10.1007/s10484-015-9293-x. 4. ^Dan Jackson. (2017). Discussion on Quantifying publication bias in meta-analysis. Biom, vol. 74 (3), 795-796. doi:10.1111/biom.12819. Qeios, CC-BY 4.0 · Article, October 7, 2020 Qeios ID: 2C2WG8 · https://doi.org/10.32388/2C2WG8 14/14

measure. In particular, note the outcomes they studies, and how the outcomes were studied. We will input those in the GRADE tables. Hence it is important to put these items right away in a table to make more sense. Use a spreadsheet such as Google Docs to write these. Here is our Google Docs document where we have documented these elements. You have to do this based on the full text of the article and within the full text, read the methods and results sections carefully. All information should be available there. In particular, collect the following information: Author-year Population Intervention (as much details as you can) Comparator (Who was compared)

Outcomes
In the outcomes, pay attention to how the outcome was measured. Did they measure the outcome as a binary variable, or did they measure the outcome as a continuous variable?
These elements are the very basic elements to start with.
Based on the above three articles, we have filled in the following table: <!--td {border: 1px solid #ccc;}br {mso-data-placement:same-cell;}--> This step is essential because in GRADE the focus is on outcomes. If there are multiple studies that study the same outcome measure in the same way, then you should either pool the results of the studies together OR at least you should put them under the same outcome/outcome measurement group. If this is not possible, as we see in this example, then each study will be listed under one specific outcome measure, and we will select individual results or results that are not pooled. In our case, we see that there are 11 outcomes.
Step 5: Now set up the study questions, the PICO, and outcomes in GRADE First, click "Add Management Question", and fill in the boxes as follows: When you save this box, you will see the question and if you click on the question, you will see a box like the one below: Now click "Add outcome", when you do you will see the following set of eight boxes. Fill or address each one of the eight boxes for EVERY outcome-outcome measurement pair you have created in the spreadsheet (see Table 2). In our case this will mean doing this 11 times over! After filling in each time, press "Add outcome" to add a new outcome.
Qeios, CC-BY 4.0 · Article, October 7, 2020 Click "Save" on the blue button to return to the main page and continue with the next item.
4: Inconsistency too has three options: not serious, serious, and very serious. If you are appraising the outcome on the basis of single studies for a single outcome, then write "not serious". If on the other hand, you find different measures even for a single study have different results, then the entry could be "serious". For many studies, evaluate the outcomes reported and write whether you find that the direction and magnitude of the results in the individual studies vary a lot and enter "very serious". As an alternative, for an outcome, you can study the results of all studies and take a call to call it serious or very serious if you find gross inconsistencies in the values of the results and the direction of the results themselves.
5: Indirectness. -Assess how the outcome was measured. Did the authors obtain data directly from the participants in some way or did the authors or investigators depended on third party observations or indirect observations? If the authors obtained data directly from the participants, then the risk is "not serious". In our case, for example, the authors directly administered the instrument to the participants and obtained data from them, hence "not serious".