Education

There are several reasons to conduct a power analysis to determine sample size before applying for grant funding. This article describes how to conduct a simple power analysis using a free online tool for a study that compares two independent groups.

Power Analysis

What is a Power Analysis and why is it needed for a Quantitative Research Study Grant Application, such as the Legacy Grant?

A power analysis is a statistical test that estimates the number of participants needed for a research study that compares two or more groups. A power analysis is often required for grant applications involving quantitative research studies and is highly recommended when applying for the Stanford Alumnae Legacy Grant.

It is helpful to have a statistician estimate the power analysis. This is because there are significant variations in study designs that alter the “power” to detect a significant difference between groups. If a study is “underpowered” there will not be enough patients to answer the research question.  Conversely, if the study is “overpowered” the study will have more patients than are needed. Both issues are considered poor research design. The power analysis is designed to mirror the Goldilocks porridge test and find a sample size that is “just right” to answer the research question.

This article describes the information that is needed to conduct a simple power analysis.  To find out the sample size for a study that compares a continuous outcome between two groups of participants, a researcher would need to know two things: standard deviation and effect size. Detailed examples using a free online tool are provided below.

Example: A researcher plans to test the efficacy of a new non-pharmacologic intervention to shorten the duration of delirium in adults 65 years and older in the hospital. The research team wants to know how effective the new intervention will be, compared to standard treatment. This research study will enroll older patients with delirium and randomize them into 2 independent groups: intervention (non-pharmacological treatment) and control (standard care).

Measurement: First, the research team needs to decide what instrument will be used to measure delirium and check whether the tool has published validity and reliability data. In the case of delirium, there are several tools available and most score delirium as either positive (has delirium) or negative (no delirium). This is a binary yes/no metric. Next, the research team will need to measure the difference in duration of delirium between the “non-pharmacologic treatment” intervention compared to the “standard care.” In other words, they want to know how quickly the delirium resolved in one group compared to the other. This outcome time could be measured in minutes, hours or days, which is a good example of a continuous variable. In this study, the outcome is measured in hours.

Standard Deviation: The standard deviation (SD) is can be a difficult number to find unless there is prior data from other published studies or the researcher already has some pilot data.  The standard deviation is used in the power analysis / effect size formula, and it represents the variability within the group.  In this case, the different ranges of time in hours needed for older patients (over 65 years) with delirium to regain their normal baseline mental status following different treatments, with distance from the mean. For example: if the delirium group mean was 30 hours, with an SD of ± 15 hours, patients’ delirium could last anywhere from 15 to 45 hours.

Effect Size: Effect size reflects the magnitude of the hypothesized relationship, or in our case how important is the difference between intervention and control groups. In an environment where the goal is to generate new knowledge, the research team may not know beforehand how well the new non-pharmacologic treatment works. Therefore, they may use clinical data or similar prior publications to estimate the anticipated effect size.

Large Effect Size (0.8): If the research team believes that the treatment is going to work well and that it will dramatically shorten the duration of delirium, compared to the control group, this is described as a large effect size, and is numerically described as 0.8.

Moderate Effect Size (0.5): If the research team thinks the new intervention will shorten delirium duration by a moderate amount, they will select a moderate effect size, numerically described as 0.5.

Small Effect Size (0.2): If the research team thinks that the new treatment will help slightly compared to the control group, they select a small effect size, numerically described as 0.2.

The effect size has a direct impact on sample size. With a large effect size, the impact is often described as “visible to the naked eye” or obvious, and only a small sample is needed.  Whereas, a moderate effect size (0.5) or a small effect size (0.2) will require significantly larger samples to detect differences between groups.

The formula for the effect size is shown below:

Before approaching a statistician, a researcher might want to do some basic power calculations on their own. This is easier to do today because there are online tools that simplify the process – the tools hide the mathematical calculations in the background.

One example is: ClinCalc(1):  https://clincalc.com/stats/samplesize.aspx

The examples below were created online with ClinCalc for two independent groups with a continuous variable.

Suggestion: Open ClinCalc online and plug-in the numbers below for 2 independent groups. (3)

For this hypothetical example, assume that delirium lasts 40 hours with standard treatment (Group 2 in ClinCalc). Additionally, assume there is some literature to suggest that the new treatment reduces delirium to 30 hours (Group 1 in ClinCalc).  That sounds very promising for the new treatment. However, the researcher does not yet know how many patients with delirium will be needed to show statistical significance (alpha .05 in ClinCalc). The examples below compare different group effect sizes.

————————————————————————————————————————————————————————–

Large effect size example:

This example has 2 independent study groups with a continuous variable metric (delirium duration in hours). Assume that the literature suggests that with the new non-pharmacologic intervention, the delirium will last an average of 30 hours with a standard deviation (SD) of 10 hours.  This compares with a mean for standard care of 40 hours. In this example, the members of the two groups will have limited overlap so this is considered a large effect size.

Group 1: 30 hours mean ± 10 hours SD                  (intervention group)

Group 2: 40 hours                                                         (control group)

– set the alpha at .05

– set the power (Beta -1) at 0.8 (80%)

Group Size with a large effect size: 16 subjects in each group. Total 32 patients.

————————————————————————————————————————————————————————–

Moderate effect size example:

This example has 2 independent study groups with a continuous variable metric (delirium duration in hours). Now imagine that the literature indicates that with the new intervention delirium will still last an average of 30 hours, but with a larger standard deviation of 15 hours.  In other words, there is potentially more overlap between patients in the two groups

Group 1: 30 hours mean ± 15 hours SD                  (intervention group)

Group 2: 40 hours                                                         (control group)

– set alpha at .05

– set power (Beta -1) at 0.8 (80%)

Group Size with a moderate effect size: 35 subjects in each group. Total 70 patients

————————————————————————————————————————————————————————–

Small effect size example:

This example has 2 independent study groups with a continuous variable metric (delirium duration in hours).  Suppose that the literature says with the new intervention delirium will still last an average of 30 hours with a much wider standard deviation of 25 hours.  In other words, there is a lot of overlap between patients in the two groups

Group 1: 30 hours mean ± 25 hours SD                  (intervention group)

Group 2: 40 hours                                                         (control group)

– set alpha at .05

– set power (Beta -1) at 0.8 (80%)

Group Size with a small effect size: 98 subjects in each group. Total 196 patients

————————————————————————————————————————————————————————–

Summary of Effect Size and Sample Size Example

Using the examples above, it appears that the researchers could use a total sample size as low as 32, 70, or as high as 196 for this delirium study.

Effect Size, Sample Size and Grant Funding

The use of an online tool such as ClinCalc can be helpful when designing a quantitative research study because it allows the researcher to play with some “what if” scenarios that will be helpful when applying for funding. For example, the Stanford Alumnae Legacy Grant awards a maximum of $10,000. Using the examples above, this award may provide adequate funding for a sample size of 32 (large effect size), or 70 (moderate effect size), but it may be insufficient for the sample size of 196 (small effect size).  This also depends on the methodology and other resources that will be used. However, it is important to understand that grant reviewers expect sample size issues to be presented with rationales in the application.

Other Resources for Estimating Sample Size

There are many software programs that can be used to estimate sample size, but most have a purchase or subscription cost. The most well-known free program is G-Power.(2) A different option is ‘PS: Power and Sample Size Calculation’ from Vanderbilt University.(3) Also, IBM SPSS has a program named ‘SamplePower’ designed to calculate sample size as an add-on to their SPSS statistics program.(4)

Conclusion

While a consultation with a statistician is optimal to calculate a sample size, this opportunity is not always available. Alternatively, before meeting with a statistician, an online tool such as ClinCalc can be used to calculate a potential sample size that will meet study objectives and good research design.

In summary, establishing a population mean and standard deviation for the variable of interest, determining effect size and calculating power analysis for sample size is highly recommended before applying for grant funding. This process will ensure that the sample size requirements match the grant award.

References

  1. Kane SP. Sample Size Calculator. ClinCalc: https://clincalc.com/stats/samplesize.aspx Updated July 24, 2019. Accessed August 26, 2019.
  2. G-Power version 3.0: http://www.psychologie.hhu.de/arbeitsgruppen/allgemeine-psychologie-und-arbeitspsychologie/gpower.html Accessed August 28, 2019.
  3. PS: Power and Sample Size Calculation version 3.1.6, http://biostat.mc.vanderbilt.edu/wiki/Main/PowerSampleSize Accessed August 28, 2019.
  4. IBM SPSS Sample Power: https://www.spss.ch/upload/1282226383_SamplePower%203.pdf Accessed August 28, 2019.

 

 

 

 

 

 

 

 

 

 

 

 

 

Article By: Mary E. Lough

Book an appointment or get a consultation.