It can be challenging to pick the right instrument to study the outcomes of patients. Taking the time to investigate if an instrument is valid and reliable will make your research or clinic findings stronger. In this issue of Discovery, we consider how instruments are psychometrically evaluated and the importance of it.
Valid and Reliable Instruments
Validity refers to the degree to which an instrument accurately measures what it intends to measure.
Reliability refers to the degree to which an instrument yields consistent results.
While researching what instruments to use in a research study, we look for instruments that claim ‘they are well validated’ and ‘have strong reliability.’ But do we ever take the time to consider what ‘well validated’ and ‘strong reliability’ truly mean? Most people do not consider how an instrument’s reliability and validity was assessed and proceed to use it in their research study or clinical site and reference it.
In 2018, Monique Bouvier, PhD, RN, PNP-BC from the SHC Office of Research collaborated with Ann Mayo, DNSc, RN, CNS, FAAN from the University of San Diego, were considering using the Canadian Acute Respiratory Illness and Flu Scale as an instrument in a research study, and were trying to determine if it was an instrument that could render reliable results for patient care. They took the time to investigate how the reliability and validity was assessed. They psychometrically evaluated the instrument prior to using it, and were surprised by the findings. They ended up writing an article about the psychometric evaluation process and published their results in the Journal of Pediatric Nursing.
The Canadian Acute Respiratory Illness and Flu Scale (CARIFS) was originally developed by Jacobs et al. (2001) for use in research to measure acute respiratory illness (ARI) disease severity in children, capturing the healthcare provider and parents’ perspective about a child’s ARI symptoms. A deep dive review of the literature showed that the CARIFS has been used in not only in the research setting, but also in the clinical setting and had been well disseminated and translated into several languages. When considering any instruments to use in clinical practice, you also want to make sure they are truly valid and reliable for whatever it is they are measuring; because they may not be capturing what you think they are. For example, an instrument can say it is collecting information on anxiety, but it is capturing information on depression.
As stated earlier, the intent of the above-mentioned article was to assess how psychometric testing was performed on CARIFS, and issues found with the validity and reliability. Several limitations with reliability and validity were discovered after taking the time to investigate the origins of the instrument. For instance, the items on the instrument were characterized in three dimensions, but they seemed to be investigator developed dimensions; therefore, the dimensions do not hold much merit when someone may reference them. Additionally, the 18 items focused on physical symptoms only, and psychological or situational factors were not considered to have any influence with symptoms. Literature has shown that symptoms are multi-dimensional; therefore, to truly capture subjective symptoms, one should consider outside factors influencing them such as situational or environmental factors. For instance, it was not captured if cold medication was given before rating the physical symptoms, or what the mental health of the patient was like. Both the above situations can influence how symptoms present, but that information is not captured as part of the tool. In regards to reliability, test-retest reliability did not occur at the same times nor under standard conditions. Therefore, the reliability of the instrument is put into question because someone would want the same conditions to make sure the instrument is stable. Most people’s symptoms differ from the morning when you just wake up to the late afternoon after a long day. Having test-retest by comparing a morning and an afternoon score makes it difficult to know if the instrument is stable for any specified time frame.
This article highlights why you need to further investigate an instrument you plan to use in your clinic, hospital, unit, or research study. If the instrument is not psychometrically sound, then the results and merit of your study or even how your patients are doing will be questioned. Make sure you are choosing an instrument that is specific for your population, and if it isn’t you can psychometrically evaluate properties of it yourself.
Tips to remember:
- Every instrument was created to address a specific population and condition. If you intend to use the instrument in a population outside of what it was designed for, you should check into the validity and reliability of the instrument for that population. If it does not exist, you should perform validation studies first.
- If you change the format of an instrument, like from paper to electronic, the reliability and validity should be assessed for the instrument in the electronic format as it was created to be delivered on paper.
- If you modify an instrument to add or subtract questions or even slightly changing the wording of questions, the instrument is no longer the same; therefore, it is no longer valid or reliable. You must assess the validity and reliability of the new instrument you created from the original one.
- Instrument creation is sometimes necessary when you cannot find a specific tool out there for what you need. You should always go about the development of your tool utilizing a specific framework to minimize bias.
Article By: Monique Bouvier