RESEARCH 301

VALIDATION STUDIES: VALIDATING NEW TOOLS & ADAPTING OLD ONES TO NEW CONTEXTS

GROUP 6 (MEDBLOCK – SECTION C | SURGBLOCK – SECTION A) | MARCH 15, 2023

VALIDATION STUDIES REMEMBER!!

• Determines the accuracy, dependability and consistency of a tool in True Positive (TP) Those with disease that tested positive
measuring what it is supposed to measure. True Negative (TN) Those without disease that tested negative
• What makes a tool “valid”? → If it measures what it is purported to False Positive (FP) Patients without disease but tested positively
measure False Negative (FN) Patients with disease but tested negatively
• There are 4 different types of validity LET’S APPLY:
FOUR DIFFERENT TYPES OF VALIDITY Using the test for rheumatoid factor (RF) that is positive in 95 of 100 rheumatoid arthritis
(RA) patients but also positive in 10 of 100 non-RA patients.
o Confirmation from a group of experts or other Given:
Face stakeholders o TP= 95
Validity o The extent to which respondents find the questions to be o FN= 5
clear and logical and to measure what the survey is o FP= 10
meant to investigate. o TN= 90
o All patients with RA = 100
Content o Checks whether all the items in the tool that should be
o All healthy patients (Non-RA) = 100
Validity included are included; identify relevance of each Sensitivity = true positives / all patients with target condition = (95/100) x
indicator and criterion % 100 = 95% sensitivity
Construct o Extent to which the items in the tool are as closely Specificity = true negatives / all healthy patients = (90/100) x 100 =
Validity associated as expected, according to theory % 90% specificity
Criterion- o Determines the ability of each criterion of the tool to Likelihood Ratio
Related measure accurately a specific concept or condition. 𝑃𝑟𝑜𝑏𝑎𝑏𝑖𝑙𝑖𝑡𝑦 𝑜𝑓 𝑎 𝑝𝑜𝑠𝑖𝑡𝑖𝑣𝑒 𝑡𝑒𝑠𝑡 𝑟𝑒𝑠𝑢𝑙𝑡 𝑤ℎ𝑒𝑛 𝑑𝑖𝑠𝑒𝑎𝑠𝑒 𝑖𝑠 𝑝𝑟𝑒𝑠𝑒𝑛𝑡 (𝑇𝑃)
LR 𝑝𝑟𝑜𝑏𝑎𝑏𝑖𝑙𝑖𝑡𝑦 𝑜𝑓 𝑡ℎ𝑒 𝑠𝑎𝑚𝑒 𝑡𝑒𝑠𝑡 𝑟𝑒𝑠𝑢𝑙𝑡 𝑤ℎ𝑒𝑛 𝑑𝑖𝑠𝑒𝑎𝑠𝑒 𝑖𝑠 𝑎𝑏𝑠𝑒𝑛𝑡 (𝐹𝑃)
Validity o To assess, compare with a “gold” or reference standard.
(+) 95
o This is the strongest form of validity
10 = 𝟗. 𝟓
MEASURES OF TEST ACCURACY LR 𝑃𝑟𝑜𝑏𝑎𝑏𝑖𝑙𝑖𝑡𝑦 𝑡ℎ𝑎𝑡 𝑖𝑛𝑑𝑖𝑣𝑖𝑑𝑢𝑎𝑙𝑠 𝑤𝑖𝑡ℎ 𝑡ℎ𝑒 𝑑𝑖𝑠𝑒𝑎𝑠𝑒 𝑤𝑖𝑙𝑙 ℎ𝑎𝑣𝑒 𝑎 𝑛𝑒𝑔𝑎𝑡𝑖𝑣𝑒 𝑡𝑒𝑠𝑡 (𝐹𝑁)
ACCURACY (-) 𝑝𝑟𝑜𝑏𝑎𝑏𝑖𝑙𝑖𝑡𝑦 𝑡ℎ𝑎𝑡 𝑖𝑛𝑑𝑖𝑣𝑖𝑑𝑢𝑎𝑙𝑠 𝑤𝑖𝑡ℎ𝑜𝑢𝑡 𝑡ℎ𝑒 𝑑𝑖𝑠𝑒𝑎𝑠𝑒 𝑤𝑖𝑙𝑙 ℎ𝑎𝑣𝑒 𝑎 𝑛𝑒𝑔𝑎𝑡𝑖𝑣𝑒 𝑡𝑒𝑠𝑡 (𝑇𝑁)
• Degree of closeness of the measurement to the true value of what is being 5
90 = 0.55 𝑜𝑟 0.6
measured
• Measured against a reference standard that confirms a patient has a MEASURES OF VALIDITY
HAS CONDITION DOES NOT HAVE
condition
(A+C) CONDITION
PROPERTIES OF DIAGNOSTIC OR SCREENING TESTS (B+D)
• Sensitivity POSITIVE TEST “A” “B”
• Specificity (A+B) (TRUE POSITIVE) (FALSE POSITIVE)
• Predictive values Aka. TYPE I ERROR
NEGATIVE TEST “C” “D”
• Likelihood ratios (C+D) (FALSE NEGATIVE) (TRUE NEGATIVE)
• Sensitivity and specificity are determined in validation studies AKA. TYPE II ERROR
• Comparing the test performance with the gold standard (reference False o Measures those testing positive who have no disease,
standard) Sensitivity and specificity are measures of the diagnostic Positive calculated by the number of people with a positive test
accuracy that are indicators of a test’s ability to distinguish between Rate with no disease divided by the number of all people
disease and absence of disease at a chosen cutoff without disease: B/(B + D)
SENSITIVITY False o is the rate of those with the disease who test negative,
• Ability of the test to detect the condition or disease Negative calculated by the number of people with a negative test
○ True positive rate Rate who have the disease divided by the number of all people
• Number of people with the disease who have a positive test divided by with disease: C/(A + C)
the number with the disease Positive o How well does it predict that someone has a disease?
○ Sensitivity = true positives / all patients with target condition Predictive o This is calculated by the number of people with a positive
• False Negatives (FN): patients with disease but tested negatively Value test who have the disease divided by the number of all
• A test with HIGH SENSITIVITY will have few FALSE NEGATIVES. the people with a positive test.
Negative o Will be the number of people with a negative test who do
Predictive not have the disease divided by all the people with a
Value negative test.
POSITIVE PREDICTIVE VALUE NEGATIVE PREDICTIVE VALUE
• PPV= TP / (TP + FP) • NPV = TN / (TN + FN)
SPECIFICITY • If there are changes in • If there are changes in prevalence
• Ability of the test to correctly identify people who do not have the condition
prevalence • Inc. Prev = Dec NPV
or disease • Inc Prev = Inc PPV • Dec Prev = Inc. NPV
○ True negative rate • Dec Prev = Dec. PPV
• Number of people without the disease who have a negative test divided HAS DOES NOT
CONDITION HAVE
the number of people without the disease
(A+C) CONDITION
○ Specificity = true negatives / all healthy patients (B+D)
• False Positive (FP): patients without disease but tested positively POSITIVE TEST “A” “B” POSITIVE
• A test that has high specificity will have a few FALSE POSITIVES (A+B) (TRUE (FALSE PREDICTIVE
POSITIVE) POSITIVE) VALUE (PPV)
Aka. TYPE I A/(A+B)
ERROR
NEGATIVE TEST “C” “D” NEGATIVE
(C+D) (FALSE (TRUE PREDICTIVE
NEGATIVE) NEGATIVE) VALUE (NPV)
AKA. TYPE II D/(C+D)
ERROR
LIKELIHOOD RATIO (LR) SENSITIVITY SPECIFICITY
• Incorporates both the sensitivity and specificity of the test A/(A+C) D/(B+D)
• Provides a direct estimate of how much test result will change the odds of EXAMPLE
having a disease
• LR is an assessment of test performance, and not of disease status
Likelihood ratio for a o How much the odds of the disease increase
positive result (LR+) when a test is positive
o LR+ = Sensitivity / (1 - Specificity)
Likelihood ratio for a o How much the odds of the disease decrease
negative result (LR-) when a test is negative
o LR- = (1 - Sensitivity) / Specificity
LIKELIHOOD RATIO INTERPRETATION
• LR = 1 represents no change
• The greater the value of LR+ above 1, the greater the likelihood of disease
• The greater below 1, the lesser the likelihood of disease

chismis buddies ☕| OLFU MED 2024|Page 1 of 3

 ▪ Word the question
▪ Sequence the questions
▪ Check questionnaire length
▪ Pre-test the questionnaire
▪ Develop the final question
• Designing a questionnaire means creating a valid and reliable questions
that address the research objectives and selecting an appropriate method
of administration
• Due to time and cost constraints of developing new questionnaires,
researchers often adapt existing questionnaires to better fit the purpose
of the study.
• Translated and/or adapted questionnaire should be identified through a
literature review, including a check that a questionnaire has not already
been developed adapted for the target population or country
Step 2: Translation and Back-translation of the Questionnaire
• Questions can first be prepared in English and then translated into the
local language using the back-translation method
• 2 steps:
DIFFERENT SCENARIOS WHEN CONDUCTING A VALIDATION 1. The original items of all the measures are translated into the local
STUDY language
1. Construct a questionnaire from scratch, and validate it against a gold 2. The local language version is then translated back to English
standard. • 5 Stages Needed to fulfill the 2 steps:
2. Adapt a questionnaire that was validated elsewhere to be used in a new o Stage 1: Translation: Two forward translation done for local language
context. o Stage 2: Synthesis: To resolve any discrepancies with the translator’s
3. Translate the questionnaire into another language for cross-cultural reports
adaptation. o Stage 3: Back-translation: Done by translators who are expert in the
Construct a questionnaire from scratch, and validate it against a gold English language
standard o Stage 4: Expert committee:
• It significant to search through various sources in order to find the most ▪ Consists of experts in fields related to are of interest and population
appropriate questionnaire for study. Once you've looked at several studied
references and received advice from experts, you can choose the best ▪ Who review translations and reach consensus on any discrepancy
reference standard that fits your needs to authenticate your questionnaire. o Stage 5: Pre-testing:
There are multiple research methods available to ensure its validity. ▪ Pre-Testing of the questionnaire for translation process conduction
Adapt a questionnaire that was validated elsewhere to be used in a on group of subjects
new context ▪ Modifications done
Step 3: Pre-testing of The Preliminary Questionnaire
• A researcher will frequently use a previously validated and published
questionnaire as a basis of reference for adapting it to the specific • Assess whether the questionnaire is acceptable to the participants.
purpose of a new study by tailoring it to fulfill the specific objectives of the • The subjects are not included in the study.
new study in order to save time and resources. • Pre-test is carried out to ensure that:
• A researcher may also need to conduct a preliminary literature review to o The right questions are being asked.
gain familiarity with the research area in order to determine what type of o All important and relevant areas are covered.
data will be collected by the questionnaire and also what new information o The questions are properly worded and flow in a logical manner.
will be generated by this new study o The questions are understandable and acceptable to the participants
Translate the questionnaire into another language for cross-cultural • Feedback and ideas from the subjects are collected and where
adaptation necessary, further formal discussions can be conducted with the expert
• A poor translation process may lead to an instrument that is not equivalent team to finalize the questionnaire
to the original questionnaire • Internal consistency reliability for the newly developed questionnaire can
• Translation of the original questionnaire is an important step for be done using Cronbach’s alpha
establishing the validity of a questionnaire. • Cronbach’s Alpha:
• This process involves initially translating the questionnaire from the o For newly developed questionnaire
foreign language into a local language and at the same time matching the o It is a measure of internal consistency reliability by comparing the
semantic feature of each item or question as closely as possible with that amount of shared variance or covariance
Cronbach’s Alpha Internal Consistency
of the original questionnaire
α ≥ 0.9 Excellent
• It is often possible for incorrect responses to be elicited from the 0.9 > α ≥ 0.8 Good
respondents by merely translating the questionnaire and then 0.8 > α ≥ 0.7 Acceptable
administering it to a group of respondents who are conversing in a 0.7 > α ≥ 0.6 Questionable
different language from that of the original questionnaire. 0.6 > α ≥ 0.5 Poor
HOW TO DEVELOP A QUESTIONNAIRE 0.5 > α Unacceptable
For criterion validity:
o Suitable questionnaires should be chosen
o Evaluates how accurately a test measures the outcome it was designed
to measure
▪ Outcome can be a disease, behavior, or performance
Convergent Validity Concurrent Validity
• Testing questionnaire are • Testing questionnaire are
correlated to another questionnaire compared to a well-established
which measures the same domain questionnaire or interview
considered as a gold standard.

Figure above: Cross-cultural adaption of a questionnaire

• Researchers in low and middle-income countries VALIDATION OF THE TESTING QUESTIONNAIRE AGAINST THE
• May take anywhere from six months to three years, depending on the GOLD STANDARD
stages of validation involved • To find a gold standard questionnaire may require an extensive literature
Step 1: Development of a Preliminary Questionnaire appropriate reference standard.
• The questionnaire to be translated and adapted should be identified • Validation of the testing questionnaire can be done through various study
through a literature review designs
• Check that a questionnaire has not already been developed or adapted GOLD STANDARD TECHNIQUES
for the target population or country • Experimental model that has been thoroughly tested and has a reputation
• A questionnaire is a specific tool or instrument for collecting data in the field as a reliable method (Cardoso [Link], 2014)
• Questionnaire development involves rigorous testing to ensure reliability • Careful review of materials and formal discussions with the expert
and validity persons or teams involve
○ It includes 9 sub-steps: • Use of various study designs: Double-blinded randomized controlled trial
▪ Decide on the information required
▪ Select the method of reaching the respondents
▪ Determine the question content

chismis buddies ☕| OLFU MED 2024|Page 2 of 3

 IMPORTANCE OF VALIDATING NEW QUESTIONNAIRES IN PRIMARY
CARE RESEARCH
• Involves questionnaires assessing:
○ Behaviors
○ Knowledge
○ Attitude
○ Practices
o Mental Depression o HPN o COPD
o Anxiety o Diabetes o Self-care
o Somatization o Heart Disease o Health risk
o Pain o Asthma o Health Screening
• To have new measures that accurately determine the presence and
severity of existing and emerging diseases in primary care.
• However, there is also a need to adapt old questionnaires to new contexts
in primary care research, especially well-established and validated
questionnaires which have been proven to be simple and effective in
determining common health care problems.

chismis buddies ☕| OLFU MED 2024|Page 3 of 3