Attribute Agreement

Analysis

Global Lean Six Sigma

Aug 2020
Attribute Agreement Analysis

Objectives
•
Understand the application of Attribute Agreement Analysis on Binary, Nominal
and Ordinal data.

•
Interpret Attribute Agreement Analysis from 3 approaches
•
Assessment of Agreement
•
Kappa Statistics
•
Kendall Coefficient
Measurement System Analysis (MSA)

Why MSA?
• One concern is the misclassification:
 “Parts” inside specs may actually measure outside the spec
 Unnecessary scrapping or reworking
• “Parts” outside the spec may actually measure inside the spec
 Ship unacceptable product to customer.
• Another concern is determining if the measurement system is capable of detecting a
difference between “parts”

Where to Start?
• The very first step is to know if the difference among good and bad parts is caused by
product variation or measurement variation.
• Start with your project Y and then any other measurements that will be utilized to
characterize the process
3
When is MSA implemented?

• As a criterion to accept new measuring equipment

• As applicable, prior to a process capability study.

• When a key characteristic or process is not capable.

• When the measurement system is suspected of being a significant source of

variation as part of the overall system (Product, Process, Gage).

• When there are major changes to the measurement system.

• When preparations are being made to conduct a Design of Experiment (DOE).

When is MSA implemented? (Cont…)

As required by any AQP project, as an example refer to following:

• AQP Transfer Project Requirements:

o Plan Phase:
 Transfer Site provides current equipment studies
 Receiving Site defines where studies are required, and reviews transfer site
results
o Evaluation Phase
 Receiving Site initiates execution Operator and Equipment MSA define in plan
phase
o Validation Phase
 Receiving Site completes and reviews MSA results
o Confirm Phase
 Receiving Site completes Cpk/Ppk studies
 Measurement System Analysis (MSA)

Two broad types of Measurement Systems

• 1. Attribute (Discrete)
• Data cannot be adequately described on a continuous scale
• Examples:
 Binary : Pass / Fail, Good /Bad, Go/No go. Kappa
 Nominal (no natural ordering) :
Defect type A, B, C…H / Lifted, Broken, Missing Kendall
 Ordinal (with natural ordering) : Low/Med/Hi, Cold/cool/
tepid/warm/hot/scalding, Satisfaction rate on a 1-5 scale

• 2. Variable (Continuous)
• Data can be described on a continuous scale
• Examples:
 Length, height, gap, thickness, Torque, Speed, Taste, surface finish, time 6
Attribute Agreement
Analysis (AAA)
Measurement System Analysis

Purposes of Study
• To determine if inspectors (appraisers) across all shifts, machines, lines, etc… use
the same criteria to discriminate “good” from “bad”
• To quantify the ability of inspectors (appraisers) or gages to accurately repeat their
inspection decisions over time
• To identify how well inspectors/gages measure a known master (possibly defined by
the customer) to ensure no misclassification occurs
• To determine areas where:
 Training is needed
 Procedures or control plans are lacking
 Standards are not clearly defined
 Gage adjustment or correlation is necessary 8
AAA General Requirements

Per Jabil procedures an Attribute Gauge R&R should be performed for appraisers
when any of the following situations may occur:
• A new appraiser is appointed.
• A new product released for production has major change in complexity or
customer´s requirement/expectations.
• At regular intervals to ensure appraiser capability is maintained. The interval
frequency should be defined based on production quality performance (bi-annual
assessment recommended but not required).
• Issues related to appraiser skills arise.
• Inspector certification / re-certification process.
Steps to perform AAA

AAA Process Steps

1. Select Measurement • # of operators (who’s involved in

System measurement)
•# of measurements (is there
2. Develop Measurement Outputs test-retest problem)
Strategy

3. Collect Samples Outputs

• Balance Good & Bad samples
• Record known standard
4. Prepare Measurement
• See next Slide
Data Sheet

5. Perform MSA
Study •Assessment of Agreement
•Binary & Nominal – Kappa
6. Analyze MSA
Results in Minitab
Outputs • Ordinal – Kappa & Kendall

7. Interpret MSA • Look at Graphical and Statistic to

Outputs
Results
Identify weakness(es)

8. Outline Next Steps Outputs • Optimization Plan

10
AAA Guidelines

• When planning to conduct an Attribute MSA, follow the Steps for AAA taught in Green Belt
training.

• The following elements require planning prior to the Attribute MSA.

• Operators selection

• Samples preparation

• Assessment process.
AAA Guidelines

• Operator Selection:

 Minimum of 2 inspectors & 2 trials each depending on what you are trying to
learn.

• Sample Size Selection

 More is better. Preferred sample size is 30 – 50 pieces, keep max within 100
pieces.

 100 samples, minimum of 2 inspectors, 2 trials will give us 95% confidence in

the Statistical calculation

12
AAA Guidelines

Sample Part Selection

 Parts in the study should represent the full range of variation of interest
 For maximum confidence in the statistic, we would like to have 50:50 mix of good/bad parts.
30:70 ratio is acceptable – beyond this level, single disagreements can have large leverage
on the Statistical calculation
- E.g. For sample size of 100, have 50 good and 50 bad parts
 When you have 3 or more categories, with one of the categories being good and the other
categories being different error modes, you should have approximately 50% of the items
being good and a minimum of 10% of the items in each of the error modes.
 When you have only 2 categories, good and bad, you should have a minimum of 20 good and
20 bad
 Have a variety of degrees of good and bad; obvious good, obvious bad, borderline good and
borderline bad samples
 You might combine some of the error modes as “other”
 The categories should be mutually exclusive or, if not, they should also be combined 13
AAA -Setup Data Collection Sheet with Minitab

1. Choose Stat > Quality Tools > Create

Attribute Agreement Analysis Worksheet.

2. In Number of Parts, enter # Part to be tested.

3. In Number of Operators, enter # Operator to

be involved in the AAA.

4. In Operator Name, enter name of inspector

involved in AAA.

5. In Number of replicates, select the # test-

retest

6. Click the Options button. Choose Randomize

runs within operators.

7. Click OK in each dialog box.

14
AAA Guidelines

Assessment Process:
 Each Inspector will perform per planned number of trials and samples
 Randomizing samples between each trial and ensure blind test (Inspector
should not be able to recognize the sample due to part identification during this
assessment)
 Attribute Gauge R&R process to be conducted in a controlled/designated area
 Inspectors follow SOP, WI, Visual Aids or IPC-A-610 to determine pass/fail
(Nominal) or classification (Ordinal) decision for each sample
 The time given to inspect each unit during the Attribute Gauge R&R shall align
with the production takt time allotted for the inspection of each unit
 The results are entered into Data collection worksheet

15
Different Analysis for Attribute Data

Binary

Pass / OK

Fail / NOK Nominal

(Failure Mode /
Failure Type) Ordinal
Assessment (With natural order /
Type A / Lifted
of Severity)
Agreement Type B / Broken 1 - Minor

2 – Moderate
:
3 – Significant
Type H / Misaligned
4 – Major

Kendall's
Kappa Coefficients
Statistics + Assessment
+ Assessment Agreement +
Agreement Kappa
16
AAA - Analysis Focus

1. Standard Attribute GR&R: Assessment of

Agreement
Expectation:
 Require a minimum score of 90%
 Obvious good and obvious bad
would be expected to be 100%.

2. Kappa Statistics: 3. Kendall's Coefficients :

 Binary : Pass / Fail, Good /Bad, Go/No  Ordinal (with natural ordering) :
go. Low/Med/Hi, Cold/cool/
 Nominal (no natural ordering): Defect tepid/warm/hot/scalding,
type A, B, C…H / Lifted, Broken, Satisfaction rate on a 1-5 scale
Missing
Consistency Correctness With Standard The general Rule Of Thumb
Reference (ROT) for interpreting Kappa
Statistics & Kendall's Coefficients :
WITHIN Within Appraiser: Assess Each Appraiser vs.
(REPEATABILITY) the consistency of Standard: Assess the • < 0.75 Measurement System
responses for each correctness of responses for needs attention
appraiser each appraiser • 0.75 – 0.9 Generally
acceptable, improvement may
BETWEEN Between Appraisers: All Appraisers vs. Standard: be needed depending on
(REPRODUCIBILITY) Assess the consistency of Assess the correctness of application and risk.
responses between responses for all appraisers • > 0.9 Excellent Measurement
appraisers System
17
 Attribute GRR
- Assessment of
Agreement
AAA - Assessment of Agreement

Standard Attribute GR&R: Assessment of

Agreement
Expectation:
 Require a minimum score of 90%
 Obvious good and obvious bad would be expected to be 100%.

Correctness With
Consistency
Standard Reference
WITHIN Within Appraiser: Each Appraiser vs.
(REPEATABILITY) Assess the consistency Standard:
of responses for each Assess the correctness of
appraiser responses for each
appraiser
BETWEEN Between Appraisers: All Appraisers vs.
(REPRODUCIBILITY Assess the consistency Standard: Assess the
) of responses between correctness of responses for
appraisers all appraisers
Example

Inspectors at a manufacturing
plant reviewed the production
parts according to the standards
given by the QE, thereby decide
on the final disposition.
To assess the consistency and
correctness of the inspectors'
ratings, the QE asks 5 inspectors
to rate the quality of 40 samples
twice. Samples were randomly
presented.

18.02.2AAA Example.mtw
Minitab Attribute Measurement Systems

• To perform an attribute agreement analysis on Binary

scale in Minitab, choose
• Open the sample data, 18.02.2AAA Example.MTW.
• Stat > Quality Tools > Attribute Agreement
Analysis
• In Data are arranged as, select Attribute
column and select the column containing Rating.
• In Samples, select the column containing the
Sample.
• In Appraisers, select the column containing the
Appraiser. DO NOT check this box
• In Known standard/attribute, select the column
containing the Standards.
• Click Results, select Percentages of assessment
agreement within and between appraisers.
• Click OK
Interpret Minitab Analysis:
Step 1: Evaluate the appraiser agreement visually

John, Ken, John, Ken &

Mary & Rose Rose have the
have most most correct
consistent ratings, while
ratings, only Keith is 85%
Keith has the matched, has
least the least correct
consistent ratings.
ratings.
Interpret Minitab Analysis
Step 2: Assess the consistency of responses for each appraiser

This output represents the percent agreement of the

appraisers and the 95% confidence interval on the
agreement of the appraisers.

Expectation:
 Require a minimum score of
90%
 Obvious good and obvious bad
would be expected to be 100%.

John, Ken, Mary & Rose scored 100%. They have consistent conclusions of each samples when
inspected twice over all 40 samples.
Keith, however scored 95%, has 38 samples with matching conclusions, out of 40 inspected. We
are 95% confident that Keith’s performance ranges between 83% and 99%
Interpret Minitab Analysis
Step 3: Assess the correctness of responses for each appraiser

John, Ken and Rose have good agreement with the

Standard i.e. 100% matched.

Keith and Mary did not match all samples with

Standard, 85% & 97.5% respectively; which we need
to investigate the differences.

Keith has Passed1 Sample but should have failed

(OK/FAIL); Failed 3 Samples but should have passed
(FAIL/OK) and made inconsistent ratings across trials
on 2 samples (Mixed); while Mary has Passed 1
Sample but should have failed. (OK/FAIL).

#OK/Fail column contains the number of samples that the appraiser passed
but should have failed—a false positive.
#Fail/OK column contains the number of samples that the appraiser failed
but should have passed—a false negative.
#Mixed column indicates the number of times that the appraiser made
inconsistent ratings across trials.
Interpret Minitab Analysis
Step 4: Assess the consistency of responses between appraisers

The 5 appraisers achieved 82.5%

agreement with 33 matching
assessments out of 40 samples.
Interpret Minitab Analysis
Step 5: Assess the correctness of responses for all appraisers

5 appraisers achieved 82.5% agreement versus Standard. There is still some work
needed as the expectation is 90%
Interpret Minitab Analysis
Assessment of Agreement - Summary

Correctness With
Consistency
Standard Reference
•All above 90% •Keith matches the true
(REPEATABILITY)

agreement result 85% of the time

WITHIN

•All Inspectors agreed •All Inspectors agreed with

between each other the true data 82.5% of the
(REPRODUCIBILITY)
BETWEEN

82.5% of the time time

Conclusion: Improvement should be focus on Keith. Consistency seems good but
not the correctness. Attention should be to understand and eliminate the diff to
Standards
Attribute GRR
- Kappa
Statistics
Attribute Agreement Analysis (AAA)
- Analysis Focus

1. Standard Attribute GR&R: Assessment of Agreement

Expectation:
 Require a minimum score of 90%
 Obvious good and obvious bad
would be expected to be 100%.

2. Kappa Statistics:
 Binary : Pass / Fail, Good /Bad, Go/No go.
 Nominal (no natural ordering): Defect type A, B, C…H / Lifted, Broken,
Missing
The general Rule Of Thumb
Correctness With
Consistency (ROT) for interpreting Kappa
Standard Reference
Statistics :
WITHIN Within Appraiser: Each Appraiser vs.
(REPEATABILITY) Assess the consistency Standard: Assess the • < 0.75 Measurement System
of responses for each correctness of responses needs attention
appraiser for each appraiser • 0.75 – 0.9 Generally
BETWEEN Between Appraisers: All Appraisers vs. acceptable, improvement may
(REPRODUCIBILITY) Assess the consistency Standard: Assess the be needed depending on
of responses between correctness of responses application and risk.
appraisers for all appraisers • > 0.9 Excellent Measurement
System
AAA: Kappa Statistics

Kappa compares the proportion of agreement between the inspectors after removing
agreement by chance.

• Kappa Statistics for 2 inspectors:

Notation:
: Proportion that the inspectors are in agreement
Kappa
: Proportion expected to occur by chance
= (P Insp1 Good)(P ) + (P
Insp2 Good Insp1 Bad )(PInsp2 Bad)

• Kappa Statistics for Multiple inspectors & Categories:

Notation:

Source: “When Quality is a Matter of Taste, Use Reliability Indexes”, D. Futrell, Quality Progress, May 1995
AAA: Kappa Statistics
- Guidelines

 Use Kappa statistics to assess the degree of agreement of the nominal (or ordinal)
ratings made by multiple Inspectors when the Inspectors evaluate the same samples.
• Kappa statistics treat all misclassifications equally
• Kappa values range from –1 to +1.
• The higher the value of kappa, the stronger the agreement. When:
• Kappa = 1, perfect agreement exists.
• Kappa = 0, agreement is the same as would be expected by chance.
• Kappa < 0, agreement is weaker than expected by chance; this rarely occurs.
The general rule of thumb for interpreting Kappa Statistics:
• < 0.75 Measurement System needs attention
• 0.75 – 0.9 Generally acceptable, improvement may be needed depending on
application and risk.
• > 0.9 Excellent Measurement System
Example – Binary (Pass / Fail)

Inspectors at a manufacturing
plant reviewed the production
parts according to the standards
given by the QE, thereby decide
on the final disposition.
To assess the consistency and
correctness of the inspectors'
ratings, the QE asks 5 inspectors
to rate the quality of 40 samples
twice. Samples were randomly
presented.

18.02.2AAA Example.mtw
Minitab Attribute Measurement Systems

• To perform an attribute agreement analysis on Binary

scale in Minitab, choose
• Open the sample data, 18.02.2.AAAExample.MTW.
• Stat > Quality Tools > Attribute Agreement
Analysis
• In Data are arranged as, select Attribute
column and select the column containing Rating.
• In Samples, select the column containing the
Sample.
• In Appraisers, select the column containing the
Appraiser. DO NOT check this box

• In Known standard/attribute, select the column

containing the Standards.
• Click Results, select Kappa & Kendall coefficients
• Click OK
• Click OK
 Interpret Minitab Analysis:
Step 1: Evaluate the appraiser agreement visually

Similar graphical display as Assessment of Agreement

 Interpret Minitab Analysis
Step 2: Assess the consistency of responses for each appraiser

Similar graphical display as Assessment of Agreement

Interpret Minitab Analysis
Step 2: Assess the consistency of responses for each appraiser

The general rule of thumb for interpreting Kappa Statistics:

• < 0.75 Measurement System needs attention

• 0.75 – 0.9 Generally acceptable, improvement may be

needed depending on application and risk.

• > 0.9 Excellent Measurement System

John, Ken, Mary & Rose achieved Kappa = 1, for both categories
of samples. They have perfect agreement.

Keith achieved Kappa = 0.89975; in general is acceptable, BUT

we want to investigate why Keith’s ratings were inconsistent.
 Interpret Minitab Analysis
Step 2: Assess the consistency of responses for each appraiser

Between Appraisers assess the consistency of responses between appraisers.

Among the 5 appraisers, they achieved

82.5% agreement with each other.

The Kappa scores of 0.89 on both

categories showed an acceptable
assessment, improvement may be
needed.
Interpret Minitab Analysis
Step 3: Assess the correctness of responses for each appraiser

John, Ken and Rose have good agreement with the

Standard.

Keith and Mary have some differences with Standard

which we need to investigate these differences.

Keith has accepted1 defective sample and rejected 3

good samples; while Mary has accepted 1 defective
sample.

Most of the kappa values are larger than 0.80, which

indicates good agreement between each appraiser and
the standard. Keith’s Kappa values are less than 0.75,
which indicates that you may need to investigate further.
Interpret Minitab Analysis
Step 4: Assess the consistency of responses between appraisers

All the kappa values are between 0.75 and

0.9.
Generally acceptable, improvement may be
needed
The 5 appraisers have similar agreements
for samples of bother categories.
Interpret Minitab Analysis
Step 5: Assess the correctness of responses for all appraisers

These results show that all the appraisers correctly matched the
standard ratings on 33 of the 40 samples. The overall kappa value
is 0.9399, which indicates strong agreement with the standard
values.
Interpret Minitab Analysis
Kappa Statistics Summary

Correctness With
Consistency
Standard Reference
• Except Keith at 82.5%, All • Keith matches the true result
others above ≥95% 85% of the time, while Mary’s
(REPEATABILITY)
WITHIN

agreement and his at 97.5% and Keith’s K=

K=0.8997 0.7497 ; Mary’s K=0.9499

• All Inspectors agreed • All Inspectors agreed with the

between each other true data 82.5% of the time
(REPRODUCIBILITY)

82.5% of the timeshould

and be focus
and their K=0.9399
BETWEEN

Conclusion: Improvement on Keith. He has both

consistencytheir K=0.8911 issues. Attention should be to understand
& correctness
and eliminate both issues.
Attribute Agreement Analysis (AAA)
- Analysis Focus

1. Standard Attribute GR&R: Assessment of Agreement

Expectation:
 Require a minimum score of 90%
 Obvious good and obvious bad
would be expected to be 100%.

2. Kappa Statistics:
 Binary : Pass / Fail, Good /Bad, Go/No go.
 Nominal (no natural ordering): Defect type A, B, C…H / Lifted, Broken,
Missing
The general Rule Of Thumb
Correctness With
Consistency (ROT) for interpreting Kappa
Standard Reference
Statistics :
WITHIN Within Appraiser: Each Appraiser vs.
(REPEATABILITY) Assess the consistency Standard: Assess the • < 0.75 Measurement System
of responses for each correctness of responses needs attention
appraiser for each appraiser • 0.75 – 0.9 Generally
BETWEEN Between Appraisers: All Appraisers vs. acceptable, improvement may
(REPRODUCIBILITY) Assess the consistency Standard: Assess the be needed depending on
of responses between correctness of responses application and risk.
appraisers for all appraisers • > 0.9 Excellent Measurement
System
Example – Nominal (Style, A, B, C, D or E)

• An organization is training five new

appraisers for the classifying essay into
different written structure:

• The appraisers‟ ability to classify

essays consistent with the standards
needs to be assessed
• Each appraiser rated fifteen essays into
one of the 5 styles (A,B,C,D,E)
• The organization also rated the essays
and supplied the “official score”
• Each essay was rated twice

18.02.3AAAforNominalandOrdinal.xlsx
 Minitab Analysis

Stat > Quality Tool > Attribute Agreement Analysis

• To perform an attribute agreement analysis on

Nominal scale in Minitab, choose
• Open the sample
data, 18.02.3AAAforNominalandOrdinal.xlsx.
• Stat > Quality Tools > Attribute Agreement
Analysis
• In Data are arranged as, select Attribute
column and select the column containing
Rating_ Type.
• In Samples, select the column containing the
Sample_1.
• In Appraisers, select the column containing the
Appraiser_1.
• In Known standard/attribute, select the
column containing the Standards Type.
• Click OK DO NOT check this box
Interpret Minitab Analysis:
Step 1: Evaluate the appraiser agreement visually

Duncan, Haynes, Holmes & Montgomery are consistent and having correct
ratings except for Simpson. Simpson is least consistent and has least correct
ratings.
Interpret Minitab Analysis
Step 2: Assess the consistency of responses for each appraiser

Kappa ratings for all appraisers are

excellent, except Simpson’s at 0.67228.
We need to investigate what issues are
affecting Simpson’s performance,
especially Sample at C, D & E ratings.
Interpret Minitab Analysis
Step 3: Assess the correctness of responses for each appraiser

All appraiser except Simpson achieved

100% matching scores between their
decisions with the Standard. Simpson
has 46.67% with 7 out of 15 matched
samples to the standard.
Interpret Minitab Analysis
Step 4: Assess the consistency of responses between appraisers

Overall an acceptable assessments between

appraisers. Kappa scores of 0.865.
But we will need determine what is the
problem that the appraisers faced when
Sample rating of D which scored a Kappa of
0.7718
Interpret Minitab Analysis
Step 5: Assess the correctness of responses for all appraisers

Overall Kappa assessment is excellent

with a score > 0.9.
Interpret Minitab Analysis
Kappa Statistics Summary

Correctness With
Consistency
Standard Reference
• Except Simpson at 77.5%, • Simpson matches the true
All others above ≥95% result 46.67% of the time and
(REPEATABILITY)
WITHIN

agreement and his his K= 0.65217

K=0.6722

• All Inspectors agreed • All Inspectors agreed with the

between each other true data 46.67% of the time
(REPRODUCIBILITY)

46.67% of the time and be focus

and their K=0.9304
BETWEEN

Conclusion: Improvement should on Simpson. He has both

consistencytheir K=0.8650 issues. Attention should be to understand
& correctness
and eliminate both issues.
 Attribute GRR
- Kendall's Coefficients
 Attribute Agreement Analysis (AAA)
- Analysis Focus

1. Standard Attribute GR&R: Assessment of Agreement

Expectation:
 Require a minimum score of 90%
 Obvious good and obvious bad
would be expected to be 100%.

2. Kappa Statistics: 3. Kendall's Coefficients :

  Ordinal (with natural
Binary : Pass / Fail, Good /Bad, Go/No
go. ordering) : Low/Med/Hi,
 Nominal (no natural ordering): Defect Cold/cool/
type A, B, C…H / Lifted, Broken, tepid/warm/hot/scalding,
Missing Satisfaction rate on a 1-5 scale
Correctness With The general Rule Of Thumb (ROT) for
Consistency
Standard Reference interpreting Kappa Statistics & Kendall's
WITHIN Within Appraiser: Each Appraiser vs. Coefficients :
(REPEATABILITY) Assess the consistency of Standard: Assess the
• < 0.75 Measurement System needs
responses for each correctness of responses for
attention
appraiser each appraiser
• 0.75 – 0.9 Generally acceptable,
BETWEEN Between Appraisers: All Appraisers vs. improvement may be needed
(REPRODUCIBILITY) Assess the consistency of Standard: Assess the depending on application and risk.
responses between correctness of responses for • > 0.9 Excellent Measurement System
appraisers all appraisers
Kendall Statistics

When using ordinal ratings, Kendall's coefficients, which account for ordering, are usually
more appropriate statistics to determine association than kappa alone. Minitab estimates
Kendall's coefficient by:

Notation
Term Description
N the number of subjects
Σ Ri2 the sum of the squared sums of ranks for each of the ranked N
subjects
K the number of appraisers
Tj Tj assigns the average of ratings to tied observation

ti the number of tied ranks in the ith grouping of ties

gj the number of groups of ties in the jth set of ranks

Reference
A. Agresti (1984). Analysis of Ordinal Categorical Data, John Wiley & Sons
54
AAA: Kendall's Coefficients
- Guidelines

• When your classifications are ordinal (ratings made on a scale), in addition to

kappa statistics, use Kendall's coefficients
• Kendall's coefficient of concordance
• Kendall's coefficient of concordance indicates the degree of association of ordinal
assessments made by multiple appraisers when assessing the same samples.
• Values range from 0 to 1 (higher value = stronger association)

• Kendall's correlation coefficient.

• When your classifications are ordinal, and have a known standard for each trial
• Determine whether an appraiser is consistent but inaccurate.
• Values range from -1 to 1 (positive value indicates positive association)
•Kendall's coefficients do not treat all misclassifications equally
The general rule of thumb for interpreting Kendall’s coefficients:
• < 0.75 Measurement System needs attention
• 0.75 – 0.9 Generally acceptable, improvement may be needed depending on application and
risk.
• > 0.9 Excellent Measurement System
Example – Ordinal (Scale from 1 - 5)

• An educational testing organization is

training five new appraisers for the
written portion of the twelfth-grade
standardized essay test:

• The appraisers‟ ability to rate

essays consistent with the
standards needs to be assessed
• Each appraiser rated fifteen essays
on a five-point scale (1,2,3,4,5)
• The organization also rated the
essays and supplied the “official
score”
• Each essay was rated twice

18.02.3AAAforNominalandOrdinal.xls
Example

1. Open file 18.02.3AAAforNominal

andOrdinal.xls
2. Import data to Minitab data
window.
3. Select ‘Rating_Type’ at Attribute
column
4. Select ‘Sample_1’ at Samples
5. Select ‘Appraiser_1’ at
Appraisers
6. Select ‘Standards_Type’ when you
have a known standard
7. If your data is Ordinal, you
must check this box
Interpret Minitab Analysis:
Step 1: Evaluate the appraiser agreement visually
 Interpret Minitab Analysis
Step 2: Assess the consistency of responses for each appraiser

Many of the kappa values are 1, Because the data are ordinal,
which indicates perfect Minitab provides the Kendall's
agreement within an appraiser coefficient of concordance values.
between trials. Some of These values are > 0.9, which
Simpson's kappa values ranges indicates a very strong association
between 0.314 to 0.9499. You within the appraiser ratings
might want to investigate why
Simpson's ratings of those
samples were inconsistent.
 Interpret Minitab Analysis
Step 3: Assess the correctness of responses for each appraiser

Most of the kappa values are

larger than 0.80, which
indicates good agreement
Because the data are ordinal, Minitab provides
between each appraiser and
the Kendall's correlation coefficients. These
the standard. A few of the
values range from 0.94977 and 1.0, which
kappa values are < 0.70,
indicate a strong association between the
which indicates that you may
ratings and the standard values.
need to investigate certain
samples or Simpson’s
performance further.
Interpret Minitab Analysis
Step 4: Assess the consistency of responses between appraisers

All the kappa values are 0.75< Kappa < 0.9,

which indicates minimally acceptable
agreement between appraisers. The
appraisers have the most agreement for
standards ranging 1 and 5, and the least
agreement for standard 4. Because the data
are ordinal, Minitab provides the Kendall's
coefficient of concordance (0.9895), which
indicates a very strong association between
the appraiser ratings.
Interpret Minitab Analysis
Step 5: Assess the correctness of responses for all appraisers

These results show that all the appraisers

correctly matched the standard ratings on
31 of the 40 samples. The overall kappa
value is 0.9676, which indicates strong
agreement with the standard values.
Because the data are ordinal, Minitab
provides the Kendall's coefficient of
concordance (0.9899), which indicates a
strong association between the ratings and
the standard values
Interpret Minitab Analysis
Kendall's Coefficients -Summary

Correctness With
Consistency
Standard Reference
• Except Simpson at • Simpson matches the true
77.5%, All others above result 77.5% of the time and
(REPEATABILITY)

≥95% agreement and his Kendall Coeff. = 0.94977

WITHIN

Simpson’s Kendall Coeff.

=0.9742

• All Inspectors agreed • All Inspectors agreed with

between each other the true data 77.5% of the
77.5% of the time and time and their Kendall Coeff.
REPRODUCIBILITY)

Conclusion: Kendall Statistics is ok across all analysis but did not meet
BETWEEN

their Kendall Coeff. =0.9899

the assessment agreement criteria of 90%. Attention should be on
Simpson to=0.9895
understand and eliminate Consistency and to Correctness
with Standard issues.
Attribute Agreement Analysis (AAA)
- Influencing Factors

INDIVIDUAL ABILITIES: ORGANIZATIONAL AND SOCIAL:

1. Visual acuity 1. Training
2. General intelligence and 2. Peer Standards
comprehension 3. Management Standards
3. Method of inspection 4. Knowledge of operator or group producing the
item
TASK: 5. Proximity of inspectors
4. Defect probability 6. Re-inspection versus immediate shipping
5. Fault Type procedures
6. Number of faults occurring
simultaneously
7. Time allowed for inspection
8. Frequency of rest periods
9. Illumination
10.Time of Day
11.Objectivity of conformance standards
12.Inspection station layout
Source: Baker, E.M. (1975). “Signal Detection Theory Analysis of Quality Control Inspector Performance.”Journal of Quality Technology, vol. 7, no. 2, 64
Attribute Agreement Analysis (AAA)
- The Vigilance Problem

Common in highly repetitive, subjective inspection activities

Example: Inspecting circuit boards (PCB)
The process of perceiving can become numbed or hypnotized by the sheer monotony of
repetition
 The “scanning” process and the “model” become dissociated
 The observer tends to see only what he / she expects to see
 This happens in spite of good visual acuity
 This becomes big problem when faults are obvious, infrequent, serious &/
unpredictable

Preventing Vigilance Problems:

 Break-up the numbing rhythm with pauses
 Introduce greater variety into the job
 Enhance faults to make them more noticeable
 Provide background and context cues
 Arrange for frequent job rotation 65
Attribute Agreement Analysis (AAA)
- Improving “Inadvertent” Errors

Sense Multipliers
• Optical magnifiers, sound amplifiers, and other devices to expand the ability of
the unaided human to sense the defects / categories
Masks
• Used to block-out the inspectors view of irrelevant characteristics so he / she can
focus on real responsibilities
Templates
• These are a combination of gage, magnifier, and mask. Example: cardboard
template placed over terminal boards
 Any extra or misplaced terminal will prevent the template from seating properly
Overlay
• These are visual aids in the form of transparent sheets on which guidelines or
tolerance lines are drawn
Reorganization of Work
• Suppose certain errors are due to fatigue, because of an inability to maintain
concentration for long periods of time
Source: Juran, J.M. (1988). Juran’sQuality Control Handbook, Fourth Edition. McGraw-Hill, New York, p18.83 66
Attribute Agreement Analysis (AAA)
- Summary

• Use attribute data only if the measurement can not be converted to

continuous data

• Operational definitions are extremely important

• Attribute measurement systems require a great deal of maintenance

• Should I use a kappa statistic or one of Kendall coefficients?

• When your classifications are nominal, use kappa.

• When your classifications are ordinal, in addition to kappa

statistics (-1 to 1), use Kendall's coefficient of concordance (0 to
1).

• When your classifications are ordinal and you have a known

standard for each trial, in addition to kappa statistics, use
Kendall's correlation coefficient (-1 to 1).

• Specific Jabil Attribute MSA example can be found in Appendix

67
Take Away

• Use attribute data only if the measurement can not be converted to

continuous data

• Operational definitions are extremely important

• Attribute measurement systems require a great deal of maintenance

• Should I use a kappa statistic or one of Kendall coefficients?

• When your classifications are nominal, use kappa.

• When your classifications are ordinal, in addition to kappa

statistics (-1 to 1), use Kendall's coefficient of concordance (0 to
1).

• When your classifications are ordinal and you have a known

standard for each trial, in addition to kappa statistics, use
Kendall's correlation coefficient (-1 to 1).

68
 THANK YOU

BUSINESS CONFIDENTIAL | ©Jabil Inc. 2019. All Rights Reserved.

Extra

Background Information of this analysis:

An automobile manufacturer wants to measure the bias and

repeatability of an automated attribute measurement
system. The system has a lower tolerance of -0.020 and an
upper tolerance of 0.020. The manufacturer ran 10 parts,
which had reference values at intervals of 0.005 from -0.050
to -0.005, through the gage 20 times.

18.02.4Autogage.mtw
Extra – Binary (Go / No GO)

Inspectors at a textile
manufacturing plant rate samples
of dyed fabric as go or no-go
(pass or fail) based on the
absence or presence of white
specks. To assess the consistency
and correctness of the inspectors'
ratings, a quality engineer asks
two inspectors to rate print
quality on 60 samples of fabric
twice. Fabric samples were
randomly presented.

18.02.5DyeCotton.mtw
Source: Minitab Support
Version Log
User Date Version # Changes

Joseph Ang 8th Aug 2020 1.0