Assessing the Accuracy of

Remotely Sensed Data

Principles and Practices, Third Edition
Assessing the Accuracy of
Remotely Sensed Data
Principles and Practices, Third Edition

by
Russell G. Congalton and Kass Green
CRC Press
Taylor & Francis Group
6000 Broken Sound Parkway NW, Suite 300
Boca Raton, FL 33487-2742

©  2019 by Taylor & Francis Group, LLC

CRC Press is an imprint of Taylor & Francis Group, an Informa business

No claim to original U.S. Government works

Printed on acid-free paper

International Standard Book Number-13: 978-1-4987-7666-0 (Hardback)

This book contains information obtained from authentic and highly regarded sources. Reasonable efforts have
been made to publish reliable data and information, but the author and publisher cannot assume responsibility
for the validity of all materials or the consequences of their use. The authors and publishers have attempted to
trace the copyright holders of all material reproduced in this publication and apologize to copyright holders if
permission to publish in this form has not been obtained. If any copyright material has not been acknowledged,
please write and let us know so we may rectify in any future reprint.

Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced, transmit-
ted, or utilized in any form by any electronic, mechanical, or other means, now known or hereafter invented,
including photocopying, microfilming, and recording, or in any information storage or retrieval system, with-
out written permission from the publishers.

For permission to photocopy or use material electronically from this work, please access www.copyright.com
(http://www.copyright.com/) or contact the Copyright Clearance Center, Inc. (CCC), 222 Rosewood Drive,
Danvers, MA 01923, 978-750-8400. CCC is a not-for-profit organization that provides licenses and registration
for a variety of users. For organizations that have been granted a photocopy license by the CCC, a separate
system of payment has been arranged.

Trademark Notice:  Product or corporate names may be trademarks or registered trademarks, and are used
only for identification and explanation without intent to infringe.

Visit the Taylor & Francis Web site at 

http://www.taylorandfrancis.com 

and the CRC Press Web site at 

http://www.crcpress.com 
 The third edition of this book is dedicated to our spouses,
Jeanie Congalton and Gene Forsburg, for their constant
support, encouragement, and love. It is also dedicated to a
very special friendship between the authors that has stood
the test of time for over 32  years and is still going strong.
Contents
Preface.................................................................................................................... xiii
Acknowledgments..................................................................................................... xv
Authors....................................................................................................................xvii

Chapter 1 Introduction...........................................................................................1
Why Map?.............................................................................................1
Why Assess the Accuracy of a Map?....................................................1
Types of Map Accuracy Assessment.....................................................2
Critical Steps in Accuracy Assessment................................................. 3
Organization of the Book...................................................................... 4

Chapter 2 The History of Map Accuracy Assessment........................................... 7

How Maps Are Made............................................................................7
History of Accuracy Assessment........................................................ 13
Positional Accuracy Assessment......................................................... 13
Thematic Accuracy Assessment.......................................................... 17

Chapter 3 Planning for Conducting an Accuracy Assessment............................ 21

What Type of Assessment?................................................................. 21
Qualitative Accuracy Assessment.................................................. 22
Between Qualitative and Quantitative Assessment........................ 22
Similarity Analysis.................................................................... 23
Error Budget..............................................................................24
Quantitative Accuracy Assessment................................................26
Positional Accuracy Assessment...............................................26
Thematic Accuracy Assessment................................................28
Why Do the Assessment?.................................................................... 29
How Will the Assessment Be Used?................................................... 30
What Special Considerations Exist?................................................... 30
Conclusions......................................................................................... 30

Chapter 4 Positional Accuracy............................................................................. 33

Introduction......................................................................................... 33
What Is Positional Accuracy?............................................................. 33
What are the Common Standards for Positional Accuracy?............... 35
How Should Positional Accuracy Assessment Be Designed and
the Samples Be Selected?.................................................................... 39
How Is Positional Accuracy Analyzed?.............................................. 41

vii
viii Contents

Review of Basic Statistics............................................................... 42

Parameters and Statistics........................................................... 43
Estimating the Dispersal of Variables............................................44
Estimating a Confidence Interval around the Estimate of the
Mean............................................................................................... 48
Statistics in Positional Accuracy Assessment................................ 49
If the Errors Are Assumed to Be Normally Distributed........... 49
If the Errors Are Assumed Not to Be Normally Distributed.......61
How Is Positional Accuracy Reported?............................................... 61
Summary............................................................................................. 67

Chapter 5 Thematic Map Accuracy Basics.......................................................... 69

Non-site-specific Assessments............................................................ 69
Site-specific Assessments.................................................................... 70
The Error Matrix............................................................................ 71
Overall, Producer’ s, and User’ s Accuracies.............................. 73
Mathematical Representation of the Error Matrix.................... 74

Chapter 6 Thematic Map Accuracy Assessment Considerations........................ 77

What Are the Thematic Map Classes to Be Assessed?....................... 78
The Classification Scheme............................................................. 79
Other Data Considerations.............................................................84
Continuous versus Non-Continuous Data.................................84
Spatial Autocorrelation.............................................................. 85
What Is the Appropriate Sample Unit?............................................... 87
Single Pixel..................................................................................... 87
Cluster of Pixels as a Single Sample Unit...................................... 88
Polygon as Single Sample Unit.......................................................90
Clusters of Sample Units................................................................92
How Many Samples Should Be Taken?.............................................. 93
Binomial Distribution.....................................................................94
Multinomial Distribution................................................................ 95
How Should the Samples Be Chosen?................................................. 98
Sampling Schemes.......................................................................... 98
Sampling Scheme Considerations..................................................99
Equal versus Minimum versus Proportional
Distribution.............................................................................. 102
Conclusions....................................................................................... 104

Chapter 7 Reference Data Collection................................................................. 107

What Should Be the Source of the Reference Data?......................... 108
Using Existing versus Newly Collected Data............................... 109
Remotely Sensed Data versus Field Visits................................... 110
Contents ix

How Should the Reference Data Be Collected?................................ 111

When Should the Reference Data Be Collected?.............................. 115
Ensuring Objectivity and Consistency.............................................. 120
Data Independence....................................................................... 120
Data Collection Consistency........................................................ 121
Quality Control............................................................................. 123

Chapter 8 Basic Analysis Techniques................................................................ 127

Kappa................................................................................................ 127
Margfit............................................................................................... 133
Conditional Kappa............................................................................. 136
Weighted Kappa................................................................................ 136
Compensation for Chance Agreement.............................................. 138
Confidence Limits............................................................................. 138
Area Estimation/Correction.............................................................. 142

Chapter 9 Analysis of Differences in the Error Matrix..................................... 145

Errors in the Reference Data............................................................. 145
Sensitivity of the Classification Scheme to Observer Variability..........147
Inappropriateness of the Remote Sensing Data Employed to
Make the Map................................................................................... 149
Mapping Error................................................................................... 150
Summary........................................................................................... 151
Appendix 9.1: Forest and Woodland Map Class Key of
Haleakalā National Park................................................................... 151

Chapter 10 Fuzzy Accuracy Assessment............................................................. 157

Expanding the Major Diagonal of the Error Matrix......................... 158
Measuring Map Class Variability..................................................... 159
The Fuzzy Error Matrix Approach................................................... 160
The Fuzzy Error Matrix............................................................... 161
Implementation of the Fuzzy Error Matrix.................................. 164
Another Fuzzy Error Matrix Example......................................... 166
Summary........................................................................................... 168

Chapter 11 Object-Based or Polygon Accuracy Assessment............................... 169

What Sample Unit Should Be Used?................................................. 169
Traditional Tally-based versus Area-based Error Matrix................. 170
Comparison of the Two Error Matrix Approaches...................... 172
Reference Data Collection/Labeling................................................. 172
Computing the Accuracy................................................................... 177
Conclusions....................................................................................... 180
x Contents

Chapter 12 An Object-based Accuracy Assessment Case Study: The Grand

Canyon National Park/Grand Canyon-Parashant National
Monument Vegetation Classification and Mapping Project.............. 181
Introduction....................................................................................... 182
Overview of the Case Study.............................................................. 182
Accuracy Assessment........................................................................ 186
What Are the Thematic Classes to Be Assessed?........................ 186
What Is the Appropriate Sampling Unit?..................................... 187
How Many Samples Should Be Taken?........................................ 187
How Should the Samples Be Chosen?.......................................... 187
What Should Be the Source of the Reference Data?.................... 191
How Should the Reference Data Be Collected?........................... 191
When Should the Reference Data Be Collected?......................... 192
How Do I Ensure Consistency and Objectivity in My Data
Collection?.................................................................................... 192
Analysis............................................................................................. 193
What Is an Error Matrix and How Should It Be Used?............... 193
What Are the Statistical Properties Associated with the Error
Matrix and What Analysis Techniques Are Applicable?...............196
What Is Fuzzy Accuracy and How Can You Conduct a
Fuzzy Accuracy Assessment?...................................................... 196
Results............................................................................................... 197
Lessons Learned............................................................................... 199

Chapter 13 The California Hardwood Rangeland Project................................... 201

Introduction....................................................................................... 201
Background....................................................................................... 201
Sample Design...................................................................................202
How Is the Map Information Distributed?...................................203
What Is the Appropriate Sample Unit?........................................204
How Many Samples Should Be Taken?........................................206
How Should the Samples Be Chosen and Distributed across
the Landscape?.............................................................................206
Samples Chosen from the 1981 Map.......................................207
Samples Chosen from the 1990 Coverage...............................208
Reference Data Collection.................................................................208
What Should Be the Source Data for the Reference Samples?...... 209
What Type of Information Should Be Collected?........................209
When Should the Reference Data Be Collected?......................... 212
Quality Control........................................................................ 213
Analysis............................................................................................. 213
Development of the Error Matrices.............................................. 213
Statistical Analysis....................................................................... 214
Analysis of Off-diagonal Samples................................................ 218
Contents xi

Crown Closure Analysis............................................................... 218

Crown Closure Map Results......................................................... 220
Cover Type Analysis.....................................................................224
Cover Type Map Results............................................................... 225
Extent............................................................................................ 228
Discussion......................................................................................... 229
Conclusions....................................................................................... 230

Chapter 14 Advanced Topics............................................................................... 233

Change Detection.............................................................................. 233
Reference Data............................................................................. 234
Sampling....................................................................................... 235
Change Detection Error Matrix................................................... 236
Two-Step Approach to Change Detection Accuracy
Assessment................................................................................... 239
Case Study....................................................................................240
Step 1: Accuracy of the Change Areas....................................240
Step 2: Change/No Change Assessment.................................. 243
Multi-layer Assessments....................................................................244
Appendix 14.1: Class Descriptions of the 2005 NLCD
Land Cover........................................................................................ 245

Chapter 15 Summary and Conclusions................................................................ 247

Appendix 1: ASPRS Positional Accuracy Standards for Digital

Geospatial Data................................................................................. 251

Foreword................................................................................................................ 254
ASPRS Positional Accuracy Standards for Digital Geospatial Data............... 255
Annex A Background and Justifications (Informative).................................... 271

Annex B Data Accuracy and Quality Examples (Normative).......................... 277

Annex C Accuracy Testing and Reporting Guidelines (Normative)................ 293

Annex D Accuracy Statistics and Example (Normative)................................. 303

Bibliography.......................................................................................................... 311
Index....................................................................................................................... 319
Preface
Our ability to map and monitor our earth using remotely sensed data has improved
rapidly over the last decade since the second edition of this book was published.
These advancements include amazing leaps forward in microelectronics, digital sen-
sor technology, object-based analysis, sensor platforms, machine learning, and cloud
computing. Therefore, the principles and especially the practices for assessing the
accuracy of these maps created from remotely sensed data have also continued to
develop and mature. The first edition of this book, published in 1999, had only eight
chapters. The second edition, published about 10  years later, expanded to 11 chap-
ters, including the very significant addition of a new chapter on positional accuracy.
This third edition contains 15 chapters, including a new chapter on better planning
for an assessment, a thoroughly revised positional accuracy chapter, a new chapter on
object-based accuracy assessment, two case studies (a polygon-based assessment of
an object-based classification map and a more classic example), and a new summary
chapter with helpful bullets emphasizing key issues. While it presents the salient
principles needed to conduct a valid accuracy assessment, the strength of this book
is the thorough presentation and discussion of the practical considerations that must
be understood for a geospatial analyst to conduct a successful assessment. These
concepts are also important for the map user to understand so as to make effective
use of any map. Not every method or approach or descriptive measure suggested
in the literature is covered here. Rather, this book is written for those who wish to
conduct a valid and successful assessment of a map to better understand the errors,
limitations, and usefulness of that map and for those who wish to be effective users
of the map. As such, this book emphasizes the practical considerations, tradeoffs,
and decisions that must be deliberated at the beginning of a mapping project and the
awareness that must be maintained throughout the project and during the accuracy
assessment. The authors have conducted many map accuracy assessments over the
last 35  years and have faced every possible situation, limitation, and difficulty imag-
inable (and some that were unimaginable). It is our hope that this book provides you
with the guidance and encouragement necessary to be successful, whether you are
assessing the accuracy of a map or using the map for a specific objective and needing
to understand its accuracy.

xiii
Acknowledgments
Every book is the responsibility of the authors but comes from a collection of ideas,
encouragement, and suggestions of countless others. By the time a third edition of a
book is published, there are many people to thank and acknowledge. The first edi-
tion of this book was inspired by Dr. John Lyon and dedicated to Dr. Roy Mead. In
addition, we are especially thankful to Dr. John Jensen, Dr. Greg Biging, Dr. Tom
Lillesand, Dr. Jim Smith, Mr. Ross Lunetta, Mr. Mike Renslow, Dr. George Lee, and
Dr. Jim Campbell for their positive feedback and support. The second edition added
a very significant chapter on positional accuracy with the help of Drs. George Lee,
Greg Biging, and Dave Maune.
This chapter has been significantly revised in the third edition with signifi-
cant input from Dr. Qassim Abdullah. Thanks also to the American Society for
Photogrammetry and Remote Sensing (ASPRS) for allowing the publication of the
new positional accuracy standards as an appendix in this edition. We would also
like to thank Dr. Michael Kearsley of the National Park Service for allowing us to
include the Grand Canyon National Park vegetation map accuracy assessment as a
case study in this edition.
Along this 30-year adventure, many colleagues at Pacific Meridian Resources,
Space Imaging Solutions, Fugro-Earthdata, Dewberry, Tukman Geospatial, and
Sanborn Solutions, as well as numerous graduate students in the Department of
Natural Resources and the Environment at the University of New Hampshire, helped
make all the editions of this book better.
Significant contributions to this third edition were made by Mark Tukman of
Tukman Geospatial, Dr. J.B. Sharma from the University of North Georgia, and
Chad Lopez of Dewberry. Kamini Yadav, Heather Grybas, and Ben Fraser, gradu-
ate students at the University of New Hampshire, also contributed extensively to
this edition. We also gratefully acknowledge all of our friends and colleagues in the
geospatial community who inspired and encouraged us on many occasions. Finally,
we would like to thank our families for the time they managed without us while we
worked on this book.

xv
Authors
Russell G. Congalton is Professor of Remote Sensing and GIS in the Department
of Natural Resources and the Environment at the University of New Hampshire. He
is responsible for teaching courses in geospatial analysis, including The Science of
Where, Remote Sensing, Photogrammetry and Image Interpretation, Digital Image
Processing, and Geographic Information Systems. Russ has authored or coauthored
more than 200 papers and conference proceedings. He is the author of 12 book chap-
ters, co-editor of a book on spatial uncertainty in natural resource databases titled
Quantifying Spatial Uncertainty in Natural Resources: Theory and Applications for
GIS and Remote Sensing, and co-author of five books, including Imagery and GIS:
Best Practices for Extracting Information from Imagery, published by Esri in 2017.
Russ served as president of the ASPRS in 2004–2005, as the National Workshop
Director for ASPRS from 1997 to 2008, and as editor-in-chief of Photogrammetric
Engineering and Remote Sensing from 2008 to 2016. He was elected a Fellow in
2007 and an Honorary Member in 2016.
Dr. Congalton received a BS (Natural Resource Management) from Rutgers
University in 1979. He earned an MS (1981) and a PhD (1984) in remote sensing
and forest biometrics from Virginia Tech. In addition to his academic position, Russ
served as chief scientist of Pacific Meridian Resources from its founding in 1988
until 2000 and then as chief scientist of Space Imaging Solutions from 2000 to
2004. From 2004 until 2015, Russ held the position of senior technical advisor with
the Solutions Group of Sanborn, the oldest mapping company in the United States.
Finally, he has been the New Hampshire View Director, part of the AmericaView
Consortium, since 2007 and has served as member, secretary, vice-chair, and chair
of the AmericaView Board of Directors.
Kass Green’s experience spans 30 years of managing and supervising GIS and
remote sensing professionals for vegetation mapping, as well as leadership in GIS
and remote sensing research and policy. In 1988, Ms. Green co-founded Pacific
Meridian Resources, a GIS/remote sensing firm, which she grew to 75 employees
in seven offices nationwide and sold to Space Imaging (now Digital Globe) in 2000.
After running half of Space Imaging for 3 years, Ms. Green decided to focus her
career on challenging remote sensing mapping and policy projects for public agen-
cies, development organizations, and nongovernmental organizations. Over the last
10 years, Ms. Green has had the pleasure of using object-oriented techniques to cre-
ate detailed vegetation maps of Grand Canyon National Park, the national parks of
Hawaii, and Sonoma County, California from high-resolution optical imagery, lidar
data, Landsat imagery, and multiple other data sets. Her work also includes inter-
national market studies and strategy papers for organizations such as the Omidyar
Network, The Bill and Melinda Gates Foundation, and the UK Department of
International Development (DFID).

xvii
xviii Authors

Ms. Green is a fellow and a lifetime honorary member of ASPRS and a past
president of both Management Association for Private Photogrammetric Surveyors
(MAPPS) and ASPRS. She is the principal author of a recently published text for
Esri Press titled Imagery and GIS: Best Practices for Extracting Information from
Imagery. Ms. Green chairs NASA’s Earth Science Applications Committee, co-
founded and chaired the Department of the Interior’s Landsat Advisory Group, and
participated in the Landsat Science Team from 2013 to 2017 as an ex officio member.
 1 Introduction

WHY MAP?
The earth’s resources are scarce. As we continue to add more people to the earth,
the scarcity of resources increases, as does their value. From land use conversion
throughout the world, to fragmentation of tropical bird habitat, to polar bear habitat
loss in the Arctic, to the droughts in Africa, and wars worldwide, people have signifi-
cantly affected the resources and ecosystems of the world. The ever-increasing world
population and need for all types of resources continue to cause the price of these
resources to increase and to intensify conflicts over resource allocation.
As resources become more valuable, the need for timely and accurate information
about the type, quantity, and extent of resources multiplies. Allocating and managing
the earth’s resources requires accurate knowledge about the distribution of resources
across space and time. To efficiently plan emergency response, we need to know
the location of roads relative to fire and police stations, hospitals, and emergency
shelters. To improve the habitat of endangered species, we need to know what the
species habitat requirements are, where that habitat exists, where the animals exist,
and how changes to the habitat and surrounding environments will affect species
distribution, population, and viability. To plan for future developments, we need to
know where people will work, live, shop, and go to school. To grow enough food for
an ever-increasing population, we need information about the spatial distribution
and yields of our agricultural regions. Because each decision (including the decision
to do nothing) impacts (1) the status and location of resources and (2) the relative
wealth of individuals and organizations who derive value from the resources, know-
ing the location of resources and how they interact spatially is critical to effectively
managing those resources and ourselves over time.

WHY ASSESS THE ACCURACY OF A MAP?

Decisions about resources require maps; and effective decisions require accurate
maps or, at least, maps of known accuracy. For centuries, maps have provided impor-
tant information concerning the distribution of resources across the earth. Maps help
us to measure the extent and distribution of resources, analyze resource interactions,
and identify suitable locations for specific actions (e.g., development or preservation),
plan future events, and monitor change. If our decisions based on map informa-
tion are to have the expected results, the accuracy of these maps must be known.
Otherwise, implementing any decisions based on these maps will result in surprises,
and often these surprises may be unacceptable.
For example, suppose that you wish to have a picnic in a forest on the edge of
a lake. If you have a map that displays forest, crops, urban, water, and barren land
cover types, you can plan the location of your picnic. If you don’t know the accuracy

1
2 A ssessing the Accuracy of Remotely Sensed Data

of the map, but the map is 100% accurate, you will be able to travel to your forest
lakeside location and, in fact, find yourself in a nice picnic spot. However, if the map
is not spatially accurate, you may find that your picnic location falls in the middle of
the lake rather than on the shore; and if the map is not labeled correctly (i.e., themati-
cally accurate), you may find yourself in a city next to a fountain or in an agricultural
field next to an irrigation ditch. Conversely, if you know the accuracy of the map,
you can incorporate the known expectations of accuracy into your planning and
create contingency plans in situations when the accuracy is low. This type of knowl-
edge is critical when we move from our lighthearted picnic example to more critical
decisions such as endangered species preservation, resource allocation, feeding our
growing populations, peace-keeping actions, and emergency response.
There are many reasons for performing an accuracy assessment. Perhaps the sim-
plest reason is curiosity—the desire to know how good a map you have made. In
addition to the satisfaction gained from this knowledge, we also need or want to
increase the quality of the map information by identifying and correcting the sources
of errors. Third, analysts often need to compare various techniques, algorithms, ana-
lysts, or interpreters to test which is best. Also, if the information derived from the
remotely sensed data is to be used in some decision-making process (i.e., geographic
information system [GIS] analysis), then it is critical that some measure of its qual-
ity be known. Finally, it is more and more common that some measure of accuracy
is included in the contract requirements of many mapping projects. Therefore, valid
accuracy is not only useful but may be required.
Accuracy assessment determines the quality of a map created from remotely
sensed data. Accuracy assessment can be qualitative or quantitative, expensive or
inexpensive, quick or time-consuming, well designed and efficient, or haphazard.
The goal of quantitative accuracy assessment is the identification and measurement
of map errors so that the map can be as useful as possible to the persons using it to
make decisions.
The central purpose of this book is to present the necessary theory and principles
for conducting a quantitative accuracy assessment along with the practical consider-
ations of how to effectively and efficiently design and implement such an assessment.
Throughout the book, we emphasize that no one single recipe exists for conducting
an accuracy assessment. Just as there is no one way to produce a map; there is no one
way to assess the accuracy of a map. Instead, this book will teach you to consider
every aspect of a mapping project and to design and implement the best possible
assessment given the strengths and limitations of each mapping project you conduct,
fund, or rely on. This book is not written to be an academic review of every possible
idea or method ever published on map accuracy assessment. Instead, it is written for
the geospatial analyst who wishes to best conduct a valid and effective assessment
of their particular mapping project. As such, the considerations and limitations of
such an assessment are emphasized here to best lead the analyst through the process.

TYPES OF MAP ACCURACY ASSESSMENT

There are two types of map accuracy assessment: positional and thematic.
Positional accuracy deals with the accuracy of the location of map features and
Introduction 3

measures how far a spatial feature on a map is from its true or reference location on
the ground (Bolstad, 2005). Thematic accuracy deals with the labels or attributes
of the features of a map and measures whether the mapped feature labels are dif-
ferent from the true or reference feature label. For example, in the picnic example,
the earth’s surface was classified as forest, water, crops, urban, or barren. We are
interested in both the accuracy of the location of the features, so that we can locate
our picnic spot in a forest on the shore of a lake, and in the thematic accuracy, so
that we truly end up in a forest and not in a city, desert, or agricultural field that
was erroneously mapped as forest.
The accuracy of any map or spatial data set is a function of both positional accu-
racy and thematic accuracy, and this book considers both. However, because the-
matic accuracy is much more complex than positional accuracy, the book devotes
considerably more attention to thematic accuracy assessment.

CRITICAL STEPS IN ACCURACY ASSESSMENT

As previously stated, there is no one single procedure for conducting either a posi-
tional or a thematic accuracy assessment. However, all accuracy assessments include
these fundamental steps:

1. Consider the factors involved in the assessment.

2. Design the appropriate sampling approach to collect the reference data.
3. Conduct the sampling.
4. Analyze the data.
5. Report the statistics/results.

Each step must be rigorously planned and implemented. First, the accuracy
assessment sampling procedures are designed, and the sample areas on the map are
selected. We use sampling because time and funding limitations preclude the assess-
ment of every spatial unit on the map. Next, information is collected from both the
map and the reference data for each sample site. Thus, two types of information are
collected from each sample:

• Reference accuracy assessment sample data: the position or map class label
of the accuracy assessment site, which is derived from data collected that
are assumed to be correct
• Map accuracy assessment sample data: the position or map class label of
the accuracy assessment site derived from the map or image being assessed

Third, the map and reference information are compared with one another, the
results of the comparison are analyzed for statistical significance and for reason-
ableness, and a report is prepared, which presents the methods and results of the
assessment. In summary, effective accuracy assessment requires (1) design and
implementation of unbiased sampling procedures, (2) consistent and accurate col-
lection of sample data, and (3) rigorous comparative analysis of the sample map and
reference data and reporting of the results.
4 A ssessing the Accuracy of Remotely Sensed Data

Because there is no one procedure for designing and implementing an accuracy

assessment, there are quite a number of important questions to ask and consider-
ations to think about when conducting a valid assessment. This book addresses the
most important ones, including:

1. Questions concerning the design of an accuracy assessment sample

approach:
• What are the map classes to be assessed, and how are they distributed
across the landscape?
• What is the appropriate sampling unit?
• How many samples should be taken?
• How should the samples be chosen?
2. Questions concerning how the reference data should be collected:
• What should be the source of the reference data?
• How should the reference data be collected?
• When should the reference data be collected?
• How do I ensure consistency and objectivity in my data collection?
3. Questions concerning how the analysis should be conducted:
• What are the different analysis techniques for continuous versus dis-
continuous map data?
• What is an error matrix, and how should it be used?
• What are the statistical properties associated with the error matrix, and
what analysis techniques are applicable?
• What is fuzzy accuracy, and how can you conduct a fuzzy accuracy
assessment?
• What is object-based accuracy, and how can you conduct an object-
based accuracy assessment?
• How is an accuracy assessment conducted on change detection maps?
• How is an accuracy assessment conducted on maps created from mul-
tiple layers of data?

ORGANIZATION OF THE BOOK

The organization of this book takes you through each of these fundamental accuracy
assessment steps as follows:

• Chapter 2 begins with a review of the history and basic assumptions of map
making and accuracy assessment.
• Chapter 3 is a new chapter in this edition and provides the reader with
frameworks for planning out accuracy assessments. An important compo-
nent of any good assessment is proper planning from the beginning of the
mapping project. This chapter aids the reader in getting started on making
an effective plan by identifying the components of the assessment neces-
sary for their project. It is expected that this chapter will be reviewed by the
analyst prior to beginning each and every accuracy assessment.
Introduction 5

• Chapter 4 is a thoroughly revised presentation on positional accuracy

assessment including a review of past standards and concluding with the
latest and most effective ASPRS Positional Accuracy Standards for Digital
Geospatial Data (ASPRS, 2014). This complete document is included as
an appendix.
• Chapters 5 and 6 provide some of the basic methods and considerations for
thematic map accuracy assessment, including the introduction of the error
matrix and a thorough review of sample design considerations.
• Chapter 7 is devoted to factors that must be taken into account during the
collection of reference data. The collection of sufficient and valid reference
data is key to the success of any thematic accuracy assessment, and this
chapter describes the many considerations, decisions, and practical details
that are part of this process.
• Chapters 8 through 11 detail thematic accuracy assessment analysis, which
is much more complex than positional accuracy assessment analysis and
hence requires more chapters. The basic analysis techniques that can be
applied to an error matrix are discussed in Chapter 8. Chapter 9 discusses
the causes of differences in the error matrix, whether from map errors
or from other non-error sources. Chapter 10 presents a solution to some
of the non-error differences in the error matrix by suggesting the use of
fuzzy accuracy assessment. Finally, Chapter 11 is a new chapter that pres-
ents the idea of an object-based accuracy assessment. Given the shift from
pixel-based to object-based classification approaches, especially for high–
spatial resolution imagery, it is necessary to explore accuracy assessment
approaches that incorporate objects and not simply pixels in the assessment.
• Chapters 12 and 13 provide two very different case studies that give the
reader some real-life examples of the complexities involved in conducting
a map accuracy assessment. Neither case study is perfect, and both have
flaws, considerations, and limitations that must be addressed. The goal
is not to have the reader follow either case study for their assessment but
rather, to use the thought process and detailed explanations provided to bet-
ter understand the entire process. Chapter 12 presents a case study using a
polygon-based accuracy assessment of an object-based vegetation map cre-
ated for Grand Canyon National Park in 2014. It reviews all the design, data
collection, and analysis methods and considerations presented in Chapters
5 through 11. Chapter 13 is a case study from the 1990s, and while dated,
it presents a classic accuracy assessment that evaluates two maps—the first
a polygon map created from manual photo interpretation, and the second
a pixel-based map created using semi-automated image analysis. This case
study contains extensive detail and explanation, beyond what would ever
be published in a peer-reviewed paper. Reading through the thought pro-
cess in this case study will help the reader to better understand the entire
assessment.
• Chapter 14 delves into more advanced topics in accuracy assessment,
including change detection accuracy assessment and multi-layer accuracy
assessment.
6 A ssessing the Accuracy of Remotely Sensed Data

• Chapter 15 summarizes the book by emphasizing lessons learned from the

authors’ implementation of dozens of accuracy assessments over the last
three decades. While mapping technologies have changed significantly over
the last 30 years and will continue to evolve rapidly, the basic principles of
mapping and accuracy assessment endure with little change. This chapter’s
action statements remind and encourage the reader regarding the key com-
ponents of any valid and effective accuracy assessment.
References
Ager, Thomas. 2004. An Analysis of Metric Accuracy Definitions and Methods of Computation.
Unpublished memo prepared for the National Geospatial-Intelligence Agency. InnoVision.
Aickin, M. 1990. Maximum likelihood estimation of agreement in the constant predictive
probability model, and its relation to Cohen’s kappa. Biometrics. Vol. 46, pp. 293–302.
American Society of Photogrammetry . 1960. Manual of Photographic Interpretation. ASP,
Washington, DC.
American Society for Photogrammetry and Remote Sensing (ASPRS) Specifications and
Standards Committee. 1990. ASPRS accuracy standards for large-scale maps.
Photogrammetric Engineering and Remote Sensing. Vol. 56, No. 7, pp. 1068–1070.
Anderson, J. R. , E. E. Hardy , J. T. Roach , and R. E. Witner . 1976. A land use and land cover
classification system for use with remote sensor data. USGS Professional Paper. Vol. 964, 28
pp.
Aronoff, Stan. 1982. Classification accuracy: A user approach. Photogrammetric Engineering
and Remote Sensing. Vol. 48, No. 8, pp. 1299–1307.
Aronoff, Stan. 1985. The minimum accuracy value as an index of classification accuracy.
Photogrammetric Engineering and Remote Sensing. Vol. 51, No. 1, pp. 99–111.
ASPRS . 1989. ASPRS interim accuracy standards for large-scale maps. Photogrammetric
Engineering & Remote Sensing. Vol. 54, No. 7, pp. 1038–1041.
ASPRS . 2004. ASPRS Guidelines, Vertical Accuracy Reporting for Lidar Data. American
Society for Photogrammetry and Remote Sensing.
ASPRS . 2014. ASPRS positional accuracy standards for digital geospatial data.
Photogrammetric Engineering and Remote Sensing. Vol. 81, No. 3, pp. A1–A26.
ASPRS and ASCE . 1994. Glossary of the Mapping Sciences. ASPRS, Bethesda, MD; ASCE,
New York.
Biging, G. and R. Congalton . 1989. Advances in forest inventory using advanced digital
imagery. Proceedings of Global Natural Resource Monitoring and Assessments: Preparing for
the 21st Century. Venice, Italy. September, 1989. Vol. 3, pp. 1241–1249.
Biging, G. , R. Congalton , and E. Murphy . 1991. A comparison of photointerpretation and
ground measurements of forest structure. Proceedings of the Fifty-Sixth Annual Meeting of the
American Society of Photogrammetry and Remote Sensing. Baltimore, MD. Vol. 3, pp. 6–15.
Bishop, Y. , S. Fienberg , and P. Holland . 1975. Discrete Multivariate Analysis: Theory and
Practice. MIT Press, Cambridge, MA. 575 pp.
Bitterlich, W. 1947. Die Winkelzahlmessung (Measurement of basal area per hectare by means
of angle measurement). Allg. Forst. Holzwirtsch. Ztg. Vol. 58, pp. 94–96.
Blaschke, T. 2010. Object based image analysis for remote sensing. ISPRS Journal of
Photogrammetry and Remote Sensing. Vol. 65, pp. 2–16.
Bolstad, Paul. 2005. GIS Fundamentals. 2nd edition. Eider Press, White Bear Lake, MN. 543
pp.
Bolstad, Paul. 2016. GIS Fundamentals: A First Text on Geographic Information Systems. 5th
Edition. XanEdu Publishers, Ann Arbor, MI. 770 pp.
Brennan, R. and D. Prediger . 1981. Coefficient kappa: Some uses, misuses, and alternatives.
Educational and Psychological Measurement. Vol. 41, pp. 687–699.
Brothers, G. L. and E. B. Fish . 1978. Image enhancement for vegetation pattern change
analysis. Photogrammetric Engineering and Remote Sensing. Vol. 44, No. 5, pp. 607–616.
Campbell, James B. 1981. Spatial autocorrelation effects upon the accuracy of supervised
classification of land cover. Photogrammetric Engineering and Remote Sensing. Vol. 47, No. 3,
pp. 355–363.
Card, D. H. 1982. Using known map categorical marginal frequencies to improve estimates of
thematic map accuracy. Photogrammetric Engineering and Remote Sensing. Vol. 48, No. 3, pp.
431–439.
Chrisman, N. 1982. Beyond accuracy assessment: Correction of misclassification. Proceedings
of the 5th International Symposium on Computer-Assisted Cartography. Crystal City, VA. pp.
123–132.
Cliff, A. D. and J. K. Ord . 1973. Spatial Autocorrelation. Pion Limited, London, England. 178
pp.
Cochran, William G. 1977. Sampling Techniques. John Wiley & Sons, New York. 428 pp.
Cohen, Jacob. 1960. A coefficient of agreement for nominal scales. Educational and
Psychological Measurement. Vol. 20, No. 1, pp. 37–40.
Cohen, Jacob. 1968. Weighted kappa: Nominal scale agreement with provision for scaled
disagreement or partial credit. Psychological Bulletin. Vol. 70, No. 4, pp. 213–220.
Colwell, R. N. 1955. The PI picture in 1955. Photogrammetric Engineering. Vol. 21, No. 5, pp.
720–724.
Congalton, Russell G. 1981. The use of discrete multivariate analysis for the assessment of
Landsat classification accuracy. MS Thesis, Virginia Polytechnic Institute and State University,
Blacksburg, VA. 111 pp.
Congalton, R. G. 1984. A comparison of five sampling schemes used in assessing the accuracy
of land cover/land use maps derived from remotely sensed data. PhD Dissertation, Virginia
Polytechnic Institute and State University, Blacksburg, VA. 147 pp.
Congalton, R. G. 1988a. Using spatial autocorrelation analysis to explore errors in maps
generated from remotely sensed data. Photogrammetric Engineering and Remote Sensing. Vol.
54, No. 5, pp. 587–592.
Congalton, R. G. 1988b. A comparison of sampling schemes used in generating error matrices
for assessing the accuracy of maps generated from remotely sensed data. Photogrammetric
Engineering and Remote Sensing. Vol. 54, No. 5, pp. 593–600.
Congalton, R. 1991. A review of assessing the accuracy of classifications of remotely sensed
data. Remote Sensing of Environment. Vol. 37, pp. 35–46.
Congalton, R. 2009. Accuracy and error analysis of global and local maps: Lessons learned
and future considerations. In: Remote Sensing of Global Croplands for Food Security. P.
Thenkabail, J. Lyon, H. Turral, and C. Biradar (Editors). CRC/Taylor & Francis, Boca Raton, FL.
pp. 441–458.
Congalton, R. 2015. Assessing positional and thematic accuracies of maps generated from
remotely sensed data. In: Remote Sensing Handbook; Vol. I: Data Characterization,
Classification, and Accuracies. P. Thenkabail (Editor). CRC/Taylor & Francis, Boca Raton, FL.
pp. 583–601.
Congalton, R. and G. Biging . 1992. A pilot study evaluating ground reference data collection
efforts for use in forest inventory. Photogrammetric Engineering and Remote Sensing. Vol. 58,
No. 12, pp. 1669–1671.
Congalton, R. and M. Brennan . 1998. Change detection accuracy assessment: Pitfalls and
considerations. Proceedings of the Sixty Fourth Annual Meeting of the American Society of
Photogrammetry and Remote Sensing. Tampa, Florida. pp. 919–932 (CD-ROM).
Congalton, R. and M. Brennan . 1999. Error in remotely sensed data analysis: Evaluation and
reduction. Proceedings of the Sixty Fifth Annual Meeting of the American Society of
Photogrammetry and Remote Sensing. Portland, OR. pp. 729–732 (CD-ROM).
Congalton, R. and K. Green . 1993. A practical look at the sources of confusion in error matrix
generation. Photogrammetric Engineering and Remote Sensing. Vol. 59, No. 5, pp. 641–644.
Congalton, R. and K. Green . 2009. Assessing the Accuracy of Remotely Sensed Data:
Principles and Practices. 2nd edition. CRC/Taylor & Francis, Boca Raton, FL. 183 pp.
Congalton, R. G. and R. D. Macleod . 1994. Change detection accuracy assessment on the
NOAA Chesapeake Bay pilot study. Proceedings of the International Symposium of Spatial
Accuracy of Natural Resource Data Bases. Williamsburg, VA. pp. 78–87.
Congalton, R. G. and R. A. Mead . 1983. A quantitative method to test for consistency and
correctness in photo-interpretation. Photogrammetric Engineering and Remote Sensing. Vol.
49, No. 1, pp. 69–74.
Congalton, R. and R. Mead . 1986. A review of three discrete multivariate analysis techniques
used in assessing the accuracy of remotely sensed data from error matrices. IEEE
Transactions of Geoscience and Remote Sensing. Vol. GE-24, No. 1, pp. 169–174.
Congalton, R. G. , R. G. Oderwald , and R. A. Mead . 1983. Assessing Landsat classification
accuracy using discrete multivariate statistical techniques. Photogrammetric Engineering and
Remote Sensing. Vol. 49, No. 12, pp. 1671–1678.
Cowardin, L. M. , V. Carter , F. Golet , and E. LaRoe . 1979. A Classification of Wetlands and
Deepwater Habitats of the United States. Office of Biological Services. U.S. Fish and Wildlife
Service. U.S. Department of Interior, Washington, DC. 103 pp.
CropScape—Cropland Data Layer, United States Department of Agriculture, National
Agricultural Statistics Service. Available online: https://nassgeodata.gmu.edu/CropScape
Czaplewski, R. 1992. Misclassification bias in aerial estimates. Photogrammetric Engineering
and Remote Sensing. Vol. 58, No. 2, pp. 189–192.
Czaplewski, R. and G. Catts . 1990. Calibrating area estimates for classification error using
confusion matrices. Proceedings of the 56th Annual Meeting of the American Society for
Photogrammetry and Remote Sensing. Denver, CO. Vol. 4, pp. 431–440.
Defense Mapping Agency). 1991. Error theory as applied to mapping, charting, and geodesy.
Defense Mapping Agency Technical Report 8400.1. Fairfax, Virginia. 71 pages plus
appendices.
Environmental Systems Research Institute, National Center for Geographic Information and
Analysis, and The Nature Conservancy . 1994. Accuracy Assessment Procedures: NBS/NPS
Vegetation Mapping Program. Report prepared for the National Biological Survey and National
Park Service. Redlands, CA, Santa Barbara, CA, and Arlington, VA, United States.
Eyre, F. H. 1980. Forest Cover Types of the United States and Canada. Society of American
Foresters, Washington, DC. 148 pp.
FEMA (Federal Emergency Management Agency) . 2003. Guidelines and Specifications for
Flood Hazard Mapping Partners.
FEMA, https://www.fema.gov/media-library/assets/documents/13948
FGDC (Federal Geographic Data Committee), Subcommittee for Base Cartographic Data .
1998. Geospatial Positioning Accuracy Standards. Part 3: National Standard for Spatial Data
Accuracy. FGDC-STD-007.3-1998: Washington, DC, Federal Geographic Data Committee. 24
pp.
Ferris State University . 2007. http://www.ferris.edu/faculty/burtchr/sure340/notes/History.pdf
Fitzpatrick-Lins, K. 1981. Comparison of sampling procedures and data analysis for a land-use
and land-cover map. Photogrammetric Engineering and Remote Sensing. Vol. 47, No. 3, pp.
343–351.
Fleiss, J. , J. Cohen , and B. Everitt . 1969. Large sample standard errors of kappa and
weighted kappa. Psychological Bulletin. Vol. 72, No. 5, pp. 323–327.
Foody, G. 1992. On the compensation for chance agreement in image classification accuracy
assessment. Photogrammetric Engineering and Remote Sensing. Vol. 58, No. 10, pp.
1459–1460.
Foody, G. M. 2009. Sample size determination for image classification accuracy assessment
and comparison. International Journal of Remote Sensing. Vol. 30, No. 20, pp. 5273–5291.
Freese, Frank. 1960. Testing accuracy. Forest Science. Vol. 6, No. 2, pp. 139–145.
Ginevan, M. E. 1979. Testing land-use map accuracy: Another look. Photogrammetric
Engineering and Remote Sensing. Vol. 45, No. 10, pp. 1371–1377.
Gong, P. and J. Chen . 1992. Boundary uncertainties in digitized maps: Some possible
determination methods. In: Proceedings of GIS/LIS’92. Annual Conference and Exposition. San
Jose, CA. pp. 274–281.
Goodman, Leo. 1965. On simultaneous confidence intervals for multinomial proportions.
Technometrics. Vol. 7, pp. 247–254.
Gopal, S. and C. Woodcock . 1994. Theory and methods for accuracy assessment of thematic
maps using fuzzy sets. Photogrammetric Engineering and Remote Sensing. Vol. 60, No. 2, pp.
181–188.
Grassia, A. and R. Sundberg . 1982. Statistical precision in the calibration and use of sorting
machines and other classifiers. Technometrics. Vol. 24, pp. 117–121.
Green, K. and R. Congalton . 2004. An error matrix approach to fuzzy accuracy assessment:
The NIMA Geocover project. A peer-reviewed chapter. In: Remote Sensing and GIS Accuracy
Assessment. R. S. Lunetta and J. G. Lyon (Editors). CRC Press, Boca Raton, FL. 304 pp.
Green, K. , K. Schulz , C. Lopez , et al. 2015. Vegetation Mapping Inventory Project: Haleakalā
National Park. National Park Service. Fort Collins, Colorado. Natural Resource Report
NPS/PACN/NRR2015/986.
https://irma.nps.gov/App/Reference/DownloadDigitalFile?code=525341&file=halerpt.pdf
Green, K. , R. G. Congalton , and M. Tukman . 2017. Imagery and GIS: Best Practices for
Extracting Information from Imagery. ESRI Press, Redlands, CA.
Greenwalt, Clyde and Melvin Schultz . 1962 and 1968. Principles of Error Theory and
Cartographic Applications. United States Air Force. Aeronautical Chart and Information Center.
ACIC Technical Report Number 96. St. Louis, MO. 60 pages plus appendices. This report is
cited in the ASPRS standards as ACIC, 1962.
Griffin, J. and W. Critchfield . 1972. The distribution of forest trees in California. USDA Forest
Service Research Paper PSW-82. Pacific Southwest Forest and Range Experiment Station,
Berkeley, CA.
Hay, A. M. 1979. Sampling designs to test land-use map accuracy. Photogrammetric
Engineering and Remote Sensing. Vol. 45, No. 4, pp. 529–533.
Hay, A. M. 1988. The derivation of global estimates from a confusion matrix. International
Journal of Remote Sensing. Vol. 9, pp. 1395–1398.
Hill, T. B. 1993. Taking the “ ” out of “ground truth”: Objective accuracy assessment. In:
Proceedings of the 12th Pecora Conference. Sioux Falls, SD. pp. 389–396.
Hopkirk, P. 1992. The Great Game. The Struggle for Empire in Central Asia. Kodansha
International. 565 pp.
Hord, R. M. and W. Brooner . 1976. Land-use map accuracy criteria. Photogrammetric
Engineering and Remote Sensing. Vol. 42, No. 5, pp. 671–677.
Hudson, W. and C. Ramm . 1987. Correct formulation of the kappa coefficient of agreement.
Photogrammetric Engineering and Remote Sensing. Vol. 53, No. 4, pp. 421–422.
Husch, B. , Beers, T. W. , and Kershaw, J. A., Jr. 2003. Forest Mensuration. 4th edition. John
Wiley & Sons, Hoboken, NJ. 443 pp.
Jensen, John. 2016. Introductory Digital Image Processing: A Remote Sensing Perspective. 4th
edition. Pearson Education, Glenview, IL. 623 pp.
Katz, A. H. 1952. Photogrammetry needs statistics. Photogrammetric Engineering. Vol. 18, No.
3, pp. 536–542.
Kearsley, M. J. , K. Green , M. Tukman , M. Reid , M. Hall , T. Ayers and K. Christie . 2015.
Grand Canyon National Park-Grand Canyon/Parashant National Monument vegetation
classification and mapping project. Natural Resource Report. NPS/GRCA/NRR—2015/913.
National Park Service. Fort Collins, CO. Published Report-2221240.
https://irma.nps.gov/DataStore/DownloadFile/520521
Landis, J. and G. Koch . 1977. The measurement of observer agreement for categorical data.
Biometrics. Vol. 33, pp. 159–174.
Lea, Chris and Anthony Curtis . 2010. Thematic accuracy assessment procedures. National
Park Service Vegetation Inventory, Version 2.0. Natural Resources Report NPS/NRPC/NRR –
2010/204. 116 pp.
Liu, C. , P. Frazier , and L. Kumar . 2007. Comparative assessment of the measures of
thematic classification accuracy. Remote Sensing of Environment. Vol. 107, pp. 606–616.
Lopez, A. , F. Javier , A. Gordo , and A. David . 2005. Sample Size and Confidence When
Applying the NSSDA. XXII International Cartographic Conference (ICC2005). Hosted by The
International Cartographic Association. Coruna, Spain. July 11–16, 2005.
Lowell, K. 1992. On the incorporation of uncertainty into spatial data systems. In: Proceedings
of GIS/LIS’92. Annual Conference and Exposition. San Jose, CA. pp. 484–493.
Lunetta, R. , R. Congalton , L. Fenstermaker , J. Jensen , K. McGwire , and L. Tinney . 1991.
Remote sensing and geographic information system data integration: Error sources and
research issues. Photogrammetric Engineering and Remote Sensing. Vol. 57, No. 6, pp.
677–687.
MacLean, M. , M. Campbell , D. Maynard , M. Ducey , and R. Congalton . 2013. Requirements
for labeling forest polygons in an object-based image analysis classification. International
Journal of Remote Sensing. Vol. 34, No. 7, pp. 2531–2547.
MacLean, M. and R. Congalton . 2013. Applicability of multi-date land cover mapping using
Landsat 5 TM imagery in the Northeastern US. Photogrammetric Engineering and Remote
Sensing. Vol. 79, No. 4, pp. 359–368.
Macleod, R. and R. Congalton . 1998. A quantitative comparison of change detection
algorithms for monitoring eelgrass from remotely sensed data. Photogrammetric Engineering
and Remote Sensing. Vol. 64, No. 3, pp. 207–216.
Malila, W. 1985. Comparison of the information contents of Landsat TM and MSS data.
Photogrammetric Engineering and Remote Sensing. Vol. 51, No. 9, pp. 1449–1457.
Massey, Richard , Temuulen T. Sankey , Kamini Yadav , Russell G. Congalton , and James
Tilton . 2018. Integrating cloud-based workflows in continental-scale cropland extent
classification. Remote Sensing of Environment. (In Review).
Maune, David (Editor). 2007. Digital Elevation Model Technologies and Applications: The DEM
Users Manual. 2nd Edition. American Society of Photogrammetry and Remote Sensing,
Bethesda, MD. 655 pp.
Mayer, K. and W. Laudenslayer (Editors). 1988. A guide to wildlife habitats in California.
California Department of Forestry and Fire Protection. Sacramento, CA.
McGlone, J. C. (Editor). 2004. Manual of Photogrammetry. American Society for
Photogrammetry and Remote Sensing, Bethesda, MD. 1151 pp.
McGuire, K. 1992. Analyst variability in labeling unsupervised classifications. Photogrammetric
Engineering and Remote Sensing. Vol. 58, No. 12, pp. 1705–1709.
Mikhail, E. M. and G. Gracie . 1981. Analysis and Adjustment of Survey Measurements. Van
Nostrand Reinhold. 340 pp.
MPLMIC . 1999. Positional Accuracy Handbook. Using the National Standard for Spatial Data
Accuracy to Measure and Report Geographic Data Quality. Minnesota Planning land
Management Information Center, St. Paul, MN. 29 pp.
National Vegetation Classification Standard . 2018. http://usnvc.org/data-standard/natural-
vegetation-classification/. Site last visited June 23, 2018.
NDEP . 2004. Guidelines for Digital Elevation Data. Version 1.0. National Digital Elevation
Program. May 10, 2004.
Pillsbury, N. , M. DeLasaux , R. Pryor , and W. Bremer . 1991. Mapping and GIS database
development for California’s hardwood resources. California Department of Forestry and Fire
Protection, Forest and Rangeland Resources Assessment Program (FRRAP). Sacramento, CA.
Pontius, R. and M. Millones . 2011. Death to Kappa: Birth of quantity disagreement and
allocation disagreement for accuracy assessment. International Journal of Remote Sensing.
Vol. 32, No. 15, pp. 4407–4429.
Prisley, S. and J. Smith . 1987. Using classification error matrices to improve the accuracy of
weighted land-cover models. Photogrammetric Engineering and Remote Sensing. Vol. 53, No.
9, pp. 1259–1263.
Radoux, J. , R. Bogaert , D. Fasbender , and P. Defourny , 2010. Thematic accuracy
assessment of geographic object-based image classification. International Journal of
Geographical Information Science. Vol. 25, No. 6, pp. 895–911.
Rhode, W. G. 1978. Digital image analysis techniques for natural resource inventories. National
Computer Conference Proceedings. pp. 43–106.
Rosenfield, G. and K. Fitzpatrick-Lins . 1986. A coefficient of agreement as a measure of
thematic classification accuracy. Photogrammetric Engineering and Remote Sensing. Vol. 52,
No. 2, pp. 223–227.
Rosenfield, G. H. , K. Fitzpatrick-Lins , and H. Ling. 1982. Sampling for thematic map accuracy
testing. Photogrammetric Engineering and Remote Sensing. Vol. 48, No. 1, pp. 131–137.
Sammi, J. C. 1950. The application of statistics to photogrammetry. Photogrammetric
Engineering. Vol. 16, No. 5, pp. 681–685.
Singh, A. 1986. Change detection in the tropical rain forest environment of northeastern India
using Landsat. In: Remote Sensing and Tropical Land Management. M. J. Eden and J. T. Parry
(Editors). John Wiley & Sons, London. pp. 237–254.
Singh, A. 1989. Digital change detection techniques using remotely sensed data. International
Journal of Remote Sensing. Vol. 10, No. 6, pp. 989–1003.
Spurr, Stephen. 1948. Aerial Photographs in Forestry. Ronald Press, New York. 340 pp.
Spurr, Stephen. 1960. Photogrammetry and Photo-Interpretation with a Section on Applications
to Forestry. Ronald Press, New York. 472 pp.
Stehman, S. 1992. Comparison of systematic and random sampling for estimating the accuracy
of maps generated from remotely sensed data. Photogrammetric Engineering and Remote
Sensing. Vol. 58, No. 9, pp. 1343–1350.
Story, M. and R. Congalton . 1986. Accuracy assessment: A user’s perspective.
Photogrammetric Engineering and Remote Sensing. Vol. 52, No. 3, pp. 397–399.
Teluguntla, P. , P. Thenkabail , J. Xiong , et al. 2015. Global Food Security Support Analysis
Data (GFSAD) at Nominal 1-km (GCAD) derived from remote sensing in support of food
security in the twenty-first century: Current achievements and future possibilities. In: Remote
Sensing Handbook; Vol. II: Land Resources Monitoring, Modeling, and Mapping with Remote
Sensing. P. Thenkabail (Editor). CRC/Taylor & Francis, Boca Raton, FL. pp. 131–159.
Tenenbein, A. 1972. A double sampling scheme for estimating from misclassified multinomial
data with applications to sampling inspection. Technometrics. Vol. 14, pp. 187–202.
Tortora, R. 1978. A note on sample size estimation for multinomial populations. The American
Statistician. Vol. 32, No. 3, pp. 100–102.
U.S. Bureau of the Budget. 1941 . 1947. National Map Accuracy Standards. Washington, DC.
United States National Vegetation Classification Database Ver. 2.02. Federal Geographic Data
Committee, Vegetation Subcommittee. Washington DC. 2018.
https://www.sciencebase.gov/catalog/item/5aa827a2e4b0b1c392ef337a
Van Genderen, J. L. and B. F. Lock . 1977. Testing land use map accuracy. Photogrammetric
Engineering and Remote Sensing. Vol. 43, No. 9, pp. 1135–1137.
Van Genderen, J. L. , B. F. Lock , and P. A. Vass . 1978. Remote sensing: Statistical testing of
thematic map accuracy. Proceedings of the Twelfth International Symposium on Remote
Sensing of Environment. ERIM. pp. 3–14.
Woodcock, C. 1996. On roles and goals for map accuracy assessment: A remote sensing
perspective. Proc: Second International Symposium on Spatial Accuracy Assessment in Natural
Resources and Environmental Sciences, USDA Forest Service Rocky Mountain Forest and
Range Experiment Station, Gen. Tech. Rep. RM-GTR-277, Fort Collins, CO. pp. 535–540.
Woodcock, C. and S. Gopal . 1992. Accuracy assessment of the Stanislaus Forest vegetation
map using fuzzy sets. In: Remote Sensing and Natural Resource Management. Proceedings of
the 4th Forest Service Remote Sensing Conference, Orlando, FL. pp. 378–394.
Yadav, Kamini and Russell G. Congalton . 2018. Issues with large area thematic accuracy
assessment for mapping cropland extent: A tale of three continents. Remote Sensing. Vol. 10,
No. 1, p. 53. doi:10.3390/rs10010053
Young, H. E. 1955. The need for quantitative evaluation of the photo interpretation system.
Photogrammetric Engineering. Vol. 21, No. 5, pp. 712–714.
Young, H. E. and E. G. Stoeckler . 1956. Quantitative evaluation of photo interpretation
mapping. Photogrammetric Engineering. Vol. 22, No. 1, pp. 137–143.
Zadeh, L. A. 1965. Fuzzy sets. Information and Control. Vol. 8, pp. 338–353.
Zar, J. 1974. Biostatistical Analysis. Prentice-Hall. 620 pp.