Chapter 10

Verification and Validation of

Simulation Models

Prof. Dr. Mesut Güneş ▪ Ch. 10 Verification and Validation of Simulation Models 10.1
 Contents
• Model-Building, Verification, and Validation
• Verification of Simulation Models
• Calibration and Validation

Prof. Dr. Mesut Güneş ▪ Ch. 10 Verification and Validation of Simulation Models 10.2
 Purpose & Overview
• The goal of the validation process is:
• To produce a model that represents
true behavior closely enough for
f
decision-making purposes
• To increase the model’s credibility to an
acceptable level
• Validation is an integral part of model
development:
• Verification: building the model
correctly, correctly implemented with
good input and structure
• Validation: building the correct model,
an accurate representation of the real
system
• Most methods are informal subjective
comparisons while a few are formal
statistical
t ti ti l procedures
d

Prof. Dr. Mesut Güneş ▪ Ch. 10 Verification and Validation of Simulation Models 10.3
 Modeling-Building,
Modeling Building, Verification & Validation

Prof. Dr. Mesut Güneş ▪ Ch. 10 Verification and Validation of Simulation Models 10.4
 Modeling-Building, Verification & Validation
• Steps in Model-Building
• Observing g the real system
y
and the interactions among
their various components
and of collecting data on
their behavior
• Construction of a
conceptual model
• Implementation of an
operational
p model

Prof. Dr. Mesut Güneş ▪ Ch. 10 Verification and Validation of Simulation Models 10.5
 Verification
• Purpose: ensure the conceptual model is reflected accurately in
the computerized representation.
• Many common-sense suggestions, for example:
• Have someone else check the model.
• Make a flow diagram that includes each logically possible action a
system can take when an event occurs.
• Closely examine the model output for reasonableness under a variety
of input parameter settings.
settings
• Print the input parameters at the end of the simulation, make sure
they have not been changed inadvertently.
• Make
M k theth operational
ti l model
d l as self-documenting
lf d ti as possible.
ibl
• If the operational model is animated, verify that what is seen in the
animation imitates the actual system.
• Use the debugger.
• If possible use a graphical representation of the model.

Prof. Dr. Mesut Güneş ▪ Ch. 10 Verification and Validation of Simulation Models 10.6
 Examination of Model Output for
Reasonableness
• Two statistics that give a quick indication of model
reasonableness are current contents and total counts
• Current content: The number of items in each component of
the system at a given time.
• Total counts: Total number of items that have entered each
component of the system by a given time.
• Compute
p certain long-run
g measures of p
performance,, e.g.
g
compute the long-run server utilization and compare to
simulation results.

Prof. Dr. Mesut Güneş ▪ Ch. 10 Verification and Validation of Simulation Models 10.7
 Examination of Model Output for
Reasonableness
• A model of a complex network of queues consisting
of many
y service centers.
• If the current content grows in a more or less linear fashion as
the simulation run time increases, it is likely that a queue is
unstable
• If the total count for some subsystem is zero, indicates no
items entered that subsystem, a highly suspect occurrence
• If the total and current count are equal to one, can indicate
that an entity has captured a resource but never freed that
resource.
resource

Prof. Dr. Mesut Güneş ▪ Ch. 10 Verification and Validation of Simulation Models 10.8
 Other Important Tools
• Documentation
• A means of clarifying
y g the logic
g of a model and verifying
y g its
completeness.
• Comment the operational model, definition of all variables and
parameters.
parameters
• Use of a trace
• A detailed p
printout of the state of the simulation model over
time.
• Can be very labor intensive if the programming language does not
support statistic collection.
collection
• Labor can be reduced by a centralized tracing mechanism
• In object-oriented simulation framework, trace support can be
integrated into class hierarchy.
hierarchy New classes need only to add little
for the trace support.

Prof. Dr. Mesut Güneş ▪ Ch. 10 Verification and Validation of Simulation Models 10.9
 Trace: Example
• Simple queue from Chapter 2
• Trace over a time interval [[0,, 16]]
• Allows the test of the results by pen-and-paper method

Definition of Variables:
CLOCK = Simulation clock
EVTYP = Event type (Start, Arrival, Departure, Stop)
NCUST = Number of customers in system at time CLOCK
STATUS = Status of server (1=busy, 0=idle)

State of System Just After the Named Event Occurs:

CLOCK = 0 EVTYP = Start NCUST=0 STATUS = 0
CLOCK = 3 EVTYP = Arrival NCUST=1 STATUS = 0
CLOCK = 5 EVTYP = Depart NCUST=0 STATUS = 0
CLOCK = 11 EVTYP = Arrival NCUST=1 STATUS = 0
CLOCK = 12 EVTYP = Arrival NCUST=2 2 STATUS = 1 There is a customer,
customer
CLOCK = 16 EVTYP = Depart NCUST=1 STATUS = 1 but the status is 0
...

Prof. Dr. Mesut Güneş ▪ Ch. 10 Verification and Validation of Simulation Models 10.10
 Calibration and Validation

Prof. Dr. Mesut Güneş ▪ Ch. 10 Verification and Validation of Simulation Models 10.11
 Calibration and Validation
• Validation: the overall process of comparing the model
and its behavior to the real system.
• Calibration: the iterative process of comparing the model
to the real system and making adjustments.
• Comparison off the
h modeld l to reall system
• Subjective tests
• People who are
knowledgeable
about the system
• Objective tests
• Requires data on the real
system’s behavior and the
output
t t off th
the model
d l

Prof. Dr. Mesut Güneş ▪ Ch. 10 Verification and Validation of Simulation Models 10.12
 Calibration and Validation
• Danger during the calibration phase
• Typically
yp y few data sets are available,, in the worst case only
y
one, and the model is only validated for these.
• Solution: If possible collect new data sets

• No model is ever a perfect representation of the system

• The modeler must weigh the possible,
possible but not guaranteed,
guaranteed
increase in model accuracy versus the cost of increased
validation effort.

• Three-step approach for validation:

• Build
B ild a model
d l that
th t has
h high
hi h face
f validity.
lidit
• Validate model assumptions.
• Compare the model input
input-output
output transformations with the real
system’s data.
Prof. Dr. Mesut Güneş ▪ Ch. 10 Verification and Validation of Simulation Models 10.13
 High Face Validity
• Ensure a high degree of realism:
• Potential users should be involved in model construction from
its conceptualization to its implementation.
• Sensitivity analysis can also be used to check a model’s
face validity.
validity
• Example: In most queueing systems, if the arrival rate of
customers were to increase,, it would be expected
p that server
utilization, queue length and delays would tend to increase.
• For large-scale simulation models, there are many input
variables and thus possibly many sensitivity tests.
tests
• Sometimes not possible to perform all of theses tests, select the
most critical ones.

Prof. Dr. Mesut Güneş ▪ Ch. 10 Verification and Validation of Simulation Models 10.14
 Validate Model Assumptions
• General classes of model assumptions:
• Structural assumptions:
p how the system
y operates.
p
• Data assumptions: reliability of data and its statistical
analysis.
• Bank
B k example:
l customer
t queueing
i and
d service
i facility
f ilit in
i a
bank.
• Structural assumptions
• Customer waiting in one line versus many lines
• Customers are served according FCFS versus priority
• Data assumptions, e.g., interarrivall time off customers, service
times for commercial accounts.
• Verify
y data reliability
y with bank managers
g
• Test correlation and goodness of fit for data

Prof. Dr. Mesut Güneş ▪ Ch. 10 Verification and Validation of Simulation Models 10.15
 Validate Input-Output Transformation
• Goal: Validate the model’s ability to predict future behavior
• The only objective test of the model.
• The structure of the model should be accurate enough to make good
predictions for the range of input data sets of interest.
• One possible approach: use historical data that have been
reserved for validation purposes only.
• Criteria: use the main responses of interest.

Input Output
System
Model is viewed as an
input-output
transformation

Input Output
Model

Prof. Dr. Mesut Güneş ▪ Ch. 10 Verification and Validation of Simulation Models 10.16
 Bank Example
• Example: One drive-in window serviced by one teller, only one
or two transactions are allowed.
• Data collection: 90 customers during 11 am to 1 pm.
• Observed service times {Si, i = 1,2, …, 90}.
• Observed interarrival times {Ai, i = 1,2, …, 90}.
• Data analysis let to the conclusion that:
• Interarrival times: exponentially distributed
with rate λ = 45/hour
Input variables
• Service times: N(1.1, 0.22)

Prof. Dr. Mesut Güneş ▪ Ch. 10 Verification and Validation of Simulation Models 10.17
 Bank Example: The Black Box
• A model was developed in close consultation with bank
management and employees
• Model assumptions were validated
• Resulting model is now viewed as a “black box”:

Model Output Variables, Y

Input Variables
Primary interest:
Poisson arrivals Y1 = teller’s utilization
Uncontrolled λ = 45/hr: X11, X12, … Y2 = average delay
variables, X Services times, Model Y3 = maximum line length
N(D2, 0.22):
0 22): X21, X22, … “black
black box
box”
f(X,D) = Y Secondary interest:
D1 = 1 (one teller) Y4 = observed arrival rate
Controlled
D2 = 1.1 min Y5 = average service time
Decision
variables, D (mean service time) Y6 = sample std. dev. of
D3 = 1 (one line) service times
Y7 = average length of time

Prof. Dr. Mesut Güneş ▪ Ch. 10 Verification and Validation of Simulation Models 10.18
 Comparison with Real System Data
• Real system data are necessary for validation.
• System responses should have been collected during the same time
period (from
f 11am to 1pm on the same day.)
• Compare average delay from the model Y2 with actual delay Z2:
• Average delay observed, Z2 = 4.3 minutes, consider this to be the true
mean value μ0 = 4.3.
• When the model is run with generated random variates X1n and X2n, Y2
should be close to Z2.

Prof. Dr. Mesut Güneş ▪ Ch. 10 Verification and Validation of Simulation Models 10.19
 Comparison with Real System Data
• Six statistically independent replications of the model,
each of 2-hour duration, are run.

Y4 Y5 Y2
Replication Arrivals/Hour Service Time [Minutes] Average Delay [Minutes]

1 51.0 1.07 2.79

2 40.0 1.12 1.12
3 45.5 1.06 2.24
4 50.5 1.10 3.45
5 53.0 1.09 3.13
6 49.0 1.07 2.38
Sample mean [Delay] 2.51
Standard deviation [Delay] 0.82

Prof. Dr. Mesut Güneş ▪ Ch. 10 Verification and Validation of Simulation Models 10.20
 Hypothesis Testing
• Compare the average delay from the model Y2 with the actual
delay Z2

• Null hypothesis testing: evaluate whether the simulation and the real
system are the same (w.r.t. output measures):

H 0: E(Y2 ) = 4.3 minutes

H1: E(Y2 ) ≠ 4.3 minutes

• If H0 is not rejected, then, there is no reason to consider the model

invalid
• If H0 is rejected, the current version of the model is rejected, and
the modeler needs to improve the model

Prof. Dr. Mesut Güneş ▪ Ch. 10 Verification and Validation of Simulation Models 10.21
 Hypothesis Testing
• Conduct the t test:
• Chose level of significance (α = 0.05) and sample size (n = 6).
• Compute the sample mean and sample standard deviation
over the n replications:
n
1 n
Y2 = ∑ Y2i = 2.51 minutes ∑ (Y 2i − Y2 ) 2
n i =1 S= i =1
= 0.82 minutes
n −1

• Compute test statistics:

Y2 − μ0 2.51 − 4.3
t0 = = = 5.34 > tcritical = 2.571 (for a 2 - sided test)
S/ n 0.82 / 6
• Hence, reject H0.
• Conclude that the model is inadequate.
• Check: the assumptions justifying a t test, that the
observations (Y2i) are normally and independently distributed.

Prof. Dr. Mesut Güneş ▪ Ch. 10 Verification and Validation of Simulation Models 10.22
 Hypothesis Testing
• Similarly, compare the model output with the observed output
for other measures:
Y4 ↔ Z4, Y5 ↔ Z5, and Y6 ↔ Z6

Prof. Dr. Mesut Güneş ▪ Ch. 10 Verification and Validation of Simulation Models 10.23
 Type II Error
• For validation, the power of the test is:
• Probability(detecting an invalid model) = 1 – β
• β = P(Type II error) = P(failing to reject H0 | H1 is true)
• Consider failure to reject H0 as a strong conclusion, the modeler
would want β to be small.
• Value of β depends on:
• Sample size, n
difference δ, between E(Y) and μ:
• The true difference,

E (Y ) − μ
δ=
σ

• In general, the best approach to control β error is:

• Specify
S if the
th critical
iti l difference,
diff δ
δ.
• Choose a sample size, n, by making use of the operating
characteristics curve (OC curve).

Prof. Dr. Mesut Güneş ▪ Ch. 10 Verification and Validation of Simulation Models 10.24
 Type II Error
• Operating characteristics
curve (OC curve).
• Graphs of the probability of a
Type II Error β(δ) versus δ for
a given sample size n

For the same error

probability with smaller
diff
difference the
h required
i d
sample size increases!

Prof. Dr. Mesut Güneş ▪ Ch. 10 Verification and Validation of Simulation Models 10.25
 Type I and II Error
• Type I error (α):
• Error of rejecting a valid model.
• Controlled by specifying a small level of significance α.
• Type II error (β):
• Error of accepting a model as valid when it is invalid.
invalid
• Controlled by specifying critical difference and find the n.
• For a fixed sample size n, increasing α will decrease β.

Statistical Terminology Modeling Terminology Associated Risk

Type I: rejecting H0 when H0 is true Rejecting a valid model α
Type II: failure to reject H0 when H1 is true Failure to reject an invalid model β

Prof. Dr. Mesut Güneş ▪ Ch. 10 Verification and Validation of Simulation Models 10.26
 Confidence Interval Testing
• Confidence interval testing: evaluate whether the
simulation and the real system
y performance
p
measures are close enough.
• If Y is the simulation output, and μ = E(Y)
• The confidence interval (CI) for μ is:

⎡ S S ⎤
⎢Y − t α , n −1 , Y + t α , n −1
⎥
⎣ 2
n 2
n⎦

Prof. Dr. Mesut Güneş ▪ Ch. 10 Verification and Validation of Simulation Models 10.27
 Confidence Interval Testing
• Validating the model: ε is a difference value
• Suppose the CI does not chosen by the analyst, that
contain μ0: is small enough to allow
valid decisions to be based
• If the best-case error is > ε, on simulations!
model needs to be refined.
h worst-case error is ≤ ε,
• Iff the
accept the model.
• If best-case error is ≤ ε,
additional replications are
necessary.
• Suppose the CI contains μ0:
• If either the best-case
best case or worst-
worst
case error is > ε, additional
replications are necessary.
• If the worst case error is ≤ ε,
worst-case
accept the model.

Prof. Dr. Mesut Güneş ▪ Ch. 10 Verification and Validation of Simulation Models 10.28
 Confidence Interval Testing
• Bank example: μ0 = 4.3, and “close enough” is ε = 1 minute of
expected customer delay.
• A 95% confidence interval, based on the 6 replications is
[1.65, 3.37] because:

S
Y ± t0.025,5
n
0.82
2.51 ± 2.571
6
• μ0 = 4.3 falls outside the confidence interval,
• the best case |3.37 – 4.3| = 0.93 < 1, but
• the worst case |1.65 – 4.3| = 2.65 > 1
Additional replications are needed to reach a decision.

Prof. Dr. Mesut Güneş ▪ Ch. 10 Verification and Validation of Simulation Models 10.29
 Using Historical Output Data
• An alternative to generating input data:
• Use the actual historical record.
• Drive the simulation model with the historical record and then
compare model output to system data.
• In the bank example,
p , use the recorded interarrival and service times
for the customers {An, Sn, n = 1,2,…}.

• Procedure and validation process: similar to the approach used

for system generated input data.

Prof. Dr. Mesut Güneş ▪ Ch. 10 Verification and Validation of Simulation Models 10.30
 Using a Turing Test
• Use in addition to statistical test, or when no statistical test is
readily applicable.
Turing Test
Described by Alan Turing in 1950. A human jugde is involved in a natural language
conversation with a human and a machine. If the judge cannot reliably tell which of the
partners is the machine,
machine then the machine has passed the test
test.

• Utilize persons’ knowledge about the system.

• For example:
p
• Present 10 system performance reports to a manager of the system.
Five of them are from the real system and the rest are “fake” reports
based on simulation output data.
• If the person identifies a substantial number of the fake reports,
interview the person to get information for model improvement.
• If the person cannot distinguish between fake and real reports with
consistency, conclude that the test gives no evidence of model
inadequacy.

Prof. Dr. Mesut Güneş ▪ Ch. 10 Verification and Validation of Simulation Models 10.31
 Summary
• Model validation is essential:
• Model verification
• Calibration and validation
• Conceptual validation
• Best to compare system data to model data,
data and make
comparison using a wide variety of techniques.
• Some techniques that we covered:
• Insure high face validity by consulting knowledgeable persons.
• Conduct simple statistical tests on assumed distributional forms.
• Conduct a Turing test.
• Compare model output to system output by statistical tests.

Prof. Dr. Mesut Güneş ▪ Ch. 10 Verification and Validation of Simulation Models 10.32