In search of the lost loss function

In search of the lost loss
function
ethics, equity and rationality in rare disease research
Stephen Senn
Consultant Statistician
(c) Stephen Senn 2018 1

Acknowledgements
Many thanks to the BBS, EFSPI and Pierre Verweij for the invitation
This work is partly supported by the European Union’s 7th Framework Programme
for research, technological development and demonstration under grant
agreement no. 602552. “IDEAL”
This research was undertaken during the period I worked for the Luxembourg
Institute of Health
Thanks to my IDeAl colleagues (in particular WP9), the FP7 and the LIH
All parties mentioned are absolved from any responsibility for what I have to say

Basic thesis
• Rare diseases present a challenge when traditional models of drug
development are considered
• Similar issues apply to small subgroups
• I think we need a fundamental examination of what we need to do
• But I am not capable of providing it!
• The value of information is central
• This requires thinking about the losses associated with
1. Imperfect decisions
2. Delays in taking action

Warning: a ragbag of thoughts
An apology
• This talk is very confusing
• In fact, it is even confusing to me
• You have little hope of finding
any structure
• The outline on the right may
help
An outline
• Conventional power calculations
• The value of information
approach
• The philosophy of John Rawls as
it may or may not apply to drug
development
• Some examples
• Lessons for ethics and perhaps
rare diseases

Approximate Formula for Sample-Size
Determination
Primitives
 : probability of type I error given no effect.
: the relevant difference.
 : probability of a type II error given that the true treatment effect
is relevant.
: the standard deviation of the outcome.
Functions of primitives
𝑍 𝛼 2 : value of standard Normal corresponding to right hand tail
probabilities of 𝛼 2 .
𝑍 𝛽 : value of standard Normal corresponding to right hand tail
probabilities of 𝛽.
Derived
n: the number of units in each treatment group.
Alternative interpretation
k: targeted signal to noise ratio or, equivalently, precision in units of
clinical relevance
 
2
2
2
2
2
2
2
2
substitute ,
2
2
n z z
k z z
n k
k
n
 
 



 
   
 
 
 
  
 


 
 
 

 =0.00125 is ‘two trials
rule’ equivalent
Alternative view of
conventional power
calculations is that they
target a given degree of ‘data
precision’
Typically between about 2.8
and 4.5
So they can be regarded as
providing a given amount of
information
Exception: frequentist
sequential trials do not do
this
In my opinion, stopping
early for efficacy is rarely a
good idea

A trap for the unwary
• It is sometimes suggested that only large trials (adequate power etc) are
worth doing
• However, consider the case of a fixed budget and many possible projects
• Now think of the problem in terms of the average size of the trials for all
projects
• All those projects that are not funded have, from one point of view, trials
with zero patients
• Where else would a statistician think it was acceptable to calculate
averages ignoring the zeros?
• So, making sure that only big trials are run may not be information efficient
from a wider perspective

What is missing?
• Cost
• In £,€,$,CHF
• In patients’ lives and quality
• In other consequences of delay
• Conventional power calculations have no cost dimension
• This means that for any degree of targeted data precision two trials
with the same value of / will give the same answer, whatever the
cost
• What is needed is a value of information perspective

Subgroups (an aside)
• In a sense small subgroups are
similar to rare diseases
• However, information may carry
over between subgroups rather
more plausibly than between
diseases
• Suggests that bias-variance trade-
offs be considered
• Also each sub-group investigation
should perhaps be regarded as
competing for funds and hence as a
mini-project
• Is it worth doing?

Value of information
The basic idea
• There are three choices facing you
• Choose A
• Choose B
• Pay to find out more about the relative values of A and B
• You may currently believe that B is better than A
• If you had to act now, you should choose B
• If you can delay action with the possibility of acquiring new
information, it might be worth doing this
• Depends on the losses involved
• Depends on cost of information

Various historical approaches (selection)
Fixed
• Raiffa and Schlaifer, 1961
• Lindley, 1997
• Stallard, 1998
• Burman and Senn 2003
• Etc, etc
Sequential
• Anscombe 1963
• Chernoff 1966
• Gittins various, starting 1979
• Etc, etc
• Pertile et al 2013
• Jobjörnssen and Christiansen
2017

Lindley’s approach:
a double optimisation
Optimal action for any given result for
any sample size
You must know what the optimal
decision would be (say choose A
or choose B) for any given result
for any given sample size
You then have to calculate the
expected value of the optimal
decision
Optimal sample size using expected value
of optimal decision per sample size
Associated with each sample size
there is a cost
This has to be subtracted from the
expected benefit of the optimal
decision
You have to search amongst the
sample sizes to find the maximum
expected benefit net of
information cost

I am not going to go into this theory
• It’s complicated
• I don’t think that advancing this theory is the main problem
• I think that the problem of deciding on appropriate loss functions and
also various perspectives , patients, society, is more important
• I am too stupid to understand let alone explain the theory fully
• So I am now going to consider a basic perspective and provide some
examples to help raise some issues

(C) Stephen Senn Ethics PSI 2010 14
A Rawlsian View (John Rawls 1921-2002)
• You are about to be born into the world
• The original position
• You don’t know who you will be
• The veil of ignorance
• How would you like society to be organised?
• Ethical arrangements require long-term perspectives
• Something similar is required in the world of insurance
• You can’t insure against a calamity that has already happened
• Utmost good faith is required

Example 1: spending priorities in society
• Try your hand at this one.
• The spending priorities of Great
Britain Ltd
• Currently spend millions on frivolous
holidays for the young in Ibiza
• We have lots of deserving elderly on
the waiting lists for hip replacement
etc.
• Shouldn’t we tax the young and
single to pay for these operations?
• You are about to start your life
• But you don’t know who you will
be
• Do you want society to be
• For “ants” only
• No holidays
• High taxes for eventual old age
• Or for “grasshoppers”
• Let’s have fun while we are young

The Original Position and Medical Research
Short termism
• Doctors back their hunches as to
what they think is best
• “Right to Try” law USA
• Signed off by president Trump 30
May 2018
• Medical Innovation Bill UK
(Saatchi bill)
• Failed to pass House of Commons
Long termism (The regulatory system)
• You don’t know whether and in
what era you fall ill
• It is in your interest that drugs are
evaluated scientifically
• You benefit from previous research
• Patients have the right to approved
medication
• The only access to unapproved
medication is through clinical trials

The two systems
A closed world of Dr and Patient
A wider world with ‘society’ as a third
player

Example 2: O’Quigley’s Continuous Reassessment
Method CRM
• Example of a one-step-ahead optimisation problem
• You are trying to target the best dose for the patient you are just
about to treat
• Once that patient has been treated you repeat the process for the next
patient
• It is possible that you could do better for later patients by allocating
some of the earlier patients sub-optimally for them
• This is usually regarded as unethical
• However, there is one problem
• How do you decide when to stop?

Example 3 Childhood melanoma
• This disease had its waiver for EMA
Paediatric Investigation Plans revoked
in 2008
• As far as I am aware the situation has not
changed
• So you have to study children if you
want to get a license
• A blow for the rights of children has
been struck
• Or has it?
• But the results of such trials will only
benefit (if at all) children who are not
yet ill and perhaps not yet born
• How will melanoma most plausibly
affect their lives?
• As a paediatric patient?
• Because their parents become patients?
• Because they get melanoma as young
adults?
• So if such legislation delays research
into melanoma in adults it may
actually be against the interests of
children

Example 4 First in Man Studies
• Suppose that the acceptable risk
to an individual is 1 in 2000
• But we believe it is 1 in 1000 for
this drug
• By having one placebo for every
active treatment and
randomising we can reduce the
risk to 1 in 2000
• Does this make it acceptable?
• Such a device reduces the risk to
the individual to acceptable
levels
• However, it does not reduce the
expected number of side-effects
per trial
• Nor the risk to the insurer
• Suggests a dual perspective
• Acceptable to an individual AND
to society
Tokyo First in Man 20

Example 5 Funding rare disease research?
• Suppose that we know that the most effective total impact for good
on the health of children would be to spend all our research budget
on the most common diseases?
• Should we abandon research in rare diseases for the foreseeable
future?
• This is a really difficult case
• At first sight the long term broader view seems to suggest ‘yes’
• But maybe the even longer Rawlsian view suggests ‘no’
• This is a very difficult issue

Trials in rare diseases
• We have to accept that the classical model is inappropriate
• Can be defended when we have a long time horizon of future patients
compared to patients in the trial
• Patients have to make choices even if we have no provided
information for them
• Pragmatic framework of Schwartz and Lellouch 1967
• Type III errors should be controlled
• Choosing the worse treatment
• We may have to accept weaker standards of evidence
• WP9 IDEAL project https://www.ideal.rwth-aachen.de/?page_id=342

Conclusions
• It is often very misleading to make decisions at the point of sickness
• Young holidays versus old hip replacements
• Sometimes, however, we concentrate on the short term
• CRM
• Sometimes longer and wider perspectives might be appropriate
• Childhood melanoma
• Nevertheless, single perspectives may not be enough
• First-in-man example (societal and individual)
• Rare diseases example
• We need more debate about what we are tying to achieve with clinical
trials

Final thought
Statistical calculation of consequences is not the be all and end all of
ethics
Nevertheless, those who ignore statistical considerations in coming to
decisions about resource allocation and information gathering in
medicine are likely to make bad decisions and this is unethical

In search of the lost loss function

More Related Content

What's hot (20)

Similar to In search of the lost loss function (20)

More from Stephen Senn (9)

Recently uploaded (20)

In search of the lost loss function