Auto-Calibration Tests for Discrete Finite Regression Functions

Abstract

Auto-calibration is an important property of regression functions for actuarial applications. Comparably little is known about statistical testing of auto-calibration. Denuit et al. (2024) recently published a test with an asymptotic distribution that is not fully explicit and its evaluation needs non-parametric Monte Carlo sampling. In a simpler set-up, we present three test statistics with fully known and interpretable asymptotic distributions.

Keywords. Auto-calibration, concentration curve, Lorenz curve, area between the curves.

1 Introduction

Recent actuarial and financial literature acknowledges the importance of the statistical concept of auto-calibration; see, e.g., Krüger–Ziegel [6], Denuit et al. [1] and Wüthrich [7]. Select an integrable response variable $Y$ and covariates $\boldsymbol{X}$ with support ${\cal X}$.

Definition 1.1

A measurable regression function $\pi:{\cal X}\to{\mathbb{R}}$ is auto-calibrated for $(Y,\boldsymbol{X})$ if

$$\pi(\boldsymbol{X})={\mathbb{E}}\left[\left.Y\right|\pi(\boldsymbol{X})\right],\qquad\text{${\mathbb{P}}$-a.s.}$$

In an actuarial pricing context this means that every price cohort $\pi(\boldsymbol{X})$ is on average self-financing for the claims $Y$, or in other words, there is no systematic cross-financing within a pricing scheme designed by the regression function $\pi$.

Surprisingly, there is no mature literature on testing for auto-calibration. Most proposals only consider binary responses, e.g., Gneiting–Resin [5] discuss a bootstrap test and Dimitriadis et al. [4] study calibration bands. Recently, Denuit et al. [2] presented an auto-calibration test that studies the difference between the concentration curve (CC) and the Lorenz curve (LC). Also this test requires simulations because the asymptotic distribution of the test statistics is not sufficiently explicitly. We take one step back here, and we present simpler test statistics with fully known and interpretable asymptotic distributions, though, in a simpler set-up.

One needs three ingredients for an auto-calibration test. (a) A regression function $\pi:{\cal X}\to{\mathbb{R}}$. This regression function $\pi$ can be fully general, i.e., we do not require that it is close (in some metric) to the conditional mean ${\mathbb{E}}[Y|\boldsymbol{X}]$, nor do we specify whether $\pi$ has been estimated from past data ${\cal D}$ or whether it has been set by an expert. (b) A pair $(Y,\boldsymbol{X})$. For simplicity, we assume that the response $Y$ is positive and square integrable. The covariates $\boldsymbol{X}$ have support ${\cal X}$. (c) An i.i.d. sample ${\cal T}=(Y_{i},\boldsymbol{X}_{i})_{i\geq 1}$ for testing. This sample should have the same law as $(Y,\boldsymbol{X})$. These three ingredients (a)-(c) are sufficient for testing for auto-calibration of $\pi$ for $(Y,\boldsymbol{X})$; if $\pi$ has been estimated from past data ${\cal D}$, we generally assume that $(Y,\boldsymbol{X})$, ${\cal T}$ and ${\cal D}$ are independent, and all subsequent statements need then be understood conditional on ${\cal D}$.

2 Tests for auto-calibration

Assume that the regression function $\pi:{\cal X}\to{\mathbb{R}}$ takes only finitely many (ordered) values $-\infty<\pi_{1}<\cdots<\pi_{K}<\infty$. This gives us a partition of the covariate space ${\cal X}$ with

$${\mathbb{P}}\left[\pi(\boldsymbol{X})=\pi_{k}\right]=p_{k}>0\qquad\text{ for all $1\leq k\leq K$.}$$ (2.1)

We assume probabilities $p_{k}>0$, otherwise the corresponding part of the covariate space can be dropped. In this finite partition case (2.1), auto-calibration of $\pi$ for $(Y,\boldsymbol{X})$ is equivalent to

$$\pi_{k}={\mathbb{E}}\left[\left.Y\right|\pi(\boldsymbol{X})=\pi_{k}\right]\qquad\text{ for all $1\leq k\leq K$.}$$

Using the tower property, auto-calibration of $\pi$ for $(Y,\boldsymbol{X})$ implies for all $1\leq k\leq K$

$$S^{(k)}:={\mathbb{E}}\left[\left(Y-\pi(\boldsymbol{X})\right)\mathds{1}_{\{\pi(\boldsymbol{X})=\pi_{k}\}}\right]={\mathbb{E}}\left[\left({\mathbb{E}}\left[\left.Y\right|\pi(\boldsymbol{X})\right]-\pi(\boldsymbol{X})\right)\mathds{1}_{\{\pi(\boldsymbol{X})=\pi_{k}\}}\right]=0,$$ (2.2)

this statement is essentially the same as Wüthrich [7, Proposition 4.1]. For a given i.i.d. sample ${\cal T}=(Y_{i},\boldsymbol{X}_{i})_{i=1}^{n}$, this motivates the statistics

$$S^{(k)}_{n}=\frac{1}{n}\sum_{i=1}^{n}\left(Y_{i}-\pi(\boldsymbol{X}_{i})\right)\mathds{1}_{\{\pi(\boldsymbol{X}_{i})=\pi_{k}\}}\qquad\text{ for $1\leq k\leq K$}.$$

Under auto-calibration of $\pi$ for $(Y,\boldsymbol{X})$, these empirical quantities $S^{(k)}_{n}$, $1\leq k\leq K$, converge to zero, ${\mathbb{P}}$-a.s., as $n\to\infty$, and we have the following central limit theorem.

Proposition 2.1

Under auto-calibration of $\pi$ for $(Y,\boldsymbol{X})$

$$\sqrt{n}\left(S^{(1)}_{n},\ldots,S^{(K)}_{n}\right)^{\top}~{}\Longrightarrow\quad{\cal N}\left(0,\,{\rm diag}\left(p_{1}\tau_{1}^{2},\ldots,p_{K}\tau_{K}^{2}\right)\right)\qquad\text{ as $n\to\infty$,}$$

with conditional variances $\tau_{k}^{2}={\rm Var}\left(\left.Y\right|\pi(\boldsymbol{X})=\pi_{k}\right)$ for $1\leq k\leq K$.

The proof of this proposition is standard and based on characteristic functions.

Test 1. Under the null hypothesis of $\pi$ being auto-calibrated for $(Y,\boldsymbol{X})$, (2.2) is a necessary condition for all $1\leq k\leq K$. We test this against the alternative that there exists a $1\leq k\leq K$ with $S^{(k)}\neq 0$. Under the null hypothesis, Proposition 2.1 gives us for $s>0$ and $n$ large

$${\mathbb{P}}\left[\max_{1\leq k\leq K}\sqrt{n}|S^{(k)}_{n}|\leq s\right]={\mathbb{P}}\left[\bigcap_{1\leq k\leq K}\{|\sqrt{n}S^{(k)}_{n}|\leq s\}\right]~{}\approx~{}\prod_{k=1}^{K}\left(2\Phi\left(\frac{s}{\sqrt{p_{k}}\tau_{k}}\right)-1\right).$$ (2.3)

Often, it is beneficial to test for the maximum of the normalized quantities $\sqrt{n}|S^{(k)}_{n}|/(\sqrt{p_{k}}\tau_{k})$, to have all terms on the same scale. This provides asymptotic limit $(2\Phi(s)-1)^{K}$.

Denuit et al. [2, formula (2.4)] consider an aggregated version of $S^{(k)}$. Namely, auto-calibration of $\pi$ for $(Y,\boldsymbol{X})$ implies for all $1\leq k\leq K$

$$T^{(k)}:={\mathbb{E}}\left[\left(Y-\pi(\boldsymbol{X})\right)\mathds{1}_{\{\pi(\boldsymbol{X})\leq\pi_{k}\}}\right]=0.$$ (2.4)

For a given i.i.d. sample ${\cal T}=(Y_{i},\boldsymbol{X}_{i})_{i=1}^{n}$, this motivates the statistics

$$T^{(k)}_{n}=\frac{1}{n}\sum_{i=1}^{n}\left(Y_{i}-\pi(\boldsymbol{X}_{i})\right)\mathds{1}_{\{\pi(\boldsymbol{X}_{i})\leq\pi_{k}\}}=\sum_{j=1}^{k}S^{(j)}_{n}.$$

The following corollary is an immediate consequence of Proposition 2.1.

Corollary 2.2

Under auto-calibration of $\pi$ for $(Y,\boldsymbol{X})$

$$\sqrt{n}\left(T^{(1)}_{n},\ldots,T^{(K)}_{n}\right)^{\top}~{}\Longrightarrow\quad{\cal N}\left(0,\left(\sum\nolimits_{j=1}^{\min\{k,m\}}p_{j}\tau^{2}_{j}\right)_{1\leq k,m\leq K}\right)\qquad\text{ as $n\to\infty$.}$$

Thus, the aggregated statistics $(T^{(k)}_{n})_{1=k}^{K}$ can asymptotically be described by a random walk

$$Z_{k}=\sum_{j=1}^{k}\sqrt{p_{j}}\,\tau_{j}\,\varepsilon_{j},$$ (2.5)

with i.i.d. standard Gaussian innovations $\varepsilon_{j}\sim{\cal N}(0,1)$ for $1\leq j\leq K$.

Test 2. Under the null hypothesis of $\pi$ being auto-calibrated for $(Y,\boldsymbol{X})$, (2.4) is a necessary condition for all $1\leq k\leq K$. We test this against the alternative that there exists a $1\leq k\leq K$ with $T^{(k)}\neq 0$. Under the null hypothesis, Corollary 2.2 gives us for $s>0$ and $n$ large

$${\mathbb{P}}\left[\max_{1\leq k\leq K}\sqrt{n}|T^{(k)}_{n}|\leq s\right]~{}\approx~{}{\mathbb{P}}\left[\max_{1\leq k\leq K}|Z_{k}|\leq s\right].$$ (2.6)

Up to one point discussed below, asymptotic approximation (2.6) gives an explicit explanation to the intractable limit in Denuit et al. [2, Proposition 3.1]. Namely, the asymptotic distribution of the test statistics in (2.6) corresponds to the maximum of the random walk (2.5) whose increments are fully determined by the probabilities $(p_{k})_{k=1}^{K}$, given in (2.1), and the conditional variances $(\tau_{k}^{2})_{k=1}^{K}$, given in Proposition 2.1. These two parameter sets can be determined from past data ${\cal D}$, being independent of the i.i.d. sample ${\cal T}$, see discussion in Section 1. The rejection area is then received by (easy) random walk simulations involving only these two (estimated) parameter sets $(p_{k})_{k=1}^{K}$ and $(\tau_{k}^{2})_{k=1}^{K}$. This seems simpler than the non-parametric Monte Carlo method used in Denuit et al. [2, Section 3.1].

3 Testing for the area between the curves

The consideration of $T^{(k)}$ is motivated by the difference of the CC and the LC. Denote by $F^{-1}_{\pi(\boldsymbol{X})}$ the left-continuous generalized inverse of the distribution function $F_{\pi(\boldsymbol{X})}$ of $\pi(\boldsymbol{X})$. The difference between the CC and the LC at probability level $\alpha\in[0,1]$ is defined by

$$U(\alpha)={\mathbb{E}}\left[\left(\frac{Y}{{\mathbb{E}}[Y]}-\frac{\pi(\boldsymbol{X})}{{\mathbb{E}}[\pi(\boldsymbol{X})]}\right)\mathds{1}_{\{\pi(\boldsymbol{X})\leq F^{-1}_{\pi(\boldsymbol{X})}(\alpha)\}}\right].$$

For a regression function $\pi$ with discrete finite range (2.1), $U(\cdot)$ only takes $K+1$ different values in the cumulative probabilities $\alpha_{k}:=\sum_{j=1}^{k}p_{j}$, and we set $\alpha_{0}=0$. Namely, we have

$$U^{(k)}:=U(\alpha_{k})={\mathbb{E}}\left[\left(\frac{Y}{{\mathbb{E}}[Y]}-\frac{\pi(\boldsymbol{X})}{{\mathbb{E}}[\pi(\boldsymbol{X})]}\right)\mathds{1}_{\{\pi(\boldsymbol{X})\leq\pi_{k}\}}\right]\qquad\text{ for $1\leq k\leq K$.}$$ (3.1)

Under unbiasedness ${\mathbb{E}}[\pi(\boldsymbol{X})]={\mathbb{E}}[Y]$, we have

$$U^{(k)}=\frac{1}{{\mathbb{E}}[Y]}\,T^{(k)}=\frac{1}{{\mathbb{E}}[\pi(\boldsymbol{X})]}\,T^{(k)}.$$

These normalized differences $U^{(k)}$ motivate the study of $T^{(k)}$ under auto-calibration of $\pi$ for $(Y,\boldsymbol{X})$, which implies the above unbiasedness. Denuit et al. [2, Proposition 3.1] do not exploit an auto-calibration test for $T^{(k)}$, but rather for $U(\alpha)$. Unfortunately, the normalized quantities $U(\alpha)$ and $U^{(k)}$ are more involved. For a given i.i.d. sample ${\cal T}=(Y_{i},\boldsymbol{X}_{i})_{i=1}^{n}$, consider

$$U^{(k)}_{n}=\frac{1}{n}\sum_{i=1}^{n}\left(\frac{Y_{i}}{\overline{y}}-\frac{\pi(\boldsymbol{X}_{i})}{\overline{\pi}}\right)\mathds{1}_{\{\pi(\boldsymbol{X}_{i})\leq\pi_{k}\}},$$

with $\overline{y}$ being the empirical mean of $(Y_{i})_{i=1}^{n}$ and $\overline{\pi}$ the empirical mean of $(\pi(\boldsymbol{X}_{i}))_{i=1}^{n}$. Dealing with $U^{(k)}_{n}$ instead of $T^{(k)}_{n}$ is more cumbersome because of these normalizations. These normalizations are mainly motivated by the fact that they imply that both the CC and the LC are calibrated to 1 for $\alpha\uparrow 1$. In statistical modeling, this then allows one to perform model selection by selecting the model that has the most convex CC, as a higher convexity implies better discrimination; see Wüthrich [7]. Similarly, in economics, a more convex LC indicates higher inequality in wealth distribution. However, for testing of auto-calibration this normalization seems not justified, and we give preference to the simpler unscaled quantity $T^{(k)}_{n}$. Note that

$$\sqrt{n}U^{(k)}_{n} = \sqrt{n}\,\frac{1}{\overline{y}}\,T^{(k)}_{n}+\sqrt{n}\left(\frac{1}{\overline{y}}-\frac{1}{\overline{\pi}}\right)\frac{1}{n}\sum_{i=1}^{n}\pi(\boldsymbol{X}_{i})\mathds{1}_{\{\pi(\boldsymbol{X}_{i})\leq\pi_{k}\}}$$ (3.2)

$$= \sqrt{n}\,\frac{1}{\overline{y}}\,T^{(k)}_{n}+\sqrt{n}\,\frac{1}{\overline{y}\,\overline{\pi}}\left(\overline{\pi}-\overline{y}\right)\frac{1}{n}\sum_{i=1}^{n}\pi(\boldsymbol{X}_{i})\mathds{1}_{\{\pi(\boldsymbol{X}_{i})\leq\pi_{k}\}}.$$

Corollary 2.2 and Slutsky’s theorem give weak convergence of the first term in (3.2) to $Z_{k}/{\mathbb{E}}[Y]$. For the second term in (3.2), one establishes weak convergence of $\sqrt{n}(\overline{\pi}-\overline{y})$, and the other terms are treated by Slutsky’s theorem. Finally, one needs to compute the covariance between the two terms in (3.2) to get the asymptotic variance of $\sqrt{n}U^{(k)}_{n}$. This is doable, but cumbersome. Therefore, we prefer to study the non-normalized quantities $T^{(k)}_{n}$.

Based on $U(\alpha)$, Denuit et al. [3, formula (4.4)] introduced the area between the curves (ABC) as a model selection criterion. The ABC is defined by

$${\rm ABC}=\int_{0}^{1}U(\alpha)\,d\alpha=\int_{0}^{1}{\mathbb{E}}\left[\left(\frac{Y}{{\mathbb{E}}[Y]}-\frac{\pi(\boldsymbol{X})}{{\mathbb{E}}[\pi(\boldsymbol{X})]}\right)\mathds{1}_{\{\pi(\boldsymbol{X})\leq F^{-1}_{\pi(\boldsymbol{X})}(\alpha)\}}\right]d\alpha.$$

Again, we prefer the unscaled version. Under the discrete finite regression function, we have

$${\rm ABC}^{\circ}:=\int_{0}^{1}{\mathbb{E}}\left[\left(Y-\pi(\boldsymbol{X})\right)\mathds{1}_{\{\pi(\boldsymbol{X})\leq F^{-1}_{\pi(\boldsymbol{X})}(\alpha)\}}\right]d\alpha=\sum_{k=1}^{K-1}p_{k+1}\,T^{(k)}.$$

For a given i.i.d. sample ${\cal T}=(Y_{i},\boldsymbol{X}_{i})_{i=1}^{n}$, this motivates the an integrated random walk statistics

$$\widehat{\rm ABC}_{n}^{\circ}=\sum_{k=1}^{K-1}p_{k+1}T_{n}^{(k)}=\sum_{k=1}^{K-1}p_{k+1}\sum_{j=1}^{k}S_{n}^{(j)}=\sum_{k=1}^{K-1}\left(1-\alpha_{k}\right)S_{n}^{(k)}.$$

Under auto-calibration of $\pi$ for $(Y,\boldsymbol{X})$, statistics $\widehat{\rm ABC}_{n}^{\circ}$ converges to zero, ${\mathbb{P}}$-a.s. Slightly modifying the terms, we propose the following weighted $L^{2}$-norm statistics of the increments

$$V^{2}_{n}:=\sum_{k=1}^{K}\left(1-\alpha_{k-1}\right)(S_{n}^{(k)})^{2},$$ (3.3)

thus, the random walk increments $S_{n}^{(k)}$ with different signs cannot compensate each other.

Corollary 3.1

Under auto-calibration of $\pi$ for $(Y,\boldsymbol{X})$

$$n\,V^{2}_{n}\quad\Longrightarrow\quad\sum_{k=1}^{K}\left(1-\alpha_{k-1}\right)p_{k}\,\tau_{k}^{2}\,\chi_{k}^{2}\qquad\text{ as $n\to\infty$},$$

where $\chi_{k}^{2}$ are i.i.d. $\chi^{2}$-distributed random variables with one degree of freedom.

Test 3. Under the above assumptions, we can test for auto-calibration of $\pi$ for $(Y,\boldsymbol{X})$ by exploiting the limiting distribution of Corollary 3.1 numerically. As in Test 2, this limiting distribution only depends on the two parameter sets $(p_{k})_{k=1}^{K}$ and $(\tau_{k}^{2})_{k=1}^{K}$.

Dropping the weighting $1-\alpha_{k-1}$ in (3.3) and scaling the individual terms $(S_{n}^{(k)})^{2}$ by $p_{k}\tau_{k}^{2}$ gives a $\chi^{2}$-test with $K$ degrees of freedom.

4 Conclusions

This letter considers statistical testing of auto-calibration. In the simplified set-up of a discrete finite regression function, we provide three different test statistics that have fully known asymptotic distributions under auto-calibration, see (2.3), (2.6) and Corollary 3.1. These three test statistics consider random walk increments, a random walk and an integrated random walk. The three test statistics can be used for statistical testing of auto-calibration in our simpler set-up; Test 2 is a modified version of Denuit et al. [2, Proposition 3.1].

In this letter, we did not cover a study of the powers of these tests. This will depend on the kind of violation of auto-calibration; in fact, we believe that it is beneficial to normalize all random walk increments to unit variance in any of the three presented tests. Another open problem is to generalize these tests to arbitrary regression functions, this seems feasible for Tests 2 and 3.

Supplementary

Proofs

Proof of Proposition 2.1. Set $\boldsymbol{S}_{n}=(S^{(1)}_{n},\ldots,S^{(K)}_{n})^{\top}$. For $\boldsymbol{r}\in{\mathbb{R}}^{K}$, consider the characteristic function

$${\mathbb{E}}\left[\exp\left\{i\sqrt{n}\boldsymbol{r}^{\top}\boldsymbol{S}_{n}\right\}\right] = {\mathbb{E}}\left[\exp\left\{\frac{i}{\sqrt{n}}\sum_{j=1}^{n}\sum_{k=1}^{K}r_{k}\left(Y_{j}-\pi(\boldsymbol{X}_{j})\right)\mathds{1}_{\{\pi(\boldsymbol{X}_{j})=\pi_{k}\}}\right\}\right]$$

$$= \prod_{j=1}^{n}\sum_{k=1}^{K}{\mathbb{E}}\left[\exp\left\{\frac{i}{\sqrt{n}}r_{k}\left(Y_{j}-\pi(\boldsymbol{X}_{j})\right)\right\}\mathds{1}_{\{\pi(\boldsymbol{X}_{j})=\pi_{k}\}}\right]$$

$$= \exp\left\{n\log\left(\sum_{k=1}^{K}p_{k}\,{\mathbb{E}}\left[\left.\exp\left\{\frac{i}{\sqrt{n}}r_{k}\left(Y-\pi(\boldsymbol{X})\right)\right\}\right|\pi(\boldsymbol{X})=\pi_{k}\right]\right)\right\}$$

$$= \exp\left\{n\log\left(\sum_{k=1}^{K}p_{k}\left(1-\frac{r^{2}_{k}}{2n}{\mathbb{E}}\left[\left.\left(Y-\pi(\boldsymbol{X})\right)^{2}\right|\pi(\boldsymbol{X})=\pi_{k}\right]+o(n^{-1})\right)\right)\right\}$$

$$= \prod_{k=1}^{K}\exp\left\{-r_{k}^{2}\,\frac{p_{k}\tau_{k}^{2}}{2}\right\}\exp\{o(1)\}\qquad\text{ as $n\to\infty$,}$$

where in the second last step we use auto-calibration of $\pi$ for $(Y,\boldsymbol{X})$. This completes the proof. $\Box$

Example

We study a gamma distribution example with $K=6$ expected response levels $(\pi_{k})_{k=1}^{K}$. Table 1 shows the selected parameters. Firstly, we choose the probabilities $(p_{k})_{k=1}^{K}$ such that the boundary levels $\pi_{k}\in\{10,15\}$ receive the smallest probabilities, and the levels in the middle $\pi_{k}\in\{12,13\}$ get the highest probabilities. This is a quite common feature in real data. Secondly, the variance parameters $(\tau^{2}_{k})_{k=1}^{K}$ are increasing in regression means $(\pi_{k})_{k=1}^{K}$. Also this is a rather common feature, e.g., a Poisson or a gamma generalized linear model (GLM) have this property. Based on these parameters, we simulate first the regression level $\pi_{k}$ using the probabilities $(p_{k})_{k=1}^{K}$. Based on this level $\pi_{k}$, we then simulate the response $Y|_{\pi(\boldsymbol{X})=\pi_{k}}\sim\Gamma(\gamma_{k},c_{k})$ with shape parameter $\gamma_{k}=3\pi_{k}$ and scale parameter $c_{k}=3$. This gives us conditional mean $\pi_{k}$ and conditional variance $\tau_{k}^{2}=\pi_{k}/3$, see Table 1. In particular, auto-calibration is fulfilled in this example because we simulate from the correct means.

$k$	1	2	3	4	5	6
$\pi_{k}$	10	11	12	13	14	15
$p_{k}$	$10/100$	$15/100$	$25/100$	$25/100$	$15/100$	$10/100$
$\tau^{2}_{k}$	10/3	11/3	12/3	13/3	14/3	15/3

Table 1: Chosen parameters for the gamma example with $K=6$.

Figure 1: Simulation of an i.i.d. sample $(Y_{i},\pi(\boldsymbol{X}_{i}))_{i=1}^{n}$ of sample size $n=1000$: (lhs) boxplot of the responses $(Y_{i})_{i=1}^{n}$ classified w.r.t. $\pi(\boldsymbol{X}_{i})=\pi_{k}$, (middle) lift plot showing the empirical level means $\overline{y}_{k}$ against their expectations $\pi_{k}$, and (rhs) statistics $S^{(k)}_{n}$ for $1\leq k\leq K$.

Based on the parameters given in Table 1, we simulate an i.i.d. sample $(Y_{i},\pi(\boldsymbol{X}_{i}))_{i=1}^{n}$ of sample size $n=1000$. Figure 1 (lhs) shows the resulting boxplot of the responses $(Y_{i})_{i=1}^{n}$ classified w.r.t. their conditional means $\pi(\boldsymbol{X}_{i})=\pi_{k}$. Remark that there is auto-calibration in this example. Figure 1 (middle) plots the empirical level means

$$\overline{y}_{k}=\frac{1}{\sum_{i=1}^{n}\mathds{1}_{\{\pi(\boldsymbol{X}_{i})=\pi_{k}\}}}\sum_{i=1}^{n}Y_{i}\,\mathds{1}_{\{\pi(\boldsymbol{X}_{i})=\pi_{k}\}},$$

against their (true) conditional expectations $\pi_{k}$; this plot is sometimes also called lift plot. Under auto-calibration, the resulting scatter plot should lie fairly much on the diagonal, and their deviation from the diagonal is described (asymptotically) by Proposition 2.1. Figure 1 (rhs) shows the resulting statistics $S^{(k)}_{n}$, for $1\leq k\leq K$. These are obtained from the lift plot by using a different normalization

$$S^{(k)}_{n}=\frac{1}{n}\sum_{i=1}^{n}\left(Y_{i}-\pi(\boldsymbol{X}_{i})\right)\mathds{1}_{\{\pi(\boldsymbol{X}_{i})=\pi_{k}\}}=\frac{\sum_{i=1}^{k}\mathds{1}_{\{\pi(\boldsymbol{X}_{i})=\pi_{k}\}}}{n}\left(\overline{y}_{k}-\pi_{k}\right),$$

the ratio on the right-hand side is an empirical estimate of $p_{k}$. The magnitude of fluctuations of these statistics $S^{(k)}_{n}$ around zero should be of order $\sqrt{p_{k}}\tau_{k}/\sqrt{n}$, see Proposition 2.1.

We repeat this simulation of an i.i.d. sample $(Y_{i},\pi(\boldsymbol{X}_{i}))_{i=1}^{n}$ $10,000$ times to study the empirical distribution of the statistics $\sqrt{n}\boldsymbol{S}_{n}=\sqrt{n}(S^{(1)}_{n},\ldots,S^{(K)}_{n})^{\top}$. For large sample sizes $n$, this empirical distribution should approximately look like the Gaussian limiting distribution given in Proposition 2.1. Our simulation has an empirical mean $\widehat{{\mathbb{E}}}[\sqrt{n}\boldsymbol{S}_{n}]$ of magnitude $10^{-2}$, thus, close to zero. The empirical covariance matrix reads as

$$\widehat{\rm Cov}(\sqrt{n}\boldsymbol{S}_{n})=\begin{pmatrix}[r]{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}{0.34}}&0.00&0.00&-0.01&0.00&0.00\\ 0.00&{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}{0.55}}&0.01&0.01&0.00&0.01\\ 0.00&0.01&{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}{1.01}}&-0.01&0.00&-0.01\\ -0.01&0.01&-0.01&{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}{1.10}}&-0.01&0.01\\ 0.00&0.00&0.00&-0.01&{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}{0.69}}&0.00\\ 0.00&0.01&-0.01&0.01&0.00&{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}{0.50}}\end{pmatrix}.$$

The off-diagonals are close to zero and the diagonal is close to true parameters

$$\left(p_{1}\tau_{1}^{2},\ldots,p_{5}\tau_{5}^{2}\right)=\left(0.33,\,0.55,\,1.00,\,1.08,\,0.70,\,0.50\right),$$

see asymptotic covariance matrix in Proposition 2.1. This confirms the limiting parameters in the weak convergence result of Proposition 2.1.

Figure 2: (lhs) Empirical densities of $\sqrt{n}S^{(k)}_{n}/(\sqrt{p_{k}}\tau_{k})$, for $1\leq k\leq K$, compared to the standard Gaussian density, and (rhs) empirical densities of the random walk $\sqrt{n}T^{(k)}_{n}$, for $1\leq k\leq K$, compared to the Gaussian random walk densities of $(Z_{k})_{k=1}^{K}$, see (2.5).

Figure 2 (lhs) shows the empirical densities of $\sqrt{n}S^{(k)}_{n}/(\sqrt{p_{k}}\tau_{k})$, for $1\leq k\leq K$ and sample size $n=1000$. They are benchmarked against the standard Gaussian density in black color. We see a quite good alignment of these empirical densities, supporting the statement of Proposition 2.1. This justifies using the asymptotic approximation (2.3) for the auto-calibration Test 1 in this example. Since the components in the maximum in (2.3) may live on different scales, we also use an alternative test statistics that evaluates the normalized quantities

$${\mathbb{P}}\left[\max_{1\leq k\leq K}\sqrt{n}\left|\frac{S^{(k)}_{n}}{\sqrt{p_{k}}\tau_{k}}\right|\leq s\right]={\mathbb{P}}\left[\bigcap_{1\leq k\leq K}\left\{\sqrt{n}\left|\frac{S^{(k)}_{n}}{\sqrt{p_{k}}\tau_{k}}\right|\leq s\right\}\right]~{}\approx~{}\prod_{k=1}^{K}\left(2\Phi(s)-1\right).$$ (S.1)

This then directly relates to the (normalized) graphs in Figure 2 (lhs).

Next, we turn our attention to the second test, involving the random walk consideration (2.5). In this case we get the random walk type empirical covariance matrix

$$\widehat{\rm Cov}(\sqrt{n}\boldsymbol{T}_{n})=\begin{pmatrix}[r]{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}{0.34}}&0.34&0.34&0.33&0.33&0.34\\ 0.34&{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}{0.89}}&0.90&0.89&0.90&0.91\\ 0.34&0.90&{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}{1.91}}&1.90&1.91&1.92\\ 0.33&0.89&1.90&{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}{2.99}}&2.99&3.01\\ 0.33&0.90&1.91&2.99&{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}{3.68}}&3.70\\ 0.34&0.91&1.92&3.01&3.70&{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}{4.21}}\end{pmatrix}.$$

Since we work with a small sample size of $n=1000$, there is still some noise involved which makes to above empirical covariance matrix not a perfect random walk covariance matrix. The random walk covariance matrix of Corollary 2.2 has diagonal entries $(0.33,0.88,1.88,2.97,3.67,4.17)$. Figure 2 (rhs) plots the empirical densities $\sqrt{n}T^{(k)}_{n}$, for $1\leq k\leq K$, and these are benchmarked against the Gaussian random walk densities (2.5) of $Z_{k}$, $1\leq k\leq K$. Again we see a rather good alignment, supporting the asymptotic approximation (2.6) for auto-calibration Test 2. Clearly, the last random walk components $\sqrt{n}T^{(K)}_{n}$ and $Z_{K}$, respectively, have the biggest variance, which implies that they will frequently determine the test statistics, see (2.6). Naturally, one could also revert index $k$ by studying the mirrored quantity, see also Wüthrich (2023) for mirroring,

$$\widetilde{T}^{(k)}={\mathbb{E}}\left[\left(Y-\pi(\boldsymbol{X})\right)\mathds{1}_{\{\pi(\boldsymbol{X})\geq\pi_{k}\}}\right]=0,$$ (S.2)

and its empirical counterpart

$$\widetilde{T}^{(k)}_{n}=\frac{1}{n}\sum_{i=1}^{n}\left(Y_{i}-\pi(\boldsymbol{X}_{i})\right)\mathds{1}_{\{\pi(\boldsymbol{X}_{i})\geq\pi_{k}\}}=\sum_{j=k}^{K}S^{(j)}_{n}.$$

If the terms $p_{k}\tau_{k}^{2}$ are increasing in $1\leq k\leq K$, this latter option may give a test with a better power, because the random walk increments will have a decreasing standard deviation.

Figure 3: Empirical density of $nV^{2}_{n}$ compared to the sum of independent scaled $\chi^{2}$-distributions given in Corollary 3.1.

Finally, Figure 3 illustrates the asymptotic result of Corollary 3.1 for a sample size of $n=1000$. The test statistics $nV_{n}^{2}$ does not consider the maximum of the increments, $\max_{1\leq k\leq K}\sqrt{n}|S_{n}^{(k)}|$, but it considers a weighted $L^{2}$-norm of all increments. In (3.3) we study a weighted $L^{2}$-norm which has been motivated by the ABC. However, in general, it is not clear why this weighting should be justified. Alternatively, we could also consider an unweighted test statistics

$$n\,\widetilde{V}^{2}_{n}=n\,\sum_{k=1}^{K}(S_{n}^{(k)})^{2}\quad\Longrightarrow\quad\sum_{k=1}^{K}p_{k}\,\tau_{k}^{2}\,\chi_{k}^{2}\qquad\text{ as $n\to\infty$},$$ (S.3)

assuming $\pi$ is auto-calibrated for $(Y,\boldsymbol{X})$. Equivalently, we could just consider a $\chi^{2}$-test

$$n\,\sum_{k=1}^{K}\frac{(S_{n}^{(k)})^{2}}{p_{k}\,\tau^{2}_{k}}\quad\Longrightarrow\quad\chi_{K}^{2}\qquad\text{ as $n\to\infty$},$$ (S.4)

where the right-hand side is a $\chi^{2}$-distributed random variable with $K$ degrees of freedom. This is the same scaling as in (S.1), however, we do not consider maximums of increments, but rather aggregated squares of the normalized random walk increments.

Summarizing, we have seen seven different test statistics that we will exploit numerically:

• From Test 1, we can study the maximum of the increments, see (2.3).

• A differently scaled version of Test 1 is given in (S.1).

• From Test 2, we can study the maximum of a random walk, see (2.6).

• An index reverted version of Test 2 is given in (S.2).

• From Test 3, we get a weighted $L^{2}$-norm of the random walk increments, see (3.3).

• An unweighted alternative of Test 3 is given in (S.3).

• Finally, we have $\chi^{2}$-test given by (S.4).

Because we have a discrete regression function $\pi$ taking finitely many values, we receive a natural partition of the covariates space, ${\cal X}=\bigcup_{k=1}{\cal X}_{k}$, and of the range of the regression function, $(\pi_{k})_{k=1}^{K}$. For continuous regression functions $\pi$, one can discretize the range of the regression function $\pi$ and then perform a $\chi^{2}$-test for auto-calibration. In the Bernoulli case this has been proposed by Hosmer–Lemeshow (1980), and the discretization is done with the help of the (empirical) quantiles of $\pi$. Our proposal is a generalization to arbitrary responses, and we present test statistics that are different (and differently aggregated and normalized) from the classical $\chi^{2}$-test in the Bernoulli case.

Next, we aim at comparing the resulting powers of the seven tests in a simulation analysis. We therefore contaminate the above model. We simulate responses

$$Y^{\delta}=Y+\delta,\quad\text{ with }\quad Y|_{\pi(\boldsymbol{X})=\pi_{k}}\sim\Gamma(\gamma_{k},c_{k}).$$ (S.5)

Thus, we introduce a global bias by shifting the means $\pi_{k}\mapsto\pi_{k}+\delta$ by a positive constant $\delta\geq 0$. This is a global shift as it affects equally all levels $\pi_{k}$, $1\leq k\leq K$.

	Test 1a	Test 1b	Test 2a	Test 2b	Test 3a	Test 3b	Test 3c
95% quantiles	2.3456	2.6310	4.2060	4.2263	5.4066	9.1198	12.5916

Table 2: Quantiles of the different test for significance level 5%.

Table 2 gives the quantiles for significance level 5% for the different tests. The quantiles of Tests 1b and 3c are directly available in standard software, the quantile of Test 1a can be found by a root search algorithm, and quantiles of Tests 2a, 2b, 3a and 3b were computed empirically by a (simple) Monte Carlo simulation.

Figure 4: Powers of the seven tests: (top-lhs) global contamination (S.5), (top-rhs) local contamination (S.6) of the lowest level $\pi_{1}$, (bottom-lhs) local contamination (S.6) of level $\pi_{4}$, (bottom-rhs) local contamination (S.6) of the highest level $\pi_{6}$.

We simulate 10,000 times (with different seeds) i.i.d. samples $(Y^{\delta}_{i},\boldsymbol{X}_{i})_{i=1}^{n}$, $n=1000$, and for a grid of contaminations $\delta\in\{0,1/20,2/20,\ldots,1\}$, see (S.5). This gives us for every simulation $1\leq t\leq 10,000$ and for every contamination level $\delta\in\{0,1/20,2/20,\ldots,1\}$ the seven test statistics. In the uncontaminated case $\delta=0$ roughly 5% of the 10,000 simulations should be above the quantiles of Table 2. This then verifies that the asymptotic results for the tests apply, i.e., that $n=1000$ is a sufficiently large sample size for these tests.

For contaminations $\delta>0$ significantly more simulations should be above the quantiles of Table 2, and the more samples there are above the corresponding quantile the bigger the power of the test. Figure 4 (top-lhs) shows the results. We see that all curves start at the significance level of 5% for $\delta=0$. Then, they increase to 1 for increasing contamination $\delta\uparrow 1$. The fastest increase is achieved by Tests 2a-2b (maximum of random walk), followed by Tests 3a-3c (squared sum of random walk increments), and the slowest increase is achieved by Tests 1a-1b (maximum of random walk increments). From this we conclude that the random walk tests (2.6) and (S.2) have the biggest power in case of a global shift, and they should be preferred to find global shifts. Intuitively this is clear, each random walk increment $S_{n}^{(k)}$ is shifted by the contamination $\delta>0$, and in the random walk these shifts are aggregated across all increments. Thus, we have an impact of $K\delta$ on the last random walk component $T_{n}^{(K)}$. This is why Tests 2a-2b are the most sensitive ones to global shifts. In our example the order of aggregation is not very relevant, and Tests 2a-2b have almost equal power.

Global shifts are one potential cause of a violation of auto-calibration, but the violation can also only occur on individual levels $\pi_{k}$, or on different levels with different signs. To test for this local failure of the auto-calibration property, we only contaminate individual levels of the regression function. Fix $\ell\in\{1,\ldots,K\}$, and consider the local contamination

$$Y^{\delta,\ell}=Y+\delta\,\mathds{1}_{\{\pi(\boldsymbol{X})=\pi_{\ell}\}},\quad\text{ with }\quad Y|_{\pi(\boldsymbol{X})=\pi_{k}}\sim\Gamma(\gamma_{k},c_{k}),$$ (S.6)

this only contaminates the responses that have conditional expectation $\pi(\boldsymbol{X})=\pi_{\ell}$.

Based on this local contamination we repeat the above simulation experiment. Since violation of auto-calibration often happens at the boundary of the range of the regression function, we contaminate the model for the smallest and biggest conditional expectations $\pi_{\ell}$, $\ell\in\{1,6\}$. These are also the least frequent levels in our example. Additionally we contaminate level $\pi_{\ell}$, $\ell=4$, being in the main body of the covariate distribution. The results are presented in Figure 4 (top-rhs and bottom). The picture now significantly changes compared to the global contamination. Tests 1b and 3c have the best behavior, both of these tests consider the normalized increments $S_{n}^{(k)}/(\sqrt{p_{k}}\tau_{k})$. From this we conclude that one should bring all random walk increments first to the same scale. This is especially true if the violation of auto-calibration takes place at rare boundary levels, $\pi_{1}$ and $\pi_{6}$ in our case. For contaminated middle levels, $\pi_{4}$ in our case, the Tests 1a-1b and 3a-3c are all almost equally good. On the other hand, one should not use the aggregated random walk versions of Tests 2a-2b, because through aggregation the impact of individual violations of auto-calibration gets diluted. Another observation is that if the violation of auto-calibration happens on the biggest level $\pi_{6}$, it cannot be found by the ABC inspired test (3.3). This comes from the scaling $1-\alpha_{K-1}=p_{K}$ which often is a small number. Therefore, we cannot generally recommend Test 3a.

We summarize our findings of the simulation example as follows:

• Global shifts can most effectively be found by the random walk Tests 2a-2b, but this requires that auto-calibration is violated in the same direction on the entire support of the regression function.

• Local violation of auto-calibration, especially in the tails of the regression function can most effectively be found by Tests 1b and 3c. Both tests consider scaled random walk increments (with unit variance), i.e., it seems beneficial that all random walk increments live on the same scale.

• The ABC inspired Test 3a can generally not be recommended, because the ABC weighting seems to prefer the lower over the upper tail of the regression function, but there is no specific reason that justifies such a weighting, compare magenta dotted lines in Figures 4 (top-rhs) and (bottom-rhs).

References

• Hosmer, D.W., Lemeshow, S. (1980). Goodness of fit tests for the multiple logistic regression model. Communications in Statistics - Theory and Methods 9, 1043-1069.

• Wüthrich, M.V. (2023). Model selection with Gini indices under auto-calibration. European Actuarial Journal 13/1, 469-477.