Finite-Sample Convergence Bounds for Trust Region Policy Optimization in Mean-Field Games

Recent work in MFG has explored the use of proximal methods due to their stability and empirical performance. Notably, Pérolat et al. (2022); Perrin et al. (2022) analyze Online Mirror Descent (OMD) from a model-specific perspective, while Yardim et al. (2023) extend this direction with a model-free approach. However, their analysis does not address population updates, operating in a restricted no-manipulation regime. These works have demonstrated the effectiveness of proximal methods in MFG, which is consistent with our research direction.

We extend this line of research by establishing finite-sample complexity guarantees providing a rigorous theoretical framework that ensures provable efficiency in solving MFG. Moreover, we explicitly incorporates the role of monotonicity in stabilizing population dynamics, a key aspect of the ergodic structure in MFG, and relax overly restrictive assumptions—such as uniformly bounded-away-from-zero policies or absolute continuity with respect to the uniform distribution. With a more flexible framework, we get refined learning guarantees under weaker assumptions, achieving an $\widetilde{O}(1/L)$ convergence rate in the optimization problem with improved sample efficiency, thus broadening the applicability of these methods. For an additional discussion on related work, we refer to Appendix A.

We propose Exact MF-TRPO and Sample-Based MF-TRPO, trust-region-based algorithms for computing approximate MFNE in the MFRL setting. Our method combines the structure of ergodic MFG with the stability of trust-region optimization to enable theoretically grounded learning in multi-agent environments. Our key contributions are:

1. 1.

   Theoretical analysis of Exact MF-TRPO with a bound on the exploitability of the learned policies, quantifying the achieved $\varepsilon_{K}$-MFNE after $K$ iterations.

2. 2.

   A sample-based variant, Sample-Based MF-TRPO, with finite-sample complexity guarantees under the $\nu$-paradigm, requiring at most $\widetilde{O}(1/\varepsilon^{6})$ environment interactions to reach an $\varepsilon$-MFNE.

3. 3.

   Numerical experiments validating the efficiency and effectiveness of our approach in representative MFRL settings.

For a finite set $\mathcal{X}$, let $\mathcal{P}(\mathcal{X})$ denote the set of probability distributions over $\mathcal{X}$. A finite MF-MDP is a tuple $(\mathcal{S},\mathcal{A},\mathsf{P}^{\hskip 0.49005pt},\mathsf{r}^{\hskip 0.49005pt},\gamma)$, where $\mathcal{S}$ is a finite state space, $\mathcal{A}$ is a finite action space, $\mathsf{P}^{\hskip 0.49005pt}\colon\mathcal{S}\times\mathcal{A}\times\mathcal{P}(\mathcal{S})\to\mathcal{P}(\mathcal{S})$ is the transition function, $\mathsf{r}^{\hskip 0.49005pt}\colon\mathcal{S}\times\mathcal{A}\times\mathcal{P}(\mathcal{S})\to\mathbb{R}$ is the reward function, $\gamma\in[0,1)$ is the discount factor. Since we consider probability distributions over a finite state space of size $|\mathcal{S}|$, they can be identified as vectors in $\mathbb{R}^{|\mathcal{S}|}$; thus, we define the inner product $\langle\cdot,\cdot\rangle$ and the Euclidean norm $\left\|\cdot\right\|_{{2}}$ accordingly.

We assume that $\mathsf{r}^{\hskip 0.49005pt}$ is continuous, thus bounded, and denote by $\left\|\mathsf{r}^{\hskip 0.49005pt}\right\|_{{\infty}}$ its upper bound, i.e., $\mathsf{r}^{\hskip 0.49005pt}(s,a,\mu)\in[0,\left\|\mathsf{r}^{\hskip 0.49005pt}\right\|_{{\infty}}]$, for $(s,a,\mu)\in\mathcal{S}\times\mathcal{A}\times\mathcal{P}(\mathcal{S}).$ Given a policy $\pi\colon\mathcal{S}\to\mathcal{P}(\mathcal{A})$ and a population profile $\mu\in\mathcal{P}(\mathcal{S})$, we define the transition operator $\mathsf{P}_{\mu}^{\hskip 0.49005pt\pi}$ as the transition matrix induced by the probability kernel, where actions are sampled as $a\sim\pi(s)$ under the mean-field parameter $\mu$, i.e.,

Denote $\lambda_{\pi,\mu}$ the stationary distribution of the Markov chain $\mathsf{P}_{\mu}^{\hskip 0.49005pt\pi}$. Let $\Pi$ to be set the policies, i.e., the set of functions from $\mathcal{S}$ to $\mathcal{P}(\mathcal{A})$. For $s\in\mathcal{S}$, and $\pi,\pi^{\prime}\in\Pi$, the Kullback–Leibler (KL) divergence between the two distributions $\pi(\cdot|s)$ and $\pi^{\prime}(\cdot|s)$ is defined as $\mathrm{KL}(\pi(\cdot|s)\|\pi^{\prime}(\cdot|s))=\sum_{a\in\mathcal{A}}\pi(a|s)\log(\pi(a|s)/\pi^{\prime}(a|s))$, if $\pi$ is absolutely continuous to $\pi^{\prime}$, $\mathrm{KL}(\pi(\cdot|s)\|\pi^{\prime}(\cdot|s))=\infty$ otherwise. We use the notation $\widetilde{O}(\cdot)$ to hide polylogarithmic factors in the asymptotic complexity.

As in Laurière et al. (2022), the discounted stationary problem within a mean-field interaction setting is designed to approximate the $N$-player game in the ergodic regime, as $\gamma\to 1$. In this setting, a representative agent in the mean-field approximation seeks to maximize the expected discounted sum of rewards while interacting with the population distribution $\mu$ under a policy $\pi$. The objective function is given by

Given an initial state $s_{0}\sim\xi$, actions are sampled at each time step as $a_{t}\sim\pi(\cdot\mid s_{t})$, with state transitions governed by the kernel $\mathsf{P}^{\hskip 0.49005pt}$, i.e., $s_{t+1}\sim\mathsf{P}^{\hskip 0.49005pt}(\cdot\mid s_{t},a_{t},\mu)$. We consider an entropy-regularized variant of the classical MFG problem, where $\eta$ denotes the entropy regularization parameter.

As MFG are an $\widetilde{O}(1/\sqrt{N})$-approximation of the $N$-player game, it is natural to introduce an additional regularization term. If this extra bias remains within the order of the existing approximation error, model fidelity is thus preserved. Regularization enhances solution stability, which is particularly beneficial given the inherent nonlinearity of MFG optimization. This is especially advantageous in RL settings, where small perturbations in the value function or policy updates can otherwise lead to erratic behavior.

The value function is defined as

​With a slight abuse of notation, we use $V^{\operatorname{MFG}}(\mu,s)$ (resp. $V^{\operatorname{MFG}}(\mu)$ and $J^{\operatorname{MFG}}(\pi,\mu)$) to denote $V^{\operatorname{MFG}}(\mu,\delta_{s})$ for $s\in\mathcal{S}$ (resp. $V^{\operatorname{MFG}}(\mu,\mu)$ and $J^{\operatorname{MFG}}(\pi,\mu,\mu)$).

Furthermore, let $Q_{\mu}^{\pi}$ denote the regularized $Q$-function, defined as

We denote $\pi_{\mu}$ the optimal policy of optimization problem $V^{\operatorname{MFG}}\left(\mu\right)$, which is unique in the entropy-regularized RL (see, e.g. Haarnoja et al., 2017; Geist et al., 2019).

Define $\mathsf{d}_{\xi,\mu}^{\hskip 0.49005pt\pi}$ (resp. $\mathsf{d}_{s,\mu}^{\hskip 0.49005pt\pi}$ and $\mathsf{d}_{\mu}^{\hskip 0.49005pt\pi}$) the occupation measure of the process $(s_{t})_{t}$ induced by this policy, under the population distribution $\mu$ and initial distribution $\xi$ (resp. $\delta_{s}$ and $\mu$), i.e.,

​with $s_{0}\sim\xi$, $a_{t}\sim\pi_{t}(\cdot|s_{t})$, and $s_{t+1}\sim\mathsf{P}^{\hskip 0.49005pt}(\cdot|s_{t},a_{t},\mu)$, and $\overline{\mathsf{d}}_{\mu,\xi}^{\hskip 0.49005pt\pi}$ its spatial marginal, i.e.,

In the RL literature, various paradigms have been proposed to structure the interaction between an agent and its environment, influencing how data is collected and utilized for policy updates. In our setting, two fundamental actions can be performed: reset and step. The reset action initializes the environment by sampling a new state from the distribution $\nu$, effectively allowing the agent to restart from a fresh initial condition. The step action, on the other hand, takes a chosen action $a$ and the mean-field distribution profile $\mu$, and progresses the environment based on the current state $s$. Specifically, it samples the next state from the transition kernel $\mathsf{P}^{\hskip 0.49005pt}(\cdot|s,a,\mu)$ and updates the environment accordingly. This interaction model aligns with previous works (Kakade, 2003; Shani et al., 2020) and provides an intermediate assumption in the spectrum of data access models in RL. It is therefore weaker than assuming full access to the true model or a generative model, where arbitrary state-action pairs can be queried, but stronger than the setting where no restarts are allowed, restricting exploration to trajectories induced by the current policy. This ensures reliable state exploration and promotes stable convergence.

Our work aims to compute a MFNE. A Nash equilibrium (NE), a fundamental concept in game theory, represents a stable state in which no player can improve her payoff by unilaterally changing her strategy, i.e., each player reacts optimally to the strategies of the others.

In the MFG framework, as the number of agents approaches infinity, this concept extends to a situation in which each agent optimally adapts its strategy to the behavior of the collective population. This leads to a mean-field fixed-point condition in which individual decisions shape and are shaped by the evolving population distribution. This formulation makes MFG a powerful tool for analyzing large-scale strategic interactions.

In this definition, the first condition ensures that no individual agent can improve their long-term objective by unilaterally deviating from the equilibrium policy $\pi_{\star}$. Given the equilibrium population distribution $\mu_{\star}$, this optimality condition guarantees that the policy $\pi_{\star}$ remains the best possible strategy for each agent, thereby ensuring individual rationality.

The second condition enforces consistency in population dynamics: when all agents follow the equilibrium policy $\pi_{\star}$, the resulting population distribution remains stable over time. This fixed-point property ensures the system’s long-term evolution does not deviate from the equilibrium state.

This notion of equilibrium, introduced by Nash (1950) and extended to the mean-field setting by Lasry & Lions (2006a, b, 2007); Huang et al. (2003, 2005, 2006a), is fundamental in game theory. In these works, the authors generalize the results of Nash (1951), proving the existence of at least one NE using the Brouwer Fixed-Point Theorem. This non-constructive result holds under mild assumptions, such as the continuity of payoff functions and the compactness of strategy spaces, forming a cornerstone for analyzing strategic interactions. However, explicitly computing such equilibria remains challenging, particularly in the multi-player regime (Austrin et al., 2011).

A key metric for evaluating the deviation from a NE is exploitability (Laurière et al., 2022). Formally, the mean-field setting, it is defined as:

This quantity measures the potential improvement an individual agent could achieve by deviating unilaterally from the learned policy $\pi$, given as mean-field parameter the stationary distribution $\lambda_{\pi,\mu}$.

For a fixed mean-field population distribution $\mu$, Exact TRPO$(\mu)$ provides a reliable approximation of the value function. The convergence of the algorithm is explicitly influenced by the regularization parameter $\eta$, which determines the optimal step size $1/(\eta(\ell+2))$.

The proposed algorithm relies on the subroutine PolicyUpdate, a key step in refining the policy at each iteration. This update mechanism is inherently tied to the regularization scheme employed in the mirror ascent formulation of TRPO. Specifically, with entropic regularization, as shown in Beck (2017), the policy update admits a closed-form solution in the form of a softmax function. This structure inherently ensures that the updated policy remains within the probabilistic simplex without requiring additional projections. This fosters smooth and stable learning while preventing overly aggressive updates that could destabilize the optimization process.

This policy update leverages a function $Q\colon\mathcal{S}\times\mathcal{A}\to\mathbb{R}$ and a policy $\pi\in\Pi$ at iteration $\ell$, leading to an explicit update rule given by

​with learning rate $\alpha_{\ell}=1/(\eta(\ell+2))$.

Shani et al. (2020) establishes error bounds for the approximation of value functions that can be directly generalized to our setting. Building on these results, we quantify the gap between the value function of the computed policy and the optimal value function under the given mean-field distribution. This allows to control the policy itself, and derive policy guarantees.

We now present Exact MF-TRPO, which iteratively solves the MFG problem (3) by updating the population distribution $\mu$ using the output of Exact TRPO$(\mu)$. This approach assumes direct access to the transition kernel and cost function, eliminating the need for stochastic approximation in policy updates.

The analysis of Exact MF-TRPO is instrumental to understand the performances of its sample-based counterpart. We do this providing precise theoretical guarantees on convergence rates without the additional complexity of sampling-induced errors. Building on this deterministic setup, we then focus on the convergence behavior of the broader Sample-Based MF-TRPO framework. We use the label “informal” to avoid overloading the main text with technical assumptions (e.g., on learning rates), rigorously stated in Appendix C.

All constants appearing in our results are explicitly defined and detailed in Appendix C, where we also provide complete proofs supporting our theoretical guarantees.

This convergence result proves its effectiveness in tackling the challenges inherent in the non-linear and non-gradient structure of MFNE. Below, we summarize the key insights derived from this result:

• •

  Convergence. We establish an exponential rate of convergence in the first term of the bound to the equilibrium population distribution $\mu_{\star}$, while the second one captures the finite-sample bias in the best-response computation.

• •

  Learning Rates constraints. The theorem provides explicit constraints on the step size $\beta_{k}$ (cf. condition (17)), ensuring stability and preventing oscillations or divergence in the optimization process.

• •

  Explicit Dependence on Model Parameters. All constants in the convergence bound are fully characterized in terms of the structural parameters of the model (cf. Appendix C).

• •

  Controlled Policy Learning Bias. The bias introduced by policy updates in Exact TRPO, bounded by $\log(L)/L$ (Proposition 4.2), remains controlled throughout the iterative process. This ensures algorithmic stability, even with approximations in policy optimization.

With these results, we can now explicitly quantify the parameter $\varepsilon$ corresponding to the proximity of our obtained solution with respect to the MFNE.

First, we adapt Exact TRPO to estimate policy updates in a data-driven manner. This approach leverages sampled trajectories to approximate the policy gradient, providing quantitative bounds on the proximity of the best response. Additionally, this TRPO formulation is particularly well-suited to the considered oracle-based framework, incorporating the $\nu$-restart modeling. This structure ensures robust exploration while maintaining stability in policy updates, aligning naturally with the trust-region optimization paradigm.

The inherent stochasticity in the updates prevents us from establishing sample complexity bounds on the last iterate of the algorithm. However, by leveraging an averaging scheme, implemented through a dedicated subroutine (cf. Remark D.1), we mitigate this variability and provide clear quantitative bounds on the desired gap. Specifically, the policy we focus on is the uniform mixture $\hat{\pi}^{\texttt{Unif},\mu}_{L}$ over the first $L+1$ policies. This averaging scheme is standard in the RL literature and, in the unregularized case, satisfies the identity

​While we do not have a statistically feasible expression for this mixture policy, it is straightforward to sample from $\hat{\pi}^{\texttt{Unif},\mu}_{L}$, using Uniform-Mixture$(\left\{\hat{\pi}_{\ell}\right\}_{\ell=0,\dots,L})$, as discussed in Remark D.1.