Adversarial Network Optimization under Bandit Feedback:
Maximizing Utility in Non-Stationary Multi-Hop Networks

We discuss the most related works here. A more comprehensive literature review is in Appendix A.

Adversarial Network Control. Adversarial networks date back to the 1990s, when Cruz (1991) gave the first adversarial dynamics network model and its scheduling algorithm. More efforts were made to allow more general arrival rates (Borodin et al., 2001, Andrews et al., 2001), link conditions (Andrews and Zhang, 2004, Andrews et al., 2007), or both (Liang and Modiano, 2018b). We also direct the readers to the references therein for more discussions. The main focus of the aforementioned papers were usually system stability, whereas ours is utility maximization. As we are aware of, existing results on utility maximization (Neely, 2010b, Liang and Modiano, 2018a) mostly assumed perfect knowledge on network conditions.

Feedback Models. Most previous works considered perfect knowledge model which assumes pre-decision knowledge on network conditions (Liang and Modiano, 2018b; a). In contrast, our paper considers bandit feedback model which only reveals the consequence of our action. A small number of previous works (Fu and Modiano, 2022, Yang et al., 2023, Huang et al., 2024) also assumed similar feedback models albeit under different names. Another feedback model whose difficulty lies in between is full-information feedback model, which requires network conditions to be revealed after decision and thus counterfactual evaluations of all actions (i.e., not only the deployed one) are allowed in hindsight. See (Neely et al., 2012) for an example.

Adversarial Networks under Bandit Feedback. Prior to our work, Huang et al. (2024), Yang et al. (2023) also studied adversarial networks under bandit feedback. However, they both assumed single-hop networks and focused on network stability. In contrast, our paper allows a general multi-hop topology and tackles the utility maximization task of optimizing an abstract, unknown, and time-varying utility function.

We first introduce our adversarial network optimization with bandit feedback model. Specifically, in a network with multiple servers and directional data links, we denote the set of all servers by ${\mathcal{N}}$ and that of all data links by ${\mathcal{L}}\subseteq{\mathcal{N}}\times{\mathcal{N}}$. Suppose that $\lvert{\mathcal{N}}\rvert$ and $\lvert{\mathcal{L}}\rvert$ are both finite. There are $\lvert{\mathcal{N}}\rvert$ commodities of jobs such that those jobs belonging to commodity $k\in{\mathcal{N}}$ are destined for server $k\in{\mathcal{N}}$. We denote $Q_{n}^{(k)}$ as the queue of unfinished commodity-$k$ jobs at server $n$, where $n\in{\mathcal{N}}$ and $k\in{\mathcal{N}}$. We assume the links do not interfere with each other.

The scheduling problem lasts for $T>0$ rounds. In round $t\in[T]$, the scheduler makes two decisions: i) arrival rates of commodity-$k$ jobs into server $n$, $\forall n\in{\mathcal{N}},k\in{\mathcal{N}}$, and ii) link rate allocations of transmitting how many commodity-$k$ jobs over data link $(n,m)$, $\forall k\in{\mathcal{N}},(n,m)\in{\mathcal{L}}$. Both decisions are made under the bandit feedback model, i.e., the scheduler makes decisions in blind and only receives feedback resulting from its actions. Below, we describe them in detail.

Arrival Rates and Utility. In every round $t$, the scheduler decides an $\lvert{\mathcal{N}}\rvert\times\lvert{\mathcal{N}}\rvert$ dimensional arrival rate matrix $\bm{\lambda}(t)$ from some fixed action set $\Lambda\subseteq\mathbb{R}_{\geq 0}^{\lvert{\mathcal{N}}\rvert\times\lvert{\mathcal{N}}\rvert}$, and consequently, $\lambda_{n}^{(k)}(t)$ jobs with commodity $k$ will be added to queue $Q_{n}^{(k)}$. The arrival rate vector $\bm{\lambda}(t)$ is associated with some abstract utility $g_{t}(\bm{\lambda}(t))$ where $g_{t}\colon\Lambda\to\mathbb{R}$ is concave (that is, the user’s marginal return diminishes gradually as the arrivals increase (Huang and Neely, 2011, Huang et al., 2012)), $L$-Lipschitz, and $[-G,G]$-bounded, where $L$ and $G$ here are known constants. Following the adversarial network assumption, we allow $g_{t}$’s to be time-dependent (though they have to be pre-determined, which is called the oblivious adversary model). Following the bandit feedback model, the scheduler has no information about $g_{t}$ before the decision, and can only observe $g_{t}(\bm{\lambda}(t))$ for the chosen $\bm{\lambda}(t)$ but not the whole $g_{t}$ after decision.

Link Rate Allocations. The capacity of each link $(n,m)\in{\mathcal{L}}$ can be time-varying. We denote the capacity of $(n,m)$ in round $t$ as $C_{n,m}(t)$. We assume the capacities are always bounded by some finite constant $M$. Due to the bandit feedback model, the scheduler cannot access $C_{n,m}(t)$ when deciding. Nevertheless, the scheduler can still decide a link allocation plan which assigns a distribution over commodities on each link, or formally denoted as $\bm{a}_{n,m}(t)\in\triangle({\mathcal{N}})$ (the $|{\mathcal{N}}|$-dimension distribution simplex, representing the portion of rates allocated to each commodity over the link). Via sending jobs from each commodity along link $(n,m)$ according to distribution $\bm{a}_{n,m}(t)$ in a round-robin manner, approximately $a_{n,m}^{(k)}(t)C_{n,m}(t)$ jobs from queue $Q_{n}^{(k)}$ will be sent along link $(n,m)$ to queue $Q_{m}^{(k)}$. Formally, we assume that after deciding link allocation plans $\{\bm{a}_{n,m}(t)\in\triangle({\mathcal{N}})\}_{(n,m)\in{\mathcal{L}}}$, the number of jobs successfully sent from $Q_{n}^{(k)}$ to $Q_{m}^{(k)}$, denoted by $\mu_{n,m}^{(k)}(t)$, are independently generated such that $\operatornamewithlimits{\mathbb{E}}[\mu_{n,m}^{(k)}(t)]=C_{n,m}(t)a_{n,m}^{(k)}(t)$ and $\mu_{n,m}^{(k)}(t)\in[0,M]$. Again, we assume a bandit fededback model, which means the scheduler is able to observe $C_{n,m}(t)$ and $\bm{\mu}_{n,m}(t)$ for all $(n,m)\in{\mathcal{L}}$ only after the decision is made at the end of round $t$.

Putting the two components together, by denoting the length of $Q_{n}^{(k)}$ at the beginning of round $t$ to be $Q_{n}^{(k)}(t)$, the network dynamics can then be characterized as follows:

where $\lambda_{n}^{(k)}(t)$ is the number of jobs with commodity $k$ that the scheduler adds to server $n$, and $\mu_{n,m}^{(k)}(t)$ is the number of jobs with commodity $k$ transmitted along data link $(n,m)$.

The objective of the scheduler is to maximize its average utility over the $T$ rounds, namely $\frac{1}{T}\operatornamewithlimits{\mathbb{E}}[\sum_{t=1}^{T}g_{t}(\bm{\lambda}(t))]$. However, a scheduling algorithm is meaningless if it cannot ensure network stability, which requires the average number of jobs remaining in the network is non-divergent when the number of rounds is large enough. Formally, the network stability requirement says

The scheduler aims to maximize its average utility subject to the network stability condition, i.e.,

In order to improve presentation and facilitate understanding, in Section 3, we first present the network stability algorithm NSO, i.e., pretending $g_{t}$ is a constant. This algorithm will serve as a key building block for the utility maximization algorithm UMO² that we introduce in Section 4.

In Figure 1, we give an overview of our main technical steps when analyzing NSO and UMO². The steps for NSO are in yellow, the ones for UMO² are in blue, and those in common are in green. In general, either analysis starts from the famous Lyapunov drift(-plus-penalty) analysis (Neely, 2010a, §4), which reveals the non-negativity of a Lyapunov drift(-plus-penalty) function – see Section 3.3 and Section 4.3 for more details. We then use online learning techniques to minimize them (i.e., making them as close to zero as possible). From here, the analyses for NSO and UMO² become different.

For the network stability algorithm NSO, we succeeded in expressing the Lyapunov drift function as a function linear in the queue lengths $\bm{Q}(t)$ and the link allocation plan $\bm{a}(t)$. While this belongs to the classical Online Linear Optimization (OLO) problem in online learning (Zinkevich, 2003), we face two unique challenges due to the potentially unbounded queue lengths and the self-bounding analysis for network stability guarantees; see Section 3.1 for more discussions. These two requirements rules out existing OLO algorithms, and thus we have to design our own algorithm crafted towards the network optimization objective. Specifically, we designed an OLO algorithm AdaPFOL (see Algorithm 2) that can handle occasionally large loss magnitudes and ensures a performance guarantee depending on all the losses. When plugging it into NSO, we are able to see that the link allocation plans perform well, as detailed in Theorem 3.5. Therefore, combining it with some reference policy assumption (1) and the Lyapunov drift analysis, we obtain the network stability guarantee of NSO in Theorem 3.6. A more detailed overview of NSO is in Section 3.1.

Regarding the utility maximization algorithm UMO², we have to decompose the Lyapunov drift-plus-penalty function into two parts. The first part is still linear in $\bm{Q}(t)$ and $\bm{a}(t)$, which we can reuse the AdaPFOL algorithm. For the second part, as the utility function $g_{t}$ is an arbitrary time-varying concave function and we only receive bandit feedback (recall Table 1), OLO cannot capture it. Instead, we model this part as a Bandit Convex Optimization (BCO) problem (Flaxman et al., 2005). Unfortunately, again due to the potentially unbounded queue lengths and the self-bounding analysis, the loss functions’ magnitudes and Lipschitzness can be very large for some rounds but we still want to adapt to them – thus, existing BCO algorithms are inapplicable either. To this end, we develop a BCO algorithm AdaBGD (Algorithm 4) which allows loss functions with large magnitudes or Lipschitzness and enjoys a performance adaptive to the loss functions. When plugging in AdaBGD to the UMO² framework, it can generate a good arrival rate sequence as we analyze in Theorem 4.4. Therefore, similar to the analysis of NSO in Theorem 3.6, if we combine the AdaPFOL in Algorithm 2 and the AdaBGD in Algorithm 4 together, we are able to derive the utility maximization guarantee of UMO² in Theorem 4.5. Again, a more detailed overview of UMO² can be found in Section 4.1.

In Algorithm 1, we present Network Stability via Online Linear Optimization (NSO), an algorithmic framework which achieves stability in adversarial multi-hop networks under bandit feedback. One key ingredient of NSO is the plug-in Online Linear Optimization (OLO) algorithm AdaPFOL. Before going into details of the AdaPFOL algorithm, we first introduce why we need it.

The design of NSO is based on the famous Lyapunov drift analysis (Neely, 2010a, §4). Conducting standard Lyapunov analysis on the network dynamics defined in Equation 1, we are able to derive

whose formal statement and proof can be found in Lemma 3.2.

Based on this inequality, a Lyapunov drift based algorithm can be constructed by minimizing the RHS of Equation 4 (Neely, 2010a, §4). As the arrival rate $\lambda$’s is regarded as a constant in this section, we may only focus on the terms related to $\mu$. Thus, minimizing RHS of Equation 4 is equivalent to

where the last step uses the assumption that $\operatornamewithlimits{\mathbb{E}}[\bm{\mu}_{n,m}(t)]=\operatornamewithlimits{\mathbb{E}}[C_{n,m}\bm{a}_{n,m}(t)]$.

For illustration purposes, let us focus on a single data link $(n,m)\in{\mathcal{L}}$. Motivated by Huang et al. (2024), we consider designing a scheduling algorithm via minimizing the following expectation:

Recall that $\bm{a}_{n,m}(t)\in\triangle({\mathcal{N}})$ can be any probability distribution from the simplex. Hence, one may view $\triangle({\mathcal{N}})$ as the action set in round $t$. Moreover, for an action $\bm{a}$ from the action set $\triangle({\mathcal{N}})$, picking it in round $t$ will incur a loss $\langle C_{n,m}(t)(\bm{Q}_{m}(t)-\bm{Q}_{n}(t)),\bm{a}\rangle$ – which is linear in $\bm{a}$. Thus, this problem belongs to the class of Online Linear Optimization (OLO) problems (Zinkevich, 2003, McMahan and Streeter, 2010, Duchi et al., 2011), whose formal definition will be presented later as Definition 3.3. Although our problem belongs to the OLO formulation, we face significantly different challenges due to our network optimization context:

1. i)

   In our task of minimizing $\operatornamewithlimits{\mathbb{E}}[\sum_{t=1}^{T}\langle C_{n,m}(t)(\bm{Q}_{m}(t)-\bm{Q}_{n}(t)),\bm{a}_{n,m}(t)\rangle]$, the magnitude of the loss $C_{n,m}(t)(\bm{Q}_{m}(t)-\bm{Q}_{n}(t))$ can occasionally be large because $\bm{Q}_{n}(t)$ or $\bm{Q}_{m}(t)$ may be unbounded. Note that, despite our system stability condition Equation 2 requires $\frac{1}{T}\operatornamewithlimits{\mathbb{E}}[\sum_{t=1}^{T}\lVert\bm{Q}(t)\rVert_{1}]$ to be small, some $\bm{Q}(t)$’s inside average and expectation are still allowed to be large. However, existing algorithms in OLO mostly require the losses to be uniformly bounded by a constant (see, e.g., (McMahan and Streeter, 2010, Cutkosky, 2020)), which means extra efforts should be made to handle occasionally large losses.

2. ii)

   Moreover, we also want our performance to depend on the geometric mean of all loss magnitudes (which in turn relates to queue lengths since the losses $C_{n,m}(t)(\bm{Q}_{m}(t)-\bm{Q}_{n}(t))$ depend on $\lVert\bm{Q}(t)\rVert$). On a high level, this can be understood as follows: The average queue length $\frac{1}{T}\operatornamewithlimits{\mathbb{E}}[\sum_{t=1}^{T}\lVert\bm{Q}(t)\rVert_{1}]$ can be controlled by the OLO performance via some other arguments (see Section 3.2). Thus, if we can additionally show that OLO performance is bounded by queue lengths, we can conduct a self-bounding analysis on the queue lengths which informally reads

   	$\displaystyle\quad\operatornamewithlimits{\mathbb{E}}\left[\sum_{t=1}^{T}\lVert\bm{Q}(t)\rVert_{1}\right]\lesssim\text{Online Learning Performance}\lesssim\operatorname{\mathcal{O}}_{T}(T)+o_{T}(T^{1/4})\operatornamewithlimits{\mathbb{E}}\left[\sum_{t=1}^{T}\lVert\bm{Q}(t)\rVert_{1}\right]^{3/4}$
   	$\displaystyle\Longrightarrow\frac{1}{T}\operatornamewithlimits{\mathbb{E}}\left[\sum_{t=1}^{T}\lVert\bm{Q}(t)\rVert_{1}\right]=\operatorname{\mathcal{O}}_{T}(1),\textit{ i.e.},\text{ the system is stabilized}.$		(6)

   More details regarding the self-bounding analysis can be found in Section 3.5. Nevertheless, it suffices to remember that a performance depending on all loss magnitudes is beneficial.

In Section 3.4, we introduce our novel OLO algorithm of AdaPFOL (Algorithm 2) that enjoys these two properties. Equipped with such an algorithm, we can minimize Equation 5 and achieve network stability guarantee by deploying it on every link $(n,m)\in{\mathcal{L}}$ for $T$ rounds with action set $\triangle({\mathcal{N}})$. This idea of deploying AdaPFOL onto every link exactly gives our NSO framework in Algorithm 1.

Therefore, we are able to analyze the network stability effect when the NSO framework is equipped with AdaPFOL in Algorithm 2, which we do in the rest of this section: We introduce our reference policy assumptions in Section 3.2, conduct the Lyapunov drift analysis in Section 3.3, introduct and analyze the novel AdaPFOL algorithm in Section 3.4, and present our final analysis in Section 3.5.

We first make the following multi-hop piecewise stability assumption, which, informally speaking, assumes that there exists a reference policy that stabilizes the system piecewisely. It is an extension of the piecewise stability assumption (Huang et al., 2024, Assumption 1) to multi-hop cases.

Intuitively, 1 means that there exists some “good” action sequence $\{\mathring{\bm{a}}(t)\}_{t\in[T]}$ making the network stable, in the sense that there are multiple windows $W_{1},W_{2},\ldots,W_{J}$ such that in expectation, for each window $W_{j}$ and for each queue $Q_{n}^{(k)}$, the average service rate it receives (whose expectation is $\sum_{(n,m)\in{\mathcal{L}}}C_{n,m}(t)\mathring{a}_{n,m}^{(k)}(t)$ in round $t$), is strictly more than its net arrival rate (which includes both external data flows $\lambda_{n}^{(k)}(t)$ and internal data flows that are forwarded from other queues $\sum_{(o,n)\in{\mathcal{L}}}C_{o,n}(t)\mathring{a}_{o,n}^{(k)}(t)$), by a constant gap of at least $\epsilon_{W}$.

Before moving on, we shall remark that the reference action sequence $\{\mathring{\bm{a}}(t)\}_{t\in[T]}$ in 1 is unknown to the scheduler. Instead, the scheduler needs to learn its own way of stabilizing the network via observations. To characterize the ability of $\{\mathring{\bm{a}}(t)\}_{t\in[T]}$ in stabilizing the network, the following lemma controls the average queue length resulting from any scheduling policy.