Hydra: Effective Runtime Network Verification

Sundararajan Renganathan Benny Rubin Hyojoon Kim

Stanford University Cornell University Princeton University
[email protected] [email protected] [email protected]

Pier Luigi Ventre Carmelo Cascone Daniele Moro

Intel Intel Intel
[email protected] [email protected] [email protected]

Charles Chan Nick McKeown Nate Foster

Intel Stanford University & Intel Cornell University
[email protected] [email protected] [email protected]
ABSTRACT 1 INTRODUCTION
It is notoriously difficult to verify that a network is behaving as At first glance, most packet-switched networks appear simple. Each
intended, especially at scale. This paper presents Hydra, a system device implements straightforward tasks like looking up headers
that uses ideas from runtime verification to check that every packet in routing tables, filtering packets using access control lists, and
is correctly processed with respect to a specification in real time. We adding or removing tunneling headers. But operating a network
propose a domain-specific language for writing properties, called In- correctly is a real challenge, especially at scale. Faults, outages,
dus, and we develop a compiler that turns properties thus specified performance degradation, and security breaches occur often in
into executable P4 code that runs alongside the forwarding code practice, for reasons ranging from simple misconfigurations to
at line rate. To evaluate our approach, we used Indus to model a pernicious hardware and software bugs. Misconfigurations and
range of properties, showing that it is expressive enough to capture bugs can appear anywhere—the control plane or the data plane,
examples studied in prior work. We also deployed Hydra checkers fixed-function switches or programmable switches, conventional
for validating paths in source routing and for enforcing slice isola- NICs or smart-NICs, the end host networking stack, and so on.
tion in Aether, an open-source cellular platform. We confirmed a Prior work has proposed methods and tools to check if a network
subtle bug in Aether’s 5G mobile core that would have been hard to correctly forwards traffic according to a formal specification. For
detect using static techniques. We also evaluated the overheads of example, static checkers take snapshots of the network forward-
Hydra on hardware, finding that it does not significantly increase ing state (e.g., device configurations or forwarding rules) to build
latency and often does not require additional pipeline stages. mathematical models of network behavior. These models can be
used to verify cloud contracts [9], to answer “what if” questions
CCS CONCEPTS about router configurations [7, 17], and to verify network-wide
• Networks → Programmable networks; In-network process- properties like connectivity, waypointing, and freedom from loops
ing; Network properties; • Software and its engineering → [4, 27, 28, 30, 38, 51]. Despite enjoying great success, they have
Domain specific languages; well-known limitations regarding scalability [26], the complexity
of collecting data plane snapshots [55], and the restriction to sta-
KEYWORDS ble configurations [7]. Moreover, there is a growing sense (e.g.,
at Google [2], Facebook [39], and Microsoft [49]) that the success
Programmable networks, runtime verification, P4.
of static checking has shifted the goalposts—the most important
ACM Reference Format: failures now often relate to switch hardware and software bugs
Sundararajan Renganathan, Benny Rubin, Hyojoon Kim, Pier Luigi Ventre, rather that simple misconfigurations.
Carmelo Cascone, Daniele Moro, Charles Chan, Nick McKeown, and Nate Fos-
Perhaps the most important limitation of static checkers, how-
ter. 2023. Hydra: Effective Runtime Network Verification. In ACM SIGCOMM
ever, is that they rely on an accurate model of the network. Hence,
2023 Conference (ACM SIGCOMM ’23), September 10–14, 2023, New York, NY,
USA. ACM, New York, NY, USA, 13 pages. https://doi.org/10.1145/3603269. static checkers can make mistakes if the abstract models they rely
3604856 upon do not reflect the “ground truth” experienced by packets trav-
eling through the data plane. For example, a static checker might
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
deduce that an end-to-end path exists (based on its model of the
for profit or commercial advantage and that copies bear this notice and the full citation forwarding state), but due to bugs in the end host networking stack
on the first page. Copyrights for components of this work owned by others than the or some other part of the network that the static checker does not
author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior specific permission model (e.g., the low-level driver code for the switches) the packet
and/or a fee. Request permissions from [email protected]. might actually follow a different path. In this situation, ironically,
ACM SIGCOMM ’23, September 10–14, 2023, New York, NY, USA rudimentary tools like ping or traceroute can successfully detect
© 2023 Copyright held by the owner/author(s). Publication rights licensed to the
Association for Computing Machinery.
a bug that the static checker cannot! This modelling limitation ex-
ACM ISBN 979-8-4007-0236-5/23/09. . . $15.00 ists irrespective of how the static checker builds its model—if any
https://doi.org/10.1145/3603269.3604856
ACM SIGCOMM ’23, September 10–14, 2023, New York, NY, USA Renganathan, et al.

aspect of the network’s behavior is not reflected in the model, then /* Variable declarations */
some bugs may go undetected. control dict < bit <8 > , bit <8 > > tenants ;
In contrast, runtime verification systems can verify the behavior tele bit <8 > tenant ;
header bit <8 > in_port ;
of the network in real time, directly in the data plane. One approach header bit <8 > eg_port ;
is to send special probe packets and check them against a model [10,
40, 43, 44, 54]. However, this technique only works if the probe /* Code blocks */
init { /* Executes at first hop */
packets test all the paths (in the topology and in the code). A second tenant = tenants [ in_port ];
approach is to attach additional information or telemetry data to }
real data packets, which are collected and analyzed offline at a telemetry { /* Executes at every hop */ }
checker { /* Executes at the last hop */
centralized server [24, 29, 46, 47, 57]. This technique is hard to scale if ( tenant != tenants [ eg_port ]) { reject ; }
for large or fast networks, because the centralized server quickly }
becomes the bottleneck.
This paper sets out to answer the following question:
Figure 1: Indus program for bare-metal multi-tenancy.
Can a network check that every packet is correctly pro-
cessed, in real-time, against a specification?

Our Approach. We present Hydra, a system that uses ideas from • We assess the expressiveness of Indus from the theoretical
the field of runtime verification [6] and applies them to networking. and practical perspectives. We show that Indus can express
Rather than analyzing idealized models or performing post-hoc all properties that can be encoded using Linear Temporal
analysis of telemetry, Hydra allows an operator to verify that each Logic over finite traces (LTL 𝑓 ) (see Section 3). We also de-
packet traversing the network is processed according to a formal velop Indus programs for a range of properties studied pre-
specification. Properties are specified in Indus, a domain-specific viously in the network verification literature (see Section 6).
language (DSL) we designed. • We evaluate the overheads of Hydra on Tofino switches [25],
Indus is designed to require little to no understanding of the for- finding that the costs of implementing Indus checkers are
warding specification, and operates at a higher level of abstraction. modest, whether measured in terms of pipeline resources,
In fact, it reads like typical imperative programming languages packet-processing latency, or throughput (see Section 6).
that operators are already familiar with. A key distinguishing fea-
ture of Indus is that it models network-wide, stateful properties 2 HYDRA BY EXAMPLE
using telemetry (comprising packet state, switch-local state, and
In this section, we present a series of examples based on real-world
control-plane state) and checkers, which are predicates over teleme-
scenarios where there is a need for verification. These examples
try that determine whether a packet should be forwarded, rejected,
showcase how Hydra takes runtime verification (RV) ideas and
or reported to the control plane. Indus operates at a higher layer of
applies them to networking, a hitherto underexplored avenue for
abstraction than existing DSLs (e.g., P4, eBPF, and DPDK), enabling
said ideas. Each example first describes the real-world scenario,
operators to focus on higher-level behaviors, without concern for
then gives an intuitive description of the property being verified,
how and where they are implemented, or what devices they are
and then presents a program that expresses the property in Indus.
compiled to.
Bare-metal multi-tenancy. In bare-metal cloud services, tenants
Hydra verifies every packet by collecting telemetry data, adding
have full control over physical servers, including the NIC and host
it to packets as they make their way through the network. Indus
networking stack. To ensure tenant isolation, the Top-of-Rack (ToR)
only requires programmers to specify what telemetry should be
switch is typically programmed with functions such as Virtual Rout-
collected at each hop and what the predicate on that telemetry
ing and Forwarding (VRF) tables and VXLAN encapsulation [5]. In
should be. By checking each packet, on switch, without the need
this setup, all traffic sent and received through a given port con-
for a central server, Hydra is inherently scalable and can enforce
nected to a physical server is expected to belong to the same tenant.
properties in real time.
If any packet crosses between supposedly isolated tenants, the
Contributions. Our contributions are as follows:
cloud provider risks losing business and trust. The Indus program
• We present Hydra, the first practical system for checking in Figure 1 enforces network-wide, per-port traffic isolation.
network-wide properties in real time at line rate (see Sec- There are two important things to note about this program. First,
tion 2). while Indus is a kind of specification language, programs looks
• We design Indus, a DSL that allows an operator to specify more like a program in a scripting language than a formula in logic.
runtime verification policies concisely (see Section 3) We chose to design a new DSL, rather than re-using an existing
• We develop a compiler for Indus that generates switch-specific logical framework (e.g., Linear Temporal Logic), to avoid the well-
checking code that executes independent of the forwarding known challenges that arise when programmers are asked to write
code (see Section 4). specifications. Second, unlike existing networking DSLs like P4,
• We demonstrate that Hydra can find bugs in real-world net- which captures the functionality of a single switch, Indus models
works by building a working prototype and using it to im- the end-to-end behavior of the network. Hence, it can be used to
plement a form of path validation for source routing, and to express network-wide properties—e.g., here, that each packet enters
detect a subtle bug in Aether [19] (see Section 5). and exits at ports associated with the same tenant.
Hydra: Effective Runtime Network Verification ACM SIGCOMM ’23, September 10–14, 2023, New York, NY, USA

sensor bit <32 > left_load = 0; control dict <( < bit <32 > , bit <32 >) , bool > allowed ;
sensor bit <32 > right_load = 0; tele bool violated = false ;
control left_port ;
control right_port ; header bit <32 > ipv4_src ;
control thresh ; header bit <32 > ipv4_dst ;
control dict < bit <8 > , bool > is_uplink ;
tele bit <32 >[15] left_loads ; init { /* Checks if packet is allowed to enter */
tele bit <32 >[15] right_loads ; if (! allowed [( ipv4_src , ipv4_dst ) ]) {
header bit <8 > eg_port ; violated = true ;
}
init { } }
telemetry { telemetry { /* Checks if packet on reverse
if ( is_uplink [ eg_port ]) { direction has been seen */
if ( eg_port == left_port ) { if ( last_hop && ! allowed [( ipv4_dst , ipv4_src ) ]) {
left_load += packet_length ; report (( ipv4_dst , ipv4_src ) ) ;
} }
elsif ( eg_port == right_port ) { }
right_load += packet_length ; checker {
} if ( violated ) { reject ; }
} }
left_loads . push ( left_load ) ;
right_loads . push ( right_load ) ;
}
checker { Figure 3: Indus program for stateful firewall.
for ( left_load , right_load in left_loads ,
right_loads ) {
if ( abs ( left_load - right_load ) > thresh ) {
report ;
}
Figure 2 shows how we can specify load-balancing in Indus in
} an intuitive manner. To keep the example simple, we focus on load
} balancing across just two ports (left_port and right_port]), but
the program generalizes to any number of ports in a straightfor-
ward manner. Note that load balancing is verified on a per-packet
Figure 2: Indus program for data center load balancing. basis, even if the implementation of load balancing is performed at
per-flow granularity. This approach is more scalable than polling
each switch for per-port utilization information and then checking
More formally, an Indus program comprises three blocks. The whether the load is imbalanced. To implement the desired function-
init block executes when a packet enters the network, at the first ality, the Indus program uses sensor variables, which aggregate
hop, before it has undergone any other processing. The telemetry telemetry data across multiple packets using switch-local state, and
block executes at every hop, including the first and last hops.1 a non-trivial telemetry block, which records the total amount of
The checker block executes only at the last hop, before the packet data transmitted on each port in tele variables. The checker block
exits the network (e.g., in the egress pipeline of the last switch). It iterates over the telemetry and flags a report if it detects an im-
executes a predicate on the collected telemetry, which can either balance above a fixed threshold. It is worth noting that the left and
come from the init or telemetry block, to determine whether the right port numbers, as well as the load imbalance threshold are
packet should be halted (“reject”), allowed to proceed, or allowed control variables. Hence, these values can be changed on the fly,
to proceed but with a report generated (“report”). without having to recompile the Indus program.
Indus supports several different kinds of variables, each related As shown in this example, we collect telemetry as the packet
to how they are used: tele variables are carried in the packet, makes its way through the network in the form of a list, and only
while control variables are switch-local state that are managed perform the check at the last hop. This provides a nice abstraction,
by the control plane. In this example, the tenants control variable similar to that of classical runtime verification, where the Indus
is realized as a table that associates switch ports to tenants. The program only needs to specify a read-only trace that is collected
tenant telemetry variable records the tenant associated with the as the packet flows through the network (telemetry block) and a
original ingress port in the packet. At the last hop, the checker predicate on that trace (checker block). Enforcing the check at the
block verifies that the ingress and egress ports were associated with last hop has the nice property of moving programmability from the
the same tenant, and rejects the packet if not. core to the edge of the network, where the functionality could be
Load Balancing. For the next example, consider a tiered data implemented on a smartNIC or even in the kernel. We elaborate on
center network with servers connected to ToR switches. Data center this design decision in Section 4.
operators typically spread traffic across multiple paths (e.g., at the Stateful Firewall. Figure 3 is a program to enforce the property
granularity of flows [1], flowlets [3] or even individual packets) that packet flows can only enter the network if a device inside
to balance the load, which reduces congestion. In our example, the network initiated the communication. To accomplish this, the
we will check that the actual usage of the uplink switch ports is control plane installs rules in the reverse direction when it sees
approximately balanced, within a given threshold. a packet in the forward direction. As described earlier, Indus pro-
grams are coupled to the network topology, which might mandate
1 In this example the telemetry block is empty but it will be used in other examples. that packets enter and leave through a designated choke point.
ACM SIGCOMM ’23, September 10–14, 2023, New York, NY, USA Renganathan, et al.

However, this Indus program is generic enough to check this prop- Linear Temporal Logic over Finite Traces (LTL 𝑓 ) [13]. Formally, it
erty in other topologies. For example, every edge switch could be says that globally (□), it is not the case that some event satisfying
a firewall, instead of all packets going through a choke point. Fol- 𝐴 (i.e., the packet being at switch 𝐴) is followed by ( d) an event
lowing standard techniques for ensuring control plane consistency, where 𝐴 eventually occurs (^). More intuitively, it says that the
the control plane could add firewall rules to all edge switches in packet must not traverse a topological loop involving switch 𝐴.
response to a single report [41]. But while formal logic is very well understood, we ultimately
We use the input packet’s contents to check if it is allowed to elected not to use it as the specification language for Hydra. First, we
reach the destination, and we carry this flag in the packet. At each did not believe that network operators would like or use formal logic.
hop, we check if a packet along the reverse direction has been Second, it was not clear how to cleanly accommodate all of the state
seen (in the telemetry block), and if not, we generate a report related to packet-processing—e.g., packet headers and metadata,
containing the IP addresses so that the control plane can install mutable state on switches, not to mention any new data we might
the corresponding rules in the allowed dictionary. Dictionaries are add to support verification [14]. Instead, we followed the second
implemented using P4 tables, as we discuss in Section 4. main approach used in runtime verification, relying on a domain-
In the init block, which executes when a packet enters the specific instrumentation language (e.g., Eagle [6] or JavaMOP [11])
network at the first hop, if the source and destination IP address to specify correctness properties. Here, the programmer writes a
tuple is not in the allowed dictionary (added by the control plane), program that monitors the execution of the program being verified,
then the packet is marked as violating the firewall rule and will using introspection features such as aspect-oriented programming.
be rejected by the checker. When a packet reaches its last hop, Ultimately, the program implements a predicate that determines
if the source and destination IP addresses are not in the allowed whether the execution should be allowed or not.
dictionary, Hydra sends a report to the control plane to add it. The design of Indus is guided by three fundamental principles.
last_hop is a built-in keyword that evaluates true if and only if a First, it provides direct access to all state in the data plane and the
packet is at its last hop before it egresses the network. control plane that could be relevant to how a packet is processed.
The three Hydra programs presented in this section are examples To put it another way, the language strives to make it easy to
of properties that could not be fully verified by a static checker. It is observe network-wide behaviors. Second, the language enforces
possible to imagine a checker that enforces that the proper tenant a strict separation between the variables that track network state,
isolation rules are installed, or a model checker that ensures that a which are read-only, and other variables, which can be read and
switch complies with the firewall rule. But this doesn’t guarantee written. This separation is to ensure that the Indus program does
correct runtime behavior: For example, a bug in the control-plane not interfere with the network’s forwarding behavior, except at the
might install an incorrect filtering rule; or a bug in the compiler edge, where it rejects packets that violate the specified property.
or data plane might not process a packet in the way the static Third, the language incorporates a number of restrictions to ensure
checker assumed. Similarly, hardware faults (memory errors, bit that programs can be compiled to high-speed packet-processing
flips on signals, failing connectors) would be undetectable by a hardware—e.g., all state must be statically allocated and it must be
static checker. Of course, Hydra programs can have bugs too. But it possible to show that all loops terminate.
is less likely that the same bug would appear in the forwarding and
the checker. This independence between forwarding and checking
is key to the value of runtime verification. 3.2 Syntax and Semantics
Indus syntax is based on familiar imperative programming con-
3 THE INDUS LANGUAGE structs (e.g., variables, conditionals, loops, etc.) and it provides a rich
Having introduced some of the main features of Hydra by example, set of data types (e.g., bitstrings, booleans, arrays, sets, dictionaries,
we now give a more precise definition of Indus, a domain-specific etc.) and operators (e.g., arithmetic, boolean, and bitwise operations)
language we use to specify network-wide properties. To a first to express network-wide correctness properties. Figure 4 defines the
approximation, an Indus program can be thought of as a classical formal syntax of a core fragment of the language. Our prototype im-
runtime monitor that is attached to each packet traversing the plementation supports a few extensions to this core language, such
network. The monitor runs alongside the forwarding code in the as for loops that iterate over multiple variables, report exceptions
data plane at line rate. It can observe the behavior of each switch that carry values, etc. We elide these from our formalization for sim-
on the network-wide path, maintain state in telemetry variables plicity. A program 𝑝 consists of a list of declarations 𝑑 followed by
that are carried along with the packet, and aggregate information an initialization block, telemetry block, and checking block. Each
across multiple packets using sensors. variable is tagged with a modifier: tele variables reside on the
packet and aggregate information along the network-wide path;
3.1 Language Design sensor variables reside on the switch and aggregate information
across multiple packets; header variables provide read-only access
Before delving into the details of Indus, it is worth asking: why
to data plane variables, such as packet headers and metadata; like-
design a new language? Generally speaking, prior work on run-
wise control variables provide read-only access to control-plane
time verification has followed one of two approaches. The first
state and other configuration information.
uses formal logic to specify correctness properties. For example,
The initialization block is executed when the packet first enters
to stipulate that a packet must not visit switch 𝐴 twice, we could
use the following formula, □ ¬(𝐴 ∧ d(^ 𝐴)), which is written in
the network. Its purpose is to perform computations that cannot
Hydra: Effective Runtime Network Verification ACM SIGCOMM ’23, September 10–14, 2023, New York, NY, USA

𝑡 ::= Types
𝑝 ::= 𝑑 𝑠 init 𝑠 tele 𝑠 check Programs
| bit⟨𝑛⟩
𝑑 ::= Declarations | bool
| tele 𝑡 𝑥 := 𝑒 | 𝑡 [𝑛]
| sensor 𝑡 𝑥 := 𝑒 | set⟨𝑡⟩
| header 𝑡 𝑥 | dict⟨𝑡 1, 𝑡 2 ⟩
| control 𝑡 𝑥
⊕ ::= Operators
𝑒 ::= Expressions | +|−|∗|/
| 𝑥 | ∼|&||
| 𝑣 | == | < | <= | ! | && | ||
| ⊕(𝑒) | ∈|∉
| 𝑒 1 [𝑒 2 ] | length | push
𝑠 ::= Statements 𝑣 ::= Values
| pass | 𝑛
| 𝑠1 ; 𝑠2 | 𝑏
| 𝑥 := 𝑒 | [𝑣]
| if (𝑒 1 ) then 𝑠 1 else 𝑠 2
exn ::= Exceptions
| for (𝑥 in 𝑒) 𝑠
| report
| exn
| reject

Figure 4: Indus syntax.

𝜑 ::= 𝐴 | ¬𝜑 | 𝜑 1 ∧ 𝜑 2 | d𝜑 | 𝜑 1 U 𝜑 2
be easily encoded using the initializers for variable declarations—
e.g., computing a function over multiple control-plane variables.
The telemetry block is executed at each hop. Often the telemetry J𝐴K𝑥 ≜ 𝐴(𝑥)
block will push data obtained from header variables into arrays J¬𝜑K𝑥 ≜ ¬J𝜑K𝑥
maintained in tele variables, but other approaches are also possible. J𝜑 1 ∧ 𝜑 2 K𝑥 ≜ J𝜑 1 K𝑥 ∧ J𝜑 2 K𝑥
The telemetry block can also update sensor variables. Finally, the J d𝜑K𝑥 ≜ ∃𝑦. 𝑠𝑢𝑐𝑐 (𝑥, 𝑦) ∧ J𝜑K𝑦
checking block is executed at the last hop. Its main purpose is to J𝜑 1 U 𝜑 2 K𝑥 ≜ ∃𝑦. 𝑥 ≤ 𝑦 ∧ 𝑦 ≤ last ∧ J𝜑 2 K𝑦 ∧
decide whether the packet is allowed to exit the network or if it ∀𝑧. 𝑥 ≤ 𝑧 ∧ 𝑧 < 𝑦 ⇒ J𝜑 1 K𝑧
needs to be rejected and/or reported to the management plane.
By design, Indus is strongly typed, which means all operations Figure 5: LTL 𝑓 syntax (top) and encoding into first-order logic
are checked to ensure that variables are used in ways consistent (bottom) [13].
with their declaration. Types are also important for ensuring termi-
nation — e.g., because arrays have a maximum size that is known
at compile time, for loops are guaranteed to terminate. As men- Hence, in examples, we will often elide the additional logic that
tioned above, the language also enforces a clear separation between would be needed to encode loop freedom in Indus.
data-plane and control-plane variables, which are read only, and
telemetry, sensor, and local variables, which can be read and written. 3.3 Expressiveness
Formally, this ensures that for packets that do not trigger a property Having defined Indus, it is natural to wonder about the class of
violation (i.e., by raising an exception), the final output packet(s) properties that it can capture. Generally speaking, questions about
will be identical to the packet(s) that would have been produced had expressiveness are settled by giving translations that map programs
the Indus program not been running at all. To put it another way, In- from one language into another—e.g., this is how we show that
dus does not interfere with the execution of packets that satisfy the Turing machines and 𝜆-calculus capture the same class of compu-
property, only those that violate it. Similarly, the telemetry, sensor, tations. We are not aware of any logic or existing language that
and local variables, which are used to implement the checking logic, precisely captures the set of properties that can be expressed using
are kept separate from the other variables. Hence, the network can- an Indus program. Among other things, the presence of header and
not subvert the property being enforced simply by injecting certain control variables, which operate as a kind of “foreign function
packets into the network or issuing control-plane commands. Indus interface” to the data plane and the control plane, as well as sensor
can be used to verify that a network is free of infinite forwarding variables, make the relationship difficult to state. Nevertheless, to
loops, but the overhead is non-trivial—one must either enforce a establish a lower bound, we prove here that Indus is rich enough
maximum length on forwarding paths, or keep track of the packet’s to express all LTL 𝑓 formulas. Along the way, we also show that it
path and periodically check for duplicates. Moreover, because loops also corresponds to first-order logic formulas over finite traces.
are almost always undesirable, many networks already offer robust Recall that LTL 𝑓 can be understood as defining predicates on
mechanisms for avoiding them—e.g., IPv4’s time-to-live (TTL) field. traces. Each trace is made up of an ordered sequence of events,
ACM SIGCOMM ’23, September 10–14, 2023, New York, NY, USA Renganathan, et al.

which are assumed to be finite. Figure 5 gives the formal syntax // Hydra Headers
of LTL 𝑓 . Formulas 𝐴 correspond to atomic predicates that either struct hydra_header_t {
hold or do not hold at a given event. For instance, atomic predicates eth_type2_t hydra_eth_type ;
bit <8 > tenant ;
could keep track of the location of the packet in the network, or }
the value of the destination address in the IPv4 header. Formulas struct hydra_metadata_t {
¬𝜑 and 𝜑 1 ∧ 𝜑 2 correspond to logical negation and conjunction bit <8 > tenant ;
respectively. Formulas d𝜑 state that 𝜑 holds in the next event—i.e.,
bool reject0 ;
}
the one that follows the current event in the ordered sequence. // Generated Init Code
Finally, formulas 𝜑 1 U 𝜑 2 state that 𝜑 1 holds at all events until apply {
...
some point at which 𝜑 2 holds. As usual, other formulas can be // look up ingress port tenant
encoded. For example both □𝜑, which states that 𝜑 always holds, tenants_in_port . apply () ;
and ^𝜑, which states that 𝜑 eventually holds, can be encoded using // initialize tele variable
hydra_header . tenant = hydra_metadata . tenant ;
the until operator. }
In their original paper on LTL 𝑓 , De Giacamo and Vardi proved ...
that formulas can be translated to first-order logic [13] over finite // Generated Checker Code
apply {
sequences. The bottom half of Figure 5 gives the translation, which // lookup output port
is parameterized on a variable 𝑥 corresponding to an index in the tenants_eg_port . apply () ;
sequence, initially the index of the first element. Hence, to prove that if ( hydra_header . tenant != hydra_metadata . tenant ) {
// reject if ingress and egress disagree
TPC can express the same set of properties as LTL 𝑓 , we simply have hydra_metadata . reject0 = true ;
to show that it can model the semantics of these first-order formulas. }
Assume that the telemetry block populates an array 𝑇 with an strip_telemetry () ; // strip telemetry at last hop
}
increasing sequence of integers as well as arrays 𝐴 corresponding to
the atomic predicates occurring in the program. With this encoding,
it is straightforward to show that the semantics of every first-order Figure 6: Generated tna code for bare-metal multitenancy.
formula used in the translation of LTL 𝑓 can be expressed in Indus.
For example, existential formulas ∃𝑥 . 𝑃 map to a for loop:
bool 𝑟 0 := 𝑓 𝑎𝑙𝑠𝑒;
for (𝑖 in 𝑇 ){ control-flow of the Indus program are not tampered with during
𝑥 := 𝑖; this process. Our compiler is written in approximately 2500 lines
(|𝑃 |)𝑟 0 ; of OCaml code. Our current prototype only supports P4, but we
𝑟 := 𝑟 || 𝑟 0 ; envision possible extensions that target other DSLs like eBPF and
} DPDK in the future.
Here (|𝑃 |)𝑟 0 denotes the translation of 𝑃 using an auxiliary variable
𝑟 0 to store the result. For the complete formalization, please see the 4.1 Code Generation
long version of this paper, which is available online [42].
The compiler takes as inputs an Indus program and a topology file
Theorem 3.1 (Expressiveness). Let 𝜑 be an LTL 𝑓 formula, 𝜋 a in which each switch is classified as an edge or non-edge switch.
trace, I a corresponding first-order interpretation, and 𝜎 the corre- The compiler then generates switch-specific code for each switch
sponding Indus store. Also let 𝑃 = J𝜑K𝑥 [𝑥 ↦→ 1] and 𝑠 = (|𝑃 |)𝑟 . The in the topology.
following are equivalent. The front-end of the compiler first lexes and parses the Indus
• 𝜋, 𝑖 |= 𝜑 program into an abstract syntax tree. Next, the type checker ensures
• I |= 𝑃 programs are well typed and respect constraints such as read-only
• ⟨𝜎, 𝑠⟩ ⇓ ⟨𝜎 ′, 𝑠 ′ ⟩ and 𝜎 ′ (𝑟 ) = true. access to control and header variables. The type checker also
constructs a symbol table for the declarations in the Indus program,
Proof. The first two cases were given by De Giacamo and which is used in the construction of the P4 headers and parsers.
Vardi [13]; the third case follows by induction. □ Finally, the compiler generates P4 code for each Indus construct.
Corollary 3.2. Every network-wide property that can be ex- Many of the abstractions found in Indus can be directly mapped
pressed in LTL 𝑓 can be expressed in Indus. onto analogous constructs in P4—e.g., assignments, conditionals, etc.
But for some other abstractions, it is not obvious how to implement
Overall, this result shows that Indus is at least as expressive as them in P4. The following list summarizes the strategies used to
the specification language used in many other runtime verification generate code in the compiler:
systems, modulo the choice of atomic predicates 𝐴.
• header variables: A header variable declaration requires an
4 THE INDUS COMPILER annotation (indicated with an @) that specifies the corre-
This section presents our compiler, which converts Indus programs sponding name in the forwarding program that tells the
to P4 code, which can then be linked with the forwarding code. Our compiler how to translate references to the variable. For
compiler is designed to make it easy to ensure that the state and example, if an Indus program needs to refer to the source
Hydra: Effective Runtime Network Verification ACM SIGCOMM ’23, September 10–14, 2023, New York, NY, USA

IP address through the ip_src variable, the required anno- control bool is_spine_switch ;
tation is hdr.ipv4.src_addr. In examples, we omit these tele bool visited_spine ;
annotations for brevity. tele bool to_reject ;
• tele variables: A telemetry variable declaration leads to init {
an extra field in a special telemetry header generated by visited_spine = false ;
the compiler. The tele variables travel with the packet as to_reject = false ;
}
telemetry and are serialized and deserialized using parsers telemetry {
and deparsers generated by the compiler. if ( is_spine_switch ) {
• sensor variables: A sensor variable declaration is imple- if ( visited_spine ) {
to_reject = true ;
mented as a P4 register. Reads and writes to sensor variables }
are translated, provided the underlying target (e.g., BMv2, visited_spine = true ;
Tofino) supports them. }
}
• control variables: A control variable declaration is mapped checker {
to a match-action table. There are two different types of if ( to_reject ) {
control variables: a non-dictionary control variable and a reject ;
}
dictionary control variable. A non-dictionary control vari- }
able is statically defined by the control plane, and can be
initialized by a default action in a single match-action ta-
ble that executes at the start of the pipeline. On the other Figure 7: Valley-free routing in Indus.
hand, a dictionary control variable requires more complex
lookups. To ensure the lookup operation returns the most
up-to-date value for each dictionary control variable, our all three code blocks, while the non-edge switches only run the
compiler creates and places a match-action table right before telemetry block. Automatically linking our compiler output blocks
the statement that contains the lookup in the translated P4 with the forwarding P4 program is future work.
code.
• Lists and loop operations: Lists are implemented as header 4.3 Last-Hop vs. Per-Hop Checking
stacks in P4, which have the semantics of a fixed length array. Our current compiler compiles Indus programs to the network
P4 does not support loops. Thus, our compiler unrolls Indus’s so that a switch at every hop collects telemetry but the check
for loops into sequential code: the loop body is executed for only runs at the last hop, or edge, switch. This approach has a
each list index that is valid. Our compiler also supports the number of advantages. First, it saves resource usage on non-edge
in operator, which translates into an expression that tests switches since running a check at a switch requires additional
if the left-hand side is equal to any valid elements of the computation. This approach is also more amenable to incremental
header stack specified on the right-hand side. deployment since Hydra can still run with switches that are not
At the final hop, before a packet exists the network, we strip fully programmable but can run telemetry and attach information
the checking headers produced by the Indus program. This ensures to packets. Another approach, however, is to execute checks at
conformance with software running on end hosts that do not rec- every hop. The main advantages of this approach are that it often
ognize the extra headers injected by Indus. To this end, the control requires less telemetry data, and packets that violate the given
plane needs to specify the set of edge ports in the network to the property can rejected (or reported) at any switch, not just at the
compiler. Then the compiler generates an extra match-action table edge. We plan to implement this approach in the future, using our
that matches on the egress port and strips the headers for packets compiler to automatically relocate checks from the edge and into
that are sent to these egress ports. A similar process is done for the network core.
injecting Indus-generated headers to packets at the first hop. In
principle, we could delegate these “last-hop” and “first-hop” tasks 5 HYDRA CASE STUDIES
to the NIC at end hosts. We leave this extension to future work. This section presents a pair of case studies that demonstrate the
practical utility of using Hydra for enforcing network-wide prop-
4.2 Linking erties using runtime verification. The first case study develops
Figure 6 shows the generated P4 code for the bare-metal multi- an application of Hydra to implement path validation in a data
tenancy example described in Section 2. The final compilation step is center network with source routing, ensuring that packets follow
to link the generated headers and parsers blocks as well as the init, “valley-free” paths. The second case study illustrates a use of Hydra
telemetry, and checker code blocks with the forwarding code for to detect a subtle bug in Aether’s implementation of application
the switch, which we assume is also written in P4. Specifically, the filtering, which provides a form of slicing.
init block must be placed at the beginning of the ingress pipeline
on first-hop switches, the telemetry block is placed at the egress 5.1 Example 1: Valley-Free Source Routing
pipeline on every switch, and the checker block is placed at the Recall that in source routing, the sender specifies the path the
end of the egress pipeline on last-hop switches. Since networks packet should take through the network. In its purest form, the
are bidirectional, the edge switches in the network end up running path is specified as a list of hops, and each switch simply pops
ACM SIGCOMM ’23, September 10–14, 2023, New York, NY, USA Renganathan, et al.

tele bit <32 > ue_ipv4_addr ;

tele bit <32 > app_ipv4_addr ;
tele bit <8 > app_ip_proto ;
tele bit <16 > app_l4_port ;
tele bit <8 > filtering_action = 0; // 1= deny ,2= allow
control dict <( bit <32 > , bit <8 > , bit <32 > , bit <16) ,bit
<8 > > filtering_actions ;

header bit <32 > inner_ipv4_src ;

/* ... Header variable declarations ... */
header bit <16 > outer_udp_dport ;

init {
Figure 8: Simple Leaf-Spine Topology if ( inner_ipv4_is_valid ) {
// this is an uplink packet
ue_ipv4_addr = inner_ipv4_src ;
app_ip_proto = inner_ipv4_proto ;
the stack and forwards the packet accordingly. Source routing has app_ipv4_addr = inner_ipv4_dst ;
many advantages—e.g., it eliminates the need for large routing ta- if ( inner_tcp_is_valid ) {
bles and complex routing protocols, since senders are responsible app_l4_port = inner_tcp_dport ;
} elsif ( inner_udp_is_valid ) {
for computing paths. One downside, however, is that source routing app_l4_port = inner_udp_dport ;
does not offer operators the same degree of control as traditional, }
destination-based forwarding schemes. With Hydra, operators can } elsif ( ipv4_is_valid ) {
// this is a downlink packet
specify and enforce policies that restrict the set of legal paths when ue_ipv4_addr = outer_ipv4_dst ;
using source routing; any packet that attempts to follow an ille- app_ip_proto = outer_ipv4_proto ;
app_ipv4_addr = outer_ipv4_src ;
gal path will be automatically dropped. For example, an important
if ( tcp_is_valid ) {
property in data center routing is that paths are valley-free, pre- app_l4_port = outer_tcp_sport ;
venting an explosion of suboptimal paths in a fat-tree topology. In } elsif ( udp_is_valid ) {
app_l4_port = outer_udp_sport ;
particular, packets may not traverse an link that goes “up” in the }
topology after they have already traversed a “down” link. }
Indus checker for source routing. Figure 8 depicts the topology filtering_action = filtering_actions [(
ue_ipv4_addr , app_ip_proto , app_ipv4_addr ,
of the simple network we instrumented with Hydra, generalizing app_l4_port ) ];
code found in the P4 Tutorial [12]. The network contains a leaf- }
spine topology with four switches. All the switches run the same P4 telemetry {}
checker {
program, which implements a simple source routing scheme, and we if ( filtering_action == 1 && ! to_be_dropped ) {
link the program with the valley-free routing checker written with reject ; report (( ue_ipv4_addr , app_ip_proto ,
Indus, shown in Figure 7. While it is possible to write a general Indus app_ipv4_addr , app_l4_port ,
filtering_action ) ) ;
program to check valley-free routing for any given fat-tree topology, }
we leverage the fact that Indus is topology-specific to write an if ( filtering_action == 2 && to_be_dropped ) {
efficient program that only requires a single control variable and report (( ue_ipv4_addr , app_ip_proto ,
app_ipv4_addr , app_l4_port ,
two bits of telemetry to ensure that a spine switch is visited at filtering_action ) ) ;
most once. This program consists of a simple state machine that }
checks if the current switch is a spine switch and marks the packet }
to be dropped if it has already visited a spine switch. Note that
the Indus program is independent of the forwarding P4 code: it
could operate on any routing protocol. And while the forwarding Figure 9: Aether application filtering in Indus.
program operates on egress ports, the Indus program operates at a
higher level, using switch-specific control plane state.
Bug caught by Hydra. In this case study, we artificially injected mobile clients; and (3) an SDN fabric of P4-programmable switches
a bug into the script used by the sender to add extra invalid hops that connects small cells to servers and the Internet [20].
to the source route. Using Mininet [34], we generated a number of The Aether software stack includes an operator-facing portal
paths and verified that Hydra allowed all possible valley free paths and API for system configuration, a 3GPP-compliant dual-mode
between hosts and successfully dropped any packets that followed 4G/5G mobile core, and ONOS, a distributed SDN controller re-
errant paths due to the bug in the sender script. sponsible for controlling the fabric switches. The fabric provides L3
connectivity by routing IPv4 packets over the spine switches using
5.2 Example 2: Application Filtering in Aether Equal Cost Multi-Path (ECMP) forwarding. It supports L2 bridging
Aether [19] is an open-source edge computing platform that offers and VLAN isolation within a rack, and other common features such
private LTE/5G connectivity. Figure 10 shows an Aether edge de- as rerouting in case of failures, learning/advertising routes via BGP,
ployment with three main elements: (1) small cells that provide LTE configuring static-routes, DHCP relay, multicast, and ACLs for filter-
or 5G access to mobile clients such as cameras, sensors, or phones; ing. A notable feature in Aether is that the switches help implement
(2) servers that run edge-applications exposing low-RTT services to the mobile core User Plane Function (UPF) [37] (i.e., with support
Hydra: Effective Runtime Network Verification ACM SIGCOMM ’23, September 10–14, 2023, New York, NY, USA

Figure 11: P4 tables demonstrating application filtering bug

different types of tables, and ONOS is responsible for translating

UPF rules into multiple table entries and updating the entries in
Figure 10: Aether architecture and topology. each leaf switch. Hence, while the slice and application filtering
configuration is conceptually simple, ensuring the correctness of
the filtering depends on the interaction of multiple software and
for GTP-tunnel encapsulation/decapsulation, downlink buffering, hardware components, each of which could be subject to different
accounting, QoS, application-filtering, and slicing). bugs. Bugs and errors could result in the installation of erroneous
Aether application filtering. We implemented a wide range of entries, which may cause traffic to violate the intended policy.
Hydra checkers for Aether (see Section 6, Table 1), but we focus Hydra checker for application filtering. Figure 9 shows the
here on UPF application filtering, which had a subtle bug we de- Indus program to verify application filtering. The init block first
tected using Hydra. Application filtering allows operators to create determines the direction of the packet and then fetches the fields of
slices that connect an isolated group of clients and give them with interest into tele variables, which it then uses to look up a control
bandwidth guarantees. Operators can define filtering rules allowing variable to know the filtering action. The filtering action is carried
clients in a slice to access some edge-applications while denying on the packet (in addition to the packet fields used in the lookup).
access to others. Internet access is considered an application, and A simple control plane application that runs atop ONOS as part
applications can be shared across slices. For example, mobile clients of the rest of the deployment configures the control dictionary
belonging to the camera-slice are allowed to communicate with an variable. At a high level, it receives the application filtering rules
edge application that analyzes video, but cannot access the Internet. from the operator at startup, listens for attach requests from mobile
Mobile clients in the phone-slice have the opposite permissions. clients, and installs the corresponding entries in the table for the
Each slice has a prioritized list of filtering rules of the form: filtering_actions variable.
priority : ip-prefix : ip-proto : l4-port : action Bug caught by Hydra. We now describe a known bug in Aether
that causes traffic to be dropped when updating application filter-
where ip-prefix, ip-port, and l4-port identify the application. ing. Figure 11 provides a simplified representation of the multiple
The action can be allow or deny, and the priority is used to dis- P4 tables used to realize the UPF function. To reduce memory uti-
ambiguate in case of overlapping rules. For example, to deny all lization (in particular, of TCAM), the Applications table is designed
traffic by default but allow access to applications using UDP port so that entries can be shared by multiple clients of the same slice.
81, the operator could use the following rules: This table determines the application for a packet by matching
• 20 : 0.0.0.0/0 : UDP : 81 : allow on the IPv4 and L4 port headers, and sets the appropriate app ID
• 10 : 0.0.0.0/0 : any : any : deny metadata for the packet. The Terminations table then uses the app
Now, to integrate with any 3GPP-compliant mobile core, Aether’s and client IDs together to determine whether to forward or drop
ONOS controller uses a standard 3GPP interface named PFCP. This the packet. This design requires ONOS to correctly manage shared
interface does not allow to specify application filtering rules globally application entries when clients connect to the network or when
for a slice. Instead, rules are sent to ONOS on a per-client basis. rules are updated in the operator portal.
This means that ONOS receives the same application filtering rules Figure 11 shows a scenario where a specific slice is first config-
for every client that connects to the network. Thus, in each slice, ured with filtering rules that deny all traffic by default (app ID 1)
there are a set of clients (identified by their IMSI, a unique number but allow traffic for apps on UDP port 81 (app ID 2), which has a
associated with a SIM card) and a list of application filtering rules higher priority. When client ID 1 connects, two rules are installed:
for each client. When a new client connects to Aether, the mobile the default drop rule for client ID 1 with app ID 1, and the allow
core looks up the slice configuration for the given IMSI and installs rule for client ID 1 with app ID 2. Thus, client ID 1 can successfully
the user plane rules on switches to terminate the GTP tunnels access applications on UDP port 81. Let’s say the operator later
and enforce application filtering. The P4 program running on the updates the filtering rules in the portal by expanding the UDP port
switch optimizes ASIC resources by splitting UPF processing across range to 81-82 and increasing the priority of that rule, and set the
ACM SIGCOMM ’23, September 10–14, 2023, New York, NY, USA Renganathan, et al.

streamline their compilation to hardware. For example, in the In-

dus checker for detecting load imbalance, we maintain a boolean
variable that records whether an imbalance has been detected on
any switch on the network-wide path, which eliminates the need
to iterate over multiple arrays in the block. Overall, our evaluation
shows that Indus, our domain-specific language, can express a wide
range of practical network properties in a concise manner.

6.2 Resource and Performance Overheads

Next, we discuss the overheads associated with deploying Hydra
(a) RTT comparison over time checkers on Intel Tofino switches.
Resource Overhead. The main resources on Tofino switches that
are relevant to Hydra are the number of pipeline stages used and
the amount of Packet Header Vector (PHV) bits used. Other stage
resources (e.g., SRAMs, TCAMs, etc.) are also important, but their
contribution is implicitly accounted for in the usage of pipeline
stages. We first measure the resource utilization of the baseline
forwarding program that runs in the Aether mobile core and then
measure the resource utilization for each of the implemented prop-
erties when linked with this program.
The baseline program is already at 12 stages. In general, de-
ploying a Hydra checker will require extra resources. However,
(b) RTT CDF comparison in this instance, each of the checkers can be executed in parallel
alongside the base program and they do not increase the number of
Figure 12: Performance overhead of Hydra stages when linked with the base program. This parallel execution
is made possible by the independence between the forwarding and
checking code. We can see that the overhead on PHV resources is
app ID as 3. When client ID 2 connects, the mobile core installs relatively modest, with higher usage for the programs that collect
client-specific rules with this updated policy, thus ONOS installs more telemetry. For instance, the properties that require the most
a new table entry with range 81-82 in the Applications table. Due PHV are source routing path validation and application filtering.
to the new higher-priority entry for app ID 3, packets from client The former carries a significant chunk of telemetry per hop while
ID 1 with UDP port 81 will now get an app ID 3 assigned by the the latter collects all its telemetry in the init codeblock. The PHV
Applications table. As a result, traffic for client ID 1 on port 81 that resource usage increases from 44.53% to 52.14% with the application
was previously allowed is now dropped since the client-app ID pair filtering checker on, a 7.6% difference.
does not exist in the Terminations table. This subtle bug is hard Performance Overhead: Setup. Next, we evaluate if Hydra intro-
to catch and even harder to pinpoint the exact location where the duces any performance overhead when deployed in practice with
packets are being dropped. our Aether testbed. We confirmed that mobile devices connected
With the checker compiled from Figure 9, Hydra detects that to the cellular network had stable Internet connectivity even when
client ID 1’s packets with UDP port 81 are actually to be dropped Hydra checkers were running. However, although Aether processes
when it should have been allowed. With the report action, such real-world traffic, the data rates in our testbed are currently not
behavior is explicitly reported to the control plane by the switch high enough to fully evaluate Hydra’s performance limits. Thus,
where the inconsistency was detected. for this evaluation, we tapped and mirrored network traffic from a
production campus network and replayed it towards leaf1 in Fig-
6 EVALUATION ure 10. As illustrated in Figure 13, we utilize an existing P4Campus
To further evaluate our design, we wrote more checkers for verify- infrastructure [31] at Princeton University. The campus network
ing a range of properties in the Aether testbed, including examples has network Test Access Point (TAP) devices installed at several
studied previously in the network verification literature. We assess vantage points in the network, which create a mirror of the traffic
the expressiveness of Indus and and overheads of our Hydra system. they see on links. We tap two /16 campus network subnets at our
Table 1 summarizes our results. border routers and send the mirrored traffic to a P4-based packet
anonymizer [32], which runs on a programmable switch. This P4
6.1 Expressiveness and Conciseness program hashes personally identifiable information like MAC and
In Table 1, we show the number of lines of Indus code required to IP addresses in a prefix-preserving manner at line rate and deliv-
specify each property and the number of lines of P4 code generated ers the anonymized traffic to the cellular network. The resulting
by our compiler. Indus enables expressing properties succinctly, anonymized packet trace’s load is around 350K packets per second.
typically requiring an order of magnitude less code compared to Ethical considerations: All packet traces were inspected and sani-
the direct implementation in P4. We optimized the programs to tized by a campus network operator. Personal data, like MAC and
Hydra: Effective Runtime Network Verification ACM SIGCOMM ’23, September 10–14, 2023, New York, NY, USA

Property Name Description LoC Tofino Overhead

Indus P4 Output Stages PHV (%)
Baseline Aether P4 program compiled in fabric-upf profile - - 12 44.53
Multi-Tenancy All traffic through a given ToR switch port, facing a 14 102 11 48.44
bare-metal server should belong to the same tenant
Datacenter uplink load Uplink ports in data center switches should load balance, to 37 194 12 48.83
balance exact equivalence, between specified ports
Stateful firewall Flows can only enter the network if a device inside initiated 23 164 12 49.21
the communication
Application filtering Clients should only be able to communicate with designated 64 126 12 52.14
applications (as identified by layer 4 ports)
VLAN isolation Packets should traverse switches in the same VLAN 21 119 11 47.85
Egress port validity Packets should only egress a switch at allowed ports 18 132 12 46.09
Routing validity The first and last hop of any packet should be a leaf switch, 21 122 12 46.09
while the rest of the hops are spine switches
Loops (4 hops) Packets should not visit the same switch twice 20 156 12 48.24
Waypointing All packets should pass through a choke point 22 154 12 47.85
Service chains Packets from switch s to switch t should pass through 26 121 12 47.26
switches (𝑤1 , 𝑤2 , ..., 𝑤𝑛 ) in that order on the way
Source routing with path A packet that is source routed through switches (𝑠, 𝑠 1 , ..., 𝑡 ) 34 211 12 51.56
validation should pass them in order
Table 1: Hydra properties.

Mirrored
ISP 1 ISP 2 traffic
Packet
appears to be no significant difference between the baseline and
broker with all checkers enabled. To further evaluate the latency overhead
system
Network Anonymized
statistically, we plotted the cumulative distribution function of our
TAPs traffic RTT measurements in Figure 12b, and also performed a t-test [23].
Line-rate Both results confirm that there is no statistical latency difference
packet anonymizer Cellular network
… with 2x2 fabric
between the baseline and with all checkers on.
Departments Data centers

Campus core
network
7 RELATED WORK
Our work adds to the growing literature on network verification.
Figure 13: TAP architecture for mirroring campus network Here, we summarize the most relevant pieces of prior work, grouped
traffic to Aether. in several topical areas.
Runtime Verification. In runtime verification (RV), a system is
instrumented to send events about its execution to a monitor. While
IP addresses, were removed or hashed before being accessed by the system executes, the monitor verifies the behavior against a
researchers. Addresses were anonymized in a consistent manner specification. When a behavior violation occurs, the monitor sends
using a one-way hash with a salt, and payloads are discarded. Our feedback to correct the behavior or halt the execution. Over time,
research was discussed and approved by Princeton’s Institutional researchers have created more expressive languages to specify prop-
Review Board (IRB). erties in runtime verification systems. Work such as Eagle [6] helped
Performance Overhead: Result. Next, we evaluate if Hydra adds popularize the use of Linear Temporal Logic to specify properties
noticeable performance overhead due to its parsing and checking in the monitor. Eagle’s powerful logic can express a diverse set of
logic. We perform a microbenchmark with and without Hydra runtime behaviors, such as requiring each request in an application
enabled and compare the two. Our throughput comparison with to have a corresponding response or limiting the size of a queue.
and without Hydra were almost identical with around 20 Gb/s. Eagle also provides significant flexibility in what it monitors: any
However, we were not able to push near to the throughput limit system can be instrumented to send a log of events to Eagle as the
of the hardware switches, which is is 6.5 TB/s. Thus, we focused structure and content of the logs are user defined.
our performance evaluation to measuring Hydra’s overhead on Static Verification for Networks. There is also a large body of
packet-processing latency. First, we generate bidirectional UDP work on static verification for networks. Early work by Xie et al. [50]
traffic at 10 Gb/s on our cellular access network testbed using proposed using static techniques to reason about reachability in IP
iperf3. The background traffic utilizes all links between the two networks. It proposed the now-standard approach of computing
spine and two leaf switches with ECMP routing. Then, we started the transitive closure of transfer functions that model the behav-
a fast ping (every 0.2 s) from one server attached to the leaf1 ior of individual devices and links. Header Space Analysis [28],
switch to another service attached to the leaf2 switch. Figure 12a Anteater [38], and Veriflow [30] emerged later, and applied this
shows the round-trip times (RTTs) during the experiment. There general approach in the context SDN. To improve the scalability of
ACM SIGCOMM ’23, September 10–14, 2023, New York, NY, USA Renganathan, et al.

their analyses, they developed optimized data structures for trans- 8 CONCLUSION
fer functions. Atomic predicates [52] replaces complex classifiers There is an important difference between catching a packet on the
with simple predicates that can be handled efficiently in backend wrong path immediately versus catching it eventually. If an intruder
solvers. NetKAT [4] is an algebraic framework based on a sound is exfiltrating confidential data, one packet may be all it takes; if a
and complete deductive system and a decision procedure based single packet passes between two “isolated” virtual tenants, trust
on automata [18]. Tools like p4v [35], Acquila [48], and Network (and business) is lost. Our approach is to check every packet as it
Optimized Datalog (NoD) [36] translate P4 code into representa- flows through the network. While it is perhaps an extreme approach,
tions that can be verified using staic techniques. Vera [45] and we think it is essential if we are to automate the closed loop control
P4-Assert [21] address the same problems using symbolic execu- of networks and minimize human intervention. Hydra programs
tion, while bf4 [15] infers control-plane constraints automatically. are easy to read and write for a large set of expressive properties.
Gravel [56] formally verifies the software of middle boxes such as Our experiences deploying Hydra programs on P4 switches and in
NATs and firewalls. A complementary line of work focuses on con- the context of an open-source cellular access network with real-
trol plane verification. Batfish [17] and Minesweeper [7] statically world traffic shows they create little overhead, yet can catch real
analyze configuration files for distributed protocols to verify reach- bugs in a live system.
ability properties automatically. Recent approaches use abstract in-
terpretation to verify simpler representations of programs [8, 16, 22].
ACKNOWLEDGMENTS
Our work builds on the extensive foundation provided by this prior
work, but uses an approach based on runtime rather than static We are grateful to Megan Jung, Bruce Spang, Owolabi Leunsen, and
techniques. the participants of the Bellairs Workshop on Network Verification
Runtime Verification for Networks. Early work on runtime for early ideas and helpful feedback. This work is supported by the
checking for networks focused on generating test or probe pack- NSF under grant FMiTF-1918396, the ONR under Contract N68335-
ets [10, 40, 54]. P4Consist proposes adding a new module to tag pack- 22-C-0411, and DARPA under Contract HR001120C0107.
ets with the path and forwarding rules used to process them [44].
However, P4Consist only adds these tags to special probe packets, REFERENCES
which are generated using a traffic generator, and verification is [1] Mohammad Al-Fares, Sivasankar Radhakrishnan, Barath Raghavan, Nelson
Huang, and Amin Vahdat. Hedera: Dynamic flow scheduling for data center net-
performed out-of-band—i.e., the tagged packets are sent to a sep- works. In USENIX Symposium on Networked Systems Design and Implementation
arate server for analysis. VeriDP follows a similar approach, but (NSDI), page 19, 2010.
focuses on detecting inconsistencies between control-plane and [2] Kinan Dak Albab, Jonathan DiLorenzo, Stefan Heule, Ali Kheradmand, Stef-
fen Smolka, Konstantin Weitz, Muhammad Timarzi, Jiaqi Gao, and Minlan Yu.
data-plane state [57]. In contrast, Hydra instead uses checkers that SwitchV: automated SDN switch validation with P4 models. In ACM Special
execute directly in the data plane, allowing it to detect violations as Interest Group on Data Communication (SIGCOMM), pages 365–379, 2022.
they occur and halt erroneous packets. Offline analysis approaches [3] Mohammad Alizadeh, Tom Edsall, Sarang Dharmapurikar, Ramanan
Vaidyanathan, Kevin Chu, Andy Fingerhut, Vinh The Lam, Francis Ma-
also face inherent scalability issues or accuracy tradeoffs due to tus, Rong Pan, Navindra Yadav, and George Varghese. Conga: Distributed
sampling. Hydra does not rely on sampling—it performs runtime congestion-aware load balancing for datacenters. In ACM Special Interest Group
on Data Communications (SIGCOMM), page 503–514, 2014.
checking on every packet. DBVal verifies assertions at runtime by [4] Carolyn Jane Anderson, Nate Foster, Arjun Guha, Jean-Baptiste Jeannin, Dexter
instrumenting the data plane [33]. However, their checks are tied Kozen, Cole Schlesinger, and David Walker. NetKAT: Semantic foundations for
to how forwarding is implemented. Thus, the verification code and networks. In ACM SIGACT-SIGPLAN Symposium on Principles of Programming
Languages, pages 113–126, 2014.
system being instrumented are not independent, which could lead [5] Manikandan Arumugam, Deepak Bansal, Navdeep Bhatia, James Boerner, Simon
to false negatives if forwarding and checking code have the same Capper, Changhoon Kim, Sarah McClure, Neeraj Motwani, Ranga Narasimhan,
bug. Aragog [53] supports defining properties parameterized on Urvish Panchal, Tommaso Pimpo, Ariff Premji, Pranjal Shrivastava, and Rishabh
Tewari. Bluebird: High-performance SDN for bare-metal cloud services. In
location, stateful variables, and temporal predicates. Additionally, it USENIX Symposium on Networked Systems Design and Implementation (NSDI),
checks every execution trace in the system. Aragog differs from Hy- pages 355–370, 2022.
[6] Howard Barringer, Allen Goldberg, Klaus Havelund, and Koushik Sen. Rule-based
dra in that it focuses on distributed network functions, rather than runtime verification. In Verification, Model Checking, and Abstract Interpretation
the data plane itself. In that sense, it is a complementary approach to (VMCAI), pages 44–57, 2004.
Hydra. Aragog requires making modest modifications to the source [7] Ryan Beckett, Aarti Gupta, Ratul Mahajan, and David Walker. A general approach
to network configuration verification. In ACM Special Interest Group on Data
code for the network functions, in order to send events of interest Communication (SIGCOMM), page 155–168, 2017.
to the verifier. Hydra checks every packet and is independent of [8] Ryan Beckett, Aarti Gupta, Ratul Mahajan, and David Walker. Abstract interpreta-
the forwarding code. tion of distributed network control planes. In ACM SIGACT-SIGPLAN Symposium
on Principles of Programming Languages (POPL), 2019.
Summary. Overall, Hydra builds on ideas that have been developed [9] Nikolaj Bjørner and Karthick Jayaraman. Checking cloud contracts in Microsoft
for years in the runtime verification and formal methods communi- Azure. In International Conference on Distributed Computing and Internet Tech-
nology (ICDCIT), pages 21–32, 2015.
ties and applies them to the problem of verifying network behavior. [10] Marco Canini, Daniele Venzano, Peter Peresini, Dejan Kostic, and Jennifer Rexford.
It provides an easy-to-use specification language for expressing A NICE way to test OpenFlow applications. In USENIX Symposium on Networked
a rich set of network-wide properties as well as a compiler that Systems Design and Implementation (NSDI), pages 127–140, 2012.
[11] Feng Chen and Grigore Roşu. Java-MOP: A monitoring oriented programming
translates these programs into executable code that can be deployed environment for Java. In Tools and Algorithms for the Construction and Analysis
on network switches. Our approach is expressive, scalable, and op- of Systems (TACAS), page 546–550, 2005.
erates in-band, detecting and blocking errant packets at line rate [12] P4 Language Consortium. P4Lang Tutorials, Fetched July 15th, 2023. https:
//github.com/p4lang/tutorials.
and in real time. [13] Giuseppe De Giacomo and Moshe Y. Vardi. Linear temporal logic and linear
dynamic logic on finite traces. In International Joint Conference on Artificial
Hydra: Effective Runtime Network Verification ACM SIGCOMM ’23, September 10–14, 2023, New York, NY, USA

Intelligence (IJCAI), page 854–860, 2013. [37] Robert MacDavid, Carmelo Cascone, Pingping Lin, Badhrinath Padmanabhan,
[14] Stéphane Demri and Ranko Lazić. LTL with the freeze quantifier and register Ajay ThakuR, Larry Peterson, Jennifer Rexford, and Oguz Sunay. A P4-based
automata. ACM Transactions on Computational Logic (TOCL), apr 2009. 5G user plane function. In ACM SIGCOMM Symposium on SDN Research (SOSR),
[15] Dragos Dumitrescu, Radu Stoenescu, Lorina Negreanu, and Costin Raiciu. bf4: page 162–168, 2021.
Towards bug-free P4 programs. In ACM Special Interest Group on Data Commu- [38] Haohui Mai, Ahmed Khurshid, Rachit Agarwal, Matthew Caesar, P. Brighten
nication (SIGCOMM), page 571–585, 2020. Godfrey, and Samuel Talmadge King. Debugging the data plane with Anteater. In
[16] Seyed K. Fayaz, Tushar Sharma, Ari Fogel, Ratul Mahajan, Todd Millstein, Vyas ACM Special Interest Group on Data Communication (SIGCOMM), page 290–301,
Sekar, and George Varghese. Efficient network reachability analysis using a 2011.
succinct control plane representation. In USENIX Conference on Operating Systems [39] Justin Meza, Tianyin Xu, Kaushik Veeraraghavan, and Onur Mutlu. A large scale
Design and Implementation (OSDI), page 217–232, 2016. study of data center network reliability. In ACM SIGCOMM Internet Measurement
[17] Ari Fogel, Stanley Fung, Luis Pedrosa, Meg Walraed-Sullivan, Ramesh Govindan, Conference (IMC), page 393–407, 2018.
Ratul Mahajan, and Todd Millstein. A general approach to network configuration [40] Andres Nötzli, Jehandad Khan, Andy Fingerhut, Clark Barrett, and Peter Athanas.
analysis. In USENIX Conference on Networked Systems Design and Implementation P4Pktgen: Automated test case generation for P4 programs. In ACM SIGCOMM
(NSDI), page 469–483, 2015. Symposium on SDN Research (SOSR), 2018.
[18] Nate Foster, Dexter Kozen, Matthew Milano, Alexandra Silva, and Laure Thomp- [41] Mark Reitblatt, Nate Foster, Jennifer Rexford, Cole Schlesinger, and David Walker.
son. A coalgebraic decision procedure for NetKAT. In ACM SIGACT-SIGPLAN Abstractions for network update. In ACM Special Interest Group on Data Commu-
Symposium on Principles of Programming Languages (POPL), pages 343–355, 2015. nications (SIGCOMM), page 323–334, 2012.
[19] Open Networking Foundation. Aether: An open source 5G connected edge [42] Sundararajan Renganathan, Benny Rubin, Hyojoon Kim, Pier Luigi Ventre,
platform, 2022. https://opennetworking.org/aether/. Carmelo Cascone, Daniele Moro, Charles Chan, Nick McKeown, and Nate Foster.
[20] Open Networking Foundation. Software Defined Fabric (SD-Fabric) Release 1.2, Hydra: Effective network verification (full version), September 2023. Available at
Fetched February 15th, 2023. https://docs.sd-fabric.org/master/release/1.2.0.html. https://www.cs.cornell.edu/~jnfoster/papers/hydra-tr.pdf.
[21] Lucas Freire, Miguel Neves, Lucas Leal, Kirill Levchenko, Alberto Schaeffer-Filho, [43] Fabian Ruffy, Jed Liu, Prathima Kotikalapudi, Vojtĕch Havel, Hanneli Tavante,
and Marinho Barcellos. Uncovering bugs in P4 programs with assertion-based Rob Sherwood, Vlad Dubina, Volodymyr Peschanenko, Anirudh Sivaraman, and
verification. In ACM SIGCOMM Symposium on SDN Research (SOSR), 2018. Nate Foster. P4Testgen: An extensible test oracle for P4. In ACM Special Interest
[22] Aaron Gember-Jacobson, Raajay Viswanathan, Aditya Akella, and Ratul Mahajan. Group on Data Communications (SIGCOMM), 2023. To appear.
Fast control plane analysis using an abstract representation. In ACM Special [44] Apoorv Shukla, Seifeddine Fathalli, Thomas Zinner, Artur Hecker, and Stefan
Interest Group on Data Communications (SIGCOMM), page 300–313, 2016. Schmid. P4consist: Toward consistent P4 SDNs. IEEE Journal on Selected Areas in
[23] William Sealey Gosset. The probable error of a mean. Biometrika, pages 1–25, Communications (JSAC), 38(7):1293–1307, 2020.
1908. [45] Radu Stoenescu, Dragos Dumitrescu, Matei Popovici, Lorina Negreanu, and
[24] Nikhil Handigol, Brandon Heller, Vimalkumar Jeyakumar, David Mazières, and Costin Raiciu. Debugging P4 programs with Vera. In ACM Special Interest Group
Nick McKeown. I know what your packet did last hop: Using packet histories to on Data Communication (SIGCOMM), page 518–532, 2018.
troubleshoot networks. In USENIX Symposium on Networked Systems Design and [46] Praveen Tammana, Rachit Agarwal, and Myungjin Lee. Simplifying datacenter
Implementation (NSDI), pages 71–85, 2014. network debugging with Pathdump. In USENIX Symposium on Operating Systems
[25] Intel. Intel® Tofino™., Fetched February 10th, 2023. https://www.intel.com/ Design and Implementation (OSDI), page 233–248, 2016.
content/www/us/en/products/network-io/programmable-ethernet-switch/ [47] Praveen Tammana, Rachit Agarwal, and Myungjin Lee. Distributed network mon-
tofino-series.html. itoring and debugging with SwitchPointer. In USENIX Symposium on Networked
[26] Karthick Jayaraman, Nikolaj Bjørner, Jitu Padhye, Amar Agrawal, Ashish Bhar- Systems Design and Implementation (NSDI), pages 453–456, 2018.
gava, Paul-Andre C Bissonnette, Shane Foster, Andrew Helwer, Mark Kasten, Ivan [48] Bingchuan Tian, Jiaqi Gao, Mengqi Liu, Ennan Zhai, Yanqing Chen, Yu Zhou,
Lee, Anup Namdhari, Haseeb Niaz, Aniruddha Parkhi, Hanukumar Pinnamraju, Li Dai, Feng Yan, Mengjing Ma, Ming Tang, Jie Lu, Xionglie Wei, Hongqiang Harry
Adrian Power, Neha Milind Raje, and Parag Sharma. Validating datacenters at Liu, Ming Zhang, Chen Tian, and Minlan Yu. Aquila: A practically usable verifi-
scale. In ACM Special Interest Group on Data Communication (SIGCOMM), page cation system for production-scale programmable data planes. In ACM Special
200–213, 2019. Interest Group on Data Communications (SIGCOMM), page 17–32, 2021.
[27] Peyman Kazemian, Michael Chang, Hongyi Zeng, George Varghese, Nick McKe- [49] Xin Wu, Daniel Turner, Chao-Chih Chen, David A. Maltz, Xiaowei Yang, Lihua
own, and Scott Whyte. Real time network policy checking using header space Yuan, and Ming Zhang. Netpilot: Automating datacenter network failure miti-
analysis. In USENIX Symposium on Networked Systems Design and Implementation gation. ACM SIGCOMM Computer Communication Review (CCR), 42(4):419–430,
(NSDI), page 99–112, 2013. August 2012.
[28] Peyman Kazemian, George Varghese, and Nick McKeown. Header space analysis: [50] Geoffrey G. Xie, Jibin Zhan, David A. Maltz, Hui Zhang, Albert G. Greenberg,
Static checking for networks. In USENIX Symposium on Networked Systems Design Gísli Hjálmtýsson, and Jennifer Rexford. On static reachability analysis of IP
and Implementation (NSDI), pages 113–126, 2012. networks. In IEEE Conference on Computer Communications (INFOCOM), pages
[29] Anurag Khandelwal, Rachit Agarwal, and Ion Stoica. Confluo: Distributed moni- 2170–2183, 2005.
toring and diagnosis stack for high-speed networks. In USENIX Symposium on [51] Hongkun Yang and Simon S Lam. Real-time verification of network properties
Networked Systems Design and Implementation (NSDI), pages 421–436, 2019. using atomic predicates. IEEE/ACM Transactions on Networking, 24(2):887–900,
[30] Ahmed Khurshid, Xuan Zou, Wenxuan Zhou, Matthew Caesar, and P. Brighten 2015.
Godfrey. Veriflow: Verifying network-wide invariants in real time. In USENIX [52] Hongkun Yang and Simon S. Lam. Real-time verification of network properties
Symposium on Networked Systems Design and Implementation (NSDI), pages 15–27, using atomic predicates. IEEE/ACM Transactions on Networking, 24(2):887–900,
2013. April 2016.
[31] Hyojoon Kim, Xiaoqi Chen, Jack Brassil, and Jennifer Rexford. Experience-driven [53] Nofel Yaseen, Behnaz Arzani, Ryan Beckett, Selim Ciraci, and Vincent Liu. Ara-
research on programmable networks. ACM SIGCOMM Computer Communication gog: Scalable runtime verification of shardable networked systems. In USENIX
Review (CCR), 51(1):10–17, January 2021. Symposium on Operating Systems Design and Implementation (OSDI), pages 701–
[32] Hyojoon Kim and Arpit Gupta. Ontas: Flexible and scalable online network 718, 2020.
traffic anonymization system. In ACM SIGCOMM Workshop on Network Meets AI [54] Hongyi Zeng, Peyman Kazemian, George Varghese, and Nick McKeown. Auto-
& ML (NetAI), pages 15–21, 2019. matic test packet generation. In Emerging Networking EXperiments and Technolo-
[33] K Shiv Kumar, K Ranjitha, PS Prashanth, Mina Tahmasbi Arashloo, U Venkanna, gies (CoNEXT), page 241–252, 2012.
and Praveen Tammana. DBVal: Validating P4 data plane runtime behavior. In [55] Hongyi Zeng, Shidong Zhang, Fei Ye, Vimalkumar Jeyakumar, Mickey Ju, Junda
ACM SIGCOMM Symposium on SDN Research (SOSR), 2021. Liu, Nick McKeown, and Amin Vahdat. Libra: Divide and conquer to verify
[34] Bob Lantz, Brandon Heller, and Nick McKeown. A network in a laptop: Rapid forwarding tables in huge networks. In USENIX Symposium on Networked Systems
prototyping for Software-Defined Networks. In ACM SIGCOMM Workshop on Design and Implementation (NSDI), page 87–99, 2014.
Hot Topics in Networks (HotNets), 2010. [56] Kaiyuan Zhang, Danyang Zhuo, Aditya Akella, Arvind Krishnamurthy, and
[35] Jed Liu, William Hallahan, Cole Schlesinger, Milad Sharif, Jeongkeun Lee, Robert Xi Wang. Automated verification of customizable middlebox properties with
Soulé, Han Wang, Călin Caşcaval, Nick McKeown, and Nate Foster. P4v: Practical gravel. In USENIX Symposium on Networked Systems Design and Implementation
verification for programmable data planes. In ACM Special Interest Group on Data (NSDI), pages 221–239, 2020.
Communication (SIGCOMM), page 490–503, 2018. [57] Peng Zhang, Hao Li, Chengchen Hu, Liujia Hu, Lei Xiong, Ruilong Wang, and
[36] Nuno P. Lopes, Nikolaj Bjørner, Patrice Godefroid, Karthick Jayaraman, and Yuemei Zhang. Mind the gap: Monitoring the control-data plane consistency in
George Varghese. Checking beliefs in dynamic networks. In USENIX Conference software defined networks. In Emerging Networking EXperiments and Technologies
on Networked Systems Design and Implementation (NSDI), page 499–512, 2015. (CoNEXT), page 19–33, 2016.