LEARN WORK IT

INFORMATION TECHNOLO GY (NE T WORK )

C I S C O ACI BLO GS VMWARE N SX BLO G S CISCO ROUT ING B LO G

C I S CO SW ITCHIN G BLO G IT INS TITU TES CONTACT US

TERMS & CONDIT ION

24. ACI Endpoint

Learning
 JUNE 18, 2021  LEAVE A COMMENT

ACI Endpoint
Learning
Cisco ACI and endpoints

Cisco ACI uses endpoints to forward traf�c. An endpoint

consists of one MAC address and zero or more IP
addresses. Each endpoint represents a single
networking device

In a traditional network, three tables are used to

maintain the network addresses of external devices:
1) A MAC address table for Layer 2 forwarding,
2) A Routing Information Base (RIB) for Layer 3
forwarding.
3) An ARP table for the combination of IP addresses and
MAC addresses.

Cisco ACI replaced the MAC address table and ARP table
with a single table called the endpoint table. This
change implies that Cisco ACI learns that information
in a different way than in a traditional network. Cisco
ACI learns MAC and IP addresses in hardware by
looking at the packet source MAC address and source IP
address in the data plane instead of relying on ARP to
obtain a next-hop MAC address for IP addresses. This
approach reduces the number of resources needed to
process and generate ARP traf�c. It also allows
detection of IP address and MAC address movement
without the need to wait for GARP as long as some
traf�c is sent from the new host.

L3Out and regular endpoints

Cisco ACI mainly uses the endpoint table instead of the

MAC address and ARP tables, it still uses the RIB and the
ARP table. This capability is especially for L3Out
communication, because the maximum number of IP
addresses on a single endpoint (one MAC address) is
limited, and there can be a huge number of IP addresses
behind a single next-hop MAC address (external router)
on a L3Out connection.

It is not ef�cient to maintain all outside IP addresses as

separate /32 or /128 endpoints. Cisco ACI must know
how to reach these IP addresses as pre�xes through
routing protocols such as Open Shortest Path First
(OSPF), which is the same behavior as for traditional
routers. However, Cisco ACI needs to know only the
next hop (external router) for those pre�xes. Because of
this consideration, Cisco ACI uses a behavior similar to
that in traditional networks for L3Out connectivity. The
Cisco ACI L3Out domain learns the MAC address only
from the data plane. IP addresses are not learned from
the data plane in an L3Out domain; instead, Cisco ACI
uses ARP to resolve next-hop IP and MAC relationships
to reach the pre�xes behind external routers.

Local endpoints and remote

endpoints

A leaf switch has two types of endpoints: local

endpoints and remote endpoints. Local endpoints for
LEAF1 reside directly on LEAF1 (For example, directly
attached), whereas remote endpoints for LEAF1 reside
on other leaf endpoints

Both local and remote endpoints are learned from the

data plane, remote endpoints are merely a cache, local
to each leaf. Local endpoints are the main source of
endpoint information for the entire Cisco ACI fabric.
Each leaf is responsible for reporting its local endpoints
to the Council Of Oracle Protocol (COOP) database,
located on each spine switch, which implies that all
endpoint information in the Cisco ACI fabric is stored in
the spine COOP database. Because this database is
accessible, each leaf does not need to know about all the
remote endpoints to forward packets to the remote leaf
endpoints. Instead, a leaf can forward packets to spine
switches, even if the leaf does not know about a
particular remote endpoint. This forwarding behavior is
called spine proxy.

Because of spine proxy, Cisco ACI packet forwarding

will work without remote endpoint learning. Spine
proxy enables leaf switches to forward traf�c directly to
the COOP database located on the spine switches.
Remote endpoint learning helps Cisco ACI forward
packets more ef�ciently by allowing leaf switches to
send packets directly to a destination leaf switch
without using the resources on the spine switch that
would be used to lookup endpoints in the COOP
database, which contains all the fabric endpoint
information.
Remote endpoints are learned from data-plane traf�c,
as are local endpoints. Therefore, only leaf switches
with actual communication traf�c create a cache entry
for remote endpoints (conversational learning) to
forward the packets directly toward the destination leaf.
Remote endpoints have either one MAC address or one
IP address per endpoint, instead of a MAC address and
IP address combination as is the case for local
endpoints. One reason for this difference is that the IP
to MAC next-hop resolution can be performed on the
destination leaf, and the next-hop MAC address is not
required just to reach the destination leaf. This behavior
also helps each leaf save its resources for these caches.
Also, the aging timer for a remote endpoint is shorter
than for a local endpoint because a remote endpoint is
just a cache and should not be present after the
conversation has ceased and the original local endpoint
on another leaf has disappeared.

Command-Line Interface (CLI)

output for a local and a remote
endpoint on a leaf switch

N O T E : The endpoint IP address learning behavior is

based on the assumption that unicast routing is enabled
on the bridge domain. If unicast routing is not enabled,
a Cisco ACI leaf cannot perform routing and cannot
learn any IP addresses. It learns only MAC addresses
and performs switching.

Local endpoint learning

Cisco ACI learns the MAC (and IP) address as a local

endpoint when a packet comes into a Cisco ACI leaf
switch from its front-panel ports.

A Cisco ACI leaf switch follows these steps to learn a

local endpoint MAC address and IP address:

1. The Cisco ACI leaf receives a packet with a source

MAC Address (MAC A) and sources IP Address (IP A).
2. The Cisco ACI leaf learns MAC A as a local endpoint.
3. The Cisco ACI leaf learns IP A tied to MAC A if the
packet is an ARP packet.
4. The Cisco ACI leaf learns IP A tied to MAC A if the
packet is routed.

If the packet is switched and not an ARP packet, the

Cisco ACI leaf never learns the IP address but only the
MAC address.

Remote endpoint learning

Cisco ACI learns a MAC or IP address as a remote

endpoint when a packet comes into a Cisco ACI leaf
switch from another leaf switch through a spine switch.
When a packet is sent from one leaf to another leaf,
Cisco ACI encapsulates the original packet with an
outer header representing the source and destination
leaf Tunnel Endpoint (TEP) and the Virtual Extensible
LAN (VXLAN) header, which contains the bridge
domain or VRF information of the original packet.

A Cisco ACI leaf switch follows these steps to learn a

remote endpoint MAC or IP address:

1. The Cisco ACI leaf receives a packet with source

MAC A and sources IP A from a spine switch.
2. The Cisco ACI leaf learns MAC A as a remote
endpoint if VXLAN contains bridge domain
information.
3. The Cisco ACI leaf learns IP A as a remote endpoint if
VXLAN contains VRF information.

If packet is Layer 2 traf�c without any routing on Cisco

ACI. Therefore, only the MAC address (Src MAC S in the
�gure) is learned as a local endpoint on LEAF1 and a
remote endpoint on LEAF2.

Endpoint movement and bounce

entries
There are several scenarios in which an endpoint
moves between two Cisco ACI leaf switches, such as a
failover event or a virtual machine migration in a
hypervisor environment. Cisco ACI data-plane endpoint
learning detects these events quickly and updates the
Cisco ACI endpoint database on a new leaf. In addition
to data-plane learning, Cisco ACI uses bounce entries to
manage the old endpoint information on the original
leaf.
When a new local endpoint is detected on a leaf, the leaf
updates the COOP database on the spine switches with
its new local endpoint. If the COOP database has already
learned the same endpoint from another leaf, COOP will
recognize this event as an endpoint move and report
this move to the original leaf that contained the old
endpoint information. The old leaf that receives this
noti�cation will delete its old endpoint entry and create
a bounce entry, which will point to the new leaf. A
bounce entry is basically a remote endpoint created by
COOP communication instead of data-plane learning.

NOTE: The difference between a bounce entry and a

remote endpoint is in whether or not the leaf rewrites
the outer source IP address of the packet. When a packet
uses a normal remote endpoint, the Cisco ACI leaf uses
its own TEP address as the outer source IP address, so
the remote leaf learns this packet with its own TEP.
When a packet uses a bounce entry, the Cisco ACI leaf
doesn’t rewrite the outer source IP address, so the
remote data-plane learning will behave as if the packet
came from the originating leaf rather than the
intermediate “bounce” leaf.

The endpoint retention timer value (aging interval) for a

bounce entry is 630 seconds by default. You can tune
this value by going to Tenant > Networking > Protocol
Policies > End Point Retention, where you can also �nd
the other endpoint retention timers.

LEAF1 learns the remote endpoint location for IP2

pointing to LEAF2 from the data plane.
In the second step, the endpoint with MAC2 and IP2 on
LEAF2 moves to LEAF4, and the new local endpoint is
created on LEAF4. This new local endpoint is reported
to the COOP database on the spine switches, which in
turn will notify LEAF2 about this move, and LEAF2 will
install bounce entries for MAC2 and IP2. Bounce entries
are basically the same as remote endpoints. Hence two
bounce entries, for the MAC and IP addresses are
created.

At this point, LEAF1 still has the old remote endpoint for
IP2, which still points to the old location: LEAF2. If a
packet is sent from LEAF1 to IP2 at this time, LEAF1
forwards it to LEAF2, instead of LEAF4, based on its
remote endpoint cache. Because of the bounce entries,
LEAF2 is already prepared for this sort of forwarding
from leaf switches with old remote endpoints. LEAF2
will then bounce the packet to the new LEAF4 based on
its bounce entries. This bounce entry is a backup
mechanism for this type of scenario. Therefore, the
bounce entry will not be used if the new traf�c from IP2
on LEAF4 reaches LEAF1 before LEAF1 sends packets to
IP2 because the old remote endpoint on LEAF1 will be
updated directly by the data-plane traf�c from the new
leaf.
The advantage of this implementation is scale. No
matter how many leaf switches have learned endpoint
information, only three components will need to be
updated after an endpoint moves. The three
components are the COOP database, the new leaf
switch to which the endpoint has moved, and the old
leaf switch from which the endpoint has moved.
Eventually, all other leaf switches in the fabric will
update their information about the location of the
endpoint through data-plane traf�c. However, there are
a few corner cases where the other leaf switches keep
the outdated remote endpoint information even after
the bounce entry ages out. This could cause traf�c to be
black-holed by sending the traf�c to a leaf that doesn’t
have the destination endpoint or the bounce entry.

Endpoint Announcement
Enhancement

The following enhancement regarding endpoint

announcement was introduced from Cisco APIC
Release 3.2(2l).
CSCvj17665 EP Announce support for stale IP remote
endpoints.
With this endpoint announcement feature, ACI will
send an announcement message to all leaf switches
when a bounce entry ages out to ensure the IP remote
endpoints on the other leaf switches have the same
information as the bounce entry and to delete the
outdated IP remote endpoints if any.

The command syntax to manually clear a

particular remote IP endpoint is shown here:

LEAF1# clear system internal epm endpoint key vrf <>

ip <>

The command syntax to manually clear all

remote endpoints (both MAC and IP) in one
VRF instance is shown here:

LEAF1# clear system internal epm endpoint vrf <>

remote
Example:
LEAF1# clear system internal epm endpoint key vrf
TK:VRF1 ip 192.168.2.2
LEAF1# clear system internal epm endpoint key vrf
TK:VRF1 remote

Bridge domain–level con�guration

options
Unicast Routing

The Unicast Routing option has been implemented

since the �rst release of Cisco ACI. It’s located at
Tenants > Networking > Bridge Domains in the APIC
GUI

Enables IP unicast routing on the

bridge domain

If this feature is not enabled, the subnets con�gured

under the bridge domain are not pushed down to leaf
switches, and routing will not occur. In addition, a
bridge domain with unicast routing disabled will not
learn any IP address as an endpoint. Thus, that bridge
domain will be used only for Layer 2 communications,
and endpoints in that bridge domain should have their
default gateways outside Cisco ACI.
If unicast routing is enabled without any bridge domain
subnets con�gured, IP information in the bridge domain
can still be learned through ARP in the data plane, but
no routing will occur because there will be no SVI to
perform routing on the bridge domain. This approach is
not recommended.
When unicast routing is disabled, both MAC and IP
endpoint information is �ushed for the BD.
GARP-based EP Move Detection
Mode

This option is located at Tenant > Networking > Bridge

Domain . This option is disabled by default. The option
is available only on bridge domains in which ARP
�ooding is enabled.

Although Cisco ACI can detect MAC and IP address

movement between leaf switch ports, leaf switches,
bridge domains, and EPGs, it does not detect the
movement of an IP address to a new MAC address if the
new MAC address is from the same interface and same
EPG as the old MAC address.
When the GARP-based detection option is enabled,
Cisco ACI will trigger an endpoint move based on GARP
packets if the move occurs on the same interface and
same EPG. If a GARP packet comes from the same
interface and same EPG, then endpoint learning is
triggered only when Unicast Routing, ARP Flooding,
and “GARP based detection” are all enabled for the
bridge domain.
Although this scenario has not been widely seen across
our customer base, in some cases customers do change
their IP to MAC bindings and need to enable GARP-
based detection.

Limit IP Learning To Subnet

The Limit IP Learning To Subnet option was originally

called Enforce Subnet Check for IP Learning. It was
introduced in APIC Release 1.1(1j)

I t i s l o c a t e d a t Te n a n t > N e t w o r k i n g > B r i d g e
Domain

Beginning with APIC Releases 2.3(1e) and 3.0(1k), this

option is enabled by default with the following
enhancement:
CSCvb16668: Enforce Subnet Check should be enabled
by default
Prior to these releases, this option was disabled by
default.
If this option is enabled, the fabric will learn only IP
addresses for subnets con�gured on the bridge domain.

If this option is disabled or enabled on a bridge domain

that was already con�gured, the following happens:
● Cisco ACI �ushes all endpoint IP addresses learned on
the bridge domain.
● Cisco ACI pauses MAC and IP address learning for 120
seconds.

If this option is enabled on a bridge domain that had the

option disabled, the following happens:
● Cisco ACI doesn’t �ush endpoint IP addresses that
belong to the subnet. (Endpoint IP addresses that do not
belong to the bridge domain subnet are �ushed.)
● MAC or IP address learning is not paused for 120
seconds.

If this option is disabled on a bridge domain that had

the option enabled, the following happens:
● Cisco ACI doesn’t �ush endpoint IP addresses learned
on the bridge domain.
● MAC or IP address learning is not paused for 120
seconds.

From the leaf, run the command v s h – c ‘ s h o w

s y s t e m i n t e r n a l e p m v l a n v l a n - i d d e t a i l ’ and
look for the Learn Enable option.

Endpoint Retention Policy

The Endpoint Retention Policy con�guration is located

at T e n a n t > P o l i c i e s > P r o t o c o l > E n d P o i n t
R e t e n t i o n and is referred from a Bridge Domain (BD)
or a VRF (Figure 34). By default, a BD or a VRF refers to
the default policy de�ned in the common tenant is
used.
Endpoint Retention Policy

This option is use d to sp e cify the life cycle of

endp oints using the following values:
● Hold Interval: The amount of time in seconds that
endpoint learning is disabled in a bridge domain due to
EP Loop Protection (BD Learn Disable) or Endpoint Move
Dampening that is triggered based on the Move
Frequency below. The default interval is 300 seconds.
● Bounce Entry Aging Interval: The amount of time in
seconds until a bounce entry in the endpoint table on a
leaf node expires. The default interval is 630 seconds.
● Local End Point Aging Interval: The amount of time in
seconds that a leaf node can keep each local endpoint
in its endpoint table without further updates. The
default interval is 900 seconds. If 75 percent of the
interval is reached, the leaf node sends three ARP
requests to verify the presence of the endpoint. If no
response is received, the endpoint is deleted.
● Remote End Point Aging Interval: The amount of time
in seconds that a leaf node can keep each remote
endpoint in its endpoint table without further updates.
The default interval is 300 seconds.
● Move Frequency: The maximum number of endpoint
moves that are allowed per second within a bridge
domain on each leaf node. The number is counted as
the total movement of any endpoint in the given BD,
whether it is a single endpoint �ap, a simultaneous
move of multiple endpoints, or a combination of both. If
the number of movements per second is exceeded the
Move Frequency, the Hold Interval (described above) is
triggered, and learning the new endpoint in the BD is
disabled until the Hold Interval expires. The feature is
called BD Move Frequency or Endpoint Move
Dampening. The default is 256.
With the default con�guration parameters, Endpoint
Move Dampening disables endpoint learning on the
bridge domain for 300 seconds if the number of
endpoint moves is more than 256 times per second.

VRF-level configuration
options
IP Data-plane Learning

The IP Data-plane Learning option was introduced in

APIC Release 4.0(1h). This option is located at Tenant >
Networking > VRFs. This option is enabled by default.
This option enables and disables endpoint data-plane IP
learning on the VRF.

When the IP Data-plane Learning option under VRF is

disabled, endpoint learning behavior on an ACI leaf
changes as follows:
● Local MACs and remote MACs are learned via the data
plane (no change with this option).
● Local IPs are not learned via the data plane.
● Local IPs are learned from ARP/GARP/ND via the
control plane.
● Remote IPs are not learned from unicast packets via
the data plane.
● Remote IPs are learned from multicast packets via the
data plane.

When the IP Data-plane Learning option is disabled,

existing remote IP endpoints are �ushed immediately
while bounce entries are retained and age out normally.
Existing local IP endpoints are not �ushed either, but
they will age out eventually unless control plane
packets such as ARP keep them alive.

N O T E : When the IP Data-plane Learning option under

VRF is disabled, it is recommended to ensure IP Aging
option is enabled as well. This is to ensure Host
Tracking, which sends ARP/ND for a local IP endpoint
at 75% of its retention timer, is always triggered to
correctly track the IP status via the control plane.
Without IP Aging option enabled, local IP endpoints
may not age out correctly due to data-plane traf�c even
when the IP Data-plane Learning is disabled on the VRF.

Fabric-level configuration
options
Disable Remote EP Learn (on
border leaf)

APIC Release 3.0(1k) and later, it is located at S y s t e m >

System Settings > Fabric Wide Setting.

This option is disabled by default.

When this feature is enabled, remote IP endpoint
learning at the VRF instance is disabled on border leaf
switches. However, border leaf may still learn remote IP
endpoints from IP multicast routing packets, because of
a limitation in the Cisco ACI IP multicast routing
implementation. This exception applies only when a
second-generation switch is used as the border leaf
because Cisco ACI IP multicast routing is supported
only starting with second-generation switches. This
feature doesn’t disable remote MAC endpoint learning.

Enforce Subnet Check

The Enforce Subnet Check option was �rst introduced

in APIC Releases 2.2(2q) and 3.0(2h).
In APIC Release 2.2(2q), the option is located at

Fabric > Ac c ess Policies > Global Policies >

Fabric Wide Setting Policy
In APIC Release 3.0(2h) and later, it is located at
System > System Settings > Fabric Wide
Setting

This feature is available only on second-generation leaf

switches.
This feature enforces subnet checks at the VRF level
when Cisco ACI learns the IP address as an endpoint
from the data plane. Although the subnet check scope is
the VRF instance, this feature can be enabled and
disabled only globally under Fabric Wide Setting Policy.
You cannot enable this option only in one VRF instance.
This feature is disabled by default.

N O T E : This feature is superior to the Limit IP Learning

To Subnet option because, for local endpoint learning, it
suppresses learning from subnets outside the bridge
domain in hardware. For remote endpoint learning, it
suppresses learning of remote IP addresses not
associated with any bridge domain subnet.

On the ingress leaf (local endpoint learning):

The option enforces bridge domain–level subnet checks

for local endpoint learning. When this feature is
enabled, the Cisco ACI leaf learns an IP address and
MAC address as a new local endpoint only when the
source IP address of the incoming packet belongs to one
of the ingress bridge domain subnets.
This behavior is almost the same as Limit IP Learning
To Subnet option under the bridge domain. The
difference is that Limit IP Learning To Subnet limits
only IP learning if the source IP address of a packet
doesn’t belong to an ingress bridge domain subnet,
whereas this feature limits learning of both the MAC
address and IP address when IP learning is triggered
but yet prevented because the source IP address doesn’t
belong to an ingress bridge domain subnet. Please note
that, regardless of the source IP range, the Cisco ACI leaf
still learns MAC address if the packet is bridging traf�c
because the leaf does not check the IP header or
whether or not it has the IP header for bridging traf�c.
Thus, Enforce Subnet Check enables slightly stronger
checks than Limit IP Learning To Subnet. This check
will be enabled on all bridge domains, and you cannot
turn the checks on and off per bridge domain.
Therefore, Limit IP Learning To Subnet is not required
when this feature is enabled.

On the egress leaf (remote endpoint

learning):

This option enforces VRF-level subnet checks for

remote endpoint learning. When this feature is enabled,
the Cisco ACI leaf will learn an IP address as a remote
endpoint only when the source IP address of the
incoming packet belongs to any bridge domain subnet
in the same VRF instance on the egress leaf.
This behavior prevents IP spoo�ng scenarios, in which
an endpoint sends a packet with an unexpected source
IP address that does not belong to any of the bridge
domains on the VRF instance, such as an IP address
that exists behind the L3Out connection.
When this feature is enabled, Cisco ACI �ushes all local
IP endpoints outside bridge domain subnets and all
remote IP endpoints.

IP Aging Policy

This con�guration is disabled by default to keep the

same behavior with the older release.
For APIC Release 2.0, this option is located at F a b r i c >
Ac c ess Policies > Global Policies > IP Aging
P o l i c y . For APIC Release 3.0(1k) and later, it is located
at S y s t e m > S y s t e m S e t t i n g s > E n d p o i n t
Controls > IP Aging.

IP Aging (APIC Release 2.0)

IP Aging (APIC Release 3.0)

The IP aging policy tracks and ages unused IP

addresses on an endpoint. Tracking is performed by
using the endpoint retention policy, which is con�gured
for the bridge domain to send ARP requests (for IPv4)
and neighbor solicitations (for IPv6) at 75 percent of the
local endpoint aging interval. When no response is
received from an IP address that IP address is aged out.

Rogue EP Control

Rogue EP Control was �rst introduced in APIC Release

3.2(1l). This con�guration is disabled by default to keep
the same behavior with the older release.
This option is located at S y s t e m > S y s t e m S e t t i n g s
> Endp oint Controls > Ro gue EP Control

Rogue EP Control is used to detect an endpoint that

moves frequently and to disable endpoint learning for
the particular endpoint only. When a leaf node identi�es
an endpoint as a rogue, the endpoint becomes a static
endpoint temporarily and will be deleted after the hold
interval, so that rapid endpoint movement across
different locations can be prevented.
Note: Only traf�c from and to the endpoint is affected,
as opposed to disabling endpoint learning the bridge or
shutting down the interface from which the endpoint
was learned, which could affect healthy endpoints
learned from the same interface or the same bridge
domain. Rogue EP Control also raises a fault that helps
an administrator to identify the rogue endpoint.
Dete ction criteria can b e configure d by using
the following values:
● Rogue EP Detection interval: to specify the time in
seconds to detect rogue endpoints. The default is 60
seconds. The supported range is 30 to 3600 seconds.
● Rogue EP Detection Multiplication Factor: The
endpoint is declared rogue if the endpoint moves more
than this number within the Rogue EP Detection
interval. The default is 4. The supported range is 2 to 10.
● Hold Interval: the amount of time the endpoint is
being handled as a rogue and kept as the static
endpoint. After this interval, the endpoint is deleted.
The default is 1800 seconds. The supported range is
1800 to 3600.

For example, if the Rogue EP Control is enabled with the

default con�guration parameters above, the ACI fabric
declares an endpoint rogue if the endpoint moves more
than four times in 60 seconds and disables learning for
the endpoint for 1800 seconds. The rogue endpoint will
be static on the leaf node, interface, and VLAN where it
was detected right before the declaration of rogue.
After the hold-interval, rogue endpoints will be deleted.
Rogue endpoints can be deleted even before the hold
interval at F a b r i c > I n v e n t o r y > P o d _ n u m b e r >
L e af_name > Cle ar Ro gue Endp oints