This document covers the design and configuration of an Arista DCS-7280SR3K series border router at an internet edge. It addresses hardware platform constraints (TCAM allocation, counter feature groups), QoS policy design, MPLS/LDP configuration, and all BGP-layer concerns: session parameters, bestpath policy, peer groups, routing control functions (RCF), convergence optimizations, and BMP telemetry. The platform uses a Jericho2C ASIC and runs EOS with the service routing protocols model multi-agent prerequisite — required for internet-scale BGP.
Platform: DCS-7280SR3K series (Jericho2C ASIC) Role: Internet Border / Transit / Peering
CLI Prompt with Timestamp
EOS prompt includes hostname and real-time timestamp for CLI session auditing.
prompt %H_%D{%y.%m.%d-%H:%M:%S}%P
| Token | Expands to | Example |
|---|---|---|
%H | Hostname | <hostname> |
%D{...} | strftime date/time | 25.07.03-14:23:01 |
%P | Privilege indicator | # |
Result: <hostname>_25.07.03-14:23:01#
Hardware TCAM Allocation
⚠️ Consult Arista SE before modifying TCAM profiles. Total key budget is fixed per ASIC. Misconfiguration causes silent TCAM overflow. Verify with
show hardware capacityafter any profile change.
Two profiles defined: BR-BLUE and BR-RED. Active system profile is BR-RED.
Rationale for keeping two profiles: Both profiles remain defined on the device to enable fast rollback. If the active profile (RED) causes unexpected TCAM behavior after a reboot, the operator can switch back to the previous profile (BLUE) with a single
system profilechange and reboot — no config reconstruction required. Only thesystem profileline needs to change; the profile definitions themselves stay intact.
hardware tcam
profile BR-RED
feature evpn mpls irb
key size limit 160
!
feature evpn mpls multi-homing
key size limit 160
!
feature l2-protocol forwarding
sequence 95
key size limit 160
key field dst-mac vlan-tag-format
action redirect-to-cpu snoop
packet ipv4 forwarding bridged
packet ipv6 forwarding bridged
packet non-ip forwarding bridged
packet non-ip forwarding bridged sub-interface
!
feature mpls
sequence 5
key size limit 160
action drop redirect set-ecn
packet ipv4 mpls ipv4 forwarding mpls decap
packet ipv4 mpls ipv6 forwarding mpls decap
packet mpls ipv4 forwarding mpls
packet mpls ipv6 forwarding mpls
packet mpls non-ip forwarding mpls
!
feature mpls pop ingress
sequence 90
!
feature sflow subintf
action sample
packet ipv4 forwarding bridged
packet ipv4 forwarding routed
packet ipv6 forwarding bridged
packet ipv6 forwarding routed
packet mpls forwarding bridged decap
packet non-ip forwarding bridged
!
feature traffic-policy cpu ipv4
key size limit 160
key field dst-ip ip-frag ip-protocol l4-dst-port l4-src-port src-ip tcp-control ttl
action count set-drop-precedence set-policer
!
feature traffic-policy cpu ipv6
key field dst-ipv6 ipv6-next-header l4-dst-port l4-src-port src-ipv6-high src-ipv6-low tcp-control ttl
action count set-drop-precedence set-policer
!
feature traffic-policy port ipv4
sequence 45
key field dscp dst-ip-label ip-frag ip-fragment-offset ip-length ip-protocol l4-dst-port-label l4-src-port-label src-ip-label tcp-control ttl
action count drop redirect set-drop-precedence set-dscp set-tc set-unshared-policer
packet ipv4 forwarding routed
packet mpls ipv4 forwarding bridged
packet mpls ipv4 forwarding mpls
packet mpls ipv4 forwarding routed decap
!
feature traffic-policy port ipv6
sequence 55
key field dst-ipv6-label hop-limit icmp-type-code ipv6-length ipv6-next-header ipv6-traffic-class l4-dst-port-label l4-src-port-label src-ipv6-label tcp-control
action count drop redirect set-drop-precedence set-dscp set-tc set-unshared-policer
packet ipv6 forwarding routed
packet mpls ipv6 forwarding bridged
packet mpls ipv6 forwarding mpls
packet mpls ipv6 forwarding routed decap
!
feature traffic-policy port ipv4 egress
key size limit 160
key field dscp dst-ip-label ip-frag ip-protocol l4-dst-port-label l4-src-port-label src-ip-label tcp-control
action count drop set-dscp
packet ipv4 forwarding routed
!
feature traffic-policy port ipv6 egress
key field dst-ipv6-label ipv6-next-header l4-dst-port-label l4-src-port-label src-ipv6-label tcp-control
action count drop set-dscp
packet ipv6 forwarding routed
system profile BR-RED
Verification
show hardware tcam profile
show hardware tcam profile detail
show hardware capacity
Hardware Counter Features
Hardware counters consume entries from a shared pool. Some features are mutually exclusive.
hardware counter feature traffic-policy cpu
hardware counter feature traffic-policy out
hardware counter feature traffic-policy in
hardware counter feature ip in
hardware counter feature subinterface out
hardware counter feature subinterface in
Mutual exclusivity
⚠️ Verify against the Arista TOI for your exact EOS version. Counter bank conflicts and supported combinations change between releases.
Checking active counters
Use show hardware counter feature to verify which counter banks are active and which ASIC engine slots they occupy. All configured features must show up; any down or absent entry means the bank failed to allocate (resource conflict or reboot required).
show hardware counter feature
Feature Direction Counter Resource (Engine) Status Detail
------------------------- ---------------- -------------------------------- ------------- ----------------------------------------------------
ACL-IPv4 out Jericho2C: 18 up
ACL in Jericho2C: 8, 19 up
Traffic policy CPU n/a Jericho2C: 10, 23 up
Traffic policy out Jericho2C: 24 up
Traffic policy in Jericho2C: 9, 20 up
IPv4, IPv6 in Jericho2C: 0 up
QoS policer in Jericho2C: 21, 22 up Configured with QoS policy-map class police action.
Queue out Jericho2C: 16 up Not user-configurable.
Subinterface out Jericho2C: 1 up
Subinterface in Jericho2C: 2 up
VOQ in Jericho2C: 17 up Not user-configurable.
Additional verification:
show traffic-policy counters interface <intf>
Enabling a counter feature may require reboot. A feature listed as “not active” after reboot indicates a resource conflict.
Counter poll interval
traffic-policies
counter interface poll interval 15 seconds
Load-interval
load-interval default 5
Sets interface rate averaging window to 5 seconds (default is 300s). Provides near-real-time rates for show interfaces, and SNMP ifInOctets rate calculation.
Platform Sand / VoQ
platform sand mdb profile l3-xxl
platform sand voq subinterface allocation 4096
platform sand qos rewrite dscp mpls encapsulation hierarchical next-hop disabled
Also required:
service routing protocols model multi-agent
service routing protocols model multi-agent is a prerequisite for all internet-edge BGP functionality described in this document. The single-agent BGP model lacks the control-plane scale needed for a full routing table and cannot run multi-protocol BGP at the required rate. Once enabled, this setting survives configuration rollback — it can only be reverted by reboot. Requires reboot; plan alongside the MDB profile change.
MDB profiles
Source: Arista DCS-7280R3K Series datasheet. All values for 7280R3K Series only (not 7280R3). Active profile highlighted.
| Profile | ARP | MAC | IPv4 Unicast | +FlexRoute | IPv6 Unicast | Multicast | TP ACL IPv4 | TP ACL IPv6 | ECMP |
|---|---|---|---|---|---|---|---|---|---|
l3 | 112k | 256k | 2250k | +2048k | 683–750k | 128k | 561k | 187k | 512-way |
l3-xl | 112k | 192k | 2850k | +1536k | 833–950k | 96k | 561k | 187k | 512-way |
l3-xxl ← MLAG | 112k | 192k | 2850k | +1536k | 833–950k | 96k | 374k | 125k | 512-way |
l3-xxxl (no MLAG) | 80k | 384k | 3950k | +3072k | 1100–1317k | 192k | 30k | 10k | 512-way |
balanced-xl | 96k | 256k | 1850k | +2048k | 567–617k | 128k | 561k | 187k | 512-way |
l3-host-xl | 192k | 256k | 1650k | +2048k | 500–550k | 128k | 561k | 187k | 512-way |
⚠️
l3-xxlvsl3-xl: same IPv4/IPv6 scale but Traffic Policy ACL capacity is reduced — 374k IPv4 prefixes (vs 561k) and 125k IPv6 (vs 187k). This is the trade-off for the additional unicast route capacity overl3. Verify that the deployed traffic-policy field-sets stay within these limits.
⚠️ Profile change requires reboot. Changing profile also resets TCAM resource allocation — all TCAM profiles must be re-verified after a profile change.
VoQ subinterface allocation
voq subinterface allocation 4096 — reserves VoQ entries for subinterfaces. Changes require reboot. Must be set before subinterfaces are created.
DSCP rewrite at MPLS encap
platform sand qos rewrite dscp mpls encapsulation hierarchical next-hop disabled — prevents MPLS push from overwriting DSCP already set by ingress traffic-policy. Required when DSCP pipe mode is used through MPLS tunnels.
QoS Subinterface
qos subinterface tx-queue count 8
qos subinterface scheduling parent round-robin
8 egress queues per subinterface. Round-robin parent scheduling ensures fairness between customer subinterfaces on a shared physical port. tx-queue count 8 must match the number of queues defined in the QoS profile applied to that subinterface.
Queue Monitor
Queue monitor is enabled in production for tracking queue depth and detecting microbursts.
queue-monitor length
queue-monitor length log 10
queue-monitor length log 10 — log queue depth events when threshold exceeded, with 10-second hysteresis.
QoS Profiles
Profiles define per-interface egress scheduling. Applied via service-profile <name> on the interface.
Trust model:
qos trust dscp— honour incoming DSCP (CORE/INFRA links)no qos trust— ignore DSCP; traffic-policy sets TC at ingress (customer/transit ports)
Queue allocation
| TC / Queue | DSCP | Traffic type | Queue type | CORE | CUST/TRANSIT |
|---|---|---|---|---|---|
| TC7 / Q7 | 56 (CS7) | Network control (BGP/BFD) | Strict priority (reserved) | strict priority | strict priority |
| TC6 / Q6 | 48 (CS6) | Network control | Strict priority (shaped) | shape 10% | shape 10% |
| TC5 / Q5 | 46/40 (EF) | Voice / real-time | Strict priority (shaped) | shape 50% | — |
| TC4 / Q4 | 26 (AF31) | Business / priority data | Round-robin (no priority) | 30% | 30% |
| TC3 / Q3 | 16 (AF21) | InetPlus / B2C, B2B clients | Round-robin (no priority) | 30% | 30% |
| TC2 / Q2 | 45 (NQB) | Non-queuing / VoIP signalling | Round-robin (no priority) | 19% | 19% |
| TC1 / Q1 | 0 (BE) | Best-effort internet (transit) | Round-robin (no priority) | 20% | 20% |
| TC0 / Q0 | 8 (CS1) | Scavenger / bulk | Round-robin (no priority) | 1% | 1% |
Queue type = the scheduler class for that tx-queue. On Sand/Jericho there are only two: strict priority (SP) and round-robin (RR). By default all queues are SP. Setting no priority on a queue demotes it — and every lower-numbered queue — to RR (see the boundary rule below). Here no priority on Q0–Q4 makes them the RR block, leaving Q5–Q7 as the SP block. See the limitations below for why the SP queues are shaped.
Queue scheduling model and limitations
Sand/Jericho QoS is more constrained than the per-queue table suggests. The relevant limits:
8 fixed queues (Q0–Q7), 1:1 with traffic class. TCn always maps to Qn; the queue index cannot be remapped. Only the DSCP→TC mapping is configurable (via
qos map, see previous section) — there is no way to add queues or change the TC-to-queue binding.Q7 is reserved for the control plane. It is hard-wired to TC7 (the TC7↔Q7 mapping is not editable) and always runs strict priority. BGP/BFD self-generated traffic is marked CS7 (see SGT QoS) precisely so it lands here. Historically Q7 was entirely non-configurable; as of EOS 4.33.0F (“Configurable Transmit Queue 7”) a limited set of Q7 knobs is exposed on Sand —
shape rateandlatency maximum(max-latency tail-drop threshold) only. Q7 stays strict priority; there is no bandwidth/priority control. Two limitations matter for this design:- Front-panel ports and LAGs only — not sub-interfaces. The customer/transit sub-interfaces on this router cannot carry Q7 config; only the physical port / port-channel can.
- Only on 8-Qpair interfaces. Verify with
show platform fap mapping interface ethernet <X>— theBaseQPair/QPairs value must be8.
Config is via
tx-queue 7under the interface (or aqos profileattached withservice-profile); remove withno tx-queue 7/default tx-queue 7. Platform-supported: 7500R/R2/R3, 7280R/R2/R3 (incl. this 7280SR3K), 7800R3.Only two scheduler classes exist: strict priority and round-robin. There are no intermediate priority tiers — a queue is either ahead of every RR queue, or in the RR group. All SP queues are serviced before any RR queue.
The SP/RR boundary is monotonic by queue number — SP must be the top contiguous block. All queues are SP by default. The moment you set any queue to
no priority, every lower-numbered queue is forced to RR as well, regardless of its own configuration. You cannot have an SP queue sitting below an RR queue. In practice this means RR is always the bottom block[Q0..Qk]and SP the top block[Qk+1..Q7]; the design’s split (RR Q0–Q4, SP Q5–Q7) is the only legal shape for that boundary. Trying to configure, say, Q3 as SP while Q4 is RR silently yields Q3 running RR.SP queues can starve RR queues, so they must be shaped. Because SP service is exhaustive, an unbounded EF (Q5) or management (Q6) queue could consume the whole port and starve data. That is why Q5 is
shape rate 50 percentand Q6shape rate 10 percent— a hard ceiling bounding the SP queues so headroom is left for the RR group. Q7 is left unshaped because control-plane volume is low and must never be dropped.bandwidth percentis an RR-only weight; SP queues cannot use it. SP queues take whatever they need up to their shaper; only RR queues honourbandwidth percent. This is why the CORE/CUST columns show a shape value for Q5–Q6 but a bandwidth percentage for Q0–Q4.RR bandwidth is a guaranteed minimum, normalised to ≤100%. The RR shares here sum to exactly 100% (1+20+19+30+30). If the configured sum exceeds 100%, EOS normalises each queue’s operational share to
configured ÷ cumulative. Below 100%, an RR queue may burst into unused capacity unless separately capped withshape rate.Subinterface QoS is a two-level hierarchy. The parent scheduler round-robins between subinterfaces (
qos subinterface scheduling parent round-robin); the 8 tx-queues are the child schedulers within each subinterface.qos subinterface tx-queue count 8must match the queue count in the applied profile (see QoS Subinterface).Egress queue depth is observed via VOQ, not a directly configurable buffer. Sand is a VOQ (ingress-buffered) architecture; the
VOQcounter is not user-configurable. Usequeue-monitor length(see Queue Monitor) to catch microbursts.
QPROF-CORE
qos profile QPROF-CORE
qos trust dscp
tx-queue 0
no priority
bandwidth percent 1
tx-queue 1
no priority
bandwidth percent 20
tx-queue 2
no priority
bandwidth percent 19
tx-queue 3
no priority
bandwidth percent 30
tx-queue 4
no priority
bandwidth percent 30
tx-queue 5
shape rate 50 percent ! EF — voice/RT, shaped to prevent starvation
tx-queue 6
shape rate 10 percent ! CS6 — management
QPROF-TRANSIT-UPSTREAM / QPROF-TRANSIT-CUSTOMER
qos profile QPROF-TRANSIT-UPSTREAM
no qos trust ! traffic-policy sets TC; DSCP from transit ignored
tx-queue 0
no priority
bandwidth percent 1
tx-queue 1
no priority
bandwidth percent 20
tx-queue 2
no priority
bandwidth percent 19
tx-queue 3
no priority
bandwidth percent 30
tx-queue 4
no priority
bandwidth percent 30
tx-queue 6
shape rate 10 percent
Profile-to-interface mapping
| Profile | Trust | Applied to |
|---|---|---|
| QPROF-CORE | DSCP | Core/backbone interfaces, MPLS links |
| QPROF-TRANSIT-UPSTREAM | None | IP transit provider ports |
| QPROF-TRANSIT-CUSTOMER | None | Customer IP transit ports |
| QPROF-UNKNOWN | None | Fallback / unclassified |
QoS Maps (DSCP → Traffic Class)
Applied when qos trust dscp is active. On untrusted interfaces, traffic-policy sets TC directly.
qos map dscp 1 2 3 4 5 6 7 17 18 19 20 21 22 23 24 25 27 28 29 30 31 32 33 34 35 36 37 38 39 41 42 43 44 47 49 50 51 52 53 54 55 57 58 59 60 61 62 63 to traffic-class 0
qos map dscp 45 to traffic-class 2 ! NQB → TC2 (signalling / non-queuing)
qos map dscp 16 to traffic-class 3 ! AF21 → TC3 (InetPlus / managed)
qos map dscp 26 to traffic-class 4 ! AF31 → TC4 (business / priority)
DSCP 0 (BE) → TC1 (BE internet), DSCP 8 (CS1) → TC0 (scavenger), DSCP 40/46 (EF) → TC5, DSCP 48 (CS6) → TC6, and DSCP 56 (CS7) → TC7 are handled by hardware default maps (not overridden here). The explicit list above catches all unmapped DSCPs → TC0.
Traffic Policies
Traffic policies are the primary QoS enforcement mechanism. Applied per-interface as ingress or egress classifiers.
Naming convention. Three distinct object types, three distinct prefixes — so a name tells you what it is at a glance:
| Prefix | Object | Example |
|---|---|---|
M- | traffic-policy match clause (a rule) | M-V4U-BGP-PEERS |
PFX- | prefix field-set (IPv4/IPv6 address data) | PFX-V4-ADMIN, PFX-V6-BOGONS |
L4P- | L4-port field-set (port-number data) | L4P-AMP-SRC-PORTS |
The V4/V6 token marks address family. A match clause and the field-set it references now never share a name (previously both were FS-…, e.g. the NTP rule and its prefix set collided).
Policy inventory
| Policy | Direction | Applied to | Key actions |
|---|---|---|---|
| TP-COPP | CPU (all VRFs) | Global | Police ICMP/NTP/DNS, count BGP/BFD/LDP, drop unknown |
| TP-TRANSIT-UPSTREAM | Ingress | Transit provider ports | Drop bogons, block IPv6 exthdr/frag, re-mark, AMP mitigation |
| TP-TRANSIT-CUSTOMER | Ingress | Customer transit | Drop bogons, anti-AMP, NQB to TC2, scavenger to TC0 |
| TP-TRANSIT-PNI | Ingress | Private peering / PNI | Drop bogons, allow BGP on LL, AMP mitigation |
TP-COPP
traffic-policies
cpu traffic-policy TP-COPP vrf all
traffic-policy TP-COPP
match M-V4U-ICMP ipv4
protocol icmp
actions
count
police rate 1024 kbps
match M-V6U-ICMP ipv6
protocol icmpv6
actions
count
police rate 1024 kbps
match M-V4U-BFD ipv4
source prefix field-set PFX-V4-BFD-PEERS PFX-V4-LL PFX-V4-ROUTERS-LO
protocol udp source port all destination port 3784-3785, 7784
actions
count
match M-V4U-BGP-PEERS ipv4
protocol neighbors bgp
actions
count
match M-V4M-LDP ipv4
destination prefix 224.0.0.2/32
protocol udp source port all destination port ldp
actions
count
match M-V4U-LDP ipv4
source prefix field-set PFX-V4-ROUTERS-LO
protocol tcp source port all destination port ldp
protocol udp source port all destination port ldp
actions
count
match M-V4U-SSH ipv4
source prefix field-set PFX-V4-ADMIN
protocol tcp source port all destination port ssh netconf-ssh
actions
count
match M-V4U-SNMP ipv4
source prefix field-set PFX-V4-ADMIN
protocol udp source port all destination port snmp
actions
count
match M-V4U-GRPC ipv4
source prefix field-set PFX-V4-ADMIN
protocol tcp source port all destination port gnmi
actions
count
match M-V4U-NTP ipv4
source prefix field-set PFX-V4-NTP
protocol udp source port ntp destination port all
actions
count
police rate 1024 kbps
match M-V4U-TCP-ESTABLISHED ipv4
protocol tcp flags established
actions
count
match M-V4U-TCP-SYN-ACK ipv4
protocol tcp flags ack syn
actions
count
police rate 1024 kbps
match M-V4U-DISCARD ipv4
actions
count
drop
match ipv4-all-default ipv4
match ipv6-all-default ipv6
AMP source ports field-set
field-set l4-port L4P-AMP-SRC-PORTS
17, 19, 111, 123, 137, 161, 389, 520, 751, 1434, 1900, 5353, 6881, 11211, 27015, 27960
! qotd, chargen, portmap/rpc, ntp, netbios, snmp, ldap, rip,
! kerberos, ms-sql, ssdp, mdns, bittorrent, memcached, srcds/quake
TP-TRANSIT-UPSTREAM (key elements)
Bogon and malformed-packet filtering is enforced in the hardware dataplane via traffic-policy, rather than in the BGP control plane via route-map. This drops invalid traffic at line rate before it reaches the routing process, and also prevents bogon-destined packets already in the forwarding table from being forwarded.
traffic-policy TP-TRANSIT-UPSTREAM
match M-V4U-BOGONS ipv4
destination prefix field-set PFX-V4-BOGONS
actions
count
drop
match M-V6U-BOGONS ipv6
destination prefix field-set PFX-V6-BOGONS
actions
count
drop
match M-V6U-EXTHD-BLOCK ipv6
protocol 0, 43, 60 ! HBH / routing / dest options
actions
count
drop
match M-V6U-FRAG-BLOCK ipv6
protocol 44
actions
count
drop
match M-V4U-UDP-FRAGMENT ipv4
protocol udp
fragment
actions
count
set dscp 8
set traffic class 0
match M-V4U-AMP-SOURCE-PORTS ipv4
protocol udp source port field-set L4P-AMP-SRC-PORTS
actions
count
set dscp 8
set traffic class 0
match M-V4U-SCAVENGER-BULK ipv4
protocol tcp source port all destination port ftp-data rsync ftps-data
actions
count
set dscp 8
set traffic class 0
match M-V4U-NBQ ipv4
dscp 34, 45
actions
count
set dscp 45
set traffic class 2
match M-V4U-INETPLUS ipv4
destination prefix field-set PFX-V4-INETPLUS
actions
count
set dscp 16
set traffic class 3
match M-V4U-INET ipv4
actions
count
set dscp 0
set traffic class 1
match ipv4-all-default ipv4
actions
count
set dscp 8
set traffic class 0
SGT QoS — Self-Generated Traffic
Control-plane traffic originated by the router must be explicitly DSCP-marked.
router bfd
qos dscp 56 ! BFD → CS7
router bgp <ASN>
bgp transport qos dscp 56 ! BGP TCP sessions → CS7
snmp-server qos dscp 26
dns qos dscp 26
ntp qos dscp 26
sflow qos dscp 26
logging qos dscp 26
management ssh
qos dscp 26
management api gnmi
transport grpc MGMT
qos dscp 26
transport grpc default
qos dscp 26
| Protocol | DSCP | CS/AF | TC | Rationale |
|---|---|---|---|---|
| BGP TCP | 56 | CS7 | 7 (strict) | Session drops cause routing black-holes |
| BFD | 56 | CS7 | 7 (strict) | Sub-second detection — cannot tolerate jitter |
| SNMP | 26 | AF31 | 4 | OAM, not real-time |
| DNS | 26 | AF31 | 4 | Router DNS queries |
| NTP | 26 | AF31 | 4 | Clock sync |
| sFlow | 26 | AF31 | 4 | Telemetry |
| Syslog | 26 | AF31 | 4 | Operational logging |
| SSH | 26 | AF31 | 4 | CLI access |
| gNMI/gRPC | 26 | AF31 | 4 | Streaming telemetry |
MPLS LDP
mpls ip
mpls ldp
router-id interface Loopback0 force ! stable RID; force = use even if non-primary
igp sync delay 2 ! RFC 5443: hold IS-IS max-metric 2s after LDP UP before restoring normal metric
interface disabled default ! LDP not enabled by default on interfaces
neighbor hello-redundancy ! dual-hello on LAG members
no shutdown
!
LDP must be explicitly enabled per interface. interface disabled default is critical — it prevents LDP from attempting sessions on all interfaces including external ones.
igp sync delay 2 is part of IGP–LDP synchronisation (RFC 5443). When an LDP session goes down, IS-IS advertises the affected link at maximum metric so traffic reroutes away from the label-less path. When the LDP session re-establishes, IS-IS holds the maximum metric for 2 seconds before restoring the normal metric — giving LDP time to complete label exchange before traffic is attracted back to the link.
interface EthernetX
mpls ip
mpls ldp interface
interface Loopback0
mpls ldp interface
MPLS ICMP
mpls icmp ttl-exceeded tunneling ! include original IP header in TTL-exceeded ICMP
mpls oam standard ietf ! RFC 4379 MPLS ping/trace
mpls icmp ipv6 source-interface Loopback0
mpls icmp ip source-interface Loopback0
| Parameter | Effect | Without it |
|---|---|---|
ttl-exceeded tunneling | ICMP includes original inner IP header | traceroute shows only MPLS hops |
oam standard ietf | RFC 4379 MPLS ping/trace | Vendor-specific OAM, incompatible with third-party |
source-interface Loopback0 | ICMP sourced from stable loopback | ICMP from transit link — unreachable if link fails |
Tunnel-ribs
Tunnel-ribs aggregate multiple label signalling protocols for BGP next-hop resolution with explicit preference ordering.
tunnel-ribs
tunnel-rib TR-SR-OVER-LDP
source-protocol bgp labeled-unicast preference 30 ! highest preference
source-protocol isis segment-routing preference 40
source-protocol ldp preference 50 ! lowest preference / fallback
Lower preference number = higher priority.
| Protocol | Preference | Description |
|---|---|---|
| bgp labeled-unicast | 30 | BGP-LU (inter-AS or explicit paths) |
| isis segment-routing | 40 | SR-ISIS node SIDs |
| ldp | 50 | LDP label bindings (fallback) |
The name TR-SR-OVER-LDP reflects a migration scenario: SR-ISIS is preferred, LDP serves as fallback. Removing LDP from the rib when migration is complete requires no BGP policy changes.
sFlow
sflow sample 512 ! 1:512 sampling rate
sflow destination <SFLOW-COLLECTOR> ! collector (e.g. Akvorado)
sflow source <LOOPBACK0> ! Loopback0 — stable, matches router-id
sflow sample input subinterface
sflow sample output subinterface
sflow extension bgp ! add BGP attrs to flow records
sflow qos dscp 26
sflow interface disable default ! explicit opt-in per interface
no sflow hardware acceleration
sflow run
Enable per-interface:
interface EthernetX
sflow enable
sflow egress enable
sflow extension bgp adds BGP attributes to flow records: peer AS, origin AS, AS path, next-hop, communities. Requires BGP running; tools like Akvorado use these for AS-level traffic matrices.
no sflow hardware acceleration is required alongside it: hardware acceleration offloads sampling to the ASIC for higher raw sample rates, but conflicts with BGP attribute annotation and per-subinterface sampling (sflow sample input/output subinterface) simultaneously.
The trade-off is intentional — richer per-flow records for AS-level traffic analysis outweigh the raw rate benefit.
Maintenance Mode
Two maintenance modes: SOFT (graceful drain with BGP prepend + GSHUT) and HARD (immediate withdraw of all external routes).
On boot, MNT-UN-SOFT runs automatically for 120 seconds to allow BGP reconvergence before accepting traffic.
Structure
| Component | Type | Purpose |
|---|---|---|
| MNT-UN-SOFT | Unit | Soft drain — on-boot + manual |
| MNT-UN-HARD-EXT | Unit | Hard withdraw — emergency |
| MNT-GR-SOFT-VRF-DEFAULT-INT | Group | iBGP peers for soft drain |
| MNT-GR-SOFT-VRF-DEFAULT-EXT | Group | eBGP peers for soft drain |
| MNT-GR-HARD-VRF-DEFAULT-EXT | Group | eBGP peers for hard withdraw |
maintenance
profile bgp MNT-PR-SOFT-EXT
initiator route-map RP-ALL-EXT-SOFT-MNT-OUT out
!
profile bgp MNT-PR-HARD-EXT
initiator route-map RP-ALL-EXT-HARD-MNT-OUT out
initiator route-map RP-ALL-EXT-HARD-MNT-IN in
!
profile bgp MNT-PR-SOFT-INT
initiator route-map RP-ALL-INT-SOFT-MNT-OUT out
!
profile unit MNT-UN-SOFT
on-boot duration 120
!
unit MNT-UN-SOFT
group bgp MNT-GR-SOFT-VRF-DEFAULT-EXT
group bgp MNT-GR-SOFT-VRF-DEFAULT-INT
!
unit MNT-UN-HARD-EXT
group bgp MNT-GR-HARD-VRF-DEFAULT-EXT
Route maps
! SOFT EXTERNAL — prepend own ASN x5 + GSHUT community (RFC 8326)
route-map RP-ALL-EXT-SOFT-MNT-OUT permit 10
set as-path prepend auto repeat 5
set community community-list CMST-SPECIAL-GSHUT additive
! SOFT INTERNAL — prepend with dummy 65500 ASN x5 + maintenance large-community
route-map RP-ALL-INT-SOFT-MNT-OUT permit 10
set as-path prepend 65500 repeat 5
set community community-list CMST-SPECIAL-GSHUT additive
set large-community large-community-list CMLR-SPECIAL-MNT-SOFT additive
! HARD EXTERNAL — deny all
route-map RP-ALL-EXT-HARD-MNT-OUT deny 10
route-map RP-ALL-EXT-HARD-MNT-IN deny 10
! HARD INTERNAL — add large-community then deny (except own aggregates)
route-map RP-ALL-INT-HARD-MNT-OUT permit 10
set large-community large-community-list CMLR-SPECIAL-MNT-HARD additive
continue 20
route-map RP-ALL-INT-HARD-MNT-OUT deny 20
match ip address prefix-list PL-V4U-AS<ASN>
route-map RP-ALL-INT-HARD-MNT-OUT deny 30
match ipv6 address prefix-list PL-V6U-AS<ASN>
Operations
! Enter soft maintenance
configure
maintenance
unit MNT-UN-SOFT
quiesce
! Exit maintenance
configure
maintenance
unit MNT-UN-SOFT
no quiesce
show maintenance
show maintenance unit MNT-UN-SOFT
show bgp neighbor | include maintenance
BGP Transport & Session Parameters
router bgp <ASN>
bgp asn notation asdot
router-id <LOOPBACK0>
update wait-install ! hold updates until FIB install complete
bgp transport pmtud ! Path MTU Discovery on BGP TCP
bgp transport qos dscp 56 ! BGP TCP → CS7
bgp always-compare-med ! compare MED across ASes
update wait-install
When a large routing table is received, BGP can converge and begin advertising routes before the hardware FIB has finished programming them. update wait-install breaks this race: outbound BGP updates are held until the FIB install is confirmed, preventing the router from attracting traffic it cannot yet forward. This increases convergence time under FIB pressure — monitor hardware programming status on Jericho2 with show platform sand l3 summary | grep backlog. A non-zero backlog value means routes are still queued for hardware programming and BGP updates are being held.
show platform sand l3 summary | grep backlog
Routes: 1050278 backlog: 0 unprogrammed: 0
Adjacencies: 327 backlog: 0 unprogrammed: 0
Routes: 242681 backlog: 0 unprogrammed: 0
Adjacencies: 327 backlog: 0 unprogrammed: 0
Routes: 59 backlog: 0 unprogrammed: 0
Adjacencies: 23 backlog: 0 unprogrammed: 0
Egress rewrite chains in backlog: resource-full 0, no-interface 0
route refresh demarcated stale-path removal
neighbor <peer-group> route refresh demarcated stale-path removal
Route Refresh (RFC 2918) lets a BGP speaker request its peer to re-advertise its full routing table — used after an inbound policy change to re-evaluate all received routes. The problem: basic Route Refresh has no convergence signal. The local router cannot tell when the peer has finished re-sending, so paths that the peer silently stopped advertising (without an explicit withdraw) can persist in the RIB indefinitely.
RFC 7313 (Enhanced Route Refresh) adds two markers: BoRR (Begin of Route Refresh) sent before the re-advertisement starts, and EoRR (End of Route Refresh) sent when complete. With stale-path removal, EOS tracks which paths were re-advertised during the refresh window and, upon receiving EoRR, purges any paths from that peer that were not re-sent. This enforces a clean post-refresh RIB state equivalent to a hard reset, but without dropping the session.
Practically: after changing an inbound filter, trigger clear ip bgp <peer> soft in — paths that no longer match the new policy and that the peer does not re-advertise will be removed automatically when the refresh completes.
out-delay
| Peer type | out-delay | Rationale |
|---|---|---|
| iBGP (border full-mesh) | 3s | Batch concurrent updates from multiple upstreams before propagating to iBGP |
| iBGP (RR client / PE) | 1s | Faster reconvergence for PE nodes |
| eBGP transit / peering | 6s | Absorb micro-instabilities before exporting |
rib-in pre-policy retain all
neighbor <transit-peer-group> rib-in pre-policy retain all
By default, EOS discards the pre-policy adj-RIB-In after applying inbound filters — only the post-policy result is kept in memory. rib-in pre-policy retain all keeps the full raw table as received from the peer, before any filters are applied.
Two consequences:
Policy changes without route-refresh. When an inbound RCF or route-map is modified, EOS can re-evaluate all received routes immediately from the cached pre-policy table — no need to request a Route Refresh from the peer. For a transit session carrying 900k+ prefixes, a Route Refresh triggers a full re-advertisement that takes minutes and causes a reconvergence event across the network. With retain all, the policy change takes effect locally in seconds without touching the peer.
BMP and gNMI pre-policy export. BMP (see BMP Monitoring section) streams both pre-policy and post-policy adj-RIB-In to collectors. Without retain all, there is no pre-policy data to export — only the post-policy view is available. Same applies to gNMI adj-rib-in-pre subscriptions.
Cost: roughly doubles BGP memory consumption for those peer sessions. Apply selectively to transit and peering sessions where policy re-evaluation matters; do not apply globally. Monitor with show bgp memory.
BGP Bestpath
router bgp <ASN>
maximum-paths 4 ecmp 4
bgp additional-paths install
bgp always-compare-med
no bgp bestpath as-path multipath-relax
| Parameter | Behaviour |
|---|---|
maximum-paths 4 ecmp 4 | Install up to 4 equal-cost paths in the FIB and perform ECMP across them. Without this, only the single best path is forwarded. |
bgp additional-paths install | Pre-install the best backup path in the FIB alongside the primary. Enables sub-second failover (BGP PIC) without waiting for BGP re-convergence — see §23. |
bgp always-compare-med | Compare MED between routes from different ASes. Without this, MED is only compared within the same AS-PATH. |
no bgp bestpath as-path multipath-relax | ECMP only across paths with identical AS-path, not just equal length. Prevents unintended load-balancing between different transit upstreams and between fabric sites with distinct AS-paths. |
Peer Groups & Peers
All peers use named peer-groups. Direct per-peer config is limited to IP address, description, and optional BFD/passive/password.
Peer group taxonomy
| Group | AF | Type | Key attributes |
|---|---|---|---|
BGR-V4U-INT-IBGPFM-RR | IPv4 | iBGP full-mesh RR | update-source Lo0, add-paths, out-delay 3 |
BGR-V6U-INT-IBGPFM-RR | IPv6 | iBGP full-mesh RR | same |
BGR-V4U-INT-RRC-CLIENT | IPv4 | iBGP RR client | route-reflector-client, out-delay 1 |
BGR-V6U-INT-RRC-CLIENT | IPv6 | iBGP RR client | same |
BGR-V4U-EXT-TRANSIT-* | IPv4 | eBGP upstream transit | remove-private-as, out-delay 6, rib-in retain |
BGR-V6U-EXT-TRANSIT-* | IPv6 | eBGP upstream transit | same |
BGR-V4U-EXT-RS-* | IPv4 | IXP route server | no enforce-first-as, maximum-routes 0, out-delay 6 |
BGR-V6U-EXT-RS-* | IPv6 | IXP route server | same |
BGR-V4U-EXT-PEER-* | IPv4 | Direct peering | out-delay 6, max-routes per peer |
BGR-V6U-EXT-PEER-* | IPv6 | Direct peering | same |
BGR-V4U-EXT-CUST-* | IPv4 | Customer / IP transit | send-community, passive where applicable |
BGR-V6U-EXT-CUST-* | IPv6 | Customer / IP transit | same |
Peer group definition example
router bgp <ASN>
neighbor BGR-V4U-EXT-TRANSIT-PROVIDER peer group
neighbor BGR-V4U-EXT-TRANSIT-PROVIDER remote-as <upstream-ASN>
neighbor BGR-V4U-EXT-TRANSIT-PROVIDER send-community standard large
neighbor BGR-V4U-EXT-TRANSIT-PROVIDER maximum-routes 1048576 warning-limit 80 percent warning-only
neighbor BGR-V4U-EXT-TRANSIT-PROVIDER route refresh demarcated stale-path removal
neighbor BGR-V4U-EXT-TRANSIT-PROVIDER remove-private-as all replace-as
neighbor BGR-V4U-EXT-TRANSIT-PROVIDER out-delay 6 changes
neighbor BGR-V4U-EXT-TRANSIT-PROVIDER rib-in pre-policy retain all
neighbor X.X.X.X peer group BGR-V4U-EXT-TRANSIT-PROVIDER
neighbor X.X.X.X description TRANSIT-PROVIDER-A
remove-private-as all strips private ASNs from the AS-path before sending to an upstream. replace-as goes further: instead of leaving a gap, it substitutes the local AS number in place of each removed private ASN. This ensures the AS-path is never empty or shorter than expected when the upstream receives it — some ISPs reject routes with an empty AS-path or apply stricter filtering to them.
maximum-routes 0 (unlimited) is sometimes recommended for full-table transit peers to avoid session drops as the internet table grows. 1048576 warning-limit 80 percent warning-only is used here instead: the session never drops, but a syslog warning fires at ~838k prefixes — providing early notice of abnormal peer behaviour (route leak, misconfigured customer) before it becomes a capacity problem.
IXP route server
neighbor BGR-V4U-EXT-RS-IXNAME peer group
neighbor BGR-V4U-EXT-RS-IXNAME remote-as <RS-ASN>
no neighbor BGR-V4U-EXT-RS-IXNAME enforce-first-as ! RS doesn't prepend its ASN
neighbor BGR-V4U-EXT-RS-IXNAME maximum-routes 0
neighbor BGR-V4U-EXT-RS-IXNAME remove-private-as all replace-as
BFD on eBGP peers
neighbor <IP> bfd damping
neighbor <IP> bfd interval 300 min-rx 300 multiplier 4
! damping: suppress BFD flaps from triggering BGP session churn
! 300ms × 4 = 1.2s detection time
The standard internet-edge recommendation is 500 ms × 3 (1.5 s detection). Timers here are tightened to 300 ms × 4 (1.2 s) to detect link failures faster. bfd damping is added to prevent rapid BFD oscillations on unstable links from cycling the BGP session — without it, a flapping physical link would repeatedly reset BGP, causing unnecessary route churn across the network.
RCF — Routing Control Functions
RCF replaces route-maps for BGP policy. Functions are written in EOS RCF language and applied as rcf in <function>() / rcf out <function>().
Policy by peer type
| Peer type | Inbound RCF | Outbound RCF | Key logic |
|---|---|---|---|
| iBGP border | allow_all() | rp_nhs_cisco_style() | Accept all; set NH to self Cisco-style |
| iBGP RR client | RP_V4U_INT_RRC_REDUCED_IN() | RP_V4U_INT_RRC_REDUCED_OUT() | Reduced table to PEs |
| Upstream transit | RP_V4U_TRANSIT_<X>_IN() | RP_V4U_TRANSIT_<X>_OUT() | Reject bogons/own; export own + communities |
| IXP route server | RP_V4U_RS_<IX>_IN_<LOC>() | RP_V4U_RS_<IX>_OUT_<LOC>() | Filter bogons/private; export own with LC |
| Direct peer | RP_V4U_PEER_<X>_IN_<IXP>() | RP_V4U_PEER_<X>_OUT_<IXP>() | Strict max-prefix; own prefixes out |
| Customer transit | RP_V4U_CUST_<X>_IN() | RP_V4U_CUST_<X>_OUT() | Customer’s own prefixes in; default + table out |
RCF application example
address-family ipv4
neighbor BGR-V4U-EXT-TRANSIT-<PROVIDER-A> rcf in RP_V4U_TRANSIT_<PROVIDER-A>_IN()
neighbor BGR-V4U-EXT-TRANSIT-<PROVIDER-A> rcf out RP_V4U_TRANSIT_<PROVIDER-A>_OUT()
neighbor BGR-V4U-EXT-PEER-<PEER-A-IX> rcf in RP_V4U_PEER_<PEER-A>_IN_IX_<LOC>()
neighbor BGR-V4U-EXT-PEER-<PEER-A-IX> rcf out RP_V4U_PEER_<PEER-A>_OUT_IX_<LOC>()
neighbor BGR-V4U-EXT-CUST-<CUSTOMER> rcf in RP_V4U_CUST_<CUSTOMER>_IN()
neighbor BGR-V4U-EXT-CUST-<CUSTOMER> rcf out RP_V4U_CUST_<CUSTOMER>_OUT()
neighbor BGR-V4U-EXT-TOOLS-SCRUB rcf in deny_all()
neighbor BGR-V4U-EXT-TOOLS-SCRUB rcf out RP_V4U_TOOLS_SCRUB_OUT()
BGP Missing-policy Safety
RFC 8212 defines that a BGP speaker should deny routes in both directions when no policy is configured — EOS does not implement this by default and requires explicit configuration. action deny is per-direction: a peer with only an inbound policy will not have its outbound silently blocked. This is intentional — monitoring and looking-glass peer-groups carry no outbound policy by design and should not be affected. The alternative action deny-in-out would deny both directions whenever either direction lacks a policy, which is too broad for a multi-purpose border router.
address-family ipv4
bgp missing-policy direction in action deny
bgp missing-policy direction out action deny
address-family ipv6
bgp missing-policy direction in action deny
bgp missing-policy direction out action deny
A new peer-group without RCF assignments will have 0 prefixes in/out — this is intentional and forces explicit policy before a session becomes operational.
BGP Add-paths (iBGP)
Add-paths (RFC 7911) enables BGP Prefix Independent Convergence (PIC). Without it, a next-hop failure triggers a full BGP re-convergence scan across the entire routing table — on a full internet table this takes seconds. With bgp additional-paths install, a backup path is pre-installed in the FIB alongside the primary. When the primary next-hop fails, the dataplane switches to the backup immediately without waiting for BGP to rerun bestpath selection.
router bgp <ASN>
bgp additional-paths install
! iBGP border full-mesh
neighbor BGR-V4U-INT-IBGPFM-BORDER additional-paths receive
neighbor BGR-V4U-INT-IBGPFM-BORDER additional-paths send backup
! RR clients — reflect best + backup to PEs
neighbor BGR-V4U-INT-RRC-CLIENT additional-paths receive
neighbor BGR-V4U-INT-RRC-CLIENT additional-paths send backup
| Mode | Behaviour | Use case |
|---|---|---|
any | Send all paths with unique path-id | RR reflecting all paths |
backup | Send best + first non-best only | Border routers — backup for fast-reroute |
all | All best paths per ECMP group | ECMP load balancing |
send backup installs the backup path in FIB for sub-second failover when the primary NH fails — before BGP re-converges.
NH Resolution via Tunnel-rib
BGP next-hops resolved through MPLS tunnels. Applied as a second router bgp stanza.
router bgp <ASN>
address-family ipv4
next-hop resolution ribs tunnel-rib TR-SR-OVER-LDP system-connected
address-family ipv6
next-hop resolution ribs tunnel-rib TR-SR-OVER-LDP system-connected
address-family vpn-ipv4
next-hop resolution ribs tunnel-rib TR-SR-OVER-LDP system-connected
address-family vpn-ipv6
next-hop resolution ribs tunnel-rib TR-SR-OVER-LDP system-connected
BMP Monitoring
BMP (BGP Monitoring Protocol, RFC 7854) streams BGP session state and adj-RIB updates to external collectors for offline analysis, route leak detection, and policy auditing. Two collectors are deployed for redundancy. Both pre-policy and post-policy exports are enabled — pre-policy shows exactly what the peer advertised before any filters, post-policy shows what was accepted. bgp rib bestpaths disabled reduces collector load by suppressing loc-RIB bestpath streaming; the collectors reconstruct bestpath from the per-peer adj-RIB feeds. rib-in pre-policy retain all on transit peer-groups (§18) is a prerequisite for meaningful pre-policy export.
router bgp <ASN>
monitoring station bmp-collector-site-a
update-source Loopback0
statistics
connection address <BMP-COLLECTOR-1-IP>
connection mode active port 1790
connection keepalive 30 3 10 ! interval 30s, retries 3, timeout 10s
export-policy received routes pre-policy post-policy
export-policy bgp rib bestpaths disabled
!
monitoring station bmp-collector-site-b
update-source Loopback0
statistics
connection address <BMP-COLLECTOR-2-IP>
connection mode active port 1790
connection keepalive 30 3 10
export-policy received routes pre-policy post-policy
export-policy bgp rib bestpaths disabled
Export policy
| Parameter | Value | Meaning |
|---|---|---|
received routes pre-policy | enabled | Stream adj-rib-in-pre (raw from peer) |
received routes post-policy | enabled | Stream adj-rib-in-post (after our filters) |
bgp rib bestpaths | disabled | Do not stream loc-rib bestpaths (reduces load) |
BMP message types (RFC 7854)
| Message | Trigger | Contains |
|---|---|---|
| Route Monitoring | BGP UPDATE send/receive | BGP UPDATE with path attributes |
| Statistics Report | Periodic | Prefix counters, rejected routes |
| Peer Down | Session down | Peer state, reason code |
| Peer Up | Session established | OPEN messages, capabilities |
| Initiation | BMP session start | Router sysDescr, sysName |
show bgp monitoring station
show bgp monitoring station bmp-collector-site-a
show bgp monitoring statistics
show bgp monitoring peer
BGP-LG Role
EOS has a built-in bgplg role for read-only BGP lookups. The role definition is explicit and uses CLI command ACLs.
User
username bgplg privilege 15 role bgplg secret sha512 <hash>
Role definition
role bgplg
5 permit mode exec command terminal width
6 permit mode exec command terminal length
10 permit mode exec command ping
20 permit mode exec command traceroute
30 permit mode exec command show ip bgp
40 permit mode exec command show ipv6 bgp
50 permit mode exec command show bgp
60 permit mode exec command show route
70 deny command .* ! deny everything else
iBGP peer group for bgplg
A dedicated iBGP peer-group BGR-ALL-INT-IBGP-BGPLG-LG is used to feed a local route-reflector session for the looking glass. This allows the bgplg user to query a local BGP view without exposing the full RIB or requiring a separate session to a route server.
router bgp <ASN>
neighbor BGR-ALL-INT-IBGP-BGPLG-LG peer group
neighbor BGR-ALL-INT-IBGP-BGPLG-LG remote-as <ASN>
neighbor BGR-ALL-INT-IBGP-BGPLG-LG update-source Loopback0
no neighbor BGR-ALL-INT-IBGP-BGPLG-LG additional-paths send
neighbor BGR-ALL-INT-IBGP-BGPLG-LG send-community standard extended large
neighbor BGR-ALL-INT-IBGP-BGPLG-LG maximum-routes 10
neighbor BGR-ALL-INT-IBGP-BGPLG-LG route refresh demarcated stale-path removal
maximum-routes 10 — safety limit; the LG peer-group is for query, not full table distribution.
Show Tech-Support Optimization
On a router carrying a full internet routing table, the default show tech-support output includes BGP RIB and routing table commands that can take 20+ minutes to complete and produce several hundred MB of text. This makes the bundle impractical for routine collection and can block the CLI during time-sensitive troubleshooting.
EOS allows individual show commands to be excluded from the tech-support bundle via management tech-support. The commands that generate disproportionately large output on internet-edge routers are the full BGP RIB dumps and routing table listings — these are better captured on-demand when needed rather than included in every tech-support.
management tech-support
show commands exclude
show ip bgp
show ip bgp detail
show ip bgp neighbors detail
show ipv6 bgp
show ipv6 bgp detail
show ipv6 bgp neighbors detail
show ip route
show ipv6 route
show ip route detail
show ipv6 route detail
For regular offline capture, schedule a daily compressed tech-support to flash:
schedule DAILY-TECH-SUPPORT
interval 1440
timeout 120
command show tech-support | gzip > flash:tech-support-latest.log.gz
The interval 1440 fires once per day (minutes). timeout 120 prevents the job from blocking indefinitely if the system is under load. The output file is overwritten each run — for retention, use a syslog-forwarded copy or replace the filename with a datestamped path via a bash alias.