Bmp on Soap & Bitshttps://soapandbits.net/tags/bmp/Recent content in Bmp on Soap & BitsHugoenMon, 06 Jul 2026 00:00:00 +0000Arista 7280SR3K Internet Edge Border Router: Design and Configuration Referencehttps://soapandbits.net/posts/2026-07-09-arista-border/Mon, 06 Jul 2026 00:00:00 +0000https://soapandbits.net/posts/2026-07-09-arista-border/Complete design and configuration guide for an Arista DCS-7280SR3K internet edge border router: TCAM allocation, hardware counters, QoS, MPLS/LDP, BGP scale and convergence, RCF policy, BMP telemetry, and operational optimizations for Jericho2C-based platforms.This document covers the design and configuration of an Arista DCS-7280SR3K series border router at an internet edge. It addresses hardware platform constraints (TCAM allocation, counter feature groups), QoS policy design, MPLS/LDP configuration, and all BGP-layer concerns: session parameters, bestpath policy, peer groups, routing control functions (RCF), convergence optimizations, and BMP telemetry. The platform uses a Jericho2C ASIC and runs EOS with the service routing protocols model multi-agent prerequisite — required for internet-scale BGP.

Platform: DCS-7280SR3K series (Jericho2C ASIC) Role: Internet Border / Transit / Peering

CLI Prompt with Timestamp

EOS prompt includes hostname and real-time timestamp for CLI session auditing.

prompt %H_%D{%y.%m.%d-%H:%M:%S}%P
Token Expands to Example
%H Hostname <hostname>
%D{...} strftime date/time 25.07.03-14:23:01
%P Privilege indicator #

Result: <hostname>_25.07.03-14:23:01#

Hardware TCAM Allocation

⚠️ Consult Arista SE before modifying TCAM profiles. Total key budget is fixed per ASIC. Misconfiguration causes silent TCAM overflow. Verify with show hardware capacity after any profile change.

Two profiles defined: BR-BLUE and BR-RED. Active system profile is BR-RED.

Rationale for keeping two profiles: Both profiles remain defined on the device to enable fast rollback. If the active profile (RED) causes unexpected TCAM behavior after a reboot, the operator can switch back to the previous profile (BLUE) with a single system profile change and reboot — no config reconstruction required. Only the system profile line needs to change; the profile definitions themselves stay intact.

hardware tcam
   profile BR-RED
      feature evpn mpls irb
         key size limit 160
      !
      feature evpn mpls multi-homing
         key size limit 160
      !
      feature l2-protocol forwarding
         sequence 95
         key size limit 160
         key field dst-mac vlan-tag-format
         action redirect-to-cpu snoop
         packet ipv4 forwarding bridged
         packet ipv6 forwarding bridged
         packet non-ip forwarding bridged
         packet non-ip forwarding bridged sub-interface
      !
      feature mpls
         sequence 5
         key size limit 160
         action drop redirect set-ecn
         packet ipv4 mpls ipv4 forwarding mpls decap
         packet ipv4 mpls ipv6 forwarding mpls decap
         packet mpls ipv4 forwarding mpls
         packet mpls ipv6 forwarding mpls
         packet mpls non-ip forwarding mpls
      !
      feature mpls pop ingress
         sequence 90
      !
      feature sflow subintf
         action sample
         packet ipv4 forwarding bridged
         packet ipv4 forwarding routed
         packet ipv6 forwarding bridged
         packet ipv6 forwarding routed
         packet mpls forwarding bridged decap
         packet non-ip forwarding bridged
      !
      feature traffic-policy cpu ipv4
         key size limit 160
         key field dst-ip ip-frag ip-protocol l4-dst-port l4-src-port src-ip tcp-control ttl
         action count set-drop-precedence set-policer
      !
      feature traffic-policy cpu ipv6
         key field dst-ipv6 ipv6-next-header l4-dst-port l4-src-port src-ipv6-high src-ipv6-low tcp-control ttl
         action count set-drop-precedence set-policer
      !
      feature traffic-policy port ipv4
         sequence 45
         key field dscp dst-ip-label ip-frag ip-fragment-offset ip-length ip-protocol l4-dst-port-label l4-src-port-label src-ip-label tcp-control ttl
         action count drop redirect set-drop-precedence set-dscp set-tc set-unshared-policer
         packet ipv4 forwarding routed
         packet mpls ipv4 forwarding bridged
         packet mpls ipv4 forwarding mpls
         packet mpls ipv4 forwarding routed decap
      !
      feature traffic-policy port ipv6
         sequence 55
         key field dst-ipv6-label hop-limit icmp-type-code ipv6-length ipv6-next-header ipv6-traffic-class l4-dst-port-label l4-src-port-label src-ipv6-label tcp-control
         action count drop redirect set-drop-precedence set-dscp set-tc set-unshared-policer
         packet ipv6 forwarding routed
         packet mpls ipv6 forwarding bridged
         packet mpls ipv6 forwarding mpls
         packet mpls ipv6 forwarding routed decap
      !
      feature traffic-policy port ipv4 egress
         key size limit 160
         key field dscp dst-ip-label ip-frag ip-protocol l4-dst-port-label l4-src-port-label src-ip-label tcp-control
         action count drop set-dscp
         packet ipv4 forwarding routed
      !
      feature traffic-policy port ipv6 egress
         key field dst-ipv6-label ipv6-next-header l4-dst-port-label l4-src-port-label src-ipv6-label tcp-control
         action count drop set-dscp
         packet ipv6 forwarding routed
   system profile BR-RED

Verification

show hardware tcam profile
show hardware tcam profile detail
show hardware capacity

Hardware Counter Features

Hardware counters consume entries from a shared pool. Some features are mutually exclusive.

hardware counter feature traffic-policy cpu
hardware counter feature traffic-policy out
hardware counter feature traffic-policy in
hardware counter feature ip in
hardware counter feature subinterface out
hardware counter feature subinterface in

Mutual exclusivity

⚠️ Verify against the Arista TOI for your exact EOS version. Counter bank conflicts and supported combinations change between releases.

Checking active counters

Use show hardware counter feature to verify which counter banks are active and which ASIC engine slots they occupy. All configured features must show up; any down or absent entry means the bank failed to allocate (resource conflict or reboot required).

show hardware counter feature 
Feature                   Direction        Counter Resource (Engine)        Status        Detail                                              
------------------------- ---------------- -------------------------------- ------------- ----------------------------------------------------
ACL-IPv4                  out              Jericho2C: 18                    up                                                                
ACL                       in               Jericho2C: 8, 19                 up                                                                
Traffic policy CPU        n/a              Jericho2C: 10, 23                up                                                                
Traffic policy            out              Jericho2C: 24                    up                                                                
Traffic policy            in               Jericho2C: 9, 20                 up                                                                
IPv4, IPv6                in               Jericho2C: 0                     up                                                                
QoS policer               in               Jericho2C: 21, 22                up            Configured with QoS policy-map class police action. 
Queue                     out              Jericho2C: 16                    up            Not user-configurable.                              
Subinterface              out              Jericho2C: 1                     up                                                                
Subinterface              in               Jericho2C: 2                     up                                                                
VOQ                       in               Jericho2C: 17                    up            Not user-configurable.

Additional verification:

show traffic-policy counters interface <intf>

Enabling a counter feature may require reboot. A feature listed as “not active” after reboot indicates a resource conflict.

Counter poll interval

traffic-policies
   counter interface poll interval 15 seconds

Load-interval

load-interval default 5

Sets interface rate averaging window to 5 seconds (default is 300s). Provides near-real-time rates for show interfaces, and SNMP ifInOctets rate calculation.

Platform Sand / VoQ

platform sand mdb profile l3-xxl
platform sand voq subinterface allocation 4096
platform sand qos rewrite dscp mpls encapsulation hierarchical next-hop disabled

Also required:

service routing protocols model multi-agent

service routing protocols model multi-agent is a prerequisite for all internet-edge BGP functionality described in this document. The single-agent BGP model lacks the control-plane scale needed for a full routing table and cannot run multi-protocol BGP at the required rate. Once enabled, this setting survives configuration rollback — it can only be reverted by reboot. Requires reboot; plan alongside the MDB profile change.

MDB profiles

Source: Arista DCS-7280R3K Series datasheet. All values for 7280R3K Series only (not 7280R3). Active profile highlighted.

Profile ARP MAC IPv4 Unicast +FlexRoute IPv6 Unicast Multicast TP ACL IPv4 TP ACL IPv6 ECMP
l3 112k 256k 2250k +2048k 683–750k 128k 561k 187k 512-way
l3-xl 112k 192k 2850k +1536k 833–950k 96k 561k 187k 512-way
l3-xxl ← MLAG 112k 192k 2850k +1536k 833–950k 96k 374k 125k 512-way
l3-xxxl (no MLAG) 80k 384k 3950k +3072k 1100–1317k 192k 30k 10k 512-way
balanced-xl 96k 256k 1850k +2048k 567–617k 128k 561k 187k 512-way
l3-host-xl 192k 256k 1650k +2048k 500–550k 128k 561k 187k 512-way

⚠️ l3-xxl vs l3-xl: same IPv4/IPv6 scale but Traffic Policy ACL capacity is reduced — 374k IPv4 prefixes (vs 561k) and 125k IPv6 (vs 187k). This is the trade-off for the additional unicast route capacity over l3. Verify that the deployed traffic-policy field-sets stay within these limits.

⚠️ Profile change requires reboot. Changing profile also resets TCAM resource allocation — all TCAM profiles must be re-verified after a profile change.

VoQ subinterface allocation

voq subinterface allocation 4096 — reserves VoQ entries for subinterfaces. Changes require reboot. Must be set before subinterfaces are created.

DSCP rewrite at MPLS encap

platform sand qos rewrite dscp mpls encapsulation hierarchical next-hop disabled — prevents MPLS push from overwriting DSCP already set by ingress traffic-policy. Required when DSCP pipe mode is used through MPLS tunnels.

QoS Subinterface

qos subinterface tx-queue count 8
qos subinterface scheduling parent round-robin

8 egress queues per subinterface. Round-robin parent scheduling ensures fairness between customer subinterfaces on a shared physical port. tx-queue count 8 must match the number of queues defined in the QoS profile applied to that subinterface.

Queue Monitor

Queue monitor is enabled in production for tracking queue depth and detecting microbursts.

queue-monitor length
queue-monitor length log 10

queue-monitor length log 10 — log queue depth events when threshold exceeded, with 10-second hysteresis.

QoS Profiles

Profiles define per-interface egress scheduling. Applied via service-profile <name> on the interface.

Trust model:

  • qos trust dscp — honour incoming DSCP (CORE/INFRA links)
  • no qos trust — ignore DSCP; traffic-policy sets TC at ingress (customer/transit ports)

Queue allocation

TC / Queue DSCP Traffic type Queue type CORE CUST/TRANSIT
TC7 / Q7 56 (CS7) Network control (BGP/BFD) Strict priority (reserved) strict priority strict priority
TC6 / Q6 48 (CS6) Network control Strict priority (shaped) shape 10% shape 10%
TC5 / Q5 46/40 (EF) Voice / real-time Strict priority (shaped) shape 50% —
TC4 / Q4 26 (AF31) Business / priority data Round-robin (no priority) 30% 30%
TC3 / Q3 16 (AF21) InetPlus / B2C, B2B clients Round-robin (no priority) 30% 30%
TC2 / Q2 45 (NQB) Non-queuing / VoIP signalling Round-robin (no priority) 19% 19%
TC1 / Q1 0 (BE) Best-effort internet (transit) Round-robin (no priority) 20% 20%
TC0 / Q0 8 (CS1) Scavenger / bulk Round-robin (no priority) 1% 1%

Queue type = the scheduler class for that tx-queue. On Sand/Jericho there are only two: strict priority (SP) and round-robin (RR). By default all queues are SP. Setting no priority on a queue demotes it — and every lower-numbered queue — to RR (see the boundary rule below). Here no priority on Q0–Q4 makes them the RR block, leaving Q5–Q7 as the SP block. See the limitations below for why the SP queues are shaped.

Queue scheduling model and limitations

Sand/Jericho QoS is more constrained than the per-queue table suggests. The relevant limits:

  • 8 fixed queues (Q0–Q7), 1:1 with traffic class. TCn always maps to Qn; the queue index cannot be remapped. Only the DSCP→TC mapping is configurable (via qos map, see previous section) — there is no way to add queues or change the TC-to-queue binding.

  • Q7 is reserved for the control plane. It is hard-wired to TC7 (the TC7↔Q7 mapping is not editable) and always runs strict priority. BGP/BFD self-generated traffic is marked CS7 (see SGT QoS) precisely so it lands here. Historically Q7 was entirely non-configurable; as of EOS 4.33.0F (“Configurable Transmit Queue 7”) a limited set of Q7 knobs is exposed on Sand — shape rate and latency maximum (max-latency tail-drop threshold) only. Q7 stays strict priority; there is no bandwidth/priority control. Two limitations matter for this design:

    • Front-panel ports and LAGs only — not sub-interfaces. The customer/transit sub-interfaces on this router cannot carry Q7 config; only the physical port / port-channel can.
    • Only on 8-Qpair interfaces. Verify with show platform fap mapping interface ethernet <X> — the BaseQPair/QPairs value must be 8.

    Config is via tx-queue 7 under the interface (or a qos profile attached with service-profile); remove with no tx-queue 7 / default tx-queue 7. Platform-supported: 7500R/R2/R3, 7280R/R2/R3 (incl. this 7280SR3K), 7800R3.

  • Only two scheduler classes exist: strict priority and round-robin. There are no intermediate priority tiers — a queue is either ahead of every RR queue, or in the RR group. All SP queues are serviced before any RR queue.

  • The SP/RR boundary is monotonic by queue number — SP must be the top contiguous block. All queues are SP by default. The moment you set any queue to no priority, every lower-numbered queue is forced to RR as well, regardless of its own configuration. You cannot have an SP queue sitting below an RR queue. In practice this means RR is always the bottom block [Q0..Qk] and SP the top block [Qk+1..Q7]; the design’s split (RR Q0–Q4, SP Q5–Q7) is the only legal shape for that boundary. Trying to configure, say, Q3 as SP while Q4 is RR silently yields Q3 running RR.

  • SP queues can starve RR queues, so they must be shaped. Because SP service is exhaustive, an unbounded EF (Q5) or management (Q6) queue could consume the whole port and starve data. That is why Q5 is shape rate 50 percent and Q6 shape rate 10 percent — a hard ceiling bounding the SP queues so headroom is left for the RR group. Q7 is left unshaped because control-plane volume is low and must never be dropped.

  • bandwidth percent is an RR-only weight; SP queues cannot use it. SP queues take whatever they need up to their shaper; only RR queues honour bandwidth percent. This is why the CORE/CUST columns show a shape value for Q5–Q6 but a bandwidth percentage for Q0–Q4.

  • RR bandwidth is a guaranteed minimum, normalised to ≤100%. The RR shares here sum to exactly 100% (1+20+19+30+30). If the configured sum exceeds 100%, EOS normalises each queue’s operational share to configured ÷ cumulative. Below 100%, an RR queue may burst into unused capacity unless separately capped with shape rate.

  • Subinterface QoS is a two-level hierarchy. The parent scheduler round-robins between subinterfaces (qos subinterface scheduling parent round-robin); the 8 tx-queues are the child schedulers within each subinterface. qos subinterface tx-queue count 8 must match the queue count in the applied profile (see QoS Subinterface).

  • Egress queue depth is observed via VOQ, not a directly configurable buffer. Sand is a VOQ (ingress-buffered) architecture; the VOQ counter is not user-configurable. Use queue-monitor length (see Queue Monitor) to catch microbursts.

QPROF-CORE

qos profile QPROF-CORE
   qos trust dscp
   tx-queue 0
      no priority
      bandwidth percent 1
   tx-queue 1
      no priority
      bandwidth percent 20
   tx-queue 2
      no priority
      bandwidth percent 19
   tx-queue 3
      no priority
      bandwidth percent 30
   tx-queue 4
      no priority
      bandwidth percent 30
   tx-queue 5
      shape rate 50 percent     ! EF — voice/RT, shaped to prevent starvation
   tx-queue 6
      shape rate 10 percent     ! CS6 — management

QPROF-TRANSIT-UPSTREAM / QPROF-TRANSIT-CUSTOMER

qos profile QPROF-TRANSIT-UPSTREAM
   no qos trust                 ! traffic-policy sets TC; DSCP from transit ignored
   tx-queue 0
      no priority
      bandwidth percent 1
   tx-queue 1
      no priority
      bandwidth percent 20
   tx-queue 2
      no priority
      bandwidth percent 19
   tx-queue 3
      no priority
      bandwidth percent 30
   tx-queue 4
      no priority
      bandwidth percent 30
   tx-queue 6
      shape rate 10 percent

Profile-to-interface mapping

Profile Trust Applied to
QPROF-CORE DSCP Core/backbone interfaces, MPLS links
QPROF-TRANSIT-UPSTREAM None IP transit provider ports
QPROF-TRANSIT-CUSTOMER None Customer IP transit ports
QPROF-UNKNOWN None Fallback / unclassified

QoS Maps (DSCP → Traffic Class)

Applied when qos trust dscp is active. On untrusted interfaces, traffic-policy sets TC directly.

qos map dscp 1 2 3 4 5 6 7 17 18 19 20 21 22 23 24 25 27 28 29 30 31 32 33 34 35 36 37 38 39 41 42 43 44 47 49 50 51 52 53 54 55 57 58 59 60 61 62 63 to traffic-class 0
qos map dscp 45 to traffic-class 2   ! NQB → TC2 (signalling / non-queuing)
qos map dscp 16 to traffic-class 3   ! AF21 → TC3 (InetPlus / managed)
qos map dscp 26 to traffic-class 4   ! AF31 → TC4 (business / priority)

DSCP 0 (BE) → TC1 (BE internet), DSCP 8 (CS1) → TC0 (scavenger), DSCP 40/46 (EF) → TC5, DSCP 48 (CS6) → TC6, and DSCP 56 (CS7) → TC7 are handled by hardware default maps (not overridden here). The explicit list above catches all unmapped DSCPs → TC0.

Traffic Policies

Traffic policies are the primary QoS enforcement mechanism. Applied per-interface as ingress or egress classifiers.

Naming convention. Three distinct object types, three distinct prefixes — so a name tells you what it is at a glance:

Prefix Object Example
M- traffic-policy match clause (a rule) M-V4U-BGP-PEERS
PFX- prefix field-set (IPv4/IPv6 address data) PFX-V4-ADMIN, PFX-V6-BOGONS
L4P- L4-port field-set (port-number data) L4P-AMP-SRC-PORTS

The V4/V6 token marks address family. A match clause and the field-set it references now never share a name (previously both were FS-…, e.g. the NTP rule and its prefix set collided).

Policy inventory

Policy Direction Applied to Key actions
TP-COPP CPU (all VRFs) Global Police ICMP/NTP/DNS, count BGP/BFD/LDP, drop unknown
TP-TRANSIT-UPSTREAM Ingress Transit provider ports Drop bogons, block IPv6 exthdr/frag, re-mark, AMP mitigation
TP-TRANSIT-CUSTOMER Ingress Customer transit Drop bogons, anti-AMP, NQB to TC2, scavenger to TC0
TP-TRANSIT-PNI Ingress Private peering / PNI Drop bogons, allow BGP on LL, AMP mitigation

TP-COPP

traffic-policies
   cpu traffic-policy TP-COPP vrf all

   traffic-policy TP-COPP
      match M-V4U-ICMP ipv4
         protocol icmp
         actions
            count
            police rate 1024 kbps
      match M-V6U-ICMP ipv6
         protocol icmpv6
         actions
            count
            police rate 1024 kbps
      match M-V4U-BFD ipv4
         source prefix field-set PFX-V4-BFD-PEERS PFX-V4-LL PFX-V4-ROUTERS-LO
         protocol udp source port all destination port 3784-3785, 7784
         actions
            count
      match M-V4U-BGP-PEERS ipv4
         protocol neighbors bgp
         actions
            count
      match M-V4M-LDP ipv4
         destination prefix 224.0.0.2/32
         protocol udp source port all destination port ldp
         actions
            count
      match M-V4U-LDP ipv4
         source prefix field-set PFX-V4-ROUTERS-LO
         protocol tcp source port all destination port ldp
         protocol udp source port all destination port ldp
         actions
            count
      match M-V4U-SSH ipv4
         source prefix field-set PFX-V4-ADMIN
         protocol tcp source port all destination port ssh netconf-ssh
         actions
            count
      match M-V4U-SNMP ipv4
         source prefix field-set PFX-V4-ADMIN
         protocol udp source port all destination port snmp
         actions
            count
      match M-V4U-GRPC ipv4
         source prefix field-set PFX-V4-ADMIN
         protocol tcp source port all destination port gnmi
         actions
            count
      match M-V4U-NTP ipv4
         source prefix field-set PFX-V4-NTP
         protocol udp source port ntp destination port all
         actions
            count
            police rate 1024 kbps
      match M-V4U-TCP-ESTABLISHED ipv4
         protocol tcp flags established
         actions
            count
      match M-V4U-TCP-SYN-ACK ipv4
         protocol tcp flags ack syn
         actions
            count
            police rate 1024 kbps
      match M-V4U-DISCARD ipv4
         actions
            count
            drop
      match ipv4-all-default ipv4
      match ipv6-all-default ipv6

AMP source ports field-set

field-set l4-port L4P-AMP-SRC-PORTS
   17, 19, 111, 123, 137, 161, 389, 520, 751, 1434, 1900, 5353, 6881, 11211, 27015, 27960
   ! qotd, chargen, portmap/rpc, ntp, netbios, snmp, ldap, rip,
   ! kerberos, ms-sql, ssdp, mdns, bittorrent, memcached, srcds/quake

TP-TRANSIT-UPSTREAM (key elements)

Bogon and malformed-packet filtering is enforced in the hardware dataplane via traffic-policy, rather than in the BGP control plane via route-map. This drops invalid traffic at line rate before it reaches the routing process, and also prevents bogon-destined packets already in the forwarding table from being forwarded.

traffic-policy TP-TRANSIT-UPSTREAM
   match M-V4U-BOGONS ipv4
      destination prefix field-set PFX-V4-BOGONS
      actions
         count
         drop
   match M-V6U-BOGONS ipv6
      destination prefix field-set PFX-V6-BOGONS
      actions
         count
         drop
   match M-V6U-EXTHD-BLOCK ipv6
      protocol 0, 43, 60        ! HBH / routing / dest options
      actions
         count
         drop
   match M-V6U-FRAG-BLOCK ipv6
      protocol 44
      actions
         count
         drop
   match M-V4U-UDP-FRAGMENT ipv4
      protocol udp
      fragment
      actions
         count
         set dscp 8
         set traffic class 0
   match M-V4U-AMP-SOURCE-PORTS ipv4
      protocol udp source port field-set L4P-AMP-SRC-PORTS
      actions
         count
         set dscp 8
         set traffic class 0
   match M-V4U-SCAVENGER-BULK ipv4
      protocol tcp source port all destination port ftp-data rsync ftps-data
      actions
         count
         set dscp 8
         set traffic class 0
   match M-V4U-NBQ ipv4
      dscp 34, 45
      actions
         count
         set dscp 45
         set traffic class 2
   match M-V4U-INETPLUS ipv4
      destination prefix field-set PFX-V4-INETPLUS
      actions
         count
         set dscp 16
         set traffic class 3
   match M-V4U-INET ipv4
      actions
         count
         set dscp 0
         set traffic class 1
   match ipv4-all-default ipv4
      actions
         count
         set dscp 8
         set traffic class 0

SGT QoS — Self-Generated Traffic

Control-plane traffic originated by the router must be explicitly DSCP-marked.

router bfd
   qos dscp 56              ! BFD → CS7

router bgp <ASN>
   bgp transport qos dscp 56   ! BGP TCP sessions → CS7

snmp-server qos dscp 26
dns qos dscp 26
ntp qos dscp 26
sflow qos dscp 26
logging qos dscp 26

management ssh
   qos dscp 26

management api gnmi
   transport grpc MGMT
      qos dscp 26
   transport grpc default
      qos dscp 26
Protocol DSCP CS/AF TC Rationale
BGP TCP 56 CS7 7 (strict) Session drops cause routing black-holes
BFD 56 CS7 7 (strict) Sub-second detection — cannot tolerate jitter
SNMP 26 AF31 4 OAM, not real-time
DNS 26 AF31 4 Router DNS queries
NTP 26 AF31 4 Clock sync
sFlow 26 AF31 4 Telemetry
Syslog 26 AF31 4 Operational logging
SSH 26 AF31 4 CLI access
gNMI/gRPC 26 AF31 4 Streaming telemetry

MPLS LDP

mpls ip

mpls ldp
   router-id interface Loopback0 force   ! stable RID; force = use even if non-primary
   igp sync delay 2                       ! RFC 5443: hold IS-IS max-metric 2s after LDP UP before restoring normal metric
   interface disabled default             ! LDP not enabled by default on interfaces
   neighbor hello-redundancy              ! dual-hello on LAG members
   no shutdown
   !

LDP must be explicitly enabled per interface. interface disabled default is critical — it prevents LDP from attempting sessions on all interfaces including external ones.

igp sync delay 2 is part of IGP–LDP synchronisation (RFC 5443). When an LDP session goes down, IS-IS advertises the affected link at maximum metric so traffic reroutes away from the label-less path. When the LDP session re-establishes, IS-IS holds the maximum metric for 2 seconds before restoring the normal metric — giving LDP time to complete label exchange before traffic is attracted back to the link.

interface EthernetX
   mpls ip
   mpls ldp interface

interface Loopback0
   mpls ldp interface

MPLS ICMP

mpls icmp ttl-exceeded tunneling     ! include original IP header in TTL-exceeded ICMP
mpls oam standard ietf               ! RFC 4379 MPLS ping/trace
mpls icmp ipv6 source-interface Loopback0
mpls icmp ip   source-interface Loopback0
Parameter Effect Without it
ttl-exceeded tunneling ICMP includes original inner IP header traceroute shows only MPLS hops
oam standard ietf RFC 4379 MPLS ping/trace Vendor-specific OAM, incompatible with third-party
source-interface Loopback0 ICMP sourced from stable loopback ICMP from transit link — unreachable if link fails

Tunnel-ribs

Tunnel-ribs aggregate multiple label signalling protocols for BGP next-hop resolution with explicit preference ordering.

tunnel-ribs
   tunnel-rib TR-SR-OVER-LDP
      source-protocol bgp labeled-unicast preference 30   ! highest preference
      source-protocol isis segment-routing preference 40
      source-protocol ldp preference 50                   ! lowest preference / fallback

Lower preference number = higher priority.

Protocol Preference Description
bgp labeled-unicast 30 BGP-LU (inter-AS or explicit paths)
isis segment-routing 40 SR-ISIS node SIDs
ldp 50 LDP label bindings (fallback)

The name TR-SR-OVER-LDP reflects a migration scenario: SR-ISIS is preferred, LDP serves as fallback. Removing LDP from the rib when migration is complete requires no BGP policy changes.

sFlow

sflow sample 512                   ! 1:512 sampling rate
sflow destination <SFLOW-COLLECTOR>       ! collector (e.g. Akvorado)
sflow source <LOOPBACK0>        ! Loopback0 — stable, matches router-id
sflow sample input subinterface
sflow sample output subinterface
sflow extension bgp                ! add BGP attrs to flow records
sflow qos dscp 26
sflow interface disable default    ! explicit opt-in per interface
no sflow hardware acceleration
sflow run

Enable per-interface:

interface EthernetX
   sflow enable
   sflow egress enable

sflow extension bgp adds BGP attributes to flow records: peer AS, origin AS, AS path, next-hop, communities. Requires BGP running; tools like Akvorado use these for AS-level traffic matrices.

no sflow hardware acceleration is required alongside it: hardware acceleration offloads sampling to the ASIC for higher raw sample rates, but conflicts with BGP attribute annotation and per-subinterface sampling (sflow sample input/output subinterface) simultaneously.

The trade-off is intentional — richer per-flow records for AS-level traffic analysis outweigh the raw rate benefit.

Maintenance Mode

Two maintenance modes: SOFT (graceful drain with BGP prepend + GSHUT) and HARD (immediate withdraw of all external routes).

On boot, MNT-UN-SOFT runs automatically for 120 seconds to allow BGP reconvergence before accepting traffic.

Structure

Component Type Purpose
MNT-UN-SOFT Unit Soft drain — on-boot + manual
MNT-UN-HARD-EXT Unit Hard withdraw — emergency
MNT-GR-SOFT-VRF-DEFAULT-INT Group iBGP peers for soft drain
MNT-GR-SOFT-VRF-DEFAULT-EXT Group eBGP peers for soft drain
MNT-GR-HARD-VRF-DEFAULT-EXT Group eBGP peers for hard withdraw 
maintenance
   profile bgp MNT-PR-SOFT-EXT
      initiator route-map RP-ALL-EXT-SOFT-MNT-OUT out
   !
   profile bgp MNT-PR-HARD-EXT
      initiator route-map RP-ALL-EXT-HARD-MNT-OUT out
      initiator route-map RP-ALL-EXT-HARD-MNT-IN in
   !
   profile bgp MNT-PR-SOFT-INT
      initiator route-map RP-ALL-INT-SOFT-MNT-OUT out
   !
   profile unit MNT-UN-SOFT
      on-boot duration 120
   !
   unit MNT-UN-SOFT
      group bgp MNT-GR-SOFT-VRF-DEFAULT-EXT
      group bgp MNT-GR-SOFT-VRF-DEFAULT-INT
   !
   unit MNT-UN-HARD-EXT
      group bgp MNT-GR-HARD-VRF-DEFAULT-EXT

Route maps

! SOFT EXTERNAL — prepend own ASN x5 + GSHUT community (RFC 8326)
route-map RP-ALL-EXT-SOFT-MNT-OUT permit 10
   set as-path prepend auto repeat 5
   set community community-list CMST-SPECIAL-GSHUT additive

! SOFT INTERNAL — prepend with dummy 65500 ASN x5 + maintenance large-community
route-map RP-ALL-INT-SOFT-MNT-OUT permit 10
   set as-path prepend 65500 repeat 5
   set community community-list CMST-SPECIAL-GSHUT additive
   set large-community large-community-list CMLR-SPECIAL-MNT-SOFT additive

! HARD EXTERNAL — deny all
route-map RP-ALL-EXT-HARD-MNT-OUT deny 10
route-map RP-ALL-EXT-HARD-MNT-IN deny 10

! HARD INTERNAL — add large-community then deny (except own aggregates)
route-map RP-ALL-INT-HARD-MNT-OUT permit 10
   set large-community large-community-list CMLR-SPECIAL-MNT-HARD additive
   continue 20
route-map RP-ALL-INT-HARD-MNT-OUT deny 20
   match ip address prefix-list PL-V4U-AS<ASN>
route-map RP-ALL-INT-HARD-MNT-OUT deny 30
   match ipv6 address prefix-list PL-V6U-AS<ASN>

Operations

! Enter soft maintenance
configure
  maintenance
    unit MNT-UN-SOFT
      quiesce

! Exit maintenance
configure
  maintenance
    unit MNT-UN-SOFT
      no quiesce

show maintenance
show maintenance unit MNT-UN-SOFT
show bgp neighbor | include maintenance

BGP Transport & Session Parameters

router bgp <ASN>
   bgp asn notation asdot
   router-id <LOOPBACK0>
   update wait-install            ! hold updates until FIB install complete
   bgp transport pmtud            ! Path MTU Discovery on BGP TCP
   bgp transport qos dscp 56      ! BGP TCP → CS7
   bgp always-compare-med         ! compare MED across ASes

update wait-install

When a large routing table is received, BGP can converge and begin advertising routes before the hardware FIB has finished programming them. update wait-install breaks this race: outbound BGP updates are held until the FIB install is confirmed, preventing the router from attracting traffic it cannot yet forward. This increases convergence time under FIB pressure — monitor hardware programming status on Jericho2 with show platform sand l3 summary | grep backlog. A non-zero backlog value means routes are still queued for hardware programming and BGP updates are being held.

show platform sand l3 summary | grep backlog
  Routes:       1050278  backlog:  0  unprogrammed:  0
  Adjacencies:  327      backlog:  0  unprogrammed:  0
  Routes:       242681  backlog:  0  unprogrammed:  0
  Adjacencies:  327     backlog:  0  unprogrammed:  0
  Routes:       59  backlog:  0  unprogrammed:  0
  Adjacencies:  23  backlog:  0  unprogrammed:  0
Egress rewrite chains in backlog: resource-full 0, no-interface 0

route refresh demarcated stale-path removal

neighbor <peer-group> route refresh demarcated stale-path removal

Route Refresh (RFC 2918) lets a BGP speaker request its peer to re-advertise its full routing table — used after an inbound policy change to re-evaluate all received routes. The problem: basic Route Refresh has no convergence signal. The local router cannot tell when the peer has finished re-sending, so paths that the peer silently stopped advertising (without an explicit withdraw) can persist in the RIB indefinitely.

RFC 7313 (Enhanced Route Refresh) adds two markers: BoRR (Begin of Route Refresh) sent before the re-advertisement starts, and EoRR (End of Route Refresh) sent when complete. With stale-path removal, EOS tracks which paths were re-advertised during the refresh window and, upon receiving EoRR, purges any paths from that peer that were not re-sent. This enforces a clean post-refresh RIB state equivalent to a hard reset, but without dropping the session.

Practically: after changing an inbound filter, trigger clear ip bgp <peer> soft in — paths that no longer match the new policy and that the peer does not re-advertise will be removed automatically when the refresh completes.

out-delay

Peer type out-delay Rationale
iBGP (border full-mesh) 3s Batch concurrent updates from multiple upstreams before propagating to iBGP
iBGP (RR client / PE) 1s Faster reconvergence for PE nodes
eBGP transit / peering 6s Absorb micro-instabilities before exporting

rib-in pre-policy retain all

neighbor <transit-peer-group> rib-in pre-policy retain all

By default, EOS discards the pre-policy adj-RIB-In after applying inbound filters — only the post-policy result is kept in memory. rib-in pre-policy retain all keeps the full raw table as received from the peer, before any filters are applied.

Two consequences:

Policy changes without route-refresh. When an inbound RCF or route-map is modified, EOS can re-evaluate all received routes immediately from the cached pre-policy table — no need to request a Route Refresh from the peer. For a transit session carrying 900k+ prefixes, a Route Refresh triggers a full re-advertisement that takes minutes and causes a reconvergence event across the network. With retain all, the policy change takes effect locally in seconds without touching the peer.

BMP and gNMI pre-policy export. BMP (see BMP Monitoring section) streams both pre-policy and post-policy adj-RIB-In to collectors. Without retain all, there is no pre-policy data to export — only the post-policy view is available. Same applies to gNMI adj-rib-in-pre subscriptions.

Cost: roughly doubles BGP memory consumption for those peer sessions. Apply selectively to transit and peering sessions where policy re-evaluation matters; do not apply globally. Monitor with show bgp memory.

BGP Bestpath

router bgp <ASN>
   maximum-paths 4 ecmp 4
   bgp additional-paths install
   bgp always-compare-med
   no bgp bestpath as-path multipath-relax
Parameter Behaviour
maximum-paths 4 ecmp 4 Install up to 4 equal-cost paths in the FIB and perform ECMP across them. Without this, only the single best path is forwarded.
bgp additional-paths install Pre-install the best backup path in the FIB alongside the primary. Enables sub-second failover (BGP PIC) without waiting for BGP re-convergence — see §23.
bgp always-compare-med Compare MED between routes from different ASes. Without this, MED is only compared within the same AS-PATH.
no bgp bestpath as-path multipath-relax ECMP only across paths with identical AS-path, not just equal length. Prevents unintended load-balancing between different transit upstreams and between fabric sites with distinct AS-paths.

Peer Groups & Peers

All peers use named peer-groups. Direct per-peer config is limited to IP address, description, and optional BFD/passive/password.

Peer group taxonomy

Group AF Type Key attributes
BGR-V4U-INT-IBGPFM-RR IPv4 iBGP full-mesh RR update-source Lo0, add-paths, out-delay 3
BGR-V6U-INT-IBGPFM-RR IPv6 iBGP full-mesh RR same
BGR-V4U-INT-RRC-CLIENT IPv4 iBGP RR client route-reflector-client, out-delay 1
BGR-V6U-INT-RRC-CLIENT IPv6 iBGP RR client same
BGR-V4U-EXT-TRANSIT-* IPv4 eBGP upstream transit remove-private-as, out-delay 6, rib-in retain
BGR-V6U-EXT-TRANSIT-* IPv6 eBGP upstream transit same
BGR-V4U-EXT-RS-* IPv4 IXP route server no enforce-first-as, maximum-routes 0, out-delay 6
BGR-V6U-EXT-RS-* IPv6 IXP route server same
BGR-V4U-EXT-PEER-* IPv4 Direct peering out-delay 6, max-routes per peer
BGR-V6U-EXT-PEER-* IPv6 Direct peering same
BGR-V4U-EXT-CUST-* IPv4 Customer / IP transit send-community, passive where applicable
BGR-V6U-EXT-CUST-* IPv6 Customer / IP transit same

Peer group definition example

router bgp <ASN>
   neighbor BGR-V4U-EXT-TRANSIT-PROVIDER peer group
   neighbor BGR-V4U-EXT-TRANSIT-PROVIDER remote-as <upstream-ASN>
   neighbor BGR-V4U-EXT-TRANSIT-PROVIDER send-community standard large
   neighbor BGR-V4U-EXT-TRANSIT-PROVIDER maximum-routes 1048576 warning-limit 80 percent warning-only
   neighbor BGR-V4U-EXT-TRANSIT-PROVIDER route refresh demarcated stale-path removal
   neighbor BGR-V4U-EXT-TRANSIT-PROVIDER remove-private-as all replace-as
   neighbor BGR-V4U-EXT-TRANSIT-PROVIDER out-delay 6 changes
   neighbor BGR-V4U-EXT-TRANSIT-PROVIDER rib-in pre-policy retain all

   neighbor X.X.X.X peer group BGR-V4U-EXT-TRANSIT-PROVIDER
   neighbor X.X.X.X description TRANSIT-PROVIDER-A

remove-private-as all strips private ASNs from the AS-path before sending to an upstream. replace-as goes further: instead of leaving a gap, it substitutes the local AS number in place of each removed private ASN. This ensures the AS-path is never empty or shorter than expected when the upstream receives it — some ISPs reject routes with an empty AS-path or apply stricter filtering to them.

maximum-routes 0 (unlimited) is sometimes recommended for full-table transit peers to avoid session drops as the internet table grows. 1048576 warning-limit 80 percent warning-only is used here instead: the session never drops, but a syslog warning fires at ~838k prefixes — providing early notice of abnormal peer behaviour (route leak, misconfigured customer) before it becomes a capacity problem.

IXP route server

neighbor BGR-V4U-EXT-RS-IXNAME peer group
neighbor BGR-V4U-EXT-RS-IXNAME remote-as <RS-ASN>
no neighbor BGR-V4U-EXT-RS-IXNAME enforce-first-as  ! RS doesn't prepend its ASN
neighbor BGR-V4U-EXT-RS-IXNAME maximum-routes 0
neighbor BGR-V4U-EXT-RS-IXNAME remove-private-as all replace-as

BFD on eBGP peers

neighbor <IP> bfd damping
neighbor <IP> bfd interval 300 min-rx 300 multiplier 4
! damping: suppress BFD flaps from triggering BGP session churn
! 300ms × 4 = 1.2s detection time

The standard internet-edge recommendation is 500 ms × 3 (1.5 s detection). Timers here are tightened to 300 ms × 4 (1.2 s) to detect link failures faster. bfd damping is added to prevent rapid BFD oscillations on unstable links from cycling the BGP session — without it, a flapping physical link would repeatedly reset BGP, causing unnecessary route churn across the network.

RCF — Routing Control Functions

RCF replaces route-maps for BGP policy. Functions are written in EOS RCF language and applied as rcf in <function>() / rcf out <function>().

Policy by peer type

Peer type Inbound RCF Outbound RCF Key logic
iBGP border allow_all() rp_nhs_cisco_style() Accept all; set NH to self Cisco-style
iBGP RR client RP_V4U_INT_RRC_REDUCED_IN() RP_V4U_INT_RRC_REDUCED_OUT() Reduced table to PEs
Upstream transit RP_V4U_TRANSIT_<X>_IN() RP_V4U_TRANSIT_<X>_OUT() Reject bogons/own; export own + communities
IXP route server RP_V4U_RS_<IX>_IN_<LOC>() RP_V4U_RS_<IX>_OUT_<LOC>() Filter bogons/private; export own with LC
Direct peer RP_V4U_PEER_<X>_IN_<IXP>() RP_V4U_PEER_<X>_OUT_<IXP>() Strict max-prefix; own prefixes out
Customer transit RP_V4U_CUST_<X>_IN() RP_V4U_CUST_<X>_OUT() Customer’s own prefixes in; default + table out

RCF application example

address-family ipv4
   neighbor BGR-V4U-EXT-TRANSIT-<PROVIDER-A>   rcf in  RP_V4U_TRANSIT_<PROVIDER-A>_IN()
   neighbor BGR-V4U-EXT-TRANSIT-<PROVIDER-A>   rcf out RP_V4U_TRANSIT_<PROVIDER-A>_OUT()
   neighbor BGR-V4U-EXT-PEER-<PEER-A-IX>       rcf in  RP_V4U_PEER_<PEER-A>_IN_IX_<LOC>()
   neighbor BGR-V4U-EXT-PEER-<PEER-A-IX>       rcf out RP_V4U_PEER_<PEER-A>_OUT_IX_<LOC>()
   neighbor BGR-V4U-EXT-CUST-<CUSTOMER>         rcf in  RP_V4U_CUST_<CUSTOMER>_IN()
   neighbor BGR-V4U-EXT-CUST-<CUSTOMER>         rcf out RP_V4U_CUST_<CUSTOMER>_OUT()
   neighbor BGR-V4U-EXT-TOOLS-SCRUB             rcf in  deny_all()
   neighbor BGR-V4U-EXT-TOOLS-SCRUB             rcf out RP_V4U_TOOLS_SCRUB_OUT()

BGP Missing-policy Safety

RFC 8212 defines that a BGP speaker should deny routes in both directions when no policy is configured — EOS does not implement this by default and requires explicit configuration. action deny is per-direction: a peer with only an inbound policy will not have its outbound silently blocked. This is intentional — monitoring and looking-glass peer-groups carry no outbound policy by design and should not be affected. The alternative action deny-in-out would deny both directions whenever either direction lacks a policy, which is too broad for a multi-purpose border router.

address-family ipv4
   bgp missing-policy direction in  action deny
   bgp missing-policy direction out action deny
address-family ipv6
   bgp missing-policy direction in  action deny
   bgp missing-policy direction out action deny

A new peer-group without RCF assignments will have 0 prefixes in/out — this is intentional and forces explicit policy before a session becomes operational.

BGP Add-paths (iBGP)

Add-paths (RFC 7911) enables BGP Prefix Independent Convergence (PIC). Without it, a next-hop failure triggers a full BGP re-convergence scan across the entire routing table — on a full internet table this takes seconds. With bgp additional-paths install, a backup path is pre-installed in the FIB alongside the primary. When the primary next-hop fails, the dataplane switches to the backup immediately without waiting for BGP to rerun bestpath selection.

router bgp <ASN>
   bgp additional-paths install

   ! iBGP border full-mesh
   neighbor BGR-V4U-INT-IBGPFM-BORDER additional-paths receive
   neighbor BGR-V4U-INT-IBGPFM-BORDER additional-paths send backup

   ! RR clients — reflect best + backup to PEs
   neighbor BGR-V4U-INT-RRC-CLIENT additional-paths receive
   neighbor BGR-V4U-INT-RRC-CLIENT additional-paths send backup
Mode Behaviour Use case
any Send all paths with unique path-id RR reflecting all paths
backup Send best + first non-best only Border routers — backup for fast-reroute
all All best paths per ECMP group ECMP load balancing

send backup installs the backup path in FIB for sub-second failover when the primary NH fails — before BGP re-converges.

NH Resolution via Tunnel-rib

BGP next-hops resolved through MPLS tunnels. Applied as a second router bgp stanza.

router bgp <ASN>
   address-family ipv4
      next-hop resolution ribs tunnel-rib TR-SR-OVER-LDP system-connected
   address-family ipv6
      next-hop resolution ribs tunnel-rib TR-SR-OVER-LDP system-connected
   address-family vpn-ipv4
      next-hop resolution ribs tunnel-rib TR-SR-OVER-LDP system-connected
   address-family vpn-ipv6
      next-hop resolution ribs tunnel-rib TR-SR-OVER-LDP system-connected

BMP Monitoring

BMP (BGP Monitoring Protocol, RFC 7854) streams BGP session state and adj-RIB updates to external collectors for offline analysis, route leak detection, and policy auditing. Two collectors are deployed for redundancy. Both pre-policy and post-policy exports are enabled — pre-policy shows exactly what the peer advertised before any filters, post-policy shows what was accepted. bgp rib bestpaths disabled reduces collector load by suppressing loc-RIB bestpath streaming; the collectors reconstruct bestpath from the per-peer adj-RIB feeds. rib-in pre-policy retain all on transit peer-groups (§18) is a prerequisite for meaningful pre-policy export.

router bgp <ASN>
   monitoring station bmp-collector-site-a
      update-source Loopback0
      statistics
      connection address <BMP-COLLECTOR-1-IP>
      connection mode active port 1790
      connection keepalive 30 3 10        ! interval 30s, retries 3, timeout 10s
      export-policy received routes pre-policy post-policy
      export-policy bgp rib bestpaths disabled
   !
   monitoring station bmp-collector-site-b
      update-source Loopback0
      statistics
      connection address <BMP-COLLECTOR-2-IP>
      connection mode active port 1790
      connection keepalive 30 3 10
      export-policy received routes pre-policy post-policy
      export-policy bgp rib bestpaths disabled

Export policy

Parameter Value Meaning
received routes pre-policy enabled Stream adj-rib-in-pre (raw from peer)
received routes post-policy enabled Stream adj-rib-in-post (after our filters)
bgp rib bestpaths disabled Do not stream loc-rib bestpaths (reduces load)

BMP message types (RFC 7854)

Message Trigger Contains
Route Monitoring BGP UPDATE send/receive BGP UPDATE with path attributes
Statistics Report Periodic Prefix counters, rejected routes
Peer Down Session down Peer state, reason code
Peer Up Session established OPEN messages, capabilities
Initiation BMP session start Router sysDescr, sysName 
show bgp monitoring station
show bgp monitoring station bmp-collector-site-a
show bgp monitoring statistics
show bgp monitoring peer

BGP-LG Role

EOS has a built-in bgplg role for read-only BGP lookups. The role definition is explicit and uses CLI command ACLs.

User

username bgplg privilege 15 role bgplg secret sha512 <hash>

Role definition

role bgplg
   5  permit mode exec command terminal width
   6  permit mode exec command terminal length
   10 permit mode exec command ping
   20 permit mode exec command traceroute
   30 permit mode exec command show ip bgp
   40 permit mode exec command show ipv6 bgp
   50 permit mode exec command show bgp
   60 permit mode exec command show route
   70 deny command .*                        ! deny everything else

iBGP peer group for bgplg

A dedicated iBGP peer-group BGR-ALL-INT-IBGP-BGPLG-LG is used to feed a local route-reflector session for the looking glass. This allows the bgplg user to query a local BGP view without exposing the full RIB or requiring a separate session to a route server.

router bgp <ASN>
   neighbor BGR-ALL-INT-IBGP-BGPLG-LG peer group
   neighbor BGR-ALL-INT-IBGP-BGPLG-LG remote-as <ASN>
   neighbor BGR-ALL-INT-IBGP-BGPLG-LG update-source Loopback0
   no neighbor BGR-ALL-INT-IBGP-BGPLG-LG additional-paths send
   neighbor BGR-ALL-INT-IBGP-BGPLG-LG send-community standard extended large
   neighbor BGR-ALL-INT-IBGP-BGPLG-LG maximum-routes 10
   neighbor BGR-ALL-INT-IBGP-BGPLG-LG route refresh demarcated stale-path removal

maximum-routes 10 — safety limit; the LG peer-group is for query, not full table distribution.

Show Tech-Support Optimization

On a router carrying a full internet routing table, the default show tech-support output includes BGP RIB and routing table commands that can take 20+ minutes to complete and produce several hundred MB of text. This makes the bundle impractical for routine collection and can block the CLI during time-sensitive troubleshooting.

EOS allows individual show commands to be excluded from the tech-support bundle via management tech-support. The commands that generate disproportionately large output on internet-edge routers are the full BGP RIB dumps and routing table listings — these are better captured on-demand when needed rather than included in every tech-support.

management tech-support
   show commands exclude
      show ip bgp
      show ip bgp detail
      show ip bgp neighbors detail
      show ipv6 bgp
      show ipv6 bgp detail
      show ipv6 bgp neighbors detail
      show ip route
      show ipv6 route
      show ip route detail
      show ipv6 route detail

For regular offline capture, schedule a daily compressed tech-support to flash:

schedule DAILY-TECH-SUPPORT
   interval 1440
   timeout 120
   command show tech-support | gzip > flash:tech-support-latest.log.gz

The interval 1440 fires once per day (minutes). timeout 120 prevents the job from blocking indefinitely if the system is under load. The output file is overwritten each run — for retention, use a syslog-forwarded copy or replace the filename with a datestamped path via a bash alias.

]]>