[NLNOG] FW: Forwarding issues related to MACs starting with a 4 or a 6 (Was: [c-nsp] Wierd MPLS/VPLS issue)

Mon Dec 5 18:34:40 CET 2016

Bedankt voor het delen Robert!

Vooral de laatste vier entries in de lijst zijn heel bijzonder. :-)

On 5 Dec 2016, 18:28 +0100, Robert Heuvel <RHeuvel at atom86.net>, wrote:
> Ola,
>
> Wij hebben hetzelfde probleem vastgesteld op een Arista 7150S-52 met EOS 4.12.3.1 (transient MPLS traffic via Layer2).
> Arista EOS 4.16.9M heeft dit voor ons opgelost.
>
> MACs die niet getransporteerd werden:
> 4X:XX:XX:XX:XX:XX
> 6X:XX:XX:XX:XX:XX
> X4:XX:XX:XX:XX:XX
> X6:XX:XX:XX:XX:XX
> XX:XX:XB:XX:6X:XX
> XX:XX:XF:XX:4X:XX
>
> Misschien zijn er nog meer geweest, maar zijn nog niet gevonden…
>
> Dank gaat uit naar:
> Richard van Looijen van Flowmailer voor het vaststellen van het probleem.
> Edwin Kalle van 2hip voor het wijzen op onderstaande tread.
> En natuurlijk Job Snijders voor zijn mail van vrijdagavond, waardoor voor ons alles in 1 keer op zijn plaats viel en we de tussenliggende switch gingen bekijken…
>
> Mvg,
> Robert Heuvel
> atom86
>
>
> On 02/12/2016, 21:45, "NLNOG on behalf of Job Snijders" <nlnog-bounces at nlnog.net on behalf of job at instituut.net> wrote:
>
> Alloah,
>
> TL;DR: Cisco Nexus 92160 switches kunnen packets met bepaalde specifieke
> payloads niet forwarden. Er komt geen software fix, de ASIC is niet goed
> gedesigned.
>
> Stel je hebt twee routers aan elkaar geknoopt via een VLAN over zo'n
> Nexus 9000 switch, en je doet VPLS tussen deze routers, dan kan het zijn
> dat packets waarbij de payload een ethernet frame binnen die VPLS
> instance is, waarbij het destination MAC begint met een 4 of een 6,
> gedropped worden.
>
> Als je precies weet wat voor payloads je stuurt over deze switches dan
> kun je er wel omheen werken (in een enterprise omgeving bijvoorbeeld),
> maar als je als service provider VLANs van A naar B verkoopt dan weet je
> natuurlijk niet wat de klant over het circuit gaat sturen, en dan kan
> dit flink bijten.
>
> Met vriendelijke groeten,
>
> Job
>
> ----- Forwarded message from Job Snijders <job at instituut.net> -----
>
> Date: Fri, 2 Dec 2016 15:32:13 +0100
> From: Job Snijders <job at instituut.net
> To: nanog at nanog.org
> Subject: Forwarding issues related to MACs starting with a 4 or a 6 (Was: [c-nsp] Wierd MPLS/VPLS
> issue)
>
> Hi all,
>
> Ever since the IEEE started allocating OUIs (MAC address ranges) in a
> randomly distributed fashion rather then sequentially, the operator
> community has suffered enormously.
>
> Time after time issues pop up related to MAC addresses that start with a
> 4 or a 6. I believe IEEE changed their strategy to attempt to
> purposefully higher the chance of collisions with MAC squatters, to
> encourage people to register and pay the fee.
>
> The forwarded email at the bottom is yet another example of a widely
> deployed, but fundamentally broken ASIC. The switch can't forward VPLS
> frames which contain a payload where the inner packet is destined to a
> MAC starting with a 4 or a 6. This is with the switch operating in pure
> layer-2 mode, it doesn't know what MPLS or VPLS even are. The switch is
> dropping packets on the floor, based on their _payload_. Try selling
> such circuits to customers "discounted layer-2 service, some flows might
> not be forwarded".
>
> Had IEEE continued the sequential OUI allocations, it probably would've
> taken many years before we ever reached MACs starting with a 4 or a 6,
> but instead, in 2012 the first linecards started rolling out of
> factories with MACs burned in which start with a 4 or a 6, and this took
> some vendors by surpise.
>
> There have been quite some issues, both in hardware and software:
>
> Brocade produced a 24x10GE linecard to the market in 2013/2014, with
> limited FIB scale, meant for a BGP-free MPLS core, but the card can't
> keep flows together on LACP bundles if the inner packets in a pseudowire
> were destined for a 4 or 6 MAC. The result: out of order delivery,
> hurting performance.
>
> Cisco ASR 9k's had a bug where if a payload started with a 6, it assumed
> it would be an IPv6 packet, compare the calculated packet-length with
> the packet-length in the packet and obviously fail because an ethernet
> packet is not an IPv6 packet. The result: packets dropped on the floor.
> (Fixed in 4.3(0.32)I)
>
> The Nexus 9000 issue described at the top of this mail. Brocade IronWare
> had an issue related to packet reordering for flows inside pseudowires,
> fixed in 2013/2014. There are probably many more examples out there in
> the wild, slowly driving operators insane.
>
> At this moment, some issues related to MACs starting with a 4 or a 6 can
> be mitigated if you enable Pseudowire Control-Word (RFC 4385) _AND_
> Flow-Aware Transport (RFC 6391). You need both to mitigate certain issues
> in multi-vendor networks (for instance if you have Cisco edge + Juniper
> core). But what to do when the ASIC won't forward the payload? As ISP
> you often don't control the payload.
>
> Unfortunatly, I don't think we've seen the end of this. The linecards
> bought in 2012 will trickle down to the grey/second-hand market about
> now, often without accompanying support contracts. In a world with
> increased complexity in our interconnectedness, and lack of visibility
> into the underlaying infrastructure (think remote peering, cloud
> connectivity, resellers reselling layer-2) it will hurt when some
> flows inexplicably fail to arrive.
>
> Dear IEEE, please pause assigning MAC addresses that start with a 4 or a
> 6 for the next 6 years. Or at least, next time you change the policy,
> consult the operational community. This 4/6 MAC issue was well
> documented in BCP128 back in 2007. The control-word drafts mentioned
> that there would be dragons related to 4 and 6 back in 2004.
>
> Dear Vendors, take this issue more serious. Realise that for operators
> these issues are _extremely_ hard to debug, this is an expensive time
> sink. Some of these issues are only visible under very specific, rare
> circumstances, much like chasing phantoms. So take every vague report of
> "mysterious" packetloss, or packet reordering at face value and
> immediately dispatch smart people to delve into whether your software or
> hardware makes wrong assumptions based on encountering a 4 or a 6
> somewhere in the frame.
>
> And you, my fellow operators, please continue to publicly document these
> issues and possible workarounds.
>
> Kind regards,
>
> Job
>
> resources:
>
> c-nsp thread "Wierd MPLS/VPLS issue": https://puck.nether.net/pipermail/cisco-nsp/2016-December/thread.html
> https://www.nanog.org/meetings/nanog57/presentations/Tuesday/tues.general.SnijdersWheeler.MACaddresses.14.pdf
> BCP128: https://tools.ietf.org/html/bcp128
>
> ----- Forwarded message from Simon Lockhart <simon at slimey.org> -----
>
> Date: Fri, 2 Dec 2016 11:44:21 +0000
> From: Simon Lockhart <simon at slimey.org
> To: cisco-nsp at puck.nether.net
> Subject: Re: [c-nsp] Wierd MPLS/VPLS issue
>
> On Wed Nov 23, 2016 at 12:01:20PM +0000, Simon Lockhart wrote:
> > On Fri Nov 04, 2016 at 03:40:05PM +0000, Simon Lockhart wrote:
> > > To me, everything *looks* right, it's just that some VPLS traffic traversing
> > > the new link gets lost.
> >
> > For those who are interested...
> >
> > Well, I finally got to the bottom of this, and have pushed it to Cisco TAC
> > for a fix...
>
> Cisco TAC finally accepted the issue. Bug CSCvc33783 has been logged.
> Nexus BU has investigated.
>
> Response is...
>
> "[...] unfortunately this is an ASIC limitation on the Nexus 9000
> switches and is therefore not fixable."
>
> If you want a Layer 2 switch that will forward all valid Ethernet
> frames, I'd suggest avoiding the Nexus 9000 range...
>
> Simon
> _______________________________________________
> cisco-nsp mailing list cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
>
> ----- End forwarded message -----
>
> ----- End forwarded message -----
> _______________________________________________
> NLNOG mailing list
> NLNOG at nlnog.net
> http://mailman.nlnog.net/listinfo/nlnog
>
>
> _______________________________________________
> NLNOG mailing list
> NLNOG at nlnog.net
> http://mailman.nlnog.net/listinfo/nlnog
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nlnog.net/pipermail/nlnog/attachments/20161205/2cb80b01/attachment-0001.html>