[NLNOG] Curious problem with connections from Ziggo customers to Linode nodes in some data centers

[Boudewijn Visser (nlnog)]

Hi Stefan, all,

> 
> > Filtering for the traffic captured on the server side :
> > (ip.src_host == 192.46.232.6 && ip.dst_host == 84.28.119.251 ) ||( ip.dst_host == 192.46.232.6 && ip.src_host == 84.28.119.251 )
> > 
> > So it seems your Ziggo public IP is 84.28.119.251 .
> > And filtering for the capture from the inside client side
> > (ip.src_host == 192.46.232.6 && ip.dst_host == 192.168.0.107 ) ||( ip.dst_host == 192.46.232.6 && ip.src_host == 192.168.0.107 )
> > 
> > I see an OK session using source port 50006 , and then a session that seems to have severe packet loss issues with source port 50007 .
> > 
> > See al the TCP retransmissions for the source-port 50007 session - and rarely that a packet gets through.
> > 
> > If you still can use this client (same public IP) try
> > curl --local-port 50006 http://192.46.232.six
> > curl --local-port 50007 http://192.46.232.six
> > 
> > that should replicate the problem exactly, first one always OK, second one always major problems.
> > Note : some socket timeouts when trying multiple times shorty after each other.(bind failure socket already in use )
> 
> That appears to be correct. What does that mean though? 

A heads-up that the bind failure is normal and to be expected here.

I was sufficiently intrigued by the problem that I wanted to look at it soon, but didn't have(/take) the time during working hours to write a more extensive explanation .
A tcp socket is defined by the four-tuple of (source ip, source port , destination ip, destination port) . 
When it is not fully and completely closed from both sides it hangs around for some fairly long timeout in case any long delayed packets still come in - and even if the program that opened it was quit.

So trying to re-use the exact same socket (because of same source/dest pairs) when it was waiting for those timeouts leads to the bind failure error .

So that error is not related to your problem .

If indeed the problem is exactly reproducible (always working with source port x,y,z  - and always failing with source ports u,v,w ) for a fixed source and destination - that is strong indication that the problem is indeed a broken link that is part of a load sharing/load balancing path somewhere between client and your server.
Given that you've (only - how sure ?) had this issue with Ziggo clients likely in their network. 

The exact source/destination IP and port pairs are helpful/necessary for any engineer who can take a look  .

[..]

> > (some caveats : NAT in the Ziggo modem may change source port , esp with repeated tests )
> > 
> > I think that to get anything more it will need a quite senior Ziggo network engineer to investigate further.
> 
> That’s what I’m afraid of as well, and Ziggo customer support doesn’t even want (doesn’t have the expertise?) to talk about the issue I guess.

People that understand this sort of networking problems are indeed not working at first line customer support ;-)
And of course - getting a ticket in as a non-customer is big hassle anywhere.
(I'm not a customer of Ziggo either - I was having a coffee at a place that happened to be a Ziggo customer).



> 
> My lack of deep networking expertise prevents me from understanding and diagnosing this issue properly. I have installed a work-around by introducing a proxy so that all our traffic goes to the Frankfurt server via London. This way our customers can at least use our service again. It would be awesome if someone, “for the greater good”, with more understanding than I have, could pick this up and put it to the right people, but I find myself forced to switch priorities now. 
> 
> Thanks again for your assistance (all of you)!

Unfortunately I don't think any Ziggo networking staff are on the nlnog.net list .
I'm not sure if I know someone in the right place to ping and see if he can put it on the right person's desk. I'll give it (him) a try.

Best regards, Boudewijn