Infraly, LLC - Infraly - Chicago Partial Blip – Incident details

Traffic has been traversing the network over the past 24 hours without issues. As a result, we are marking this as resolved

The cables have been replaced, and the network is traversing traffic as expected. We will continue to monitor logs over the next 24 hours for interface errors. As stated previously, we sincerely apologize for any inconvenience this incident has caused.

We currently have someone on-site who is prepping these changes. Please note that this is expected to occur in the next 15 minutes.

As promised in our previous message, we will be performing replacements of these DACs with backups from our storage at an off-peak time. This is scheduled to occur at 10:00 UTC on Wednesday, April 30th. We will provide an update when this is completed.

Our network team has concluded the investigation, and we have determined the root cause to be failing direct attached copper (DAC) cables between our spines and edge routers. Due to the severity of failing cables, the impact was felt more drastically throughout the network. As a result, we plan to replace the failing cables in the early morning of Wednesday, April 30th, at off-peak hours to minimize disruptions. We will provide a time for this in the coming hours. We apologize for any inconvenience this incident has caused.

Our network team is investigating deeper, but we can confirm that one of our core spine switches flapped all LACP interfaces. Physical interfaces remained online.

The flapping of the LACP interfaces caused some connections between our top-of-rack switches and carriers to flap that were connected to said spine switch. More info will be posted soon.